spirosgyros.net

<Understanding System Layers: W, X, and Z in Software Architecture>

Written on

There are a few standard ideas I’ve shared with engineers throughout my career, and I believe it’s high time to document them for broader access. This is the first of those ideas!

A few years back, during an engineering meeting, we attempted to structure our system and discovered it naturally divided into three layers. Over time, we realized this three-layer structure was not only beneficial for that one project but also provided a useful framework for discussing engineering systems in general. It not only names the components but also facilitates conversation about their relationships, common challenges, and potential solutions. Looking back, this has been one of the most impactful tools in engineering architecture I’ve encountered.

Today, I want to share this framework with you: first, I will explain the core concept, then how everything operates smoothly, and finally address the four most frequent issues along with strategies for resolution.

Table of Contents

  • The Core Idea: Understanding the W, X, Z layers and their functions
  • Healthy Relationships: How these teams collaborate effectively
  • Common Problems (And Solutions): X-Z Mixing, Z-Z Mixing, W-Z Mixing, Horizontal Disconnects

> If you’re curious about the missing “Y”: Initially, Brian Stoler, Reza Behforooz, and I identified X, Y, and Z layers, but later discovered a W layer existed, while the Y layer was nonexistent. Thus, we settled on W, X, and Z.

The Core Idea: Systems Have Three Layers

The Z layer represents the user-facing aspect of the system, encompassing a multitude of features. New Z components are frequently tested while older ones are phased out.

The primary challenge for teams involved in the Z layer is to discern what customers genuinely want (often referred to as product-market fit). Consequently, those engaged with the Z layer prioritize rapid experimentation and customer adoption metrics. Their focus tends to be short-term, often spanning months rather than years, and while they utilize infrastructure, they may lack expertise in it and often prefer not to delve into it.

The X layer serves as the infrastructure supporting the Z layer, communicating in the same terminology — for instance, using concepts like “users” or “feeds” instead of “integers” or “cores.”

The X layer’s responsibility is to streamline experimentation for Z teams, ensuring that they can pursue customer-centric inquiries without being bogged down by unrelated issues, excessive coordination, or the risk of one Z component disrupting the entire system.

Consequently, X layer professionals emphasize isolating issues and gauging success through the flexibility of Z teams. Their perspective is long-term, spanning years, unlike the Z teams who focus on shorter timelines.

The W layer, situated beneath the X layer, often presents primitives that differ significantly from those relevant to the Z layer. For example, it may involve storage systems that manage bytes, while Z teams concentrate on users and feeds. In many respects, the W layer operates similarly to the X layer, but its clientele consists of a smaller number of infrastructure-savvy teams that share a similar long-term perspective.

W layer teams cultivate close relationships with their clients, emphasizing performance and reliability, measuring success through the ability of X teams to scale seamlessly. Unlike the X layer, W-layer infrastructure can be procured from third parties.

Distinct from these three layers are horizontal functions like security, privacy, legal, health, and site reliability. These functions don’t constitute layers per se but rather act as custodians of shared resources within the organization (such as reputation or security). They collaborate closely with W and X layer teams to ensure that critical behaviors are incorporated into the infrastructure, allowing Z layer teams to focus on their respective tasks without needing to address these concerns directly. They aim to ensure that the system can gracefully handle failures, measuring success by the health of the resources they oversee.

> The concept of “three layers” is not exhaustive: each layer can contain sublayers. For example, an X layer might have its own foundational layer (the “W of the X”) that facilitates building higher X layers, a mid-layer of shared primitives (the “X of the X”), and a top layer (the “Z of the X”) that embodies Z layer logic, reused by multiple Z teams. Furthermore, if you’re sourcing a W layer externally, that provider will also have W, X, and Z layers within their organization, making you a customer of their Z layer.

Healthy Relationships

When operations run smoothly, the dynamics resemble this:

  • W-layer teams concentrate on scalability, reliability, and security, maintaining close communication with X-layer teams and anticipating future demands. They monitor system performance metrics, ensuring that while overall load increases, negative impacts remain minimal. If acquiring technology rather than developing it, they focus on seamless integration into the X-layer ecosystem.
  • X-layer teams increasingly adopt a mindset akin to W-layer teams, considering aggregate load from their users. They categorize customers into large accounts with intimate relationships and smaller ones with more self-service interactions. They closely observe the types of experiments Z teams wish to conduct and develop new features to facilitate those endeavors.
  • Z-layer teams engage in extensive experimentation, often unaware of the W or X layers, focusing primarily on numerous ideas for product enhancements and a relentless pursuit of product-market fit. Occasionally, they encounter concepts necessitating fundamentally new infrastructure, leading to frustration when it’s not available.
  • Horizontal teams evolve into central reviewers and embedded support within W, X, and Z layer teams, ensuring collaboration and communication. They actively assist teams in navigating potential failure scenarios and managing escalations effectively.

> Remember: Escalation is a healthy practice!

Escalation is often misconstrued as a negative indicator or something to be avoided. However, it serves several important functions: it brings decision-makers together, facilitates resolution, and clarifies the issues at hand.

If escalation is avoided during prolonged discussions, it merely prolongs indecision, causing stress and inefficiency. Therefore, it’s crucial to escalate matters promptly for the benefit of your team, management, and overall project success.

Common Problems (and How to Handle Them)

This model reveals four prevalent issues — and strategies for addressing them. Most stem from inadequate separation between layers: X-Z Mixing, Z-Z Mixing, W-Z Mixing, and Horizontal Disconnect.

Problem #1: X-Z Mixing

The Problem: Z-level teams often waste time figuring out modifications for corresponding X-level systems. This may require extensive consulting from X teams or necessitate Z teams managing X-level issues themselves. The result is frustration on both sides due to delays in experimentation and fulfillment of requests.

How it Happens: This issue typically arises when a system initially lacks clear layers, beginning as a single Z. As more Z components are added, they often intermingle with X-level logic, leading to a convoluted structure. Even if an X team is formed later, they still grapple with the complexity of this merged logic.

How to Resolve It: Establish a clear contract — an API boundary — between X and Z layers. Follow these steps: 1. Establish X teams as distinct entities focused on supporting Z teams. 2. Encourage X teams to adopt an enterprise software mindset, categorizing their users and understanding their needs. 3. Design a future vision for the X layer, engaging Z teams for input and ensuring the new API meets their requirements. 4. Initiate a traditional planning process centered around customer satisfaction and pain point alleviation.

Problem #2: Z-Z Mixing

The Problem: Z-level teams often coordinate extensively with each other to prevent unintended disruptions to features, leading to frustration and production incidents.

Why It Happens: Similar to X-Z Mixing, this results from indistinct primitives within the system, causing seemingly minor changes to escalate into larger collaborative efforts.

How to Resolve It: Evaluate existing X-Z separation. If issues persist, reassess the X-layer API to clarify exposed nouns and verbs. If the problem lies with deployment, X teams should adopt a more W-like approach, enhancing system isolation and robustness.

Problem #3: W-Z Mixing

The Problem: Z teams become entangled in W-layer specifics, complicating operations and making production changes challenging.

How It Happens: Initially, Z systems may not define proper nouns and verbs, leading to direct interactions with lower levels. Alternatively, scaling may reveal previously unnoticed complexities in W-layer systems.

How to Resolve It: Confirm whether the issue genuinely exists. If so, address the usability concerns from the perspective of W-layer abstraction. Consider automating tasks or establishing an intermediate X layer to streamline operations.

Problem #4: Horizontal Disconnect

The Problem: Horizontal teams (security, etc.) are often excluded from design processes, resulting in failures that can have dire consequences.

How It Happens: This disconnect can arise from a lack of awareness of risks, insufficient processes to facilitate collaboration, or inadequate support from leadership.

How to Resolve It: Solutions depend on the root cause. For knowledge gaps, bring in experts to educate teams. For lack of leadership support, advocate for prioritizing resource protection at the senior level.

Reframing the issue as “playbooks” rather than “risks” can also be effective, emphasizing the need for decision-making protocols in critical situations.

In conclusion, understanding and addressing these layers and their interactions can greatly enhance system design and operational efficiency.

Want to Connect? Yonatan Zunger is a Distinguished Engineer at Twitter. Previously, he held similar roles at Humu and Google, overseeing projects ranging from high-capacity search to planet-scale storage and social networking.

Share the page:

Twitter Facebook Reddit LinkIn

-----------------------

Recent Post:

Conquering Procrastination: Effective Strategies for Success

Explore actionable strategies to overcome procrastination and boost productivity, transforming

Exploring the Possibility of Resurrecting Extinct Species

An exploration of gene-editing technologies and their potential to bring back extinct animals and cure genetic diseases.

Never Invite the Vampire In: The Psychology of Human Behavior

Exploring how subconscious behavior influences decisions and the dangers of manipulation.

Unlock Your Health Potential: Essential Tips for Men

Discover unconventional health strategies for men to enhance well-being and vitality.

# Understanding Long-Covid Recovery Rates and Influencing Factors

Examining the factors influencing long-Covid recovery and what studies reveal about patient outcomes over time.

Innovative Market Strategies for Marine Conservation Efforts

Exploring a market-based approach to help nations meet 30x30 conservation goals, reducing costs and promoting international cooperation.

The Unseen Cost of Progress: Prioritizing Mental Health in Tech

Exploring the hidden impacts of technology on mental well-being and the need for mindful design in the tech industry.

Exploring the Nature of Altruism: Is Selflessness Possible?

A deep dive into whether true altruism exists, examining motives behind helping behavior.