Skip to content
How your team accidentally built a distributed monolith

How your team accidentally built a distributed monolith

6 min read architecture

Transitioning to microservices often leads to unintended consequences like distributed monoliths and cascading failures. This guide breaks down the most dangerous service architecture anti-patterns, why they happen, and the exact steps you need to take to prevent them....

Subscribe to listen
audio-thumbnail
How your team accidentally built a distributed monolith
0:00
/0
Clinical Summary
Diagnosis

Applying local separation principles across network boundaries creates fragile distributed monoliths, replacing atomic in-memory functions with high-latency HTTP requests and JSON serialization.

Prescription
  • Modular Monoliths: Enforce clean boundaries via strict internal modules to preserve nanosecond calls and atomic Postgres transactions.
  • Compute Isolation: Extract services only when components have drastically different infrastructure needs, such as dedicated GPU instances.
  • Distributed Tracing: Rely on trace logs and deployment schedules to detect deep, synchronous call chains and hidden monolithic coupling.
Side Effects

Distributing architecture replaces free database rollbacks with complex saga patterns, partial network failures, and eventual data consistency nightmares.

Script

Inside a single program, one function calling another takes about five nanoseconds. The CPU grabs the instruction. It executes. It returns an answer. Or it throws an error. It's atomic, predictable, and practically instantaneous.

But the moment you draw a line between those two functions and move one of them across a network boundary, the physics of your system fundamentally change. That five-nanosecond call is now a network hop. It takes a few milliseconds. It can drop packets. It can time out. It can succeed halfway, die on the return trip, and leave your data in a broken, zombified state.

This is the fundamental physics problem of distributed architecture. Pretending a network boundary is just a local function call with a little extra latency is exactly how teams accidentally build distributed monoliths.

And here is the hardest part to swallow. A distributed monolith isn't created by careless developers making reckless decisions. It's built by very good engineers. It's built by teams meticulously applying local "clean code" separation principles to network boundaries where those principles absolutely don't belong.

Picture this. You're looking at a standard, monolithic e-commerce codebase. The invoicing logic feels entirely separate from the shopping cart logic. It has different business rules. It has different stakeholders. To keep the codebase clean, to enforce that perfect single-responsibility principle, you extract the invoicing logic into its own independent service.

You pat yourself on the back. The architecture diagram looks great on a whiteboard. Everything has its own isolated box.

A whiteboard diagram showing a monolithic service being split into multiple independent microservices with arrows indicating network calls between them.

But look at what happens at runtime. A simple user checkout now triggers four sequential network hops. The checkout service calls the inventory service to reserve the item. The inventory service calls the pricing service to calculate tax. The pricing service calls the new invoicing service.

What used to be a safe, guaranteed, in-memory local call is now a two-hundred-millisecond timeout risk. Worse, there's no atomic database rollback. If that third hop fails, what happens? The inventory is reserved, the user's credit card is charged, but the invoice is never created. You now have a partial failure.

This is the boiling frog problem of service architecture. You don't wake up one day and decide to build a fragile, distributed mess. You make a series of individually sound choices. A clean separation here. An independent deployment there. You extract a new service each time a domain feels distinct enough to stand on its own. Slowly, the water gets hotter.

The Two Anti-Patterns of the Distributed Monolith

Let's ruthlessly dissect the two specific anti-patterns that emerge from this exact mistake. Ignore the rest of the standard architecture diagrams for a minute. Focus entirely on chatty services and deep call chains.

First, chatty services

When you slice a system based simply on parts feeling distinct, you often separate components that actually need to talk to each other constantly to get any real work done. You end up with Service A asking Service B for permission, checking a status, updating a record, and asking for a receipt, all in the span of one single user request.

You've replaced a fast, in-memory function call with a high-latency, uncompressed HTTP request. You're serializing JSON, opening a TCP connection, doing a TLS handshake, sending the payload, and waiting for the deserialized response. Multiply that by thousands of requests per minute. You've introduced massive latency and infrastructure cost for zero organizational benefit.

The code might be cleaner to read, but the system is fundamentally worse to operate.

Second, call chains

This is the real nightmare. Service A calls B, which calls C, which calls D. Deep call chains are a glaring indicator that you've built a distributed monolith.

When call chains get too deep, partial failures aren't just a risk. They are a mathematical guarantee. If Service C times out while waiting for Service D, what exactly does Service B do? Does it retry? Does it fail the whole request?

You're suddenly forced to write complex saga patterns. You have to write compensating transactions to handle rollbacks. You're spending engineering cycles building safety nets that a standard Postgres database handles for you, automatically, for free.

There's a popular architectural ideal that says a service controls its own data. It implies you never reach into another service's database. But achieving this decoupling without introducing eventual consistency nightmares is incredibly painful. You can't just run a cross-table SQL join anymore. You have to pull data over the wire and join it in memory, or duplicate it and keep it synced via event streams. The ideal glosses over the brutal reality of distributed data.

When Clean Separation Becomes a Liability

So, what's the exact moment a clean separation turns into an architectural liability? It happens the moment you prioritize "feeling distinct" over the actual, physical requirements of deployment and scaling.

Single responsibility tells us to decouple things that change for different reasons. But applying that logic to a network topology is a category error. Network boundaries are physical realities. They aren't organizational folders.

How to Diagnose the Problem

How do you diagnose if you've accidentally built a distributed monolith before it takes down production? You don't look at the codebase. You look at your distributed traces and your deployment logs.

Pull up a trace for a core user action. Does a single click fan out into a deep, synchronous call chain across four different services? If one of those downstream services spikes in latency, does the entire user-facing request grind to a halt?

Now look at your deployments. Do you frequently have to deploy three services at the exact same time to release one single feature? Do your teams have to coordinate their release schedules?

If the domains still require coordinated deployments, or if they secretly share a database under the hood, you don't have independent services. You just have a monolith spread across a network, communicating over expensive and fragile HTTP boundaries.

The Right Reasons to Create a Service Boundary

If "feeling distinct" is the wrong reason to split a service, what's the right metric for a boundary? The answer is organizational friction and compute isolation.

Service architecture is an organizational scaling tool. It's not for solo developers. It's not for early-stage startups trying to find product-market fit. It requires a dedicated platform team just to manage the resulting operational overhead.

You reach for independent services when you have forty or more engineers stepping on each other's toes in a single codebase. You use it when independent deployability becomes a hard organizational requirement because your release trains are constantly blocked by unrelated feature teams.

You also reach for it when a specific component has drastically different infrastructure needs. If your system does standard CRUD operations, but one piece handles heavy video encoding or machine learning inference, you split that piece out. You split it so you can scale the GPU instances independently from the web servers. You split it for the hardware, not for the neatness of the code.

If you don't have those specific scaling problems, extracting services is premature optimization. You're paying a massive cognitive tax. You're confronting distributed tracing. You're managing multiple CI/CD pipelines. You're handling network failures. And you're doing it all for absolutely zero practical gain.

A Superior Alternative: The Modular Monolith

The worst anti-patterns stem from organizational coupling and poor domain understanding, not just network physics. The closest, and often superior, alternative to this mess is a well-structured modular monolith. Teams often prefer it, and for good reason.

A modular monolith enforces clean boundaries via strict modules. It prevents tangled dependencies. But crucially, it keeps the actual function calls in-memory. It runs in nanoseconds instead of milliseconds. It preserves atomic database transactions.

Don't split a system just to keep the codebase tidy. You can keep it tidy with modules. Only introduce a network boundary when the pain of coordinating human engineers completely outweighs the pain of managing distributed data consistency.

This is TAKEYOURPILLS.TECH. Go ship something.

References

/