How Spotify broke podcast video ingestion for two months

Picture this. You ship a small quality upgrade to your media processing pipeline. Nothing major, just a slightly heavier video codec to improve playback. Later in the day, a routine background cron job kicks off to re-process some old files. Meanwhile, a minor scheduling bug in your new hardware is quietly eating about ten percent of your compute pool. None of these things are critical on their own. They are mundane, everyday engineering realities. But they are all pointing at the exact same shared compute cluster.

The combined load silently erodes your capacity headroom. Then, a normal wave of daily traffic hits. And suddenly, your users cannot upload anything for four hours.

This is not a theoretical exercise. This is exactly what took down Spotify’s video podcast ingestion system on June 24. Creators were hitting publish, and their videos just sat there. What usually took minutes stretched into hours.

We naturally want to blame outages on massive, unpredictable traffic spikes. It makes us feel better as engineers. It lets us point to external factors we cannot control. But Spotify’s ingestion did not break because of a massive spike. It broke because four separate, completely ordinary factors pooled together to exhaust their transcoding capacity.

It is a textbook example of a Swiss cheese failure. Every layer of defense has small holes. When those holes align, the system fails.

The Four Factors That Stacked Up

To understand how this happens, we need to look at the four factors that stacked up.

1. First, the team had recently rolled out an improvement to their video transcoding. It delivered better video quality at lower bitrates. This is a common trade-off. Lower bitrates save money on content delivery networks and improve buffering for the end user on mobile connections. But compressing that video requires significantly more compute cycles. The team tested the quality, but they failed to fully account for that added processing demand in their capacity planning. The baseline floor for compute was raised.

2. Second, they had recently migrated their infrastructure to more powerful hardware. The irony is that this new hardware was supposed to give them more headroom. But a software bug in their custom resource scheduler caused the system to underutilize the available processing capacity. That bug alone reduced total throughput by roughly ten percent.

3. Third, a scheduled batch processing job was running in the background. On occasion, Spotify needs to re-process existing, older episodes to ensure they are compatible with changes to their playback systems. This routine job was consuming additional capacity alongside the regular processing for brand new episodes.

4. Fourth, because of the first three hidden regressions, the system simply lacked the buffer to handle a normal delivery wave. The headroom was gone.

A Tale of Two Queues

So, how does a low-priority background job take down real-time user uploads? This points directly to the architecture of their queueing system. Spotify used a priority-based queue. They had a medium-priority tier for time-sensitive new episodes, and a low-priority tier for background updates to older episodes.

But a priority queue only works if the workers pulling from it can actually process the jobs faster than they arrive. If your workers are bogged down by a heavier codec and crippled by a scheduling bug, the queue just backs up. Priority does not magically create compute out of thin air. Both queues were drinking from the same constrained well.

This Was Not a "Perfect Storm"

If we look at this through a builder's lens, framing this incident as a "perfect storm" is a massive dodge. Marketing and PR teams love the perfect storm excuse because it implies the engineering was flawless but the universe conspired against them.

But three of these four factors are completely controllable engineering decisions. You chose to deploy a heavier codec without scaling your compute. You chose to run a background job without strict resource limits. You built a custom resource scheduler that failed to allocate capacity. This is not a storm. It is a predictable exhaustion of resources resulting from basic capacity planning misses.

Architectural Flaws and Tight Coupling

Why did a routine batch job lack strict Quality of Service limits? A low-priority background job should never be permitted to steal compute capacity from time-sensitive, user-facing operations. They need hard resource caps.

And the architectural flaw goes deeper. Why was the upload ingestion API so tightly coupled to the downstream transcoding capacity? During the outage, the system failed to confirm to users that their uploads were actually received and queued.

Think about this from the user's perspective. A creator records a video. They hit upload. The progress bar finishes, but the episode does not show up on their dashboard. What do they do? They assume their internet dropped or the browser glitched. So they try again. And again. Every time they retry, they throw another massive, high-resolution video file at a system that is already choking to death.

A resilient system completely decouples the front door from the factory floor. When a user uploads a file, the ingestion layer should validate the payload, drop it into persistent storage like an S3 bucket, and immediately return an HTTP 200 success response. The user gets their confirmation. The system queues the heavy processing asynchronously. Your ingestion API should never care if the downstream transcoding workers are currently busy, broken, or overloaded. The upload succeeds regardless.

The Four-Hour Monitoring Gap

This tight coupling also obscured the failure from the engineers. The timeline of this incident shows a massive gap in situational awareness. The first automated alerts fired in their internal monitoring at 13:30 UTC. But formal incident response did not begin until 17:34. That is four hours between the system knowing something was wrong and humans treating it as an outage.

Why the delay? The early alerts were treated as noise. Engineers actually investigated those early alerts and manually stopped the background batch job around 16:35, hoping to free up some capacity. That means a human looked at the dashboard, took a manual action, and still did not declare an incident. They thought they had applied a minor course correction. An hour later, the medium-priority queue completely overflowed the critical thresholds, and creators started complaining on social media.

If your alerts fire four hours before an incident is recognized as a full capacity failure, your monitoring strategy lacks actionable severity mapping. This happens when you alert on server metrics instead of the end-to-end user experience. If you measure CPU usage or queue depth, you just see a busy server. If you measure the actual time-to-publish metric, you see a complete outage immediately.

Spotify's immediate fix for this was blunt force. They deployed a patch for the scheduling bug, and then they increased their transcoding capacity by roughly sixty-seven percent overnight. They threw massive amounts of compute at the queue bottleneck to stop the bleeding.

The Core Lesson: Isolation and Managed Services

By the next morning, the backlog was cleared. But throwing hardware at the problem is a costly band-aid. It clears the queue, but it does not solve the underlying lack of backpressure and resource isolation.

The architecture lesson here is about knowing when to build custom infrastructure and when to offload it. Spotify is operating at a massive scale, which forces them into custom solutions. They build custom resource schedulers and maintain complex priority queues for their own hardware clusters. But that custom scheduler alone cost them ten percent of their throughput due to a single bug.

For the rest of us building systems at normal scale, you do not need to take on this operational burden. You use a managed, event-driven architecture. A direct upload to an object store that triggers a managed service like AWS MediaConvert or Step Functions handles burst capacity automatically. The managed service absorbs the spike. It handles the backpressure. It scales to zero when you do not need it. Most importantly, it physically decouples the upload ingestion layer from the heavy processing layer out of the box.

You only pull that complexity in-house when the sheer volume of your traffic makes managed SaaS unit costs higher than the engineering salaries required to maintain a custom infrastructure team. Until you cross that exact threshold, let the cloud provider handle the scaling.

And if you do manage your own processing pipelines, the rules remain the same. Strictly isolate your background batch processing from your synchronous, user-facing ingestion. Keep your queues entirely separate. Enforce hard resource limits on anything running in the background. Alert on the user experience, not just the server load. And always acknowledge the upload before you start the heavy lifting.

This is TAKEYOURPILLS DOT TECH. Go ship something.