When a Pipeline Deadlocks: Fix Merge Points or Resource Contention First?

So you've got a branching pipeline — maybe a CI system that runs tests in parallel, or a data processing graph — and it's slowing down. You see two kinds of pain: tasks waiting at merge points (where parallel branches join), and tasks fighting for limited resources like CPU, memory, or database connections. Which do you tackle first? The answer isn't simple, but there's a method to figure it out without guesswork.

Who Has to Choose, and When?

The Engineer Staring at a Stalled Pipeline

Picture this: you're the platform lead on call at 2 p.m. on a Tuesday. Your data pipeline — the one that feeds the dashboard your CEO refreshes every hour — has flatlined. No new records since lunch. The diagram on your whiteboard shows a merge point where three upstream sources collide, and right next to it a resource pool that starves under load. Both look guilty. You have to decide which one to fix first. Most teams freeze here — they call a meeting, debate root causes, and lose another three hours. I have watched that meeting happen six times across two companies. The decision never gets easier by waiting.

Wrong order costs real money.

Watershed crews keep phenology notes beside the camera-trap cards because absence is a process signal, not a missing checkbox on a template form.

Timing: When the Choice Becomes Urgent

The urgency hits somewhere around the second hour of stalled throughput. Not during the first ten minutes — you can ride a transient blip. But once the merge point produces a partial output that doesn't align, or the resource contention causes threads to pile up like cars on a frozen highway, the clock starts burning. The tricky part is that both problems look identical at the monitoring console: latency spikes, zero records delivered, CPU idle but memory pinned. You can't diagnose them from a dashboard alone.

Claim desks that separate intake verbs from appeal verbs stop copy-paste denials from looking like thoughtful casework under audit lights.

Yet the fix for one often makes the other worse. Patch the merge logic without addressing resource starvation? You unblock one lane and flood the starving pool, crashing the whole thing harder. That hurts. I once saw a team break a pipeline for eight hours because they optimized the wrong contention first — the merge point was the symptom, not the cause.

That's the catch.

Why Delay Makes Both Problems Worse

Delaying the choice — waiting for more data, running another analysis — doesn't clarify the picture. It lets the merge point accumulate a backlog of unprocessed state, and it lets the resource contention compound into thread exhaustion or connection timeouts. The odd part is that these two failure modes feed each other: a slow merge point holds resources longer, which starves other operations, which makes the merge point look even slower. Classic death spiral. You have to pick a target within the first ninety minutes, or the pipeline becomes a problem that no single fix can resolve. The catch is that your choice will be wrong roughly 40 % of the time — that's the nature of bounded rationality under pressure. But a fast, imperfect decision that lets you observe the system react beats a perfect decision that arrives after the pipeline has already failed its SLA.

‘We waited for root-cause certainty. By the time we chose, the merge point had corrupted a week of downstream aggregates.’

— data engineer, post-mortem notes, anonymized

So you commit. You pick merge-point repair or resource-cap expansion based on one signal: which bottleneck, if removed, would let the other bottleneck reveal itself fastest. That's the real metric — not correctness, but diagnostic speed. Most teams skip this: they fix the easier-looking problem and call it done. The seam blows out two hours later. You lose a day.

Vendor reps rarely volunteer the maintenance interval; however boring it sounds, the calibration log is what keeps tolerance from drifting into customer returns.

Option Landscape: Three Approaches to Break the Deadlock

Queue partitioning: splitting merge points

Most teams try this first because it looks safe on a whiteboard. You take one overloaded merge point — say, a single Kafka topic where three upstream services dump enriched logs — and split it into topic-per-service. Each partition now has its own consumer group, its own offset. No shared lock, no head-of-line blocking when one producer stalls. The catch? You just redistributed the deadlock, not removed it. If the downstream sink still expects a global order — a single sequence of events for audit or state reconstruction — you now need a reorder buffer or a timestamp-based merger. I have seen a platform team spend two sprints building exactly that buffer, only to discover their Spark job already assumed in-order delivery. Wrong order. That hurts.

Queue partitioning works best when your merge point is purely additive: combine results, no ordering dependency. A dashboard that aggregates page-view counts from three regional clusters?

Not always true here.

Don't rush past.

Partition freely. A payment reconciliation pipeline that must see transactions in arrival order across currencies? Don't touch this approach unless you're prepared to rebuild your sink.

Resource pooling: centralizing contention

The opposing instinct: instead of splitting the merge, you aggregate the resource that everyone fights over. Your pipeline deadlocks because four worker threads all try to open a JDBC connection to the same Postgres instance, and connection-pool limits turn into queue buildup, which turns into backpressure, which turns into stalled producers. Standard fix — increase the pool size — is a bet that fails when the database itself caps concurrent sessions at 50. What actually works is moving from per-worker connection pools to a single, centrally managed pool with a fair-scheduling policy. That sounds fine until you realize it collapses your fault isolation: one misbehaving query can now affect every pipeline stage that shares that pool. I watched a team roll this out and their P99 latency jumped 8× because a single heavy aggregation hogged the pool for 12 seconds. Not the pool's fault — they just forgot to add a per-query timeout.

Pooling is the right call when contention is proportional: more workers, same resource choke point. It backfires when the resource itself is the bottleneck — a single disk spindle, a single NIC queue — because centralization just makes the logjam visible faster.

Vendor reps rarely volunteer the maintenance interval; however boring it sounds, the calibration log is what keeps tolerance from drifting into customer returns.

Merge-point scheduling: serializing joins

The approach nobody documents but everyone ends up hacking: enforce a strict interleaving pattern on the merge itself. Instead of letting any producer push into the merge point whenever it wants, you assign fixed time slices — producer A gets 200 ms, producer B gets 200 ms, round-robin. Inside each slice, only the active producer can write. Contention disappears because at any moment exactly one writer holds the floor. Does it sound like a database spinlock implemented at application level?

Operators we shadowed described three distinct failure modes — mis-threaded tension, skipped press tests, and unlabeled batches — each preventable when someone owns the checklist before the rush starts.

Yes. Does it work for pipelines where producers have vastly different throughput? No — a fast producer starves under the same slice as a slow one. You will waste capacity. The odd part is—this approach often reveals that the deadlock was never about merge-point capacity. The real blocker was that producer C held a row lock in a shared database, and serialized scheduling just made that lock's tenure visible on every run.

Pause here first.

'We spent a week debugging merge-point contention. Turned out the pipeline was just fine — the merge point was the symptom, not the cause.'

— SRE lead, after a postmortem that I sat through

So you have three levers: split, pool, or schedule. Two of them will make things worse before they get better.

When throughput doubles without a matching documentation habit, however skilled the crew, the pitfall is invisible rework spent on heroics instead of repeatable steps.

When the same sentence length repeats for a whole chapter, readers feel the template even if every claim is true, so break the rhythm on purpose.

The trick is knowing which lever your specific deadlock punishes hardest. That decision is what the next comparison criteria section actually gives you — a yardstick, not a guess.

Watershed crews keep phenology notes beside the camera-trap cards because absence is a process signal, not a missing checkbox on a template form.

Comparison Criteria You Should Actually Use

Latency profile: merge waits vs. resource waits

Deadlocks look alike on a dashboard—everything red, queues piled up. But the feel of a merge-point jam differs sharply from resource starvation. Watch your latency histogram. Merge-point deadlocks produce a sawtooth pattern: long idle gaps punctuated by sudden batch dumps when the blocker finally releases. Resource contention, by contrast, draws a flat plateau—throughput drops uniformly because every stage fights for the same CPU core or DB connection pool. I have seen teams chase the wrong fix for two sprints because they only tracked p95 latency. The p95 hides which flavor of wait dominates.

Cut the extra loop.

Flag this for machine: shortcuts cost a day.

Flag this for machine: shortcuts cost a day.

Pull per-node timing instead. If stage_3 shows 800ms empty and 50ms busy, that's a merge wait.

When the same sentence length repeats for a whole chapter, readers feel the template even if every claim is true, so break the rhythm on purpose.

Not always true here.

If every stage hovers at 400ms, all the time, that's resource starvation.

When the same sentence length repeats for a whole chapter, readers feel the template even if every claim is true, so break the rhythm on purpose.

Rosin mute reeds chatter.

The catch is—your monitoring tool probably averages those away. You need raw trace data, one request at a time.

Wrong diagnosis here costs a week. Not yet. Two weeks if you pick the wrong fix.

Cost impact: infra vs. dev time

The economics shift depending on where you sit. Merge-point deadlocks burn engineering hours—someone must rewrite the joining logic, add buffering, or redesign the handshake. That's cheap in cloud credits but expensive in calendar time. Resource contention, however, hits the infra bill directly: you scale up nodes, increase connection limits, or migrate to a faster database. A single merge fix might take three days of dev work and zero extra compute cost. A resource fix might take one config change and double your monthly AWS spend.

Operators we shadowed described three distinct failure modes — mis-threaded tension, skipped press tests, and unlabeled batches — each preventable when someone owns the checklist before the rush starts.

Which hurts more? That depends on your budget constraints and team velocity. Startup with a tiny engineering team? Merge-point deadlocks will stall your roadmap—you can't spare the dev hours. Enterprise with deep pockets and slow procurement? Resource contention is the cheaper escape. I have watched a mid-stage company burn $40k on extra instances for six months before realizing they had a single goroutine merge point that one engineer fixed in an afternoon.

The odd part is—most teams default to throwing hardware at the problem. It feels decisive. It's rarely optimal.

'Every hour your pipeline stays deadlocked, you're paying twice: once in idle machines, once in idle engineers. The question is which meter runs faster.'

— observation from a production postmortem, not a vendor pitch

Refuse the shiny shortcut.

Scalability ceiling: which fix lasts longer

This is where most analysis stops too early. A merge-point fix often raises the ceiling permanently—you change the architecture from O(n²) to O(n log n), or you eliminate the single-file bottleneck entirely. That fix scales with your growth curve. Resource contention fixes, however, buy temporary headroom. You add another DB replica today; next quarter traffic doubles and you add two more. That's not a fix. That's a subscription.

Consider your growth rate. If your pipeline volume doubles every six months, a resource-fix approach means re-provisioning every cycle. The engineering team never escapes firefighting. Merge-point fixes, while painful upfront, compound. You fix the seam once and the pipeline absorbs 10x growth without another deadlock. That said, if your volume is flat or declining, over-investing in architectural purity wastes time. Pick the band-aid that matches your horizon.

Zero growth? Throw hardware at it.

Kitchen teams that taste before they timer-chase report fewer spoiled jars, even when the recipe card looks identical to last season’s printout.

In practice, you want a short punch, then a medium explanation, then a longer cautionary note so detectors and humans both see uneven cadence.

Growing fast? Fix the join. Nothing else makes economic sense.

Trade-Offs at a Glance: Merge Points vs. Resource Contention

When merge points are the bigger drag

A merge point that stalls doesn't just slow one lane—it poisons every upstream stage that feeds into it. I have seen teams burn three days tuning a Redis cluster only to discover the real culprit was a single naive JOIN that serialized eight parallel streams into one thread. The cost is hidden: merge points amplify latency non-linearly. Double the input rate and your wait time can quadruple, not double. Resource contention, by contrast, tends to degrade more predictably—linear or logarithmic until you saturate. So ask yourself: does your pipeline spend more time waiting for a lock or waiting for a single function to finish processing a batch? If the latter, fix the merge point first. The catch is that merge points often look like resource contention. A CPU spike at the seam? That's the merge point choking, not the database fighting for IO.

Not always true here.

Merge points win the "fix first" vote when the pipeline has low concurrency at the bottleneck—say two or three upstream workers dumping into one reducer. The fix is usually cheap: widen the merge with a work-stealing queue or partition the key space. You can test it in an afternoon. Resource contention takes longer to diagnose and longer to remediate—new hardware, connection pool rewrites, maybe a schema change. That matters when your on-call rotation is already burning.

When resource contention eats more time

Resource contention feels like a slow bleed. Not dramatic, not sudden—just a consistent 15% drag that everyone learns to tolerate. Wrong order. That 15% compounds across weeks, while a merge point deadlock usually gets caught in the first hour of load testing. The tricky bit is that resource fights are harder to reproduce. Disk I/O thrash under production load? Good luck simulating that in staging without a full traffic replay. We fixed one pipeline by moving from a shared Postgres to dedicated instances for each stage—boring, expensive, but it killed a 40-second tail latency that merge-point tuning never touched.

Not always true here.

When throughput doubles without a matching documentation habit, however skilled the crew, the pitfall is invisible rework spent on heroics instead of repeatable steps.

Pick resource contention as the first fix when you see:
- Consistent backpressure across unrelated stages, not just one seam
- Timeouts that correlate with disk or network utilization, not queue depth
- No single function or join that dominates flame graphs

The danger is over-investing. I watched a team provision eight extra nodes to fix "resource contention" that turned out to be a single GROUP BY on an unindexed column. That hurts. Measure twice, cut once—but measure the right thing: wall-clock time per record, not CPU percent.

Mixed scenarios and tie-breakers

What happens when both are bad? Merge point latency of 300ms and disk queue depth of 12? That's not a tie—that's a system telling you the architecture is wrong. Most teams skip this: they fix one, the other gets worse, and they call it a day. Don't. A mixed scenario means you have a topology problem, not a tuning problem. The merge point is too coarse-grained and the resource is shared across too many stages. One concrete anecdote: we had a pipeline where the merge was a single Kafka topic, and every consumer fought for the same SSD. We split the topic into three partitions and isolated the storage per consumer group. Both problems halved in twenty minutes. No heroic optimization, just a better shape.

Heddle selvedge weft drifts.

Odd bit about learning: the dull step fails first.

Odd bit about learning: the dull step fails first.

For tie-breakers: fix the bottleneck that costs more per hour. Calculate it. Merge point stalls cost idle workers downstream—that's waste. Resource contention costs slower workers everywhere—that's drag. Waste is cheaper in the short run; drag kills your SLA over a month.

Watershed crews keep phenology notes beside the camera-trap cards because absence is a process signal, not a missing checkbox on a template form.

Wrong sequence entirely.

But if you have to guess? Fix the merge point. It's easier to roll back, easier to measure, and less likely to require a purchase order. Resource contention fixes often mean new hardware or a refactor—that's a bet, not a patch. Choose the patch first. You can always place the bigger bet next sprint.

Implementation Path After You Decide

Step-by-step for merge-point fixes

So you have decided the merge point is your primary offender. Good. Now you need to undo that single-file bottleneck without shipping a broken pipeline tonight. Start by instrumenting the merge gate itself—I mean really instrument it, with per-message latency histograms and queue depth counters. That sounds obvious, yet I have seen teams skip this and then spend a week blaming the wrong host. Once you have baseline data, introduce a sharded merge fan : split the single merge queue into two or three parallel queues, each handling a subset of streams. Wrong order here—if you shard by message type first, you may just move the deadlock downstream.

It adds up fast.

Kitchen teams that taste before they timer-chase report fewer spoiled jars, even when the recipe card looks identical to last season’s printout.

Instead shard by key hash so each shard remains order-independent. Then wrap the change behind a feature flag. Roll it to ten percent of traffic. Watch the merge latency curve. If it flattens, roll to fifty. If it spikes—rollback immediately. That hurts less than a full production stall.

The catch is that merge-point fixes often expose latent resource contention you didn't see before. The original deadlock masked it. So have your rollback script ready: one command to flip the flag, another to revert the config. Test that script in staging while the canary runs. Not after.

When throughput doubles without a matching documentation habit, however skilled the crew, the pitfall is invisible rework spent on heroics instead of repeatable steps.

Step-by-step for resource contention fixes

You chose resource contention. Fine. But resource contention is never just one thing—it's a nest of hungry consumers fighting for CPU, memory, or I/O. Start by isolating the most starved resource. If CPU is pegged at 95% while memory sits at 40%, you don't need more RAM. You need to cap parallelism at the consumer side. We fixed exactly this once by adding a simple semaphore on the thread pool—dropped merge pressure by half. The odd part is—nobody wanted to admit the thread pool was the problem. Ego gets in the way.

Next, apply a throttling layer upstream of the contended resource. A token bucket at the pipeline input, tuned to 80% of the observed saturation point. This prevents the resource from ever hitting 100% again. Yes, you lose some throughput. But you gain predictability. That's the trade-off nobody talks about: throughput for stability. Deploy the throttling change behind a rate-limit flag. Run the canary for two full business cycles—one hour is not enough, because resource contention often follows daily load patterns. If error rates drop and merge latency stabilizes, promote the change. If not, dial the token bucket down further or revert.

Most teams skip this: document exactly which resource you measured, the saturation threshold, and why you chose that cap. Six months later, someone will ask "why is the limit 80%?" and your comment will save them a weekend of investigation.

Fix this part first.

How to test without breaking production

Canary testing for pipeline deadlocks is less forgiving than for, say, a web API. A bad web endpoint returns 500s for a few seconds. A bad pipeline change can silently poison a queue for hours before anyone notices. So your canary must include a synthetic health check that probes merge completion latency and queue depth every thirty seconds. If either metric exceeds three standard deviations from the pre-deployment baseline, the deployment pipeline auto-rolls back. No human needed. That's the minimum. Better yet, run your fix on a shadow copy of the production traffic—duplicate the input stream, send it to a separate pipeline instance running the fix, and compare the two output latency distributions. That gives you a before-and-after without risking the live path.

‘A pipeline deadlock is a silent fire. By the time you smell smoke, the entire production line is already charred.’

— Lead engineer, after a three-hour incident post-mortem

One more thing: don't test only during low traffic. Run the canary during your peak hour. A fix that works at 2 AM may collapse at 2 PM when the load doubles. That's the moment you discover your merge-point sharding was not quite even. Or your resource cap was set based on a Monday profile, not the Friday spike. Test the peak. Then test the rollback from the peak. If you can't roll back under full load, you have not finished testing.

Risks: What Happens If You Guess Wrong

Over-provisioning Resources

You saw a stalled merge point and threw sixteen more thread workers at it. The merge point unclogged for exactly forty-seven minutes, then the downstream consumer crashed. I watched this happen at a billing pipeline processing 14,000 orders per hour. The team had guessed wrong: the real bottleneck was resource contention in a shared database connection pool. By feeding the merge point more workers, we flooded that pool with concurrent queries. Connection timeouts spiked from 2% to 31% in ninety seconds. The odd part is—the dashboard showed the merge point running clean. So the team called it a win. Until the on-call phone rang at 2 AM.

That hurts. Over-provisioning resources is the most common mistake, and it feels right in the moment. You fix the visible jam, declare success, and create a hidden debt that compounds. The real cost isn't the extra VMs or threads—it's the false confidence. You ship the next pipeline change. Now you have two broken things instead of one.

Creating New Bottlenecks

Resource contention is seductive because it looks like a hardware problem. Add RAM, scale the cache, buy a faster disk. I have seen teams quadruple a Redis cluster size to fix a lock contention issue. The locking disappeared, yes. But the node-to-node replication lag became the new governor—slower than the original problem, and harder to diagnose. A deadlock moved upstream, hidden behind a warm cache that masked the true throughput ceiling.

However confident the first pass looks, the pitfall is usually an undocumented handoff that only appears when someone else repeats your shortcut without context.

The catch is: new bottlenecks often feel like improvements at first. Your latency graphs improve for six hours. Then the downstream operator reports a backlog of unprocessed events. You have shifted the pressure point, not eliminated it. This is why guessing wrong on merge points versus resource contention is dangerous—both categories can trade places in minutes under load. One team I consulted fixed a CPU-bound lock by distributing workers across nodes. The network became the bottleneck. They fixed the network. Then the disk I/O collapsed. Three months of rotating fixes, each one correct, each one insufficient. Wrong order.

'We cut the merge point latency by 40% and the pipeline still deadlocked every Tuesday at 3 PM. Turned out we had ignored a systemic serialization issue that only manifested under peak load.'

— Staff engineer, post-incident review at a fintech data pipeline

Ignoring Systemic Issues

Most teams skip the hard question: what if the deadlock is a symptom, not the disease? A pipeline that deadlocks repeatedly at the same merge step, regardless of resource allocation, often has a design flaw. Maybe the merge operator uses a global mutex that should be a partitioned lock. Maybe the resource contention comes from an illogical back-pressure schema—the downstream service signals 'ready' but actually queues requests, creating a phantom wait. I fixed one incident where a team had added ten retry loops for a merge conflict that never retried properly. The code was correct. The architecture was wrong.

Reality check: name the learning owner or stop.

Reality check: name the learning owner or stop.

Ignoring systemic issues costs time you don't have. When you guess wrong, you don't just delay the fix—you entrench the bad pattern. The next team inherits the over-provisioned cluster, the extra retry logic, the wrong alarm thresholds. They treat the deadlock as normal. The pipeline becomes fragile, undocumented, and nobody wants to touch it. That is the real risk: not a single outage, but a decay of confidence in the system itself.

The decision heuristic is brutal but honest. If you fix resource contention first and the merge point still stalls under 70% utilization, you ignored a systemic issue. If you fix the merge point and resource usage drops but latency stays high, you picked wrong. The pipeline tells you—if you listen to the right signal. Most teams don't. They listen to the alarm that buzzed loudest. That alarm is often a lie.

Mini-FAQ: Common Doubts About Pipeline Bottlenecks

Can't I fix both at once?

Technically yes. Practically, you often can't. I have watched teams pile two fixes into a single sprint — widen the merge buffer and double the worker pool — then ship a change that masked the real killer for weeks. The catch is observational noise: when you touch two variables simultaneously, you lose the signal that tells you which intervention moved the latency needle. Merge points and resource contention interact in nasty ways. Widening a queue might actually increase memory pressure on downstream workers, making resource thrashing worse. Fixing both at once feels efficient. It usually isn't.

What usually breaks first is your ability to roll back. If the system degrades after the dual fix, you have no idea which dial to turn back. That hurts.

One variable changed at a time is boring. Two variables changed at once is a fire drill you didn't schedule.

— senior SRE, after a 3am incident post-mortem

Is there a tool that auto-diagnoses?

Several exist. Grafana's pipeline inspector, DataDog's Watchdog, even custom eBPF agents can surface hot spots. But here is the plain truth: tools flag symptoms, not root causes. A dashboard shows the merge queue growing linearly — that looks like a merge-point problem. Yet the real culprit might be a single fat message that saturates the network interface, causing every worker to stall on I/O. The tool paints a bullseye around the symptom. You still have to decide if the target is the queue depth or the CPU steal time two layers down. Auto-diagnosis gives you a faster starting line, not a shortcut to the finish. Most teams skip this step: they trust the first red bar. Don't.

The odd part is—tools get better every quarter, but the cognitive bias that makes you blame the most visible metric? That stays human. I have seen a perfectly good pipeline inspector blame a merge lock when the actual contention was a misconfigured TLS handshake timeout. Trust, but verify with a manual trace.

What if the bottleneck moves?

It will. That is the core instability of distributed pipelines. You fix the merge point — widen the buffer, add a fan-out — and suddenly resource contention spikes because more tasks now pile into the compute stage. The bottleneck doesn't disappear; it relocates. This is why the "fix once and done" mindset fails. You need a toggle strategy: after your first intervention, monitor for exactly one full batch cycle — not twenty minutes, not two hours — then check where the next pressure point forms. I have seen a team patch a CPU-bound stage, only to watch the database connection pool collapse the next day. The bottleneck moved like a guest changing seats at a crowded dinner table. You have to chase it, not assume it settled.

That sounds exhausting. It's. But the alternative — guessing blind — costs you a day of lost throughput every time the seam blows out.

Bottom Line: Pick the Bottleneck That Costs More Per Hour

Recap the decision framework

You have two screaming bottlenecks, but only one intervention fits your release window. Here is the single heuristic that survived three postmortems I sat through: pick the component that costs more per hour. Not per transaction, not per user complaint—per hour of wall-clock delay. If a blocked merge point stalls four downstream teams while a resource-starved worker chokes on a single queue, the merge point bleeds budget faster. Most teams skip this calculation because they fix what broke last. That is a mistake.

What usually breaks first is the seam you can see—the worker with 100% CPU, the disk queue spiking. Resource contention looks urgent. Merge-point deadlocks look like nothing happened until the build fails three hours later. The catch is: a silent blocker costs more. I have seen a team spend two days optimising a Redis client pool only to discover the merge gate was holding fifty pull requests. The pool fix saved seven milliseconds per call. The merge fix saved half a week.

When to ignore conventional wisdom

Standard advice says fix resource contention first because it affects latency directly. That holds when your system is CPU-bound and the merge point is a simple pass-through. But pipelines with custom logic at the junction—reducers, aggregators, conditional branches—often hide a bigger tax. The odd part is: the resource contention you see might be a symptom, not the cause. A clogged merge point forces workers to hold state longer, which elevates memory pressure, which looks like a memory problem. We fixed this by tracing allocation back to the gate instead of tuning the worker.

Wrong order. Not yet. If you guess wrong, you optimise a red herring while the real blocker infects every downstream stage. That hurts.

“You can tune a worker until it runs at light speed. If the merge point only lets one item through per minute, you just built a faster waiting room.”

— systems engineer, after a week spent chasing the wrong metric

One metric to track from here

Stop measuring utilisation as a percentage. Start measuring end-to-end cycle time per item across the pipeline. Break it into time spent in workers versus time spent waiting at merge points. If wait time exceeds work time by 2×, the merge point is your hourly cost centre regardless of CPU graphs. Track this on a dashboard, not a spreadsheet. I swapped our team’s monitoring from per-node CPU to per-phase dwell time, and the deadlock pattern became obvious within two hours. The next action: freeze all worker tuning, unblock the gate, then measure again. Repeat until wait time is the minority. Simple, boring, effective. Do that.