What to Fix First in a Multi-Tenant Inference Workflow: Request Batching or Model Replication?

So you've got a multi-tenant inference service. Maybe it's a chatbot API used by startups, or a vision model serving e-commerce catalogs. Users pile in—some fast, some slow—and GPU utilization looks like a rollercoaster. Two knobs tempt you. Request batching packs multiple inputs into one forward pass to max out GPU compute. Model replication spins up separate copies of the model to isolate tenants and slash tail latency. Which one do you twist primary? It depends. But most groups burn weeks optimizing the wrong thing. This article gives you a decision framework so you don't.

Who Must Choose and By When?

The clock: latency SLOs vs. overhead targets

The deadline is not next quarter. It's the moment your opening tenant sends a request and the p99 response time breaches 200 milliseconds. I have watched crews treat batching and replication like a menu they can browse at leisure — then their GPU memory fills, latency spikes, and the infra lead gets a 2 AM page. You don't have six weeks to prototype both. The clock runs on two conflicting meters: your latency SLO (service-level objective) and your spend cap. If you can tolerate 50ms of queuing delay per request, batching might buy you throughput cheap. If your SLO demands sub-30ms responses, replicating models across more GPUs is the only move. The catch is most groups discover their true latency floor at peak traffic — and by then, swapping strategies means re-deploying the entire inference graph. That hurts.

The budget is what forces the choice. GPU hours, money, patience. The three resources that never align.

The cast: ML engineers, platform groups, infra leads

This is not a decision for everyone — and that's why it often goes wrong. The ML engineer wants model accuracy and low latency; they will argue for replication because it feels safer. The platform team owns the serving infrastructure and sees idle GPU cycles; they will push batching to squeeze utilization. The infra lead holds the credit card and the SLO contract — they care about expense per inference and uptime. Three roles, three clocks, one bottleneck. The tricky bit is these three rarely talk during the same sprint. I have seen an ML engineer merge a model change that broke the lot scheduler, simply because nobody told them the platform was running dynamic batching with a 128-request window. The fix took two days. The trust took longer.

Who decides? The person who can kill a deployment. Usually the infra lead — but only if they understand the model's compute graph. Otherwise, the platform team holds the keys.

“We replicated primary because latency was sacrosanct. Then we burned through our monthly GPU budget in eleven days.”

— platform engineer, after a misaligned launch

The budget: GPU hours, money, and patience

What breaks opening is not the model — it's the monthly spend forecast. Replication scales linearly: double the GPU count, double the expense, flat latency. Batching scales sub-linearly: you pay for memory but save on compute, yet latency grows with lot size. The trap is thinking you can afford replication indefinitely. Most multi-tenant setups start with 1–2 GPUs per model; at ten tenants, replication demands 10–20 GPUs. That's a $30,000–$60,000 monthly bill before you serve a single happy customer. Batching, by contrast, might hold 10 tenants on 2 GPUs — but the p99 jumps from 15ms to 120ms the moment a heavy group arrives. Patience runs out when the initial tenant complains. The real question: can your SLO tolerate a 6× latency variance? If yes, run. If no, replicate — but only until you hit the expense ceiling, then you must run anyway.

Wrong order? You lose a day. Skip the step entirely? You lose the tenant.

Option Landscape: What’s on the Table?

Dynamic batching with timeout heuristics

Most crews start here because it’s the cheapest lever to pull. You configure an inference server—say, Triton or TorchServe—to collect incoming requests into a single group until either the lot reaches a maximum size or a timeout fires, whichever hits initial. The trick is the timeout: set it too short and you lot only one or two requests, defeating the purpose; set it too long and the slowest tenant in the pool makes everyone wait. I once watched a team set a 50-millisecond window and still miss the P99 latency target by 200 milliseconds—because one tenant’s payloads were tiny while another’s were enormous. The mechanic is simple: you pack more work into each GPU cycle, raising throughput per card. The catch? It only works when tenants arrive fast enough to fill the bucket. Sparse traffic? You pay the latency tax for a near-empty lot.

That hurts when you’re wrong.

Per-tenant model replicas (K8s + pod per tenant)

Here you spin up one dedicated inference pod per tenant, each pod loading its own copy of the model into GPU memory. Kubernetes handles the rolling deploys; you pin each replica to a specific node or namespace. The isolation is beautiful—no noisy neighbor can spike your tenant’s latency because another tenant’s request floods the group queue. I have seen a fintech operation run twelve replicas for twelve clients, each hitting a consistent 15-millisecond response time. But the GPU memory spend multiplies: twelve replicas means twelve copies of the model parameters, and if your model is 7 GB, that’s 84 GB just for weights. The mechanic is straightforward—parallelism over sharing—but the waste is real. Most crews underestimate the memory overhead until the cloud bill arrives. Wrong order: buying replicas before confirming batching can't saturate your GPU utilization.

Not yet a solution for more than a handful of tenants.

Hybrid: batched replicas with tenant-level queues

This approach acknowledges that neither extreme works alone. You run a small number of replicas—say, four—each handling a group of requests, but you insert a per-tenant queue in front of the lot scheduler. When tenant A sends a burst, their requests queue separately; the scheduler draws from each queue round-robin, batching the collected requests from multiple tenants together. The mechanic: you keep the GPU utilization high (because batches cross tenants) while preventing any single tenant from starving others (because queues drain independently). The odd part is—the tuning is brutal. Queue depth, lot timeout, replica count—three knobs that interact nonlinearly. We fixed this by adding a priority watermark: if a tenant’s queue exceeds 50 requests, that tenant’s requests jump the line for the next lot. That broke the fairness guarantee for two weeks until we rebalanced. The trade-off? Higher engineering spend for a system that can scale from three tenants to forty without rewriting the scheduler.

‘Batching buys you throughput; replication buys you isolation. The hybrid buys you a headache—but also a ceiling lift.’

— senior infra lead, postmortem chat

Flag this for machine: shortcuts cost a day.

Apiary supers, queen cages, smoker fuel, varroa boards, and nectar flows punish calendar-only beekeeping.

Ember nexus clamps seize overnight.

Flag this for machine: shortcuts cost a day.

Not covered: model parallelism, pruning, quantization

These techniques matter—but they target a different bottleneck. Sharding a model across GPUs (model parallelism) or shrinking its footprint (pruning, quantization) reduces per-request compute cost. That's a separate optimization pass, one you apply after you have stabilized the batching-vs-replication decision. Trying to shrink a model before you know whether your latency problem is caused by queue congestion or compute overload wastes weeks. Choose your primary fix from the options above; reshape the model only when the workflow is otherwise stable and you need the last 20% of latency reduction. Most groups skip this order—and then wonder why quantizing a 13B model still leaves P99 at 400 milliseconds. It was never the math; it was the wait.

Comparison Criteria: How to Pick Your opening Fix

Latency SLOs: P50 vs. P99 breakpoints

The fastest path to a wrong choice is treating all latencies as equal. I have seen crews chase P50 improvements while their P99 tails were already hitting ten-second cliffs — and wondering why users were complaining. Batching shines when your latency budget is relaxed at the tail: you can pack more requests into a single GPU pass, accept a 30–50% increase in P99, but keep P50 flat. Your users might not even notice. Replication, by contrast, is the only sane play when your SLO demands sub-100ms response times across both median and tail. It trims queuing delay at the cost of idle GPU cycles. The catch is—most teams discover their actual P99 breakpoint only after they ship. Run a controlled experiment: push group size until P99 crosses your threshold, then count how many replicas that buys you. That number tells you where to fix opening.

Wrong order is expensive. If you replicate before batching, you burn memory on copies that could have been one larger run.

GPU memory pressure: run size vs. number of replicas

Memory is the invisible ceiling. A single replica with group size 32 might fit inside 16 GB VRAM — barely. Push to 64 and you OOM instantly. Replication avoids that ceiling by spreading load, but each replica carves out its own memory island. I have watched teams burn through a cluster budget simply because they replicated opening, then had no room for larger batches later. The diagnostic is straightforward: check your `torch.cuda.max_memory_allocated` versus total VRAM. If you have 20% headroom, lot initial. If you're pinned at 95% utilization with lot size 8, you can't grow — replicate. One concrete anecdote: a team serving a 7B parameter model ran out of memory at lot size 16 on A10G cards. They added four replicas, slashed latency, and only later realized that halving the model precision would have let them group 32. That took three hours to fix.

Memory pressure also dictates cascading trade-offs. Larger batches demand contiguous VRAM blocks — fragmentation can kill you faster than total capacity.

Tenant isolation: noisy neighbors and fairness

The dirty secret of multi-tenant inference is that one aggressive tenant can poison the well for everyone. Batching treats all requests equally: a single client sending 200 concurrent calls will push smaller tenants into the same queue. Replication gives you per-tenant islands — assign one replica to high-priority traffic, another to bursty group jobs. That isolation comes at a cost: you lose statistical multiplexing, so average GPU utilization drops from 85% to maybe 60%. The trick is identifying your noisiest tenant. Measure request arrival variance per tenant over a 24-hour window. If one tenant spikes 10× above its baseline, replication buys you a firewall. If variance is uniform across tenants, batching plus a weighted queue is simpler and cheaper.

‘Noisy tenants don’t break your system; they expose where you chose batching over isolation.’

— engineering lead after a weekend PagerDuty storm

Fairness is not a toggle — it's a sliding scale of operational pain. Replication gives perfect fairness and mediocre utilization. Batching gives great utilization and ugly fairness when traffic tilts.

Operational complexity: knobs to tune

Every knob is a future incident. Batching introduces three tuning levers: max lot size, batch timeout, and queuing policy. Tune them wrong and you either starve slow requests or waste GPU cycles waiting for a full batch. Replication adds at least five: number of replicas, autoscaling threshold, cooldown period, traffic router weights, and health-check semantics. I have seen a team spend two weeks debugging a replica that kept crashing because the health-check interval was too short — and they never needed it. The editorial rule of thumb: if your team has fewer than three people on inference, batch primary. You can get batching right in an afternoon. Replication takes days and a decent observability stack. That said, replication’s complexity is up-front; once tuned, it mostly runs. Batching’s complexity is recurring — every model update, every traffic shift, every GPU driver upgrade can shift the optimal batch size.

Which beast can you afford to feed? Choose the one whose failure mode you can debug at 3 AM.

Trade-Offs: When Batching Wins and When Replication Does

High GPU compute utilization: batching primary

If your GPU is sitting at 40% utilization during inference—idle cycles bleeding money—the fix is almost always request batching. I have seen teams burn weeks on replica tuning when the real culprit was a single inefficient forward pass. Batching coalesces those idle GPU threads into dense matrix operations. The catch is that batching adds latency: you wait for enough requests to fill the batch window. For workloads where GPU compute is the bottleneck (think dense transformer decoders, image generation, or large embedding models), batching wins by a landslide. You go from 40% utilization to 85%, and total throughput doubles or triples. That hurts—in a good way.

Wrong order? Batching opening. Replication second.

Strict per-tenant P99 latency: replication primary

Now flip the script. Your multi-tenant SLA demands every tenant sees a P99 under 50ms, and one tenant’s flood of requests must not starve another’s. Batching here is a trap—packing four tenant-A requests behind three tenant-B requests inflates tail latency for both. Replication gives each tenant its own dedicated replica or a shared pool with strict concurrency limits. The odd part is—this feels wasteful. More GPUs, higher cost, lower aggregate utilization. But when the penalty for a single P99 spike is a lost enterprise customer, replication is the cheaper bet. I fixed this exact problem at a previous gig: we had a single replica with dynamic batching hitting 80% utilization but blowing past 200ms every third minute. Splitting into three per-tenant replicas—each with conservative batching—brought P99 back under 40ms. Utilization dropped to 55%, but churn dropped to zero.

‘Batching optimizes the machine. Replication optimizes the tenant. Pick which one you're willing to upset.’

— overheard from an infrastructure lead, mid-postmortem

Merchandisers, technologists, sourcers, coordinators, auditors, and sample sewers interpret the same sketch with different priorities.

Mycelium agar plates collapse overnight.

Odd bit about learning: the dull step fails opening.

Odd bit about learning: the dull step fails initial.

Mixed workloads: hybrid with careful queue sizing

Reality rarely picks one extreme. You have a chat model (bursty, latency-sensitive) running alongside a batch scoring job (steady, throughput-hungry). The hybrid path: replicate the chat model across two lightweight GPUs with tiny batch windows (max 4, flush every 20ms), then route the scoring job to a single large GPU with aggressive batching (batch size 64, window 200ms). The tricky bit is queue sizing. Let the chat queue spill into the scoring GPU, and you defeat the purpose—the scoring batch stalls, the chat latency spikes. Set per-queue capacity with a hard cap. When the chat queue hits its limit, reject or spill to a fallback, never let it bleed into the throughput pool. Most teams skip this: they merge both workloads onto one queue and wonder why neither SLA holds. Don't.

That said, a simple decision matrix helps: if your workload has

• high GPU utilization + loose latency → batching initial
• low utilization + tight latency → replication primary
• high utilization + tight latency → hybrid with per-workload replicas

The fourth cell—low utilization plus loose latency—means you have a different problem (wrong model size, bad hardware), and neither batching nor replication will save you.

Cost cap: batching (fewer GPUs) vs. replication (more GPUs)

When your budget is the wall you hit opening, batching is the orthodox choice. Fewer GPUs, better per-card throughput, lower absolute spend. Replication burns dollars linearly—each new replica is another GPU rental. The catch is that cost caps often hide latency requirements. A common pattern: leadership sets a $5k monthly GPU budget, and the team optimizes for throughput to hit that number, only to discover the real constraint is P99 under 100ms. I have seen this exact squint happen three times. The fix is not to choose batching over replication; it's to calculate the cost-per-tenant under each strategy. Batching might push 100 tenants through 2 GPUs at $2k/month, but each tenant's P99 sits at 200ms. Replication pushes 10 tenants per GPU across 5 GPUs at $5k/month, but P99 stays under 30ms. Which number does your contract penalize? If the SLA fine for missing P99 exceeds the extra $3k, replication wins. Do the math before you buy the GPUs.

Implementation Path After You Choose

How to implement dynamic batching with CUDA graphs

Start by instrumenting your inference server to collect request inter-arrival times per tenant. I have seen teams skip this and hardcode a 50ms timeout — that burns throughput when traffic is bursty and inflates latency when it’s sparse. The better path: set a maximum batch size that fits your GPU memory without spilling to host, then expose a configurable `max_batch_time` that defaults to a conservative 20ms. Integrate that with CUDA graphs by recording a fixed graph for each batch size you expect (1, 2, 4, 8, 16) and replaying the closest match rather than rebuilding the graph per request. What usually breaks opening is the kernel launch overhead — CUDA graphs trim that by 80% in my experience, but only if you pre-compile for the actual shapes you see, not the ones you assumed. You will want to pin tensor memory once, at startup, to avoid cudaMalloc stalls inside the hot path. The catch is that mixed tenant queries with wildly different sequence lengths kill the graph reuse benefit; padding to a common length is the common fix, but that wastes FLOPS. Monitor your batch utilization ratio — if it drops below 0.6 after two weeks, your timeout is too conservative or your traffic pattern shifted.

How to set up per-tenant model replicas using Kubernetes

opening, assign each tenant a dedicated deployment with a `minReplicas: 1` and a `targetCPUUtilizationPercentage` of 70 on the HPA. Don't share a single deployment with pod anti-affinity — that gives you co-location, not isolation. The odd part is that many teams replicate the entire model per tenant even when the model weights are identical; a better approach is a shared weight store mapped into each pod via a read-only hostPath, cutting memory overhead by 5–8x. Then layer a gRPC load balancer per tenant prefix (e.g., `/v1/tenant-a/*`) so requests never route to the wrong replica. Most teams skip this: set a `terminationGracePeriodSeconds` of 10 seconds and enable the Kubernetes preStop hook to drain in-flight requests. Without that, you drop 3–5% of requests during rolling updates. We fixed this by adding a readiness probe that fails if the batch queue depth exceeds 100 — not perfect, but it prevents cascading overshoot when one replica falls behind.

Now monitor replica count variance across tenants. If Tenant A scales from 1 to 6 pods while Tenant B sits idle, your HPA thresholds are too sensitive or you need per-tenant CPU reservations. I have seen teams chase replica counts blindly, adding pods until the interconnect becomes the bottleneck — then latency doubles because of network noise. The synthetic benchmark said 3ms p99, the production trace said 17ms. That hurts. Set a hard cap per tenant of 8 replicas until you validate the node-level throughput ceiling.

'Replication without batching is like buying more buses but never filling them — you just park empty buses at the curb.'

— overheard during a post‑mortem at a GPU‑as‑a‑service startup, after they tripled their inference cost for zero throughput gain.

How to monitor and tune batch timeouts or replica counts

Start with two dashboards: one for batch formation delay (p50, p95, max) and one for per-replica request queue depth. If your batch timeout is 30ms but the p95 formation delay is 2ms, you're waiting too long — cut the timeout to 8ms and watch the throughput climb. Conversely, if queue depth on a single replica hits 200 while batch timeout fires every request, you're under-replicated. The tricky bit is that these metrics interact: lowering batch timeout forces more frequent, smaller batches, which raises queue depth, which triggers HPA to scale out, which reduces batch opportunities again. You oscillate. We fix this by introducing a cooldown window of 90 seconds on the HPA and a batch timeout floor of 5ms — no lower, or the GPU utilization collapses below 30%. Collect data for two business cycles before changing both knobs at once; change one, wait a day, review. The second iteration is always better than the initial because the primary guess is always wrong.

Risks of Choosing Wrong or Skipping Steps

Batching-first with latency-sensitive tenants — the SLO betrayal

You decide to max out batch size before adding replicas. Throughput looks great on the dashboard. The odd part is—your high-priority tenant starts timing out during peak hours. What breaks? Batch accumulation adds a fixed latency floor. A 50ms batch window feels harmless until a stream of real-time requests arrives and every inference waits for the buffer to fill. Now your p99 spikes from 80ms to 220ms. The SLO says 150ms. You’re violating it daily by 11 AM.

The signs are specific: your batch size curves look smooth but your latency percentiles diverge. P50 stays flat; p99 climbs like a hockey stick. I have seen teams re-prioritize for two weeks before admitting the root cause — they stacked requests waiting for throughput that never arrived for that one critical tenant. The fix hurts — you flush partial batches early or you add a replica just for the latency-sensitive route. But by then you’ve already burned a sprint.

Watch for this: if your dynamic batching timeout is longer than your tightest tenant SLO, you're already in trouble. Don’t set a single batch window for all traffic.

“Batching first feels safe because throughput rises. But throughput doesn't equal latency. You serve the average, but the outlier tenant pays.”

— incident postmortem, two-team inference platform

Beekeeping nucs, drone frames, honey supers, entrance reducers, and oxalic dribbles each need a calendar and a nose.

Zinc quinoa glyph marks stock.

Replication-first with memory-bound GPUs — OOMs and thrashing

So you spin up replicas for isolation. Each model copy gets its own GPU. Sounds clean. Until your memory budget says ‘no’. Replication multiplies the memory footprint linearly — two replicas double the VRAM demand. If your model barely fits on one card, three replicas mean swapping or crashing. The GPU starts thrashing: inference latency jumps as the driver evicts and reloads weights. You get OOMs during rolling deploys. Worse, silent corruption when memory pressure forces lower precision without a fallback plan.

Reality check: name the learning owner or stop.

Reality check: name the learning owner or stop.

What usually breaks first is the cache. KV-cache memory for each replica competes for the same pool. The catch is—your monitoring shows GPU utilization at 95% but you miss that the cache hit rate dropped from 80% to 40%. Every miss costs a recompute. A single replica with shared batching might have outperformed three starving replicas.

Most teams skip this: they add replicas until the first OutOfMemory error, then tweak batch size downward. Wrong order. Profile your memory first. If your model uses >70% of VRAM at batch size 1, replication will bite you before batching does. The sign to watch is increasing ‘timeout waiting for memory’ logs — not just OOMs. Those logs mean your GPUs are thrashing, not serving.

Skipping load testing — silent regressions that compound

You chose batching. Or replication. Either way, you skipped the load test. “We’ll fix it in prod.” That sentence should terrify you. Without a realistic traffic pattern against your multi-tenant mix, you can't see how the two choices interact. Maybe batching helps one tenant but starves another because the request size distribution is bimodal. Maybe replication isolates noisy neighbors but the memory overhead pushes your batch size so low that throughput collapses. You won’t know until pager duty calls at 2 AM.

The dangerous part is silence. No crash. No OOM. Just a slow creep — inference times increase 5ms per week, then 15ms. Your SLO breach happens gradually. By the time you notice, the regressions have compounded across four deployments. Rollback? Hard, because the config changed alongside the model version. You can't untangle what caused what.

I fixed one incident where the team had zero test data simulating three tenants sending requests at different rates. They had one synthetic benchmark — uniform Poisson arrivals. Real traffic was bursty, with one tenant firing 200 requests in a second then idling for five minutes. Batching-first ruined latency for that bursty tenant. Replication would have handled it. The synthetic test showed neither failure mode. Run load tests that mirror the actual request inter-arrival distribution — not a flat average. Your first fix depends on knowing that shape.

Not considering cost — the bill shock that ends projects

Batching looks cheap on paper. You pack more requests per dollar of compute. But if you over-batch, you need more memory per request, which drives up instance tier cost. Replication looks expensive — more GPUs, more licenses. Yet replication can reduce memory fragmentation and let you use cheaper, smaller instances. I have seen a team choose batching to “save money,” quadruple their batch size, then hit memory limits that forced them into A100s when T4s would have sufficed with two replicas. Their monthly bill doubled.

The real risk isn’t choosing wrong — it’s choosing without cost modeling. Without a per-tenant cost breakdown, you can't see that batch-first saves the platform team money but bleeds the latency-sensitive tenant’s department budget through retry fees and lost user trust. Replication-first costs more upfront but avoids those hidden costs. The sign to watch for is a finance report that shows inference cost growing faster than request volume. That ratio should be flat or declining. If it climbs, your fix is bleeding cash.

Build a simple cost-per-inference model before you pick your first fix. Include the price of retries, SLO penalty clauses, and engineer hours spent debugging thrashing. Then decide. Skip that step and you get a surprise — not the good kind.

Mini-FAQ: Quick Answers on Batching vs. Replication

Can I do both at the same time?

Yes, but don't. That sounds like a power move — batching and replication running in parallel from day one. In practice, I have watched teams burn two weeks tuning both knobs simultaneously, then realize they fixed a throughput problem that didn't exist yet. The catch is you can't isolate whether batch size or replica count caused a latency spike. Wrong order. Start with one, measure hard, then add the other. Most multi-tenant workflows I debugged hit a clear bottleneck first: either the GPU sits idle between requests (batching wins) or queue depth explodes under tenant spikes (replication wins). Do both only after you have three days of steady-state metrics from the first fix.

What about model parallelism, should I consider that first?

Not yet. Model parallelism splits a single large model across devices — think 70B-parameter monsters that can't fit one GPU. If your inference workload fits comfortably inside a single A100 or H100, model parallelism adds overhead without benefit. The tricky bit is teams confuse it with replication. Replication copies the whole model; model parallelism carves it up. For a multi-tenant setup where each tenant's model is under 20B parameters, model parallelism is a distraction. I have seen one team spend a month sharding a 13B model across two T4 cards, only to find simple replication cut their p95 latency by 40% in three hours. That hurts. Save model parallelism for the day your largest tenant demands a model that genuinely exceeds VRAM — until then, fix batching or replication first.

How do I choose batch timeout values for multi-tenant?

Set it too low and you batch zero requests — no gain. Too high and the first tenant in line waits forever while you collect stragglers. The rule of thumb I use: start with 50 milliseconds per tenant class, then measure p99 tail latency over one production hour. If your p99 request has already waited 40ms in the queue before the batch fires, you're losing the speed advantage that batching was supposed to buy. What usually breaks first is tenant heterogeneity — one client sends 1KB images, another sends 4MB documents. They should not share the same batch window. Split tenants into two pools, each with its own timeout (maybe 30ms for the fast pool, 80ms for the heavy pool). That single change cut our p95 from 2.3 seconds to 890ms on a recent project. The alternative? A uniform timeout that punishes fast tenants for the sake of slow ones — poor trade-off.

What metrics should I track to know if I chose right?

Three numbers. First: GPU utilization during idle windows — if it sits below 30% between tenant bursts, replication is adding cold silicon you pay for but don't use. Second: end-to-end latency per tenant class at p50 and p99 — batching smooths p50 but can spike p99 if timeout is aggressive. Third: queue depth over a sliding 5-minute window. If queue depth exceeds your replica count by more than 3× during peak, replication is underprovisioned. If depth stays near zero while GPU utilization crawls, batching is the missing fix. I track these on a single dashboard before and after each change. One team I advised ignored queue depth entirely, added three replicas, and still saw timeouts — the root cause was a batch timeout set to 500ms, drowning throughput. Metrics told the story. They just refused to read it.

‘We doubled replicas and latency got worse. Turns out each replica was starving for requests to batch — more copies, same throughput.’

— Anecdote from a production postmortem, lightly edited for clarity

Bottom line: pick batching if your GPU has headroom and tenants trickle in. Pick replication if concurrent tenant demand exceeds what one model instance can handle without queue blow-up. Track utilization, latency, and queue depth for two business days. If those metrics converge toward a sane steady state, you chose right. If they wander — swap your approach. That's the whole decision loop. No magic. Just three numbers and a willingness to revert fast.