All posts

Engineering

The Arithmetic of Inference: How a Modern Inference API Actually Spends Your FLOPs

Jul 8, 2026 The Pipe team 13 min read

A technical deep-dive from the team behind Pipe.

A note on numbers. Unless a figure is attributed to a specific published source with a link, treat every quantitative example in this post as illustrative — order-of-magnitude figures chosen to make the arithmetic legible, not measurements from a specific deployment. Where we cite real published results, we link them inline.

Most writing about LLM serving reaches for the same three words — "fast," "scalable," "efficient" — and stops there. This post is for people who want the layer underneath: the roofline arithmetic that decides whether a token is cheap or expensive, the memory accounting that governs how many requests share a GPU, and the scheduling and routing decisions that turn a pile of accelerators into a serving system. If you build models, train draft heads, or design serving stacks, this is the mental model we use at Pipe, written out in full.

We'll move in two arcs. First, single-replica inference: why a forward pass costs what it costs, and the techniques — continuous batching, PagedAttention, chunked prefill, speculative decoding, prefill/decode disaggregation, low-precision arithmetic — that bend the cost curve. Second, the serving system: how requests are routed to warm KV caches, how the fleet scales against latency objectives, and how a disaggregated cluster is actually wired.

Part I — The physics of a single forward pass

Two regimes, one model

An autoregressive transformer serves a request in two phases with completely different performance characteristics, and conflating them is the single most common analytical error in this space.

Prefill ingests the prompt. Every prompt token is processed in parallel through the network in one shot, producing the KV cache for the whole prompt and the first output token. Because you are multiplying large activation matrices against the weights, prefill is compute-bound: it saturates the tensor cores.

Decode generates output tokens one at a time. Each step processes exactly one new token per sequence, reads the entire KV cache to attend over history, and — critically — must stream all of the model's weights out of HBM to compute a single token's worth of work. Decode is memory-bandwidth-bound: the tensor cores sit mostly idle waiting on memory.

The cleanest way to see this is the roofline model. Define arithmetic intensity as FLOPs performed per byte moved from HBM. A GPU has a ridge point — the intensity at which it transitions from bandwidth-limited to compute-limited — equal to peak-FLOP/s divided by memory-bandwidth. For an H100 SXM (roughly 989 dense FP16 TFLOP/s over 3.35 TB/s of HBM), the ridge point sits near ~295 FLOP/byte. Anything below that line is leaving compute on the table because it can't feed the ALUs fast enough.

Now do the accounting for decode at batch size 1 on a 70B model. A forward pass does about 2 × N_params FLOPs per token (~140 GFLOP) and must read about 2 × N_params bytes of FP16 weights (~140 GB). That's an arithmetic intensity of roughly 2 FLOP/byte — nearly two orders of magnitude below the ridge point. The GPU is a Ferrari in a traffic jam.

This single fact drives almost everything that follows. The entire discipline of decode optimization is a fight to raise arithmetic intensity — to make each expensive trip to HBM pay for more useful arithmetic — without blowing the latency budget or the memory budget.

The KV cache is the real resource

The lever that raises decode intensity is batching: if you stream the weights out of HBM once and apply them to B sequences at once, you amortize the weight-read across B tokens and intensity scales roughly linearly with batch size until you approach the ridge point. Batching is free FLOPs — right up until you run out of memory for the KV cache.

Here is the KV cache size for a single sequence, which every serving engineer should be able to write from memory:

kv_bytes = 2 × n_layers × n_kv_heads × d_head × seq_len × dtype_bytes
             │
             └─ the 2 is for K and V

For a Llama-70B-class model with grouped-query attention (80 layers, 8 KV heads, head dim 128) in FP16, that's:

2 × 80 × 8 × 128 × 2 bytes  =  327,680 bytes ≈ 320 KiB  per token

So an 8K-token context costs about 2.5 GiB of HBM for one sequence. On an 80 GB H100, after the ~140 GB of a 70B model has already been sharded across GPUs, KV cache is the binding constraint on how many concurrent sequences you can hold — and therefore on your maximum batch size, and therefore on your throughput. (Figures illustrative; exact values depend on model config, sharding, and reserved memory.)

Everything in the "how many requests can this box serve" question reduces to KV cache accounting. The optimizations below are, to a first approximation, all about using this scarce memory more cleverly.

Continuous batching: schedule at the token, not the request

Static batching — collect B requests, run them together to completion — wastes enormous capacity because sequences finish at different lengths. The moment the shortest sequence emits its stop token, its slot goes idle while the batch waits for the longest one, and a batch of 32 can degrade to an effective batch of a handful near the tail.

Continuous batching (also called in-flight batching) moves scheduling to the iteration level. At every decode step the scheduler re-examines the running set: finished sequences are evicted and their KV blocks freed; waiting requests are admitted into the freed slots. The batch is rebuilt every token. This keeps the GPU near its target occupancy continuously rather than in sawtooth bursts, and it is the foundational trick behind every modern engine. The vLLM anatomy writeup is the canonical description of how this is implemented in practice.

PagedAttention: virtual memory for the KV cache

Continuous batching creates a memory-management problem. If each sequence's KV cache must live in one contiguous HBM region sized to its maximum possible length, you fragment memory horribly and reserve capacity you'll almost never use.

PagedAttention borrows the operating-system solution wholesale. The KV cache is chopped into fixed-size blocks (say, 16 tokens each). A per-sequence block table maps logical token positions to physical blocks scattered anywhere in HBM — exactly like OS page tables map virtual to physical memory. Sequences grow by allocating one block at a time, so there's no need to reserve for the worst case, internal fragmentation is capped at one block, and near-100% of KV memory becomes usable. As a bonus, blocks can be shared across sequences via copy-on-write: two requests with a common prompt prefix point at the same physical blocks until one of them diverges. That sharing is the seed of prefix caching, which we return to in Part II. See the vLLM overview for the lineage.

Chunked prefill: stop letting long prompts stall the fleet

Prefill and decode fight over the same GPU. A long prompt — say 32K tokens — occupies the tensor cores for a substantial, uninterruptible window. If that prefill runs as one monolithic step, every decode request already in flight stalls behind it, and your inter-token latency spikes for everyone sharing the replica.

Chunked prefill splits the prompt into fixed-size chunks and interleaves them with ongoing decode work, forming hybrid batches: a slice of prefill plus a stack of decode tokens in the same step. Because prefill is compute-bound and decode is memory-bound, the two actually complement each other — the decode tokens soak up memory bandwidth while the prefill chunk soaks up compute, pushing overall utilization up while keeping decode latency smooth. Per the Spheron H100 analysis, the continuous-batching + PagedAttention + chunked-prefill combination reaches roughly 2,200–2,400 tokens/s for Llama 3.3 70B in FP8 at 128+ concurrent requests on an H100, about 3–5× a naive PyTorch loop.

Speculative decoding: buy tokens on credit

Decode is memory-bound, which means the GPU has spare compute at every step. Speculative decoding spends that idle compute to attack the fundamental serialization of autoregression.

A small, cheap draft model (or a lightweight draft head grafted onto the target model) proposes k tokens ahead. The large target model then verifies all k proposals in a single parallel forward pass — the same cost as generating one token, because verification is just one more compute-bound matmul over a slightly longer sequence, and compute is exactly what decode has to spare. A rejection-sampling check guarantees the output distribution is identical to the target model's: speculative decoding is lossless, not an approximation.

The expected speedup is governed by the acceptance rate α and the draft length. If each verification step accepts on average E[accepted] tokens, you get that many target-quality tokens for the price of one target forward pass (plus the draft's cost). EAGLE-3 (NeurIPS 2025), which fuses features from multiple layers of the target to draft better, reports acceptance rates around 70–80% across positions and end-to-end speedups reported in the 1.8×–6.5× range depending on task, batch size, and sampling settings — with the largest gains at small batch sizes where the GPU is most starved for work, and smaller gains at large batch where decode is already compute-saturated. That batch-size dependence is the subtle part: speculation and batching are competing ways to spend the same idle compute, so their benefits don't simply add.

Prefill/decode disaggregation: stop making one GPU do both jobs

We've established that prefill is compute-bound and decode is memory-bound. Running both phases on the same GPU pool forces a single hardware configuration and a single scheduler to serve two workloads with opposite bottlenecks — and the interference is exactly the stall problem chunked prefill only partially hides.

PD disaggregation splits them across different GPU pools. Prefill workers, tuned for compute, ingest prompts and produce KV caches; decode workers, tuned for memory bandwidth and capacity, stream out tokens. The KV cache computed during prefill is transferred over the interconnect to a decode worker, which then runs the generation loop. Each pool can be sized, batched, and even built from different hardware independently, and each phase gets a scheduler tuned to its own latency objective.

The honest framing — from the Hao AI Lab retrospective, one of the groups that originated the idea — is that "disaggregated prefill does not improve throughput" per se. The wins are latency control and cost efficiency through better utilization: you can hit tight time-to-first-token and inter-token-latency targets simultaneously instead of trading one against the other, and you stop paying for compute-heavy GPUs to do memory-bound decode. The catch is the KV transfer: moving gigabytes of cache between pools demands fast interconnect and careful scheduling, which is why systems like Mooncake make the KV cache itself the center of the architecture.

Low precision: the cheapest FLOPs are the ones you round off

Since decode is bottlenecked on moving weights out of HBM, the most direct lever is to make the weights smaller. Halving precision roughly halves the bytes moved per token and — on hardware with matching tensor cores — roughly doubles arithmetic throughput.

The progression is now well-established in production. FP8 is the default production inference precision on recent NVIDIA hardware, with quality degradation from FP16 typically in the 0.5–2% range on standard benchmarks. FP4 goes further, and this is where microscaling formats matter: naive 4-bit rounding is destroyed by activation outliers, so formats like NVFP4 attach an FP8 scale to every 16-value block plus a global FP32 tensor scale, which recovers most of the lost precision. NVIDIA reports DeepSeek-R1's MMLU dropping only 0.1% (90.8% → 90.7%) moving from FP8 to NVFP4, within measurement noise — see the Edge AI & Vision writeup on NVFP4. On Blackwell, FP4 tensor cores deliver roughly 2× the throughput of FP8 on the same silicon. The engineering discipline is knowing what to quantize: weights tolerate aggressive quantization, some activations and the KV cache are touchier, and outlier-heavy layers often stay in higher precision.

Putting Part I together

None of these techniques is independent. Speculative decoding's payoff shrinks as batch size grows; batch size is capped by KV memory; KV memory is stretched by PagedAttention and quantization; chunked prefill and PD disaggregation both exist to keep prefill from poisoning decode latency. The job of an inference platform is not to apply a checklist — it's to find the operating point on this coupled surface that satisfies each workload's latency objective at the lowest cost per token. Which brings us to the system.

Part II — The serving system

A single well-tuned replica is table stakes. The harder, and more interesting, engineering is turning a fleet of them into a service that holds latency objectives under real, skewed, bursty traffic. Three problems dominate: where to send each request, how many replicas to run, and how to wire the cluster.

Routing is a cache-hit problem, not a load-balancing problem

The instinct from web serving is to route for even load — least-connections, round-robin. For LLM inference that instinct is actively wrong, because the dominant cost variable is whether the target replica already holds the relevant KV cache.

Consider the common shapes of real traffic: a shared system prompt across every request; a multi-turn chat where turn N shares the entire prefix of turns 1..N-1; a coding agent hammering the same large file context across dozens of calls. In all of these, a request routed to a replica that already has the prefix cached skips prefill for the shared portion entirely — turning thousands of tokens of compute-bound work into a cache lookup. Route the same request to a cold replica and you pay full prefill.

So the router's real objective is prefix-cache affinity: send each request to a worker likely to already hold its prefix, while preserving enough load balance to avoid hotspots. These goals conflict — perfect affinity would pile all shared-prefix traffic onto one poor replica — so the router solves a live optimization over a cluster-wide, necessarily-stale view of which prefixes live where. This is the design behind production systems like the llm-d intelligent scheduler and Google's GKE Inference Gateway with cache-aware routing.

A sketch of the scoring logic a KV-aware router runs per request — in Rust, since that's the kind of hot-path, tail-latency-sensitive component you want in a systems language:

/// Score a candidate replica for a request. Higher is better.
/// Balances prefix-cache affinity against current load.
fn score_replica(req: &Request, replica: &ReplicaState, w: &Weights) -> f64 {
    // How many tokens of this request's prefix are already cached here?
    // Computed against a radix tree of known prefixes per replica.
    let cached_prefix = replica.prefix_overlap(&req.prompt_hash_chain);
    let affinity = cached_prefix as f64 / req.prompt_len.max(1) as f64; // 0.0..=1.0

    // Load signals: queue depth and KV-cache pressure both push us away.
    let queue_penalty = replica.pending_tokens as f64 / w.queue_norm;
    let kv_penalty    = replica.kv_utilization;                 // 0.0..=1.0

    w.affinity * affinity - w.queue * queue_penalty - w.kv * kv_penalty
}

fn pick(req: &Request, fleet: &[ReplicaState], w: &Weights) -> usize {
    fleet
        .iter()
        .enumerate()
        .max_by(|(_, a), (_, b)| {
            score_replica(req, a, w)
                .partial_cmp(&score_replica(req, b, w))
                .unwrap()
        })
        .map(|(i, _)| i)
        .expect("non-empty fleet")
}

The subtleties live in the details this sketch elides: the prefix overlap is computed against a radix tree of prompt-hash chains so that partial matches score proportionally; the cache metadata is stale (replicas evict blocks between heartbeats) so the router must tolerate misroutes gracefully; and the weights themselves are tuned against live SLO attainment, not set once.

Autoscaling against SLOs, not CPU

Web autoscalers scale on CPU utilization. That signal is meaningless here — a decode-bound replica can be 100% "busy" moving memory while its tensor cores idle. Inference autoscaling must be driven by the latency objectives that actually matter:

  • TTFT (time to first token) — dominated by prefill and by queueing delay before admission. Governs perceived responsiveness.
  • ITL / TPOT (inter-token latency / time per output token) — dominated by decode batch dynamics. Governs streaming smoothness.

These decompose cleanly onto the disaggregated architecture: TTFT is mostly a prefill-pool property, ITL is mostly a decode-pool property, so the two pools scale on different signals against different targets. When the prefill queue's projected wait threatens the TTFT target, add prefill workers; when decode batches grow large enough to stretch ITL, add decode workers. The scaling controller is watching queue depth and projected SLO attainment, not a coarse utilization gauge.

Two structural facts make this genuinely hard. First, cold start: spinning up a replica means loading tens of gigabytes of weights into HBM, so reactive scaling always lags a burst — you need predictive provisioning against demand patterns, and warm pools to absorb the lag. Second, scaling interacts with routing: adding a cold replica helps load but hurts cache affinity until it warms up, so a good controller accounts for the affinity cost of the very capacity it's adding.

The shape of a disaggregated cluster

Assembling Part I and Part II, a production inference cluster looks less like a homogeneous farm and more like a small distributed system with specialized tiers:

                         ┌──────────────────────────────┐
        request ───────▶ │  KV-aware router / scheduler  │
                         │  (prefix radix tree + load)   │
                         └───────────────┬──────────────┘
                                         │
                 ┌───────────────────────┴───────────────────────┐
                 ▼                                                 ▼
        ┌─────────────────┐        KV cache transfer     ┌──────────────────┐
        │  PREFILL POOL   │ ───────────────────────────▶ │   DECODE POOL    │
        │ compute-optim*  │      (fast interconnect)      │ bandwidth/cap-*  │
        │ chunked prefill │                               │ continuous batch │
        │ prefix cache    │                               │ paged attention  │
        └─────────────────┘                               │ speculative dec. │
                 ▲                                         └────────┬─────────┘
                 │                                                  │
                 └───────────── shared KV cache tier ───────────────┘
                        (HBM ▸ DRAM ▸ SSD, offload & reuse)
        * pools sized, batched, and scaled independently against TTFT vs ITL

The KV cache spans a memory hierarchy — hot blocks in HBM, warm blocks spilled to host DRAM, cold-but-reusable blocks on NVMe — so that expensive-to-compute prefixes survive eviction and can be pulled back on a later cache hit instead of recomputed. Modular's "Five Eras of KVCache" traces how the KV cache went from an implementation detail to the organizing principle of the whole system, which is exactly the shift this diagram encodes: the cache is the center, and compute is arranged around it.

What "fast" actually means

When we say Pipe is fast, we mean something specific and measurable: for a given model and a given traffic mix, we find the operating point on the coupled surface from Part I — precision, batch size, speculation depth, prefill/decode split — that satisfies each workload's TTFT and ITL targets at the lowest cost per million tokens, and we hold that point under bursty, prefix-skewed, real-world load using the routing and scaling machinery from Part II. Not one number on one benchmark. A policy over a design space.

That's the layer underneath the three words. If you're building models or serving systems and want to compare notes on any of this — draft-head training, KV transfer schedulers, cache-aware routing under adversarial traffic — we'd like to hear from you.

Sources & further reading

Building models or serving systems? We'd like to compare notes.

Back to all posts