Continuous vs. Static Batching in LLM Serving
Serving a single LLM inference stream leaves modern GPU compute cores mostly idle. Because autoregressive decode requires transferring model weights from VRAM into registers for every single token, single-sequence execution runs at an arithmetic intensity of roughly 1 FLOP per byte. This causes throughput to be entirely memory-bandwidth-bound.
Batching solves this by reusing loaded model weights across multiple concurrent requests. However, applying classical ML batching patterns to generative models introduces heavy padding overhead and scheduling stalls.
1. The Failure Modes of Static Batching
Static batching works well for fixed-size workloads like image classification, but it falls apart under the variable lengths of generative text. Packing requests into a rigid rectangular tensor forces the GPU to waste cycles computing attention over dummy pad tokens just to align matrix dimensions.
Because execution runs in lockstep, early-finishing requests remain trapped in memory until the slowest sequence in the batch completes, while new incoming requests queue up waiting for the pipeline to clear. Even what classical serving frameworks call "dynamic batching", which accumulates requests over a short arrival window, still results in an immutable, static execution graph once the forward pass begins.
2. Continuous Batching: Iteration-Level Scheduling
Continuous batching (iteration-level scheduling) moves scheduling decisions from the request level to the individual token generation step. Rather than locking a batch for its full lifecycle, the inference runtime evaluates the active batch state in every iteration of the decode loop:
Immediate Eviction: As soon as an individual sequence emits an end-of-sequence token (<|endoftext|>), it is evicted from the active batch immediately, releasing its allocated memory.
Dynamic Injection: During the very next iteration, the scheduler admits a new incoming request to fill the vacant slot, beginning its prefill phase during the ongoing decode steps of active sequences.
Ragged Tensor Packing: By stripping structural sequence padding, token representations are packed into flattened 1D ragged buffers, eliminating matrix operations on dummy pad tokens.
3. Memory Paging and Prefill-Decode Interference
Iteration-level scheduling creates its own challenges, primarily VRAM fragmentation and compute contention. Dynamically allocating and freeing variable-length sequences on every step rapidly fragments GPU memory. Modern systems handle this by adopting OS-style paging. That means storing the KV cache in fixed-size, non-contiguous blocks via mechanisms like PagedAttention.
Simultaneously, mixing a compute-heavy prompt prefill into an active decode iteration creates compute contention, causing sudden spikes in inter-token latency for existing streams. Serving engines resolve this through chunked prefill, slicing incoming prompts into bounded token budgets across multiple iterations to keep output streaming steady and predictable.
4. Architectural and Performance Metrics
Moving from static to continuous batching fundamentally changes how the serving engine interacts with the underlying silicon:
Scheduling Granularity: Static batching locks execution at the sequence level; continuous batching reschedules dynamically on every token step.
Hardware Saturation: Runtimes keep memory buses saturated during decode (maximizing MBU) while feeding tensor cores active tokens during prefill rather than dummy pad masks.
Latency Stability: Eliminates queue-level head-of-line blocking to smooth Time to First Token (TTFT) without degrading steady-state token streaming.
You can extract maximum value from your inference engines and expensive hardware by managing VRAM as virtualized physical pages, treating prompt prefills as preemptible workloads, and running the decode loop as an interruptible queue. Ultimately, continuous batching helps push your GPU to the physical limits of memory bandwidth and compute silicon. This ensures your bottleneck is dictated by the hardware rather than software-induced idle time.