Blog Archive

If you've been keeping up with our blog, then you probably know that the biggest scaling liability in modern Transformers is the KV cache. The reason for this is the attention mechanism retaining every prior token. Keeping track of each token in generation creates a massive mountain of data that needs to be moved back and forth repeatedly. Even with the fastest hardware, a memory-bandwidth bottleneck can happen long before the compute cores hit saturation. State Space Models (SSMs) replace the ever-growing token cache with a fixed-dimension hidden state. By compressing context recurrently rather than caching raw token representations, SSMs keep the inference memory footprint static, no matter how long the sequence runs.

Serving a single LLM inference stream leaves modern GPU compute cores mostly idle. Because autoregressive decode requires transferring model weights from VRAM into registers for every single token, single-sequence execution runs at an arithmetic intensity of roughly 1 FLOP per byte. This causes throughput to be entirely memory-bandwidth-bound. Batching solves this by reusing loaded model weights across multiple concurrent requests. However, applying classical ML batching patterns to generative models introduces heavy padding overhead and scheduling stalls.