LLM Inference Optimization: Latency, Throughput and Memory
Inference performance is mostly memory movement, context management and careful batching.
Start with the real bottleneck
LLM latency is not one number. Time to first token, time per output token, total request time and throughput answer different questions. Measure each before choosing an optimization.
Quantization trades precision for memory
Lower-bit weights reduce memory footprint and often improve bandwidth efficiency. The exact quality loss depends on the model and quantization scheme, so evaluate a representative task set rather than assuming every four-bit model behaves the same.
Context length has a cost
The KV cache grows with sequence length. Long prompts can consume large amounts of memory even when the model weights fit comfortably. Trimming irrelevant history, retrieving only useful evidence and limiting output length can make a larger difference than changing the GPU.
Batching improves throughput — until memory says no
Batching lets the accelerator process multiple sequences together, which can raise tokens per second for server workloads. But larger batches also increase memory use and can make single-request latency worse. Tune for your workload, not for one benchmark screenshot.
Speculative decoding and smaller models
A fast draft model can propose tokens that a larger model verifies, reducing the work needed per accepted token. In many products, though, the biggest win is simply choosing a smaller model that meets the quality requirement.
The optimization order
- Choose the smallest acceptable model.
- Limit unnecessary context.
- Use sensible quantization.
- Measure KV-cache pressure.
- Tune batching for your actual traffic.
- Only then explore advanced decoding tricks.