Two concepts are required for anyone who uses large language models (LLMs) or builds applications on them: Inference and the KV Cache (key-value cache). Whether the application is a coding agent, RAG (retrieval-augmented generation), an AI copilot, or an internal enterprise model deployment, each user request triggers one inference run.
For most online AI applications, training is a one-time investment. Inference is a recurring cost.
A trained model cannot serve users on its own. The inference process is what makes the model operate. To reduce AI infrastructure cost, parameter count is not the only factor to consider. The following questions also apply:
How is a token generated?
Why does the KV Cache keep growing?
What occupies GPU memory?
How does an inference server use these resources?
This article starts with the basics.
1 Why a model cannot serve users directly
A common assumption when first deploying a large model is that downloading the model file and loading it onto a GPU is sufficient. It is not. An LLM inference system that serves users online requires at least three components.
1.1 Model weights
The model itself, for example Qwen, Llama, GLM, Kimi, or DeepSeek models. These models contain billions to hundreds of billions of parameters. The weights determine what the model has learned.
1.2 Inference server
Examples: vLLM, TensorRT-LLM, SGLang. An inference server does more than run the model. It manages concurrent requests, including model loading, request scheduling, KV Cache management, batch management, GPU memory allocation, prefix caching, and token generation. The model performs the computation. The inference server schedules the requests that share it.
1.3 Hardware accelerator
Usually a GPU. The GPU executes the matrix and neural network computation.
A complete inference system can be summarized as:
Model Weights → Inference Server → GPU → User RequestThe model can also run directly on a GPU with PyTorch. This works for notebooks, experiments, and single users. When dozens, hundreds, or thousands of users send requests at the same time, a different problem appears: the GPU is fast enough, but it is not fully used. This is the reason inference servers such as vLLM exist.
In production, the system must consider three questions:
How fast can the GPU compute?
How efficiently can the GPU memory be used?
How efficiently can multiple GPUs communicate with each other?
2 An LLM generates one token at a time
Understanding the KV Cache starts with one fact: an LLM does not generate a complete answer at once. It generates one token at a time.
"Modern AI"
↓
"requires"
↓
"Modern AI requires"
↓
"high-performance"
↓
"Modern AI requires high-performance"
↓
"networks"
↓
...Final output: Modern AI requires high-performance networks
Generation continues until the model produces an EOS (End-of-Sequence) token or another stop condition is met.
This is autoregressive generation. Each time the model generates a token, the token is added to the context, and the updated context is used to predict the next token.
3 Are earlier tokens recomputed every time?
This question is the reason the KV Cache exists. Suppose the model has generated "Modern AI requires high-performance" and is about to generate the next token. The model needs the information from the preceding tokens.
Each token passes through multiple Transformer layers. One of the main components in each layer is self-attention. Attention produces three vectors:
Q (Query): what information to look for in the earlier context.
K (Key): what information a token holds that can be matched.
V (Value): the information a token provides to the model.
When a new token is generated, its Q is matched against the K of the earlier tokens. The match result is used to weight the corresponding V.
Q: what does the current token need?
K: does an earlier token contain it?
V: what information is returned?
4 Why K and V are cached
Assume 1,000 tokens have been processed. To generate token 1,001, the model refers to the earlier tokens. Without caching, the K and V of those 1,000 tokens would be computed again at every step.
Within one generation sequence, the K and V of earlier tokens do not change after they are computed. The K/V of token 1, token 2, token 3, up to token 1,000 are used repeatedly in later steps. Because they do not change, they can be stored instead of recomputed.
This is the KV Cache. Computed K and V values are stored in GPU memory and reused when later tokens are generated:
Earlier tokens: use the cached K/V.
New token: compute new K/V.
This removes a large amount of repeated computation.
The cache is not limited to one layer. Each attention layer in the model keeps its own KV Cache. A deeper model, a longer context, and more concurrent users all increase the GPU memory used by the KV Cache.
5 KV Cache size
KV Cache size depends on the following factors:
Context length: more tokens require more stored KV data.
Concurrent requests: more simultaneous users require more KV Cache copies.
Number of layers: larger models usually have more Transformer layers, and each layer produces its own K and V.
Number of KV heads.
Head dimension.
Data precision: higher precision generally requires more memory per value.
KV Cache size can be estimated as:
KV Cache ≈ 2 × Layers × KV Heads × Head Dimension × Data Type Size × Tokens2 = K and V
Layers = number of Transformer layers
KV Heads = number of KV attention heads
Head Dimension = dimension of each head
Data Type Size = bytes per value
Tokens = number of tokens in the context
Longer context increases KV Cache size. More concurrent users increase KV Cache size. One user with 2K tokens and 100 users with 32K tokens each differ by orders of magnitude in memory demand. In production, GPU memory must hold the model weights and must also reserve space for the KV Cache.
6 What GPU memory is used for
This is often overlooked in large model deployment. Take a GPU with 80 GB of memory. A model smaller than 80 GB does not necessarily run on that GPU, because GPU memory must also hold:
Model weights
KV Cache
Activations
Runtime and workspace
Other inference overhead
In online serving, the KV Cache grows as requests continue. If memory is allocated in advance for the worst case, a problem follows. Suppose each request can need up to 4 GB of KV Cache:
10 requests → 40 GB
20 requests → 80 GB
Even if many requests use only a short context, the memory is reserved in advance. The GPU still has compute capacity, but it cannot accept more requests because of how memory is allocated.
7 Allocating memory on demand
Modern inference servers address this by not reserving a large contiguous block of memory for each request. Memory is allocated according to actual use. Three techniques are common.
7.1 PagedAttention
PagedAttention follows the same approach as virtual memory in an operating system.
Traditional method: the user may need 4 GB, so 4 GB is reserved.
PagedAttention: the request receives the memory it needs now, and requests more when needed.
The KV Cache is divided into fixed-size blocks. The blocks do not need to be contiguous. The system keeps a mapping of request to KV blocks. Memory use changes from advance reservation to on-demand allocation, which reduces memory fragmentation and idle memory.
7.2 Continuous batching
In static batching, a group of requests, for example 8, starts together and the next batch starts only after all 8 are complete. Requests differ in length. One request may need 20 tokens and another 2,000 tokens. After the short request completes, the GPU waits for the long one.
In continuous batching, when a request completes, its resources are released and a new request enters immediately:
Request A → complete → Request D enters
Request B → complete → Request E enters
Request C → continues generatingThe GPU does not wait for the whole batch to finish. This keeps GPU utilization high, which matters for online AI services.
7.3 Prefix caching
Many requests share the same beginning. An AI agent request, for example, may contain:
System prompt
Agent instructions
Tool definitions
Codebase context
User question
If the first 10,000 tokens are identical in every request, recomputing them for each request wastes compute. Prefix caching computes an identical prefix once:
First request: Prefix → compute → store KV Cache
Second request: Same prefix → reuse
Third request: ReuseOnly the part that differs between requests is computed. Prefix caching applies to:
RAG
Coding agents
AI agents
Long system prompts
Enterprise knowledge bases
8 The next bottleneck after single-GPU optimization
Modern LLM inference systems use several methods to improve the efficiency of a single GPU:
PagedAttention: uses GPU memory more efficiently for the KV Cache.
Continuous batching: keeps GPU utilization high.
Prefix caching: reduces repeated computation of identical prompts.
Quantization: lowers numeric precision to reduce the memory used by the model.
These methods have one goal: a single GPU serves more requests. A question follows: what happens when the model grows and one GPU is no longer sufficient? At that point the problem changes.
