What is KV Cache (Caching previous attention keys and values)

KV Cache is a way to avoid repeated computations. During text generation, the model reprocesses the entire sequence. The cache stores previously computed keys and values, so at each new step only the last token is recalculated, not the whole context.

The technology is applied in the inference of large language models of the decoder type (GPT, Llama, Gemini). It is critically important for text autogeneration, dialogue systems, and response streaming. Without KV cache, generation time would grow quadratically with sequence length, making real-time interactive work with long contexts impossible.

The main challenge is the linear growth of memory consumption with dialogue length, leading to rapid GPU memory exhaustion. Memory fragmentation occurs due to dynamic allocation of blocks for variable-length requests. During batched processing of requests of different lengths, computational idling (padding) and inefficient utilization of the accelerator’s compute resources are inevitable.

How KV Cache works

In the decoder, the self-attention mechanism computes Key and Value matrices for each token. Without optimization, at step t the model recalculates K and V for tokens 1…t−1. KV cache stores these once-computed matrices in high-speed GPU memory. When generating the next token, only its embedding is processed from the input data, new K and V vectors are computed for it, and they are concatenated with the cache contents. The full tensor is then passed to the attention operation.

Unlike conventional result caching, which can be invalidated, here the data is physically determined by the preceding context. Compared to context compression methods, KV cache is exact and does not lose information. Unlike Multi-Query Attention, which structurally reduces memory volume through shared heads, KV cache solves the problem of computational duplication over time. Advanced inference engines like vLLM implement PagedAttention—page-based cache management analogous to OS virtual memory—which eliminates fragmentation and allows efficient sharing of a common system prompt prefix among requests through logical block copying.

KV Cache functionality

The purpose of KV cache in the Transformer architecture. KV cache eliminates redundant computations of keys and values for previously generated tokens during the autoregressive inference stage by storing them in GPU memory for reuse at subsequent decoding steps.
Key and Value matrices in the self-attention mechanism. At each layer, input embeddings are linearly projected into Query, Key, and Value matrices. Keys determine addressing, values determine the retrieved content, and the KV cache captures them after the token’s first computation.
The reason for computational redundancy without caching. When generating each new token, the model recalculates K and V for the entire preceding sequence. This leads to a quadratic increase in computational complexity relative to context length, making inference practically infeasible.
The principle of concatenation in the cache. At step t, only vectors k_t and v_t are computed for the new token, which are immediately concatenated with the already stored K_cache and V_cache tensors of dimension (t-1). The resulting tensor is fed to the input of the scaled dot-product attention operation.
The storage lifecycle on the GPU. The cache is initialized as an empty tensor before prefilling, populated with the keys and values of the prompt during a single forward pass, and then incrementally appended in the autoregressive generation loop until the maximum sequence length is reached.
Memory allocation structure. A separate KV block of dimension (2, batch_size, num_heads, seq_len, head_dim) is allocated for each transformer layer. The total memory volume is proportional to the product of the number of layers, context length, and hidden state dimension.
Impact on computational complexity. With caching, the attention computation at step t is reduced from O(t² · d) to O(t · d), where d is the head dimension. The Query-Key multiplication for the entire history is replaced by extending the matrix with a single new row.
Prefilling as the initialization phase. The initial processing of the input prompt is performed as a batch in a single pass, forming the initial K and V snapshot for all layers. This stage fully utilizes GPU parallelism and is not autoregressive, after which control passes to the decoding loop.
Decoding mode and batch processing. During token generation, the model operates in an arithmetic-intensity-limited regime. Continuous batching dynamically manages the KV caches of different-sized requests, aligning padding and minimizing memory fragmentation through mechanisms like PagedAttention.
Memory management and fragmentation. Naive reservation of rectangular blocks leads to significant internal and external fragmentation. Modern serving systems use virtual memory and page tables to map logical sequence positions to non-contiguous physical memory blocks.
PagedAttention and block addressing. The cache is divided into fixed-size blocks (pages). When the length is exceeded, a block is allocated dynamically, enabling the processing of significantly longer dialogues without the need to allocate a monolithic tensor for the maximum context length.
Multi-Query and Grouped-Query Attention. Multi-Query Attention stores a single shared KV block for all query heads, radically reducing cache size. Grouped-Query Attention (GQA) introduces a compromise, sharing K and V among groups of heads, which reduces cache volume proportionally to the number of groups.
Cache quantization and compression. To reduce VRAM usage at long lengths, KV cache in INT8 or FP8 formats is used. Values are quantized with per-channel adaptive scaling before writing and restored upon reading, introducing minimal perplexity degradation while saving up to 50% of memory.
VRAM (High-speed buffer for graphical data)
Sliding window and local attention. A number of architectures (Mistral) limit the attention window to a fixed length W. K and V beyond the window boundary are discarded, transforming the linear cache growth into a constant one, which is fundamentally important for streaming tasks with infinite generation time.
Separation of prefill and decoding cache. In scenarios with a shared prefix portion (system prompt), the prefill KV cache is computed once and shared among multiple requests, drastically reducing total memory consumption and request initialization costs.
Eviction and selective invalidation mechanisms. When branching a dialogue, regenerating, or rolling back a blacklisted token list requires excising a portion of the cache. Invalidation is implemented by truncating tensors by length or logically remapping pages without physical data copying in memory.
Cache mirroring in speculative decoding. The draft model generates a speculative sequence, storing its K and V in a shadow cache. After verification by the target model, valid cache blocks are copied verbatim to the main cache, eliminating recomputation for successfully predicted tokens.
Tensor parallelism and distributed cache. In models sharded across attention heads, each accelerator stores its own KV shard. Cross-synchronization during all-gather is absent, as local computations remain independent until the reduction stage of the attention output projection.
Tensor (Multidimensional container for numerical data)
Offloading to the CPU memory hierarchy. When working with extremely long contexts, unused cache pages are evicted to host RAM or NVMe disk using an LRU strategy. Prefetching masks latency through asynchronous prefetch of blocks predicted for imminent use.
Flash Attention compatibility. Fused Flash Attention kernels consume the block representation of the cache directly from HBM, without materializing the full attention matrix. The KV cache is loaded in chunks into SRAM, participating in the aggregation of softmax statistics via an online safe summation algorithm.
SRAM (Fast volatile random storage of bits)HBM (3D stacked memory with silicon vias)

Comparisons

KV Cache vs Multi-Query Attention. KV Cache stores separate keys and values for each attention head, whereas Multi-Query Attention shares a single set of key-value pairs among all query heads. This approach radically reduces cache volume and memory bandwidth, but at the cost of potential long-context quality degradation compared to classical MHA.
KV Cache vs PagedAttention. Standard KV Cache reserves contiguous memory for the maximum sequence length, causing fragmentation. PagedAttention introduces a virtual address space, dividing the cache into fixed-size blocks. This eliminates internal fragmentation and allows the system to process requests with dynamic batching without memory collisions, increasing throughput.
KV Cache vs Token Merging. The Token Merging method aggressively reduces the number of processed tokens by merging redundant ones, decreasing the computational complexity of attention. In contrast, KV Cache does not alter the model topology but merely avoids recomputation by preserving previously obtained projections. The merging strategy loses some information, whereas the cache guarantees the mathematical identity of the incremental generation result.
KV Cache vs Sliding Window Attention. Sliding Window Attention limits the model’s receptive field to a fixed window of the most recent tokens, keeping the cache size constant regardless of history length. KV Cache without a window accumulates the entire context, requiring linearly growing memory costs. The window method wins in resource consumption predictability but loses the ability to access distant parts of the text without additional mechanisms.

Mechanism of operation

KV cache stores the computed keys and values for each token in the multi-head attention layers, eliminating their recalculation when generating the next token. The implementation uses pre-allocation of contiguous memory blocks for key and value tensors, dynamic appending of new vectors during decoding, and position management via window shifting or a free-slot pool.

Memory management

Cache memory is allocated in blocks using systems like PagedAttention, where logical sequences are mapped to fixed-size physical blocks via an index table. This eliminates external fragmentation, allows sharing blocks between different requests when prefixes match, and lets the operating system handle pages standardly through the GPU driver without custom allocators.

Limitations

Cache volume grows linearly with sequence length and the number of layers, creating a memory bottleneck on the device. Quantitative limitations include a fixed maximum context length, the impossibility of preserving the cache between heterogeneous model architectures, and accuracy degradation under aggressive quantization of stored vectors below eight bits.

Security

Cache isolation between different user sessions is ensured at the level of the block manager’s logical address spaces, where each request receives a unique space identifier. The driver controls access to physical pages through GPU virtual memory mechanisms, preventing the reading of other parties’ key and value tensors even during parallel batched execution.

History and development

The concept appeared in early transformer decoder implementations as a way to avoid quadratic computational growth, initially by storing full matrices. Development progressed from static maximum-length reservation to dynamic block allocation with PagedAttention in vLLM, compression techniques via grouped-query attention, and multi-level caching with eviction of infrequently used blocks to host RAM.