DadgogoAI Dictionary

Dictionary entry

KV cache

noun \ˌkā-ˈvē ˈkash\

also key-value cache

TopicChips & Infrastructure Senses1

Definition of KV cache

  1. : memory that stores intermediate attention results for text a model has already processed, so it doesn't have to recompute them for every new token

    • Long conversations eat GPU memory because the KV cache keeps growing.
    • Prompt caching is built on top of the KV cache.
Advertisement

Test yourself

Which of these is the meaning of KV cache?

Related terms

attention a technique that lets a model weigh how relevant every part of the input is to every other… prompt caching reusing the already-processed beginning of a prompt across many requests, which cuts cost and… inference the act of running a trained model to produce an output, such as answering a prompt or… VRAM the memory built into a GPU; how much a model needs largely decides which graphics card can… HBM a type of three-dimensionally stacked memory that uses a very wide connection to move data… prefill-decode disaggregation an architecture for serving language models that runs the compute-heavy prefill stage, which… speculative decoding a technique for speeding up autoregressive model inference in which a faster draft process… sparse attention an attention mechanism that allows each query to attend to only a selected subset of tokens or… unified memory a memory architecture or programming model that lets a CPU and one or more accelerators access… continuous batching an inference-serving technique that adds new requests to and removes completed requests from a… agentic workload the pattern of computing work produced when an AI agent pursues a goal through many dependent… LLM serving the infrastructure and systems work of deploying a trained large language model so it can… CXL memory pooling a server architecture that uses Compute Express Link to combine memory devices into a shared… memory wall a performance limit reached when a processor or AI accelerator can perform calculations faster… KV cache compression the reduction of the memory and data-movement cost of a language model's KV cache, for example…

Cite this entry

"KV cache." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/kv-cache/. Accessed 24 Oct. 2026.

Dictionary entries near KV cache

Advertisement