Prefill-decode disaggregation – Definition & Meaning

Definition of prefill-decode disaggregation

: an architecture for serving language models that runs the compute-heavy prefill stage, which processes the input prompt, on different workers or hardware from the memory-heavy decode stage, which generates output tokens one at a time
- The serving cluster used prefill-decode disaggregation so that long prompts would not delay token generation for other users.
- After prefill finished, the system transferred the request's KV cache to a decode worker.

Origin & history

first known use2024

The DistServe research published in 2024 made prefill-decode disaggregation prominent by separating the two stages across GPUs and optimizing their placement for time to first token and time per output token. The design was subsequently adopted by open serving systems and extended to specialized and heterogeneous hardware.

Test yourself

Which of these is the meaning of prefill-decode disaggregation?

Related terms

inference the act of running a trained model to produce an output, such as answering a prompt or… KV cache memory that stores intermediate attention results for text a model has already processed, so… latency the delay between sending a request to a model and getting a response, often measured as time… throughput how much work an AI system can handle in a given time, often measured in tokens per second GPU graphics processing unit: a chip originally designed for video games that can do many… data center a large building full of servers, chips, cooling, and power equipment; AI data centers can…

Cite this entry

"Prefill-decode disaggregation." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/prefill-decode-disaggregation/. Accessed 5 Oct. 2026.