Continuous batching – Definition & Meaning

Definition of continuous batching

: an inference-serving technique that adds new requests to and removes completed requests from a model's processing batch between generation steps, keeping computing hardware busy without waiting for every sequence in a fixed batch to finish
- Continuous batching admitted a new prompt as soon as another sequence completed.
- The serving engine increased throughput by scheduling requests at each decoding step instead of forming one fixed batch.

Two meanings, one word

In everyday English

Batching means grouping items so they can be processed together.

In AI

An inference-serving technique that adds new requests to and removes completed requests from a model's processing batch between generation steps, keeping computing…

Origin & history

The technique developed from iteration-level scheduling for transformer inference and became widely known as continuous batching through high-throughput language-model serving systems in the early 2020s.

Test yourself

Which of these is the meaning of continuous batching?

Related terms

batch inference running a trained AI model on a collected group of inputs as a scheduled or asynchronous job… inference the act of running a trained model to produce an output, such as answering a prompt or… throughput how much work an AI system can handle in a given time, often measured in tokens per second latency the delay between sending a request to a model and getting a response, often measured as time… KV cache memory that stores intermediate attention results for text a model has already processed, so… prefill-decode disaggregation an architecture for serving language models that runs the compute-heavy prefill stage, which…

Cite this entry

"Continuous batching." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/continuous-batching/. Accessed 12 Oct. 2026.