LLM serving – Definition & Meaning

Definition of LLM serving

: the infrastructure and systems work of deploying a trained large language model so it can receive concurrent inference requests and return outputs with acceptable latency, throughput, reliability, and cost
- The team improved LLM serving by batching requests that reached the GPU at nearly the same time.
- Long prompts created an LLM serving bottleneck because prefill work delayed short interactive requests.

Origin & history

Model serving is an older machine-learning term for making trained models available to applications. LLM serving became a distinct systems field around 2022 and 2023 as autoregressive generation, variable-length requests, KV-cache memory, and the separate prefill and decode phases created performance problems that general model servers were not designed to solve.

Editorial references

Sources & further reading

3 sources

Efficient Memory Management for Large Language Model Serving with PagedAttention Research Paper · ACM SOSP / arXiv · 2023-09-12
Fairness in Serving Large Language Models Research Paper · USENIX OSDI · 2024
Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration Research Paper · USENIX OSDI · 2026-07

Test yourself

Which of these is the meaning of LLM serving?

Related terms

inference the act of running a trained model to produce an output, such as answering a prompt or… continuous batching an inference-serving technique that adds new requests to and removes completed requests from a… KV cache memory that stores intermediate attention results for text a model has already processed, so… prefill-decode disaggregation an architecture for serving language models that runs the compute-heavy prefill stage, which… throughput how much work an AI system can handle in a given time, often measured in tokens per second latency the delay between sending a request to a model and getting a response, often measured as time…

Cite this entry

"LLM serving." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/llm-serving/. Accessed 15 Oct. 2026.