Multi-token prediction – Definition & Meaning

Definition of multi-token prediction

: a language-model training objective that asks the model to predict several future tokens from each position instead of optimizing only for the immediately next token
- The model added multi-token prediction heads during pre-training and discarded most of them for ordinary generation.
- The team tested whether MTP improved representation learning as well as speculative decoding speed.

Origin & history

Multi-step and multi-token prediction ideas existed earlier, but the term became widely recognized in large-language-model research in 2024 through work on training models to predict multiple future tokens and through the DeepSeek-V3 architecture, which used a multi-token prediction objective.

Test yourself

Which of these is the meaning of multi-token prediction?

Related terms

token the basic unit of text a language model reads and writes: a whole word, part of a word, a… pre-training the first and most expensive stage of building a model, in which it learns general patterns… speculative decoding a technique for speeding up autoregressive model inference in which a faster draft process… inference the act of running a trained model to produce an output, such as answering a prompt or… transformer a neural network design that processes a whole sequence at once and uses attention to work out…

Cite this entry

"Multi-token prediction." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/multi-token-prediction/. Accessed 8 Oct. 2026.