Dictionary entry Trending
speculative decoding
also speculative sampling or speculative inference
Definition of speculative decoding
-
: a technique for speeding up autoregressive model inference in which a faster draft process proposes several tokens and a larger target model checks them in parallel, accepting valid proposals and replacing rejected ones
- The serving system used speculative decoding to reduce latency without retraining the main language model.
- Speculative decoding helped the target model produce several verified tokens during one decoding cycle.
Two meanings, one word
In everyday English
Speculative work is done in advance because it may prove useful, even though it may later be rejected.
In AI
A technique for speeding up autoregressive model inference in which a faster draft process proposes several tokens and a larger target model checks them in parallel…
Origin & history
The term was introduced in the 2023 ICML paper 'Fast Inference from Transformers via Speculative Decoding' by Yaniv Leviathan, Matan Kalman, and Yossi Matias. The paper described exact decoding that uses a smaller approximation model to propose tokens and a larger model to verify them in parallel without changing the target output distribution.
Test yourself
Which of these is the meaning of speculative decoding?
Cite this entry
"Speculative decoding." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/speculative-decoding/. Accessed 6 Oct. 2026.