Dictionary entry Trending
expert offloading
also MoE expert offloading or expert streaming
Definition of expert offloading
-
: an inference technique for mixture-of-experts models that keeps only some expert subnetworks in fast accelerator memory and stores the others in CPU memory or on disk, loading or running them when the model's router selects them
- Expert offloading let the workstation run a mixture-of-experts model whose full set of weights would not fit in GPU memory.
- The inference engine used an expert cache and predictive prefetching to reduce the delay caused by loading a selected expert from system memory.
Two meanings, one word
In everyday English
Moving part of a workload away from the place where it would normally be kept or processed.
In AI
An inference technique for mixture-of-experts models that keeps only some expert subnetworks in fast accelerator memory and stores the others in CPU memory or on disk…
Origin & history
The technique builds on older parameter-offloading methods, but became a distinct mixture-of-experts inference approach with the 2023 paper Fast Inference of Mixture-of-Experts Language Models with Offloading. Research and inference engines later standardized the wording expert offloading for caching selected experts in fast memory and fetching inactive experts on demand. In 2026, expert streaming became a common implementation-oriented name as the technique reached general-purpose local inference software.
Editorial references
Sources & further reading
- Fast Inference of Mixture-of-Experts Language Models with Offloading↗ arXiv · 2023-12-28
- HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference↗ arXiv · 2024-11-03
- Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models↗ ICLR 2026 · 2026
- 180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers↗ Hugging Face · 2026-11-18
Test yourself
Which of these is the meaning of expert offloading?
Cite this entry
"Expert offloading." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/expert-offloading/. Accessed 18 Nov. 2026.