Self-distillation – Definition & Meaning

Definition of self-distillation

: a form of knowledge distillation in which a model learns from outputs, representations, critiques, or other training signals produced by the same model or by an earlier version of it rather than by a separate larger teacher
- The team used self-distillation to improve the model without querying a more expensive external teacher.
- In the experiment, a critique-conditioned copy acted as the teacher while the same model learned from a shorter input.

Origin & history

first known use2018

The modern deep-learning use is commonly traced to the 2018 Born-Again Neural Networks work, which trained a new model of the same architecture from a predecessor's output and was later described as self-distillation. The term now covers several designs in which teacher and student share a model, architecture, or lineage. A September 2026 UniEvo-VL paper applied on-policy self-distillation to multimodal self-improvement.

Test yourself

Which of these is the meaning of self-distillation?

Related terms

distillation training a smaller "student" model to imitate the outputs of a larger "teacher" model… on-policy distillation a form of model distillation in which the student generates its own outputs or trajectories… post-training everything done to a model after pre-training to make it useful and safe, such as instruction… recursive self-improvement a scenario in which an AI system improves its own design, and each improved version gets… synthetic data training data generated by AI models or simulations rather than collected from the real world fine-tuning further training of an existing model on a smaller, focused dataset so it gets better at a…

Cite this entry

"Self-distillation." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/self-distillation/. Accessed 8 Oct. 2026.