40 terms

Training

How models are built and improved, from pre-training to fine-tuning and reinforcement learning.

byteificationnountrendingthe process of retrofitting a language model trained on subword tokens so that it can read and generate byte sequences, while…
catastrophic forgettingnounthe abrupt loss of previously learned abilities or knowledge when a neural network is trained on new data or tasks that overwrite…
character trainingnountraining that shapes an AI model's stable personality and values, such as curiosity, honesty, and care for the people it talks…
computenountrendingcomputing power, especially the chips and processing time used to train and run AI models
concept driftnouna change over time in the statistical relationship between a model's inputs and the outcome it is meant to predict, causing…
constitutional AInouna training approach in which a model is guided by a written set of principles (a "constitution") and uses AI feedback, rather…
continual learningnountrendingan approach in which a machine-learning system learns from a sequence or stream of changing tasks and data while trying to retain…
credit assignmentnounthe problem of determining which earlier actions, tokens, components, or agents contributed to a later reward, error, or outcome…
data labelingnounthe work of tagging raw data with correct answers or categories, such as marking objects in photos or rating chatbot replies, so…
direct preference optimizationnountrendinga method for aligning a language model with pairs of preferred and rejected responses by directly optimizing the model on those…
distillationnountrendingtraining a smaller "student" model to imitate the outputs of a larger "teacher" model, producing a cheaper model that keeps much…
distribution shiftnouna difference between the statistical distribution of the data used to train or evaluate a model and the data the model encounters…
emergent abilitiesplural nounskills that seem to appear suddenly once a model passes a certain size, even though it was not specifically trained for them
epochnounone complete pass through the entire training dataset
federated learningnouna machine-learning method in which multiple devices or organizations train a shared model from locally held data and exchange…
fine-tuningnountrendingfurther training of an existing model on a smaller, focused dataset so it gets better at a particular task, style, or domain
GFlowNetnountrendinga generative learning method that constructs objects through sequences of actions and learns to sample many different high-reward…
imitation learningnouna way of training an AI system or robot to perform a task by learning from examples of an expert's behavior, such as recorded…
instruction tuningnounfine-tuning a base model on examples of instructions paired with good responses, so it learns to follow requests instead of just…
knowledge editingnountrendingthe targeted modification of facts or factual associations stored in a trained model while trying to preserve unrelated knowledge…
LoRAnounlow-rank adaptation: a cheap fine-tuning method that freezes the original model and trains only a small add-on, so a new style or…
machine unlearningnountrendingthe process of removing the influence of specified training examples, knowledge, or capabilities from a trained machine-learning…
model collapsenouna gradual loss of quality and diversity that happens when models are trained on content produced by earlier models, so errors and…
model mergingnountrendingthe creation of one model by mathematically combining the parameters, parameter differences, adapters, or checkpoints of two or…
multi-token predictionnountrendinga language-model training objective that asks the model to predict several future tokens from each position instead of optimizing…
on-policy distillationnountrendinga form of model distillation in which the student generates its own outputs or trajectories and learns from teacher feedback on…
online continual learningnountrendinga continual-learning setting in which a model updates incrementally from a changing stream of examples or interactions, usually…
post-trainingnountrendingeverything done to a model after pre-training to make it useful and safe, such as instruction tuning, reinforcement learning, and…
pre-trainingnounthe first and most expensive stage of building a model, in which it learns general patterns from an enormous amount of unlabeled…
quantizationnounshrinking a model by storing its numbers with less precision (for example 4 bits instead of 16), which makes it smaller and…
reinforcement learningnountrendinga way of training in which a model tries actions, receives rewards or penalties, and gradually learns which behavior earns the…
reward modelnouna separate model trained to score how good a response is, used to guide reinforcement learning without a human rating every…
RLHFnounreinforcement learning from human feedback: a training method in which people rate or compare a model's answers, and the model is…
sample efficiencynountrendingthe amount of useful learning or task performance an AI system gains from each training example, observation, or interaction…
scaling lawsplural nountrendingobserved patterns showing that model performance improves in a predictable way as you increase model size, training data, and…
self-distillationnountrendinga form of knowledge distillation in which a model learns from outputs, representations, critiques, or other training signals…
sub-1-bit quantizationnouna model-compression approach that stores neural-network weights using an average of less than one bit per weight, typically by…
synthetic datanountrendingtraining data generated by AI models or simulations rather than collected from the real world
test-time adaptationnountrendinga family of methods that adjust a trained model, its internal statistics, or its predictions while it is being used on new data…
test-time trainingnountrendinga method that updates a model at deployment time using a learning signal derived from the unlabeled test input or stream before…

Other topics