Post-training is the set of techniques applied to a language model after the initial pre-training run, with the goal of turning a raw next-token-prediction model into a system that follows instructions, behaves safely, and performs well on intended tasks. Where pre-training produces a base model with broad capabilities but no particular disposition toward being helpful or harmless, post-training shapes how those capabilities are expressed. It encompasses supervised fine-tuning, reinforcement learning from human or AI feedback, reward modeling, and preference optimization, and it has become a primary axis along which frontier developers differentiate their models.
Stages and methods
A typical post-training pipeline begins with supervised fine-tuning, in which the base model is trained on curated examples of desired responses. This is commonly followed by a preference-learning stage. Reinforcement learning from human feedback trains a reward model on human comparisons of candidate outputs and then optimizes the language model against that reward signal. Related approaches substitute AI-generated feedback for some human labeling (RLAIF) or use constitutional methods in which a model critiques and revises its own outputs against a set of written principles. Direct preference optimization and similar algorithms collapse parts of this pipeline by optimizing on preference data without training a separate reward model. Reasoning-oriented models add reinforcement learning on verifiable tasks, rewarding correct final answers on problems with checkable solutions.
Significance for frontier-model differentiation
A recurring argument, associated with researchers such as Nathan Lambert, holds that pre-training scale receives disproportionate public attention while much of the observable variance among current frontier models is attributable to post-training choices, including the design of reward models and the mix of RLHF, RLAIF, and constitutional methods (RLHF (Reinforcement Learning from Human Feedback)). On this view, two models trained on comparable data and compute can diverge substantially in helpfulness, safety behavior, and task performance depending on how they are post-trained. Post-training is also where many alignment interventions are applied, making it central to debates over how model behavior is shaped and how robustly safety properties hold under adversarial pressure.
Relationship to distillation and open models
Post-training interacts with distillation, in which a smaller or newer model is trained on the outputs of a larger or more capable one; distilled models can inherit much of a teacher model's post-trained behavior at lower cost. The availability of open-weight base models has made post-training a widely accessible lever, since organizations without the resources to pre-train a frontier model can still adapt an existing base model to a domain or task through fine-tuning and preference optimization.
Relationships
- depends-on: Scaling Laws — pre-training, the stage post-training builds on
- related: RLHF (Reinforcement Learning from Human Feedback) — a central post-training method
- related: Distillation — transfers post-trained behavior between models
- related: Nathan Lambert — researcher who argues post-training drives much frontier-model variance
Provenance note: Foundational-concept page built from established descriptions of the post-training pipeline and the wiki's existing Nathan Lambert and RLHF (Reinforcement Learning from Human Feedback) pages. Created in response to a dangling
[[concepts/post-training]]reference. Citations should upgrade to primary methodological sources as relevant papers are ingested.