Synthetic data is training data produced by an AI model rather than collected from humans or the web. It overlaps with distillation, whose teacher outputs are by definition synthetic, but is a broader category that also includes model-generated demonstrations, self-instruct prompts, rewritten web data, agentic trajectories, and constitutional critiques.
Categories
Synthetic data is used at several stages of the training pipeline:
- Pretraining synthetic data. Rewrites, paraphrases, or expansions of web text generated by an LLM to improve quality or coverage. The Phi series ("textbooks are all you need") and Microsoft's Cosmopedia are commonly cited examples.
- Post-training synthetic data. Instructions, preferences, reasoning traces, and tool-use demonstrations generated by strong models for alignment fine-tuning.
- Self-play / agentic synthesis. Model-generated environments, tasks, and trajectories, used for reasoning and agent training. The Kimi K2 technical report made this approach explicit.
- Constitutional data. AI-generated critiques and revisions used as the training signal in Constitutional AI and RLAIF.
Mechanism
A typical synthetic-data pipeline proceeds in four steps. First, a seed: a small set of human-authored prompts, documents, or task specifications. Second, generation: a strong model, often the frontier model itself or a specialist teacher, produces candidate outputs such as completions, reasoning traces, preference pairs, or task decompositions. Third, filtering: heuristic filters, verifiers, or reward models remove low-quality or incorrect outputs, with rule-based checkers such as compilers and math provers working best. Fourth, training: the filtered synthetic data is used for SFT, preference optimization, or continued pretraining.
Origin papers
Several papers and corpora trace the development of the technique:
- Self-Instruct (Wang et al., 2022) — bootstrapping alignment data from a small seed using an LLM.
- Alpaca (Stanford, 2023) — popularized LLM-generated instruction data.
- Constitutional AI (Anthropic, 2022) — AI-generated critiques as alignment signal; see Constitutional AI.
- Textbooks Are All You Need (Microsoft Phi, 2023) — showed curated synthetic textbooks could outperform web data at small scale.
- FineWeb / FineWeb-Edu (HuggingFace, 2024) — an LLM-graded web corpus; mostly curated-human data with LLM labels, a bridge case.
- Cosmopedia (HuggingFace, 2024) — an open Phi-style synthetic textbook corpus.
- DeepSeek-R1 (2025) — distilled reasoning traces from R1 into smaller models, a high-profile post-training synthetic-data result; see DeepSeek-R1.
- Kimi K2 technical report (2025) — an explicit "agentic data synthesis" pipeline generating tool-use trajectories.
Current state
Every frontier post-training pipeline is majority-synthetic for SFT and preference data. Pretraining mixes typically include 10–40% synthetic or LLM-rewritten content at frontier scale, though exact fractions are undisclosed. Labs increasingly gate API output against suspected distillation, with terms of service prohibiting training competitors' models on outputs. Agentic training data, including tool-use trajectories and long-horizon task completions, is now primarily synthetic because human demonstration does not scale.
Policy and safety relevance
Synthetic data bears on several policy and safety questions. On training-data transparency, regimes such as the EU AI Act (Regulation 2024/1689) and EU General-Purpose AI Code of Practice (Final Version, 2025) require disclosure of training-data sources, which synthetic data complicates because its provenance recurses into the teacher model. On copyright, synthetic data partially sidesteps training-data copyright disputes, but only to the extent that the teacher was itself trained on licensed data, leaving the circularity unresolved. Recursive training on synthetic data also raises model collapse risk, degrading distribution coverage if not mitigated. On alignment, Constitutional AI and RLAIF replace human feedback with synthetic feedback, addressing the labor bottleneck in RLHF but introducing bias from the labeling model. Synthetic data drawn from frontier APIs is also a central channel in the Fast-Follow Problem; Epoch AI has reported alleged Claude-distillation by Moonshot, DeepSeek, and MiniMax (Source: epochai.substack.com).
"Textbooks Are All You Need" (Microsoft Research, 2023) is the founding demonstration that model-generated training data can move a frontier result: phi-1, at 1.3B parameters trained for four days on eight A100s using 6B tokens of filtered web data plus 1B tokens of GPT-3.5-generated textbooks and exercises, reached 50.6% pass@1 on HumanEval, with a 350M-parameter variant still reaching 45%. The paper argues data quality "can dramatically change the shape of the scaling laws," while flagging two caveats on itself: a section studying possible HumanEval contamination, and the note that details of the synthetic-data generation are withheld "for proprietary reasons." It also records the open question about "recursive training" narrowing model scope.
Relationships
- related: Distillation — a subset of synthetic data; see dedicated page
- depends-on: Constitutional AI — CAI is the best-documented synthetic-alignment pipeline
- supports: RLHF (Reinforcement Learning from Human Feedback) — RLAIF is synthetic-feedback RLHF
- contradicts (partially): Model Collapse — model collapse is the failure mode of unconstrained synthetic training
- related: Fast-Follow Problem — synthetic data is the main fast-follow channel
- related: AI Software Progress — synthetic data quality drives measured efficiency gains