AI software progress is the rate at which the training compute needed to reach a given AI capability level declines over time. If software progress runs at 10× per year, a model that required $10M of compute one year can be matched with $1M of compute the next. The phenomenon is also called "algorithmic progress," though The Least Understood Driver of AI Progress describes that label as a misnomer, since much of the measured gain comes from data-quality improvements rather than algorithmic innovations.
Estimates
The Least Understood Driver of AI Progress gives a best estimate of roughly 10× per year, with an 80% confidence interval of 2× to 50× per year. The error bars are wide: across the literature, estimates range from 1.1× to 300× per year. Two caveats accompany the central estimate. Most estimates fail to capture post-training improvements such as reasoning and RLHF, and estimates may overstate real-world impact because of "bench maxxing" (gains that show up on benchmarks more than in deployment).
Epoch AI’s "Algorithmic progress in language models" (Ho et al., March 2024), estimated from more than 200 language-model evaluations on Wikitext and Penn Treebank spanning 2012–2023, finds that the compute needed to reach a given performance level has halved roughly every eight months, with a 95% confidence interval of five to fourteen months — roughly 2–3× per year, at the lower end of the essay’s range and faster than the two-year doubling time associated with Moore’s law (Source: epoch.ai; arxiv.org). A complementary capability-side measure, the task "time horizon" tracked by METR — the length of software tasks agents can complete at 50% reliability — has doubled roughly every seven months (Measuring AI Ability to Complete Long Software Tasks).
Drivers
The Least Understood Driver of AI Progress identifies two underappreciated factors behind measured software progress:
- Data quality improvements: a shift from raw web data toward synthetic and processed data and distillation, which may explain the bulk of measured gains. Surge AI revenue exceeding $1B is cited as evidence of large industry investment in data.
- Scale-dependent innovations: a small number of key changes (LSTM→Transformer, Kaplan→Chinchilla scaling) that produce larger efficiency gains at higher compute scales. Much of measured progress may come from scaling up just 2–3 such innovations.
Relation to policy and other concepts
The estimated rate of software progress bears on several debates. Because software progress appears to depend on scaling compute, compute controls would also constrain software progress, which strengthens the case for compute-centric governance (Compute Governance). A rate of roughly 10× per year also implies that frontier advantages erode quickly, which is the premise of the Fast-Follow Problem.
The rate cuts both ways in the intelligence-explosion debate: higher software-progress rates make an explosion more likely, while compute bottlenecks arising from scale-dependent innovations make it less likely, leaving the net effect uncertain. The estimate is also in tension with the AI as Normal Technology thesis: if capabilities improve this fast, the prediction of decades-long diffusion is under pressure, though the scale-dependence of progress means it is not "free."
Relationships
- depends-on: The Least Understood Driver of AI Progress (source of the central 10×-per-year estimate)
- related: Measuring AI Ability to Complete Long Software Tasks (capability-side progress measure), Compute Governance (compute controls as a constraint on software progress), Fast-Follow Problem (premise: frontier advantages erode quickly), Distillation (a driver of measured data-side efficiency gains)
See also
- The Bitter Lesson — the thesis that general methods leveraging computation dominate
- Compute Governance — compute as the bottleneck and control point
- Compressed 21st Century — the prediction these rates enable
- Fast-Follow Problem
- AI as Normal Technology