Context length (also "context window") is the maximum number of tokens a model can process in a single forward pass, counting both input and generated output. It is one of the frontier axes, alongside parameters, training data, and test-time compute, along which models have scaled. Models grew from roughly 2K tokens to 1M+ over about three years, with research systems reaching 10M+.
Trajectory
The advertised context window has expanded by roughly 1,000× over six years, faster than parameter scaling over the same interval. The progression of notable models, newest on top:
| Model / milestone | Date | Context window |
|---|---|---|
| Frontier research | — | 10M+ via ring attention and specialized architectures |
| GPT-4.1 / Claude Sonnet 4.5+ / Gemini 2.5+ | 2025 | 1M+ |
| Gemini 1.5 Pro | Feb 2024 | 1M (and 10M experimental) |
| Claude 2.1 | Nov 2023 | 200K |
| Claude 2 | 2023 | 100K |
| GPT-4 | 2023 | 8K, then 32K |
| GPT-3 | 2020 | 2,048 tokens |
| GPT-2 | 2019 | 1,024 tokens |
The attention scaling problem
Standard transformer attention is O(n²) in sequence length: each token attends to every prior token. Doubling context quadruples attention compute and roughly doubles KV-cache memory. At 1M tokens this becomes prohibitive without algorithmic changes.
Several techniques enabled long context:
- Flash Attention (Dao et al., 2022) is an IO-aware kernel that reduces memory and improves wall-clock speed; it is a prerequisite for anything beyond roughly 32K.
- Ring Attention (Liu et al., 2023) splits the sequence across devices, with each device computing a block and passing KV through a ring, enabling near-linear scaling in devices.
- Sparse / local attention uses windowed or strided attention that attends to a subset of tokens, as in sliding-window models (Mistral, Gemma).
- Linear attention variants such as Performer, Mamba, and Hyena offer sub-quadratic alternatives. Hybrid architectures (Jamba, Zamba) mix state-space models with transformer layers to trade attention cost for state compression.
- KV cache compression uses quantization, eviction, and compression of cached keys/values to fit longer contexts in GPU memory.
Position encoding
A separate scaling challenge is telling the model where each token sits in the sequence. RoPE (Rotary Position Embeddings, Su et al., 2021) rotates query/key vectors by position-dependent angles and is the current default. RoPE extensions such as YaRN, LongRoPE, and Position Interpolation extend a model trained at short context to longer windows with minimal fine-tuning. Alternatives including ALiBi, NoPE, and learned positions have varying long-context properties. Anthropic's "position-agnostic" work and Gemini's long-context training reflect a pattern in which frontier labs typically train natively at long context rather than extending post-hoc.
Evaluation
The Needle-in-a-Haystack (NIAH) test (Kamradt, 2023) inserts a specific fact in a long document and asks the model to retrieve it; it became the first widely used long-context benchmark. Lost in the Middle (Liu et al., 2023) showed that retrieval accuracy is much higher at sequence boundaries than in the middle, meaning context utilization is not uniform. More demanding multi-step long-context benchmarks include RULER, LongBench, and InfiniteBench. Across frontier models, reasoning and factuality tend to degrade as context fills, even below the advertised window, so real useful context is often less than nominal.
Current state
All frontier models ship with 200K–1M+ context. Open-weights models have followed: Llama 4, Qwen3, Kimi K2, and DeepSeek-V3 all support long context. Effective context, as measured by NIAH and RULER, typically lags nominal context, and quality across the window is uneven. Inference cost and latency at long context remain significant barriers, and many workloads still run at 32K–128K in practice.
Relation to policy and safety
Agentic workflows accumulate context aggressively through tool outputs, intermediate reasoning, and observed pages, making long context a prerequisite for serious agentic capability. Document and code understanding in legal, medical, scientific, and code workflows all depend on long-context capability. Long context lets models ingest entire books or codebases as single documents, changing training-data economics and copyright surfaces. It also blurs the line between "memory" and "session" in chat products, and privacy regimes such as GDPR and CCPA interact with context persistence.
Benchmark validity depends on whether nominal context matches effective context; the lost-in-the-middle effect means many capability claims are optimistic. More context also expands the surface for Prompt Injection, since it creates more places an attacker can hide instructions.
Relationships
- depends-on: Scaling Laws — long context is a distinct scaling axis
- related: Agentic AI — long context is prerequisite for agentic capability
- related: Reasoning Models and Chain-of-Thought — reasoning traces consume context aggressively
- related: Inference Economics and Token Pricing — long-context inference has specific cost structure (KV cache)
- related: Prompt Injection — longer contexts expand attack surface
- related: AI Benchmarks and Evaluation — NIAH and successors