Qwen2.5 Technical Report (Qwen Team, December 2024)

The Qwen2.5 Technical Report is the Qwen team's account of the Qwen2.5 series of large language models, posted to arXiv as 2412.15115 on 19 December 2024 and revised on 3 January 2025 (Source: arxiv.org). It covers seven open-weight dense models — 0.5B, 1.5B, 3B, 7B, 14B, 32B and 72B parameters, each in base and instruction-tuned form with quantized variants — and two proprietary mixture-of-experts models served through Alibaba Cloud Model Studio, Qwen2.5-Turbo and Qwen2.5-Plus. The report's central claims are that pre-training data grew from 7 trillion tokens in Qwen2 to 18 trillion, that post-training combined supervised fine-tuning on over one million samples with a two-stage reinforcement-learning process, and that the open-weight flagship Qwen2.5-72B-Instruct is competitive with Llama-3-405B-Instruct, "which is around 5 times larger" (the conclusion says six times). The report also presents Qwen2.5 as the foundation for the Qwen2.5-Math, Qwen2.5-Coder and QwQ specialist models.

Models and architecture

The dense models keep Qwen2's Transformer decoder design: grouped-query attention, SwiGLU activations, rotary position embeddings, QKV bias and pre-normalised RMSNorm. The MoE models replace feed-forward layers with expert layers using fine-grained expert segmentation and shared-expert routing, following Qwen1.5-MoE. All models share a byte-level BPE tokenizer with 151,643 regular tokens, and the control-token set was expanded from 3 to 22, including two tokens for tool use (§2). Table 1 gives layer counts, query/key-value head counts, embedding tying, context and generation lengths (32K/8K for the three smallest models, 128K/8K from 7B upward) and licences: Apache 2.0 for 0.5B, 1.5B, 7B, 14B and 32B; a Qwen Research licence for 3B; and a Qwen licence for 72B.

Pre-training (§3) scaled the corpus to 18 trillion tokens with filtering, synthetic mathematics and code data from Qwen2.5-Math and Qwen2.5-Coder, hyperparameter scaling laws, and a long-context stage. Post-training (§4) added supervised fine-tuning aimed at long-sequence generation, mathematics, coding, instruction following, structured data, logical reasoning, cross-lingual transfer and system-prompt robustness; an offline reinforcement-learning stage using DPO for capabilities the reward model finds hard to judge; an online stage using GRPO guided by a reward model scored on truthfulness, helpfulness, conciseness, relevance, harmlessness and debiasing; and long-context fine-tuning.

Evaluation approach

The evaluation suite (§5) is described as "primarily automatic with minimal human interaction." Contamination control uses n-gram matching: a training sequence is removed when its longest common subsequence with any test sequence is at least 13 tokens and at least 60% of the shorter sequence. Base models are scored on general tasks (MMLU, MMLU-Pro, MMLU-redux, BBH, ARC-C, TruthfulQA, Winogrande, HellaSwag, with stated shot counts), mathematics and science (GPQA, TheoremQA, GSM8K, MATH), coding (HumanEval, HumanEval+, MBPP, MBPP+, MultiPL-E) and four multilingual groups (exams, understanding, mathematics, translation), against Llama 3 70B and 405B, Mixtral 8x22B, Gemma 2, Mistral 7B, Yi-1.5 and the previous Qwen2 and Qwen1.5 models by size class (Tables 2–5).

Instruction-tuned models are scored on MMLU-Pro, MMLU-redux, LiveBench 0831, GPQA, GSM8K, MATH, HumanEval, MBPP, MultiPL-E, LiveCodeBench (problems from May 2023 to September 2024), IFEval strict-prompt, Arena-Hard and MT-Bench (§5.2.1, Tables 6–10), against Llama 3.1 70B and 405B Instruct, Gemma 2 27B, GPT-4o mini and prior Qwen models. The report adds in-house English and Chinese automatic evaluations of instruction following, knowledge, comprehension, coding, mathematics and reasoning that include GPT-4o and Claude 3.5 Sonnet (§5.2.2, Tables 11–12); multilingual instruct evaluations (Tables 13–14); a separate evaluation of the reward model on Reward Bench, RMB, PPE and an internal Chinese preference set against Nemotron-4-340B-Reward, Llama-3.1-Nemotron-70B-Reward and Athene-RM-70B (§5.2.3, Table 15); and long-context evaluations on RULER, LV-Eval (keyword recall) and LongBench-Chat with and without the DCA + YARN extrapolation techniques (§5.2.4, Tables 16–17).

Findings as stated by the authors

The Qwen2.5-72B base model is described as outperforming its peers across a wide range of tasks and matching Llama-3-405B on several while being smaller; Qwen2.5-Plus is said to score higher still on most rows of Table 2. For instruct models, the report states that Qwen2.5-72B-Instruct surpasses Llama-3.1-405B-Instruct on MMLU-redux, MATH, MBPP, MultiPL-E, LiveCodeBench, Arena-Hard and MT-Bench, and that Qwen2.5-Plus beats Qwen2.5-72B-Instruct on 9 of 13 benchmarks (Table 6). In the mid-size class, Qwen2.5-32B-Instruct is presented as leading most tasks, Qwen2.5-14B-Instruct as rivalling GPT-4o mini, and Qwen2.5-Turbo as beating Qwen2.5-14B-Instruct on eight of ten benchmarks at lower cost (Table 7). On long context, Qwen2.5-72B-Instruct is reported as the strongest model at every length, and Qwen2.5-Turbo as reaching 100% on a 1M-token passkey retrieval task, with a MInference-based sparse attention cutting attention compute 12.5-fold at 1M tokens and speeding time to first token 3.2 to 4.3 times.

On reward models the authors record a limitation: across several benchmarks they found that optimising for one benchmark can degrade others, and that "a higher score on RM benchmarks does not necessarily correlate with superior performance of the resulting RL model," which they present as a need for more predictive reward-model evaluations (§5.2.3). The conclusion sets out future work on data quality, multimodal unification and scaling inference compute for reasoning. The report contains no safety, refusal or dangerous-capability evaluations.

Provenance and confidence

The raw file is a scrape of the arXiv HTML rendering of v2 and includes the site's page chrome; tables are preserved as markdown, figures by caption only. Comparison figures are not attributed per cell to a source, and the suite is described as applied uniformly, so baseline rows are the Qwen team's own runs as far as the report states. Confidence is medium: a single developer-authored source whose benchmark claims are of a generation since superseded by Qwen3.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

Relationships

• supports: Qwen 2.5 (primary reference for the series' architecture, training and launch evaluations)
• superseded-by: Qwen3 Technical Report (the successor generation's report, which uses Qwen2.5 specialists as synthetic-data generators)
• related: Llama 3 (Meta), Mistral Large / Mistral Medium / Mixtral, Gemma (Google open-weight models), GPT-4 Family (OpenAI) (comparison models), Alibaba / Qwen Team, MMLU, MMLU-Pro, GPQA, MATH and MATH-500, GSM8K, HumanEval, LiveCodeBench
• instance-of: Open-Weight Frontier Models