The Llama 3 Herd of Models (Meta, July 2024)

"The Llama 3 Herd of Models" is the technical report accompanying Meta's Llama 3 release, posted to arXiv on July 31, 2024 by the Llama Team at Meta AI and last revised November 23, 2024 (arXiv:2407.21783). It documents three dense Transformer language models of 8B, 70B and 405B parameters, the data and infrastructure used to train them, the post-training recipe, benchmark results against contemporaneous open and closed models, safety evaluations, and separate experiments attaching image, video and speech capabilities through adapters. The report accompanied the open release of pre-trained and post-trained weights for all three sizes under the Llama 3 Community License, together with an updated Llama Guard safety model.

Models and training

The report describes the flagship model as a 405B-parameter dense Transformer trained on 15.6 trillion tokens with an 8K-token context window, followed by a continued pre-training stage that extends the supported context to 128K tokens. The 8B and 70B models were pre-trained on roughly the same corpus; the report states they were pre-trained on multilingual data but were intended for use in English at the time of release. Meta reports choosing a dense architecture over a mixture of experts on the grounds of training stability and development simplicity, and describes a scaling-law procedure used to set model size given a compute budget.

Post-training is described as several rounds of supervised finetuning on instruction data followed by Direct Preference Optimization, with rejection sampling between rounds; the report states that Meta did not use PPO-style reinforcement learning from human feedback. Tool use, coding, reasoning, multilinguality and long-context behaviour are introduced or strengthened at this stage.

Evaluation approach and results

The report separates pre-trained-model evaluation (§5.1) from post-trained-model evaluation (§5.2). Pre-trained results are reported with 95% confidence intervals across reading comprehension (SQuAD, QuAC, RACE), code (HumanEval, MBPP), commonsense understanding (CommonSenseQA, PiQA, SiQA, OpenBookQA, Winogrande), mathematics and reasoning (GSM8K, MATH, ARC-C, DROP, WorldSense), general language tasks (MMLU, MMLU-Pro, AGIEval, BIG-Bench Hard) and long-context tasks (QuALITY, long-context GSM8K), with Mistral 7B, Gemma 7B, Mixtral 8×22B, GPT-4, Nemotron 4 340B and Gemini Ultra as comparison columns where published figures exist.

The headline post-training table (Table 2) reports the finetuned 8B, 70B and 405B models against Gemma 2 9B, Mistral 7B, Mixtral 8×22B, GPT-3.5 Turbo, Nemotron 4 340B, GPT-4 (0125), GPT-4o and Claude 3.5 Sonnet on sixteen benchmarks spanning general knowledge and instruction following (MMLU 5-shot and 0-shot CoT, MMLU-Pro, IFEval), code (HumanEval, MBPP EvalPlus), mathematics (GSM8K, MATH), reasoning (ARC Challenge, GPQA), tool use (BFCL, Nexus), long context (ZeroSCROLLS/QuALITY, InfiniteBench/En.MC, NIH/Multi-needle) and multilinguality (MGSM). Meta's summary of these results is that the 8B and 70B models outperform other models in their size classes, and that the 405B model "performs on par with leading language models such as GPT-4 … and is close to matching the state-of-the-art"; the report notes that Claude 3.5 Sonnet leads on several benchmarks in the largest size class.

The report also presents human preference evaluations comparing Llama 3 405B with GPT-4, GPT-4o and Claude 3.5 Sonnet, robustness experiments measuring how MMLU scores move with label variants, answer order, few-shot label bias and prompt format, and adversarial versions of question-answering, mathematical-reasoning and paraphrase-detection benchmarks.

Contamination analysis

Section 5.1.4 reports a contamination analysis following the method of Singh et al. (2024), giving for each key benchmark the percentage of evaluation data judged contaminated by similar data in the pre-training corpus together with an estimated performance gain attributable to that contamination, broken out by model size. Meta reports that contamination has a large estimated impact on some datasets — including AGIEval and BIG-Bench Hard — and little on others, and states that how best to run such analyses remains an open research question.

Safety

The safety section covers benchmark evaluations of violation and false-refusal rates, red teaming, and uplift testing in cyber and chemical/biological domains, alongside the release of Llama Guard 3 for input and output filtering and Code Shield for insecure-code detection. The report presents safety as a system-level property: Meta releases the base and instruction-tuned weights together with safeguards that deployers can compose, rather than relying on model-level refusal alone.

Multimodal experiments

Sections 7 and 8 describe compositional experiments that add image, video and speech understanding through separately trained encoders and cross-attention adapters, with the language-model parameters held fixed during adapter training. The report states that these multimodal models were still under development and not released at the time of publication.

Provenance and confidence

Confidence is high for the documented architecture, training scale and reported figures, which come directly from the paper. All benchmark numbers for Llama 3 are developer-reported, and the comparison columns are drawn by Meta from other developers' published figures or from Meta's own runs, so cross-model comparisons carry the harness and prompting differences the report itself notes. The paper is a Meta-authored account of Meta's models; independent replication of its headline claims is not part of the document.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

Relationships

• supports: Llama 3 (Meta) (canonical primary reference)
• depends-on: Scaling Laws
• related: Meta AI, Open-Source AI / Open-Weight Models, LLaMA: Open and Efficient Foundation Language Models (Touvron et al., Meta AI, February 2023), Llama 4 (Meta)
• instance-of: AI System Cards