Llama 3 is a family of open-weight large language models developed by Meta AI, released across 2024. It was the first instance of a major US lab shipping a dense model at the parameter scale of a frontier closed model — Llama 3.1 405B — under an open-weights license. At release, Meta reported Llama 3.1 405B as benchmark-competitive with GPT-4-class models on MMLU, HumanEval, and GSM8K. (Source: epochai.substack.com, State of AI Report 2025)
| Field | Value | |
|---|---|---|
| Developer | [[meta | Meta AI]] |
| Released | Llama 3 8B/70B: April 18, 2024. Llama 3.1 (incl. 405B): July 23, 2024. Llama 3.2 (multimodal + small): Sept 2024. Llama 3.3 70B: Dec 2024. | |
| Architectures | Dense decoder-only Transformer (no MoE in this generation) | |
| Parameter sizes | 8B, 70B, 405B (plus Llama 3.2's 1B, 3B, 11B-Vision, 90B-Vision) | |
| Training tokens | ~15T (8B/70B); 15.6T (405B) | |
| Context length | 8k → 128k (Llama 3.1+) | |
| License | Llama 3 Community License — open weights with key commercial restrictions |
Capabilities and benchmarks
At release, Meta reported Llama 3.1 405B as competitive with GPT-4-class closed models across standard evaluations. The figures below are Meta's own numbers and were not independently verified. (Source: epochai.substack.com)
| Benchmark | Llama 3.1 405B | GPT-4o | Claude 3.5 Sonnet |
|---|---|---|---|
| MMLU | 88.6 | 88.7 | 88.3 |
| HumanEval | 89.0 | 90.2 | 92.0 |
| GSM8K | 96.8 | — | 96.4 |
| MATH | 73.8 | 76.6 | 71.1 |
Before Llama 3.1, the best available open weights — Llama 2 70B and Mixtral 8x22B — lagged frontier closed models by a clear margin. After Llama 3.1, the gap narrowed enough that many enterprise deployments could substitute open weights for closed APIs. (Source: epochai.substack.com, State of AI Report 2025)
Training and architecture
Llama 3 used a dense decoder-only Transformer across all sizes. Meta opted against a mixture-of-experts (MoE) design for this generation, arguing that dense models were better understood and more deployable.
The models were trained on roughly 15 trillion tokens for the 8B and 70B sizes, expanded to 15.6 trillion tokens for the 405B. This was substantially larger than Llama 2's 2 trillion tokens. The training was roughly Chinchilla-optimal at 8B but heavily over-trained on a tokens-per-parameter basis at 70B and 405B, reflecting a shift toward inference-efficient training.
Other architectural choices included grouped-query attention across all sizes — including 8B, where Llama 2 had used multi-head attention — and a new 128k-token vocabulary, up from Llama 2's 32k. Post-training used multi-stage supervised fine-tuning, rejection sampling, and direct preference optimization (DPO); Meta did not use RLHF-PPO, in contrast to OpenAI and Anthropic practice. Llama 3.1 405B was trained on a 16,000 H100 cluster and was, as of release, the largest openly released dense model.
Safety and evaluations
Alongside Llama 3.1, Meta released Llama Guard 3 (a content-moderation model), Prompt Guard (a prompt-injection classifier), and CyberSecEval 3 (a cyber-capability benchmark). Meta also ran pre-release dangerous-capability evaluations covering CBRN and offensive cyber and published results, though with less granularity than Anthropic's RSP reports.
Availability and licensing
The Llama 3 Community License is not OSI-approved open source. Its key restrictions are:
- Monthly active user cap: services with more than 700M MAU at release require a separate Meta license. This clause specifically targets the largest cloud providers and was read as directed at Amazon, Google, and Microsoft.
- Attribution requirement: products must display "Built with Llama."
- Use restrictions: the license prohibits use to improve "any other LLM," excluding derivatives of Llama 3 — a restriction on competitor distillation.
This contrasts with the more permissive licensing of Qwen3 (Apache 2.0) and DeepSeek-V3 (MIT-style). The license became a reference point in debates over what "open" should mean in the context of Open-Source AI / Open-Weight Models, cited both as a step toward openness and as an example of licensing terms that blur the category.
Reception and context
Llama 3 has been characterized as the peak of Meta's open-weight commitment before the Llama 4 pullback. (Source: epochai.substack.com) Its release immediately preceded the Chinese models DeepSeek-V3 and Qwen3, both of which explicitly benchmarked against it and went on to use MoE architectures and more permissive licensing. The 405B release functioned as a precedent for open frontier-scale releases that subsequent labs built on.
Related models
- instance-of: General-Purpose AI (GPAI)
- superseded-by: Llama 4 (Meta) (partial — Llama 3 remained supported alongside Llama 4)
- related: Meta AI, Open-Source AI / Open-Weight Models, GPT-4 Family (OpenAI), Qwen3, DeepSeek-V3, Mark Zuckerberg