"LLaMA: Open and Efficient Foundation Language Models" is a paper submitted to arXiv on February 27, 2023 (arXiv:2302.13971) by fourteen authors at Meta AI, with Hugo Touvron, Thibaut Lavril, Gautier Izacard, Edouard Grave and Guillaume Lample marked as equal contributors. It introduces a family of foundation language models ranging from 7B to 65B parameters and states two claims in its abstract: "that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets," and that "LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B." The paper closes the abstract with "We release all our models to the research community."
The inference-budget argument
The paper's framing argument is that the scaling-law objective of Hoffmann et al. (2022) determines how to scale dataset and model size for a given training compute budget, but "disregards the inference budget, which becomes critical when serving a language model at scale." Its statement of the alternative: "given a target level of performance, the preferred model is not the fastest to train but the fastest at inference, and although it may be cheaper to train a large model to reach a certain level of performance, a smaller one trained longer will ultimately be cheaper at inference." The authors report an empirical basis for training past the Chinchilla-optimal point: "although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens."
The stated purpose of the release is access: "We believe that this model will help democratize the access and study of LLMs, since it can be run on a single GPU."
Training data
The paper reports a training mixture drawn only from publicly available sources compatible with open-sourcing, totalling roughly 1.4T tokens after tokenization, with each token used once except for Wikipedia and Books, over which approximately two epochs are performed.
| Dataset | Sampling proportion | Epochs | Disk size |
|---|---|---|---|
| CommonCrawl | 67.0% | 1.10 | 3.3 TB |
| C4 | 15.0% | 1.06 | 783 GB |
| Github | 4.5% | 0.64 | 328 GB |
| Wikipedia | 4.5% | 2.45 | 83 GB |
| Books | 4.5% | 2.23 | 85 GB |
| ArXiv | 2.5% | 1.06 | 92 GB |
| StackExchange | 2.0% | 1.03 | 78 GB |
The paper describes the preprocessing for each: five CommonCrawl dumps from 2017 to 2020 through the CCNet pipeline with line-level deduplication, fastText language identification, n-gram quality filtering and a linear classifier that discards pages not classified as Wikipedia references; GitHub filtered to Apache, BSD and MIT licensed projects; Wikipedia dumps from June–August 2022 across 20 languages in Latin or Cyrillic script; and books from Project Gutenberg together with "the Books3 section of ThePile (Gao et al., 2020)," deduplicated at book level for more than 90% content overlap. The inclusion of Books3 is the detail most often cited in the later copyright litigation over training corpora; see AI Copyright. Tokenization uses byte-pair encoding via SentencePiece, splitting all numbers into individual digits and falling back to bytes for unknown UTF-8 characters.
Architecture and compute
The models are transformers with three named modifications, each attributed to prior work: pre-normalization using RMSNorm, taken from GPT-3, normalizing the input rather than the output of each sub-layer for training stability; the SwiGLU activation function from PaLM, at dimension 2/3 · 4d rather than PaLM's 4d; and rotary positional embeddings (RoPE) from GPT-Neo, replacing absolute positional embeddings at each layer. Training uses AdamW with β₁ = 0.9, β₂ = 0.95, a cosine schedule to a final rate of 10% of maximum, weight decay 0.1, gradient clipping 1.0, and 2,000 warmup steps.
| Params | Dimension | Heads | Layers | Learning rate | Batch size | Tokens |
|---|---|---|---|---|---|---|
| 6.7B | 4096 | 32 | 32 | 3.0e−4 | 4M | 1.0T |
| 13.0B | 5120 | 40 | 40 | 3.0e−4 | 4M | 1.0T |
| 32.5B | 6656 | 52 | 60 | 1.5e−4 | 4M | 1.4T |
| 65.2B | 8192 | 64 | 80 | 1.5e−4 | 4M | 1.4T |
The authors estimate they used 2,048 A100-80GB GPUs over approximately five months to develop the models, which under their assumptions would have cost around 2,638 MWh and produced a total emission of 1,015 tCO₂eq, computed using the US national average carbon-intensity factor of 0.385 kg CO₂eq/KWh rather than any particular data center's grid.
Reported results
On zero-shot common-sense reasoning the paper reports LLaMA-65B outperforming Chinchilla-70B on all reported benchmarks but BoolQ, and PaLM-540B everywhere but BoolQ and WinoGrande, with LLaMA-13B outperforming GPT-3 on most benchmarks "despite being 10× smaller."
| Model | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA |
|---|---|---|---|---|---|---|---|---|
| GPT-3 175B | 60.5 | 81.0 | – | 78.9 | 70.2 | 68.8 | 51.4 | 57.6 |
| Gopher 280B | 79.3 | 81.8 | 50.6 | 79.2 | 70.1 | – | – | – |
| Chinchilla 70B | 83.7 | 81.8 | 51.3 | 80.8 | 74.9 | – | – | – |
| PaLM 540B | 88.0 | 82.3 | – | 83.4 | 81.1 | 76.6 | 53.0 | 53.4 |
| LLaMA 7B | 76.5 | 79.8 | 48.9 | 76.1 | 70.1 | 72.8 | 47.6 | 57.2 |
| LLaMA 13B | 78.1 | 80.1 | 50.4 | 79.2 | 73.0 | 74.8 | 52.7 | 56.4 |
| LLaMA 33B | 83.1 | 82.3 | 50.4 | 82.8 | 76.0 | 80.0 | 57.8 | 58.6 |
| LLaMA 65B | 85.3 | 82.8 | 52.3 | 84.2 | 77.0 | 78.9 | 56.0 | 60.2 |
On closed-book question answering the paper reports LLaMA-65B achieving state-of-the-art performance in zero-shot and few-shot settings on both Natural Questions and TriviaQA, and notes that LLaMA-13B is competitive on these benchmarks with GPT-3 and Chinchilla "despite being 5-10× smaller. This model runs on a single V100 GPU during inference." On TriviaQA exact match, LLaMA-65B scores 68.2 zero-shot and 73.0 at 64 shots, against Chinchilla-70B's 55.4 and 64.6 and Gopher-280B's 43.5 and 57.2.
On MMLU at five shots the paper reports LLaMA-65B at 63.4 average, "behind both Chinchilla-70B and PaLM-540B by a few percent in average, and across most domains." On mathematical reasoning it reports LLaMA-65B outperforming Minerva-62B on GSM8k "although it has not been fine-tuned on mathematical data." A single instruction-finetuning experiment following the protocol of Chung et al. (2022) produced LLaMA-I, reaching 68.9% on MMLU — better than instruction-finetuned models of moderate size, but the paper states this remains "far from the state-of-the-art, that is 77.4 for GPT code-davinci-002."
The paper also reports bias, toxicity and misinformation evaluations, including WinoGender co-reference accuracy — where the authors observe better performance on "their/them/someone" pronouns than on gendered pronouns and state this is "likely indicative of biases" — and TruthfulQA, where the 65B model reports 0.57 truthful and 0.53 truthful-and-informative.
Provenance
The arXiv abstract page and the full 27-page PDF were retrieved August 1, 2026 by direct HTTPS fetch (HTTP 200); PDF text was extracted with pdfplumber. Firecrawl was out of credits during this cycle. The capture covers pages 1–12, through the main results and the responsible-AI evaluations. The extraction does not preserve inter-word spacing in the two-column body text or table layout, and omits figures; tables above were transcribed from the extracted rows. Only v1 (submitted 27 Feb 2023) is listed on the abstract page.
Relationships
- related: Meta AI — the publisher.
- related: LLaMA 1 (Meta AI) — the models the paper introduces.
- depends-on: Scaling Laws — the Hoffmann et al. result the paper's framing argument modifies by adding the inference budget.
- related: Open-Weight Frontier Models — the release practice this paper's family is the origin case for.
- related: AI Copyright — the Books3 and CommonCrawl composition later at issue in copyright litigation.
- related: Llama 3 (Meta) — a later generation of the same family.