"LLaMA: Open and Efficient Foundation Language Models" is a paper submitted to arXiv on February 27, 2023 (arXiv:2302.13971) by fourteen authors at Meta AI, with Hugo Touvron, Thibaut Lavril, Gautier Izacard, Edouard Grave and Guillaume Lample marked as equal contributors. It introduces a family of foundation language models ranging from 7B to 65B parameters and states two claims in its abstract: "that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets," and that "LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B." The paper closes the abstract with "We release all our models to the research community."
The inference-budget argument
The paper's framing argument is that the scaling-law objective of Hoffmann et al. (2022) determines how to scale dataset and model size for a given training compute budget, but "disregards the inference budget, which becomes critical when serving a language model at scale." Its statement of the alternative: "given a target level of performance, the preferred model is not the fastest to train but the fastest at inference, and although it may be cheaper to train a large model to reach a certain level of performance, a smaller one trained longer will ultimately be cheaper at inference." The authors report an empirical basis for training past the Chinchilla-optimal point: "although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens."
The stated purpose of the release is access: "We believe that this model will help democratize the access and study of LLMs, since it can be run on a single GPU."
Training data
The paper reports a training mixture drawn only from publicly available sources compatible with open-sourcing, totalling roughly 1.4T tokens after tokenization, with each token used once except for Wikipedia and Books, over which approximately two epochs are performed.
| Dataset | Sampling proportion | Epochs | Disk size |
|---|---|---|---|
| CommonCrawl | 67.0% | 1.10 | 3.3 TB |
| C4 | 15.0% | 1.06 | 783 GB |
| Github | 4.5% | 0.64 | 328 GB |
| Wikipedia | 4.5% | 2.45 | 83 GB |
| Books | 4.5% | 2.23 | 85 GB |
| ArXiv | 2.5% | 1.06 | 92 GB |
| StackExchange | 2.0% | 1.03 | 78 GB |
The paper describes the preprocessing for each: five CommonCrawl dumps from 2017 to 2020 through the CCNet pipeline with line-level deduplication, fastText language identification, n-gram quality filtering and a linear classifier that discards pages not classified as Wikipedia references; GitHub filtered to Apache, BSD and MIT licensed projects; Wikipedia dumps from June–August 2022 across 20 languages in Latin or Cyrillic script; and books from Project Gutenberg together with "the Books3 section of ThePile (Gao et al., 2020)," deduplicated at book level for more than 90% content overlap. The inclusion of Books3 is the detail most often cited in the later copyright litigation over training corpora; see AI Copyright. Tokenization uses byte-pair encoding via SentencePiece, splitting all numbers into individual digits and falling back to bytes for unknown UTF-8 characters.
Architecture and compute
The models are transformers with three named modifications, each attributed to prior work: pre-normalization using RMSNorm, taken from GPT-3, normalizing the input rather than the output of each sub-layer for training stability; the SwiGLU activation function from PaLM, at dimension 2/3 · 4d rather than PaLM's 4d; and rotary positional embeddings (RoPE) from GPT-Neo, replacing absolute positional embeddings at each layer. Training uses AdamW with β₁ = 0.9, β₂ = 0.95, a cosine schedule to a final rate of 10% of maximum, weight decay 0.1, gradient clipping 1.0, and 2,000 warmup steps.
| Params | Dimension | Heads | Layers | Learning rate | Batch size | Tokens |
|---|---|---|---|---|---|---|
| 6.7B | 4096 | 32 | 32 | 3.0e−4 | 4M | 1.0T |
| 13.0B | 5120 | 40 | 40 | 3.0e−4 | 4M | 1.0T |
| 32.5B | 6656 | 52 | 60 | 1.5e−4 | 4M | 1.4T |
| 65.2B | 8192 | 64 | 80 | 1.5e−4 | 4M | 1.4T |
The authors estimate they used 2,048 A100-80GB GPUs over approximately five months to develop the models, which under their assumptions would have cost around 2,638 MWh and produced a total emission of 1,015 tCO₂eq, computed using the US national average carbon-intensity factor of 0.385 kg CO₂eq/KWh rather than any particular data center's grid.
Reported results
Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.
| Evaluation | Model | Score | Setting | Section |
|---|---|---|---|---|
| MBPP — pass@1 | LLaMA 7B | 17.7 | 3-shot prompts, temperature 0.1; developer-reported | Table 8 (§3.5) |
| MBPP — pass@1 | LLaMA 13B | 22.0 | developer-reported | Table 8 (§3.5) |
| MBPP — pass@1 | LLaMA 33B | 30.2 | developer-reported | Table 8 (§3.5) |
| MBPP — pass@1 | LLaMA 65B | 37.7 | developer-reported | Table 8 (§3.5) |
| MBPP — pass@1 | LaMDA 137B | 14.8 | reported by Meta in the LLaMA paper | Table 8 (§3.5) |
| MBPP — pass@1 | PaLM 8B | 5.0 | read from figures (∗); reported by Meta in the LLaMA paper | Table 8 (§3.5) |
| MBPP — pass@1 | PaLM 62B | 21.4 | reported by Meta in the LLaMA paper | Table 8 (§3.5) |
| MBPP — pass@1 | PaLM 540B | 36.8 | reported by Meta in the LLaMA paper | Table 8 (§3.5) |
| MBPP — pass@80 | LLaMA 7B | 56.2 | temperature 0.8; developer-reported | Table 8 (§3.5) |
| MBPP — pass@80 | LLaMA 13B | 64.0 | developer-reported | Table 8 (§3.5) |
| MBPP — pass@80 | LLaMA 33B | 73.4 | developer-reported | Table 8 (§3.5) |
| MBPP — pass@80 | LLaMA 65B | 76.8 | developer-reported | Table 8 (§3.5) |
| MBPP — pass@80 | LaMDA 137B | 62.4 | reported by Meta in the LLaMA paper | Table 8 (§3.5) |
| MBPP — pass@80 | PaLM 8B | 35.7 | read from figures (∗); reported by Meta in the LLaMA paper | Table 8 (§3.5) |
| MBPP — pass@80 | PaLM 62B | 63.2 | read from figures (∗); reported by Meta in the LLaMA paper | Table 8 (§3.5) |
| MBPP — pass@80 | PaLM 540B | 75.0 | reported by Meta in the LLaMA paper | Table 8 (§3.5) |
| Natural Questions — 0-shot | LLaMA 7B | 16.8 | exact match, closed book; developer-reported (13B 20.1, 33B 24.9, 65B 23.8) | Table 4 (§3.2) |
| Natural Questions — 1-shot | LLaMA 65B | 31.0 | developer-reported (7B 18.7, 13B 23.4, 33B 28.3) | Table 4 (§3.2) |
| Natural Questions — 5-shot | LLaMA 65B | 35.0 | developer-reported (7B 22.0, 13B 28.1, 33B 32.9) | Table 4 (§3.2) |
| Natural Questions — 64-shot | LLaMA 65B | 39.9 | developer-reported (7B 26.1, 13B 31.9, 33B 36.0) | Table 4 (§3.2) |
| Natural Questions — 64-shot | GPT-3 175B | 29.9 | 0-shot 14.6, 1-shot 23.0; reported by Meta in the LLaMA paper | Table 4 (§3.2) |
| Natural Questions — 64-shot | Gopher 280B | 28.2 | 0-shot 10.1, 5-shot 24.5; reported by Meta in the LLaMA paper | Table 4 (§3.2) |
| Natural Questions — 64-shot | Chinchilla 70B | 35.5 | 0-shot 16.6, 5-shot 31.5; reported by Meta in the LLaMA paper | Table 4 (§3.2) |
| Natural Questions — 64-shot | PaLM 540B | 39.6 | 0-shot 21.2, 1-shot 29.3; PaLM 8B 14.6 and PaLM 62B 27.6 at 64-shot; reported by Meta in the LLaMA paper | Table 4 (§3.2) |
| TriviaQA — 0-shot | LLaMA 65B | 68.2 | exact match on the filtered dev set; developer-reported (7B 50.0, 13B 56.6, 33B 65.1) | Table 5 (§3.2) |
| TriviaQA — 1-shot | LLaMA 65B | 71.6 | developer-reported (7B 53.4, 13B 60.5, 33B 67.9) | Table 5 (§3.2) |
| TriviaQA — 5-shot | LLaMA 65B | 72.6 | developer-reported (7B 56.3, 13B 63.1, 33B 69.9) | Table 5 (§3.2) |
| TriviaQA — 64-shot | LLaMA 65B | 73.0 | developer-reported (7B 57.6, 13B 64.0, 33B 70.4) | Table 5 (§3.2) |
| TriviaQA — 64-shot | Gopher 280B | 57.2 | 0-shot 43.5, 5-shot 57.0; reported by Meta in the LLaMA paper | Table 5 (§3.2) |
| TriviaQA — 64-shot | Chinchilla 70B | 64.6 | 0-shot 55.4, 5-shot 64.1; reported by Meta in the LLaMA paper | Table 5 (§3.2) |
Provenance
The arXiv abstract page and the full 27-page PDF were retrieved August 1, 2026 by direct HTTPS fetch (HTTP 200); PDF text was extracted with pdfplumber. Firecrawl was out of credits during this cycle. The capture covers pages 1–12, through the main results and the responsible-AI evaluations. The extraction does not preserve inter-word spacing in the two-column body text or table layout, and omits figures; tables above were transcribed from the extracted rows. Only v1 (submitted 27 Feb 2023) is listed on the abstract page.
Relationships
- related: Meta AI — the publisher.
- related: LLaMA 1 (Meta AI) — the models the paper introduces.
- depends-on: Scaling Laws — the Hoffmann et al. result the paper's framing argument modifies by adding the inference budget.
- related: Open-Weight Frontier Models — the release practice this paper's family is the origin case for.
- related: AI Copyright — the Books3 and CommonCrawl composition later at issue in copyright litigation.
- related: Llama 3 (Meta) — a later generation of the same family.