LLaMA is a family of dense decoder-only foundation language models introduced by Meta AI in a paper submitted to arXiv on February 27, 2023 (LLaMA: Open and Efficient Foundation Language Models (Touvron et al., Meta AI, February 2023)). Four sizes were trained — 6.7B, 13.0B, 32.5B and 65.2B parameters — using only publicly available data. The paper's stated results are that "LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B," and it closes its abstract with "We release all our models to the research community."
| Field | Value | |
|---|---|---|
| Developer | [[companies/meta\ | Meta AI]] |
| Paper submitted | February 27, 2023 (arXiv:2302.13971) | |
| Sizes | 6.7B, 13.0B, 32.5B, 65.2B | |
| Training tokens | 1.0T (7B, 13B); 1.4T (33B, 65B) | |
| Architecture | Dense decoder-only transformer; RMSNorm pre-normalization, SwiGLU, rotary positional embeddings | |
| Training data | Publicly available sources only (~1.4T tokens) | |
| Training hardware | ~2,048 A100-80GB GPUs over ~5 months | |
| Open weights | Yes (released to the research community) |
Design argument
The family's design follows from an argument about inference rather than training cost. The paper accepts the Chinchilla scaling result (Scaling Laws) but observes that it "disregards the inference budget, which becomes critical when serving a language model at scale," and concludes that "given a target level of performance, the preferred model is not the fastest to train but the fastest at inference." The authors report training well past the compute-optimal point on that basis, finding that "the performance of a 7B model continues to improve even after 1T tokens" against the Chinchilla recommendation of 200B tokens for a 10B model (LLaMA: Open and Efficient Foundation Language Models (Touvron et al., Meta AI, February 2023)).
Access is stated as the purpose of the release: Meta wrote that the model "will help democratize the access and study of LLMs, since it can be run on a single GPU," and reported that LLaMA-13B "runs on a single V100 GPU during inference."
Architecture and training
The architecture is a transformer with three modifications the paper attributes to prior work: RMSNorm pre-normalization on sub-layer inputs, taken from GPT-3; the SwiGLU activation from PaLM at dimension 2/3 · 4d; and rotary positional embeddings from GPT-Neo in place of absolute positional embeddings. Optimization uses AdamW with a cosine schedule, weight decay 0.1, gradient clipping 1.0 and 2,000 warmup steps.
The training mixture is drawn entirely from public sources: CommonCrawl 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, Books 4.5%, ArXiv 2.5% and StackExchange 2%. The Books component combines Project Gutenberg with "the Books3 section of ThePile" — the inclusion later at issue in copyright litigation over training corpora; see AI Copyright. Meta estimated the development run at roughly 2,638 MWh and 1,015 tCO₂eq, computed at the US average carbon intensity rather than any specific data center's grid (LLaMA: Open and Efficient Foundation Language Models (Touvron et al., Meta AI, February 2023)).
Reported benchmarks
Selected figures from the paper, all Meta's own:
| Benchmark | LLaMA-13B | LLaMA-65B | Comparison |
|---|---|---|---|
| HellaSwag (0-shot) | 79.2 | 84.2 | GPT-3 175B 78.9; PaLM 540B 83.4 |
| WinoGrande (0-shot) | 73.0 | 77.0 | GPT-3 175B 70.2; PaLM 540B 81.1 |
| BoolQ (0-shot) | 78.1 | 85.3 | Chinchilla 70B 83.7; PaLM 540B 88.0 |
| TriviaQA (0-shot exact match) | 56.6 | 68.2 | Chinchilla 70B 55.4; Gopher 280B 43.5 |
| MMLU (5-shot average) | — | 63.4 | Behind Chinchilla-70B and PaLM-540B "by a few percent" |
The paper reports LLaMA-65B outperforming Chinchilla-70B on all reported common-sense reasoning benchmarks but BoolQ, and PaLM-540B on all but BoolQ and WinoGrande. A single instruction-finetuning experiment produced LLaMA-I, reaching 68.9% on MMLU, which the paper states is still "far from the state-of-the-art, that is 77.4 for GPT code-davinci-002."
Responsible-AI evaluations are reported alongside: on WinoGender the authors observe better performance on "their/them/someone" pronouns than on gendered pronouns and state this is "likely indicative of biases"; on TruthfulQA the 65B model scores 0.57 truthful and 0.53 truthful-and-informative.
Open questions
- The paper states the models were released "to the research community" but does not describe the access-request mechanism or licence terms, and those documents have not been ingested. The circumstances under which the weights subsequently circulated beyond that channel, and the derivative-model wave that followed, are not documented here for want of a source; this is a recorded gap rather than an absence of events.
Relationships
- related: Meta AI — the developer.
- depends-on: LLaMA: Open and Efficient Foundation Language Models (Touvron et al., Meta AI, February 2023) — the paper introducing the family.
- instance-of: Open-Weight Frontier Models — an early case of frontier-adjacent weights released outside a product API.
- depends-on: Scaling Laws — the result the design argument modifies by adding inference cost.
- related: Llama 3 (Meta) — a later generation of the same family.
- related: AI Copyright — the Books3 training-data component later contested in litigation.