LLaMA: Open and Efficient Foundation Language Models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample

2023 · arXiv

touvronLLaMA2023

Problem

Framing

Open LMs lagged closed models in compute-optimal scaling and benchmark quality at fixed inference cost. LLaMA closes that gap with Chinchilla-style token budgets, a plain decoder-only transformer, and public-data training, reaching competitive zero- and few-shot results with 7B–65B models.

Currently Used Methods

Foundational

@brownGPT3_2020 — autoregressive few-shot scaling with dense decoder-only transformers.
- Limitation in context: undertrained per parameter, so smaller longer-trained models can win.
@hoffmannChinchilla2022 — compute-optimal scaling laws for parameter–token tradeoffs.
- Limitation in context: does not deliver an open public-data model family.
@suRoFormer2021 — rotary position embeddings for long-context attention.
- Limitation in context: positional encoding alone does not set training efficiency.
"PaLM: Scaling Language Modeling with Pathways" — strong large-scale dense transformer baseline.
- Limitation in context: larger closed model and non-public training recipe.
"OPT: Open Pre-trained Transformer Language Models" — open decoder-only LM replication effort.
- Limitation in context: weaker quality-per-parameter than compute-optimal training.

Proposed Method

Architecture

LLaMA is a decoder-only transformer with 7B, 13B, 33B, and 65B variants. It keeps the standard causal stack but replaces LayerNorm with RMSNorm, uses SwiGLU feed-forward blocks, and uses rotary positional embeddings. Context length is 2048.

Table 2: Model sizes, architectures, and optimization hyper-parameters.

params	dimension	n heads	n layers	learning rate	batch size	n tokens
6.7B	4096	32	32	3.0e $^{-4}$	4M	1.0T
13.0B	5120	40	40	3.0e $^{-4}$	4M	1.0T
32.5B	6656	52	60	1.5e $^{-4}$	4M	1.4T
65.2B	8192	64	80	1.5e $^{-4}$	4M	1.4T

Loss / Objective

The model uses the standard causal language-model objective.

\mathcal{L}(\theta) = - \sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})

Sampling Rule / Algorithm

At inference, the model autoregressively samples or scores the next token from the decoder distribution.

\hat{x}_t \sim p_\theta(\cdot \mid x_{<t}), \qquad \hat{x}_{1:T} = \prod_{t=1}^{T} p_\theta(\hat{x}_t \mid \hat{x}_{<t})

Training Procedure

Context length: 2048.
Batch size: 4M tokens.
Token budget: 1.0T for 7B and 13B.
Token budget: 1.4T for 33B and 65B.
Learning rate: $3.0 \times 10^{-4}$ for 7B and 13B.
Learning rate: $1.5 \times 10^{-4}$ for 33B and 65B.
Data mixture includes CommonCrawl, C4, Github, Wikipedia, Books, and ArXiv/StackExchange-style sources.

Evaluation

Datasets

Common-sense reasoning: BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA.
Reading comprehension: NaturalQuestions, TriviaQA, RACE.
World knowledge: MMLU.
Code and math: HumanEval, MBPP, GSM8K, MATH.
Robustness and bias: RealToxicityPrompts, CrowS-Pairs, WinoGender.

Metrics

Accuracy for multiple-choice tasks.
Exact match for NaturalQuestions and TriviaQA.
pass@ $k$ for code generation.
Benchmark average accuracy for MMLU.

Headline results

Common-sense reasoning, zero-shot: LLaMA-65B beats Chinchilla-70B on HellaSwag, WinoGrande, and PIQA.
MMLU, 5-shot: LLaMA-65B reaches 63.4 average.
NaturalQuestions, 1-shot trend: LLaMA-65B rises above the Chinchilla reference during training.
TriviaQA, 1-shot trend: LLaMA-65B reaches about 72 accuracy by 1.4T tokens.
The paper reports that 13B outperforms GPT-3 175B on most benchmarks, and 65B is competitive with Chinchilla-70B and PaLM-540B.

Six-panel training curves for TriviaQA, HellaSwag, NaturalQuestions, SIQA, WinoGrande, and PIQA across 7B–65B LLaMA models, with dashed Chinchilla baselines.

Ablations

Model size: larger models improve monotonically on most tasks during training.
Token budget: 33B and 65B continue improving through 1.4T tokens.
Small-model training length: 7B still improves after 1.0T tokens.
Benchmark sensitivity: WinoGrande saturates; gains correlate less cleanly with training loss.

Method Strengths and Weaknesses

Strengths

Strong quality-per-parameter: 13B surpasses GPT-3 175B on many benchmarks.
Public-data training recipe makes the results reproducible in principle.
Simple architecture changes improve efficiency without exotic routing or retrieval.
Training curves show 7B–65B models remain undertrained at smaller token budgets.

Weaknesses

No architectural novelty beyond careful transformer simplifications.
2048-token context is short for many downstream uses.
Evaluation is still benchmark-centric, with limited task adaptation analysis.
The paper reports toxicity and bias issues that instruction tuning does not solve.

Suggestions from the authors

Extend training beyond 1T tokens for smaller models.
Improve instruction tuning for stronger zero-shot usability.
Reduce toxicity and social bias in open foundation models.
Study larger-context and broader multilingual evaluation settings.