Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 2 (July 2023)).
What it changed
Meta released Llama 2 base models and chat models, from 7 to 70 billion parameters, for both research and commercial use. With them it published a detailed account of the RLHF and safety training behind the chat models, a kind of training that the authors say had often not been transparent or reproducible. The largest chat model came close to ChatGPT in Meta’s human tests.
The problem before it
Open base models, such as LLaMA, BLOOM and Falcon, could match closed base models like GPT-3. But none could replace closed chat products such as ChatGPT, Bard and Claude. Those products were heavily fine-tuned with human feedback to make them helpful and safe.
That fine-tuning needs a lot of compute and human labelling, and the authors note it was “often not transparent or easily reproducible”. So outside researchers could not easily study or improve it.
The idea
The mechanism. Train better base models, then fine-tune chat models in several rounds.
The base models (Llama 2). The same design as LLaMA, with these changes:
- 40% more training data: 2 trillion tokens from a new mix of public sources, with no data from Meta’s products. The most factual sources were sampled more often.
- Twice the context length: 4,096 tokens.
- Grouped-query attention in the two largest models (34 and 70 billion), to make them faster to run.
The chat models (Llama 2-Chat):
- Supervised fine-tuning. About 27,500 high-quality examples written by hired annotators. The authors found that fewer, better examples beat millions of examples from public datasets.
- Human preference data. Annotators wrote a prompt, compared two model answers and said how much better one was. Meta collected over 1.4 million such comparisons, for helpfulness and for safety separately.
- Two reward models, one for helpfulness and one for safety, because the two goals can conflict. During reinforcement learning, the safety score is used for risky prompts or unsafe answers, and the helpfulness score for the rest.
- Iterative RLHF, in five rounds. Each round used new preference data on the latest model. Two methods were used:
- Rejection sampling: generate several answers with the 70-billion-parameter model, keep the best by reward, and fine-tune on them. The smaller models learn from these answers too.
- PPO, the reinforcement learning method of InstructGPT, applied on top in the last rounds.
- Ghost Attention (GAtt). A data trick that teaches the model to keep following an instruction given at the start of a chat, such as “act as Napoleon”, for many turns.
Safety training added adversarial prompts with safe answers, a safety reward model, and “context distillation”: the model learns the safer answers it gives when told “You are a safe and responsible assistant”, so that it gives them without the instruction. More than 350 people tested the models for risks (“red teaming”).
The notation. The reward model is trained with -log σ(r(x, y_c) - r(x, y_r) - m(r)), where y_c is the chosen answer, y_r the rejected one, and m(r) a margin that is larger when annotators said one answer was much better.
What it showed
Reported in the paper. We read version 2 of the arXiv paper. It is a preprint. Most chat results come from Meta’s own human raters and reward models, and from GPT-4 used as a judge.
- Better base models. Llama 2 70B beat LLaMA 65B by about 5 points on MMLU (68.9%) and about 8 on BBH. It beat all open base models compared.
- Against closed models. Llama 2 70B was close to GPT-3.5 on MMLU (68.9% against 70.0%) and GSM8K (56.8% against 57.1%), but far behind on coding (HumanEval: 29.9% against 48.1%). It was well behind GPT-4 and PaLM-2-L.
- Chat helpfulness. On about 4,000 prompts, Llama 2-Chat 70B won 36% and tied 31.5% against ChatGPT. Llama 2-Chat 34B won over 75% against Vicuna-33B and Falcon 40B.
- Safety. On about 2,000 adversarial prompts, Llama 2-Chat had comparable or lower rates of unsafe answers than the other models tested. Its share of toxic outputs fell to effectively zero.
- Truthfulness. For the 70-billion-parameter model, truthful and informative answers rose from 50.2% before fine-tuning to 64.1% after. ChatGPT scored 78.5%.
- RLHF worked better than expected. The authors had preferred supervised data at first. They found that people judge between two answers more consistently than they write answers, so RLHF can lift the model beyond the best human writing.
- Unexpected behaviour. The chat model could organise facts by date after only 1,000 date-related examples, and could use tools such as a calculator without being trained to.
- Cost. Pretraining used about 3.3 million GPU-hours and an estimated 539 tonnes of CO2-equivalent, which Meta says it fully offset.
Limits, and what came later
The limits come from the paper:
- Biased evaluations. The helpfulness and safety tests use Meta’s own prompts and guidelines, which the authors say are “likely to be biased” towards Llama 2-Chat. The prompts included no coding or reasoning tasks, and human ratings are subjective.
- Mostly English. About 90% of the pretraining data is English, so the model’s use of other languages is fragile.
- Usual weaknesses. It can make up facts, give unqualified advice, and its knowledge stops at the end of pretraining.
- Too cautious at times. The safety tuning sometimes goes too far, refusing reasonable requests.
- Harmful content remains possible, especially in languages other than English.
- Not released. The 34-billion-parameter models were not released, because there was not enough time to red-team them.
- Forgetting. One round of RLHF made the model worse at writing rhyming poems, which was fixed by reusing the best answers from earlier rounds.
Llama 3 was its direct successor. Mistral 7B reported beating Llama 2 13B on all its benchmarks. OLMo aimed at the Llama 2 level as a fully open model, and Gemma benchmarked against it.
What to know first
How to read it
The paper is 77 pages, with 36 pages of main text.
- Read section 1 with Figures 1 to 4 for the overview.
- Skim section 2 on pretraining, with Tables 1 and 3.
- Read section 3 closely. Sections 3.1 and 3.2 are the most detailed public account of RLHF at this scale.
- Read section 4.2 on safety fine-tuning and 4.4 on safety results.
- Read sections 5.1 and 5.2 for the authors’ observations and limits.
Expect about 3 hours for the steps above (an estimate).
Links
- The paper: arXiv 2307.09288
- Code: facebookresearch/llama, named in the paper. GitHub now describes it as “Inference code for Llama models”. The models were released under Meta’s own licence and acceptable use policy.
- Video: none is linked here yet.