Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 6 (March 2024) of the GPT-4 Technical Report, which includes the GPT-4 System Card).
What it changed
GPT-4 reached human-level scores on many professional and academic exams, such as a simulated bar exam in the top 10% of test takers. It accepts images as well as text. Its report also showed that some results of a huge training run can be predicted from much smaller runs, and it published almost no details of how the model was built.
The problem before it
GPT-3.5 and ChatGPT were useful but unreliable. The report’s own comparison shows the gap: GPT-3.5 scored in the bottom 10% of test takers on the simulated bar exam.
Training the largest models also has a practical problem. A run as large as GPT-4 cannot be tuned by trial and error, because each attempt is too expensive. The developers needed to know in advance how such a run would turn out.
The idea
The mechanism. The report gives only an outline:
- A Transformer pre-trained to predict the next token, on public data, such as internet data, and on data licensed from third parties.
- Fine-tuned with RLHF, the method of InstructGPT.
- Images and text in, text out. The user can mix images and text in one prompt.
- Predictable scaling. OpenAI built its training infrastructure to behave the same way at many scales. It then predicted GPT-4’s results from smaller models trained with 1,000 to 10,000 times less compute, using scaling-law fits.
- Safety steps, described in the system card:
- More than 50 outside experts tested the model for risks, for example in cybersecurity, biological risk and international security.
- “Rule-based reward models”: GPT-4 itself, used as a classifier that checks answers against a written rubric. It adds a reward during RLHF for refusing harmful requests and for answering harmless ones.
What is not published. The report says that, “given both the competitive landscape and the safety implications”, it contains no further details about the architecture (including model size), hardware, training compute, dataset construction or training method.
The notation. The loss prediction fits L(C) = a C^b + c, where C is the training compute and c an irreducible loss.
What it showed
Reported in the paper. We read version 6 of the arXiv report. It is a technical report from OpenAI, not a peer-reviewed paper, and the model cannot be checked from outside. For each exam, OpenAI ran a version without the questions that appeared in training and reported the lower of the two scores.
- Exams. A simulated Uniform Bar Exam: 298 out of 400, about the 90th percentile (GPT-3.5: about the 10th). LSAT: about the 88th percentile. GRE Verbal: about the 99th. But it scored low on some: Codeforces programming contests (below the 5th percentile), AP English Literature (8th to 22nd) and AMC 10 maths (6th to 12th).
- Benchmarks. It beat earlier language models on all the benchmarks listed in Table 2, and beat even task-specific systems on all except DROP. For example, MMLU: 86.4% (GPT-3.5: 70.0%). HumanEval coding: 67.0%. GSM8K maths: 92.0%, but part of the GSM8K training set was in GPT-4’s training data.
- Other languages. On MMLU translated into 26 languages, GPT-4 beat the earlier best English score in 24 of them, including low-resource languages such as Latvian, Welsh and Swahili.
- Predictions came true. A fit to small models predicted GPT-4’s final loss accurately. A prediction of its HumanEval pass rate, registered before training ended, was also accurate, except that GPT-4 did worse than predicted on the easiest problems.
- Preferred to GPT-3.5. On 5,214 real user prompts, labelers preferred GPT-4’s answer 70.2% of the time.
- Fewer made-up facts. On OpenAI’s internal factuality tests, GPT-4 scored 19 percentage points higher than the latest GPT-3.5.
- Safer by OpenAI’s measures. Compared with GPT-3.5, it answered requests for disallowed content 82% less often. On RealToxicityPrompts, 0.73% of its outputs were toxic, against 6.48% for GPT-3.5.
Limits, and what came later
The limits come from the report and its system card:
- Still not reliable. It still “hallucinates” facts and makes reasoning errors, so its outputs need care, especially in high-stakes uses.
- Out of date. Most of its training data ends in September 2021, and it does not learn from experience.
- Overconfident after training. The pre-trained model was well calibrated: its confidence matched how often it was right. After RLHF, calibration got much worse.
- Jailbreaks. Safety training makes harmful outputs harder to get, but not impossible.
- Risks. Without its safety training, it gave more detailed guidance on harmful or illegal activities. It may help identify private individuals when combined with other data, and may lower the cost of some steps of a cyberattack. An outside evaluation found it probably not yet able to copy itself and gather resources on its own.
- Test data in training. Parts of BIG-bench were found in the training data, so those results were left out.
- Not reproducible. Without details of size, data or training, other researchers cannot check or rebuild it.
Later systems built on it. GPT-4V brought image input to the deployed model, and GPT-4o extended GPT-4-level ability across modalities. LLaVA trained an open model on visual instruction data generated by GPT-4. AutoGPT was built on the GPT-4 API. Claude 3 was benchmarked directly against it.
What to know first
How to read it
The report is 100 pages: a 14-page main report, appendices, and a system card of about 60 pages.
- Read sections 1 and 2. Section 2 says what the report leaves out.
- Read section 3 with Figures 1 and 2 on predictable scaling.
- Look at Table 1 (exams) and Table 2 (benchmarks).
- Read section 5, the limits, and section 6, risks and safety steps.
- Read section 1.1 of the system card for the main risks found.
Expect about 1.5 hours for the main report (an estimate). The system card takes longer.
Links
- The report: arXiv 2303.08774
- Code: no model code is released. The report open-sources OpenAI Evals, a framework for testing models.
- Video: none is linked here yet.