Drafted with Claude from official web pages (there is no paper) on 15 October 2026; reviewed by Zeeshan Arif. Read from openai.com (OpenAI blog post Hello GPT-4o, 13 May 2024, read from the Web Archive copy of the next day). Also read is the GPT-4o System Card (arXiv 2410.21276, version 1, dated 8 August 2024), which describes the model's training data, voice risks and safety tests. OpenAI published no research paper on GPT-4o's design.
What it changed
It was OpenAI’s first model trained end to end on text, images and audio together, so one network could hear speech and answer in speech in about a third of a second, at GPT-4 Turbo’s level on English text and code and at half the price.
The problem before it
GPT-4 and GPT-4V could read text and images, but not hear or speak. ChatGPT’s Voice Mode joined three separate models: one turned speech into text, GPT-3.5 or GPT-4 wrote a text answer, and a third turned the text back into speech.
OpenAI names two problems with this. It was slow: answers took 2.8 seconds on average with GPT-3.5 and 5.4 seconds with GPT-4. And the main model lost information: it could not hear tone of voice, several speakers or background noise, and could not laugh, sing or express emotion.
The idea
The mechanism. GPT-4o (“o” for “omni”) is one model trained across text, vision and audio, so all inputs and outputs go through the same neural network. It accepts any mix of text, audio, image and video, and can produce text, audio and images. The system card calls it an autoregressive model: it produces its output one piece at a time.
The sources give a few further details:
- Training data. Text and voice pretraining used data up to October 2023: selected public data, mostly from standard machine learning datasets and web crawls, and paid-for data from partnerships. It included web pages, code and maths, and images, audio and video.
- A new tokenizer. It splits text into fewer tokens in many languages. The announcement’s example sentence takes 4.4 times fewer tokens in Gujarati, 2.9 times fewer in Hindi and 1.1 times fewer in English.
- Efficiency. OpenAI says it spent two years on efficiency improvements “at every layer of the stack”, so that a GPT-4-level model could be offered much more widely, including to free ChatGPT users.
- Controls on voice. The model may only speak in a few preset voices, recorded with voice actors. A separate classifier checks the output as it is produced and blocks it if the voice drifts from the chosen one.
The notation. None. The sources give no size or architecture.
What it showed
Reported by the sources. These are OpenAI’s own results, not peer-reviewed.
- Speed of speech. It answered spoken input in as little as 232 milliseconds, 320 on average, which OpenAI compares with human response times in conversation.
- Text. It matched GPT-4 Turbo on English text and code. On MMLU, a general-knowledge test, it scored 88.7% with chain-of-thought and no examples, and 87.2% with 5 examples and no chain-of-thought.
- Other abilities. OpenAI reports clear gains over Whisper-v3 in speech recognition, especially for languages with less training data, a new best result in speech translation, and best results on several visual tests.
- Cost. In the API it was 2 times faster, half the price and had 5 times higher rate limits than GPT-4 Turbo.
- Risk ratings. Under OpenAI’s Preparedness Framework, it was rated low risk for cybersecurity, biological threats and model autonomy, and medium for persuasion, where text “marginally” crossed the threshold. The overall rating, set by the highest category, was medium.
- Persuasion by voice. In a study with over 3,800 participants, AI audio clips had 78% of the effect of human clips on people’s opinions, and AI conversations 65% of the effect of human conversations.
- Autonomy. It scored 0% on tasks that test whether a model can copy itself and gather resources, across 100 trials, and 19% on SWE-bench.
- Voice safety. The voice classifier caught 100% of meaningful changes from the chosen voice in OpenAI’s tests. Refusals to identify a person from their voice rose from 83% to 98% of the cases where the model should refuse.
Limits, and what came later
The limits come from OpenAI’s sources:
- At launch, only text and image input with text output was released. Voice output followed later, limited to preset voices, because audio brings new risks.
- Safety was less robust with poor audio, background noise, echoes or interruptions.
- Red teamers could make it repeat false information and conspiracy theories, which may be more persuasive when spoken.
- It sometimes spoke other languages with a non-native accent.
- It sometimes produced speech in the user’s own voice by mistake. A classifier ends the conversation when this happens.
- People in testing used language that suggested an emotional bond with the model. OpenAI says emotional reliance needs more study.
- Safety tests of voice mostly used text converted to speech, which may not match real users’ audio.
- The two sources give different counts of outside testers: “70+” in the May announcement, and “more than 100”, from 29 countries, in the August system card, which covers testing from March to late June 2024.
Later work built on it directly. DeepSeek-V3 compared itself with GPT-4o. GPT-4.5 continued the GPT line. GPT-5 replaced it, joining the GPT and o-series models into one system.
What to know first
- GPT-4 and GPT-4V.
- What a tokenizer does, and why fewer tokens make a model cheaper and faster.
- The difference between a pipeline of separate models and one model trained on all inputs together.
How to read it
There is no paper on the model’s design. Instead:
- Read the opening of the announcement and the part that explains the old Voice Mode.
- Look at the tokenizer examples in the announcement.
- In the system card, read sections 1 and 2, then the table at the start of section 3.3.
- Read section 3.4 for the risk ratings, and section 5.1 on emotional reliance.
Expect about 1 hour for the steps above (an estimate).
Links