20 January 2022 · Alignment

LaMDA

LaMDA: Language Models for Dialog Applications

R. Thoppilan, D. De Freitas, J. Hall, et al. (~60 authors)

Date: 20 January 2022
What the date marks: arXiv submission
Kind: Model
Organisation: Google
Access: internal
Threads: Alignment
Products and governance
In the guided tour: Chapter 6. Teaching models what people want
Learning phases: 2. How LLMs are trained

Notable: no later milestone in the data is documented as descending from it. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Large models were tuned for completion, not for holding a conversation.

The problem

Dialogue needs safety, factual grounding and a consistent persona, none of which come free.

What it introduced

Dialogue-specific fine-tuning with safety and groundedness objectives, plus external tool lookups for facts.

Why it matters

Google had an aligned conversational model well before ChatGPT and chose not to ship it publicly.

What modern models inherit

Safety and groundedness as explicit fine-tuning targets for chat.

What came next

Bard in 2023, then the Gemini line.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for LaMDA without weak links.

No recorded ancestors are recorded for LaMDA.

After: what it led to

No documented descendants are recorded for LaMDA without weak links.

No recorded descendants are recorded for LaMDA.

Direct links and their sources

Where it came from

Developed alongside

GPT-3, 2020 Contemporaneous large dialogue model. Source: arxiv.org

What it led to

None recorded in the sources used.