Learning path

Phase 1. Transformer architecture

25 milestones, 4 with study notes

Start with

• Attention 2014. Turning point

  To understand it, learn

  • Attention as a weighted average
  • Alignment scores between positions
  • Why the bottleneck hurt long inputs

  Helpful first

  Sequence-to-sequence models.

• Seq2seq 2014. Turning point

  To understand it, learn

  • Encoder and decoder
  • The fixed-length bottleneck
  • Autoregressive generation

  Helpful first

  Recurrent networks and LSTM gating.

• Transformer 2017. Turning point

  To understand it, learn

  • Scaled dot-product self-attention
  • Multi-head attention
  • Positional encoding
  • Residual connections and layer norm

  Helpful first

  Attention in encoder-decoder translation models.

• FlashAttention 2022. Milestone

  To understand it, learn

  • Memory hierarchy on a GPU
  • Tiling to avoid writing the attention matrix
  • Exact versus approximate attention

  Helpful first

  Self-attention and why it is quadratic.

Terms to know

• Encoder-decoder
• Gated recurrent unit (GRU)
• Attention
• Byte-pair encoding (BPE)
• Residual connection
• Layer normalisation
• Multi-head attention
• Positional encoding
• Sparse attention
• Multi-query attention
• Sliding-window attention
• Rotary position embedding (RoPE)
• Attention with linear biases (ALiBi)
• Grouped-query attention (GQA)

Turning points in this phase

• Sparse TransformerGenerating Long Sequences with Sparse Transformers2019
• RoPERoFormer: Enhanced Transformer with Rotary Position Embedding2021
• Grouped-query attentionGQA: Training Generalized Multi-Query Transformer Models2023

Phase 2. How LLMs are trained

62 milestones, 11 with study notes

Start with

• Neural LM 2003. Milestone

  To understand it, learn

  • Word embeddings as learned vectors
  • The curse of dimensionality in counting models
  • Next-word prediction as a training objective

  Helpful first

  Basic neural networks and softmax.

• GPT-1 2018. Milestone

  To understand it, learn

  • Causal (left-to-right) language modelling
  • Decoder-only Transformers
  • Pretrain then fine-tune

  Helpful first

  The Transformer's decoder half.

• BERT 2018. Turning point

  To understand it, learn

  • Masked language modelling
  • Bidirectional context
  • Why encoders suit understanding and retrieval

  Helpful first

  The Transformer's encoder half.

• GPT-2 2019. Turning point

  To understand it, learn

  • Zero-shot task transfer
  • Byte-level BPE tokenization
  • Staged release of risky models

  Helpful first

  GPT-1 and next-token prediction.

• GPT-3 2020. Turning point

  To understand it, learn

  • In-context learning
  • Few-shot versus zero-shot prompting
  • Why scale changed the interface

  Helpful first

  Scaling laws for language models.

• InstructGPT 2022. Turning point

  To understand it, learn

  • Supervised fine-tuning on demonstrations
  • Reward models from human rankings
  • PPO against a reward model
  • Alignment elicits rather than adds knowledge

  Helpful first

  GPT-3 and the 2017 preference-learning work.

• Chinchilla 2022. Turning point

  To understand it, learn

  • Compute-optimal allocation
  • Roughly 20 tokens per parameter
  • Why labs now overtrain small models

  Helpful first

  Kaplan's scaling laws.

• LLaMA 2023. Turning point

  To understand it, learn

  • Open weights versus open source
  • Overtraining for cheap inference
  • RoPE, SwiGLU and pre-norm in practice

  Helpful first

  Chinchilla scaling.

• DPO 2023. Milestone

  To understand it, learn

  • Preference pairs
  • Why DPO needs no reward model
  • The KL constraint to the base model

  Helpful first

  The InstructGPT pipeline.

• o1 2024. Turning point

  To understand it, learn

  • Test-time compute as a scaling axis
  • Reinforcement learning on reasoning
  • Hidden reasoning tokens

  Helpful first

  Chain of thought and process supervision.

• DeepSeek-R1 2025. Milestone

  To understand it, learn

  • GRPO
  • Reasoning from pure reinforcement learning
  • Distilling reasoning into small models

  Helpful first

  o1 and DeepSeekMath's GRPO.

Terms to know

• Entropy
• Language model
• Word embedding
• Pretraining and fine-tuning
• Reinforcement learning from human feedback (RLHF)
• Reward model
• Masked language model
• Next sentence prediction
• Text-to-text format
• Scaling law
• Few-shot prompting
• In-context learning
• Instruction tuning
• Context distillation
• Helpful, honest and harmless (HHH)
• Compute-optimal training
• Emergent ability
• Rule-based reward model
• Ghost Attention (GAtt)
• Group relative policy optimization (GRPO)
• Test-time compute

Turning points in this phase

• LM pretrainingSemi-supervised Sequence Learning2015
• C4C4: Colossal Clean Crawled Corpus2019
• T5Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer2019
• Scaling lawsScaling Laws for Neural Language Models2020
• RLHF for summarisationLearning to Summarize from Human Feedback2020
• GPT-3.5GPT-3.5 series2022
• GPT-4GPT-4 Technical Report2023
• Llama 2Llama 2: Open Foundation and Fine-Tuned Chat Models2023

Phase 3. API usage and prompting

35 milestones, 6 with study notes

Start with

• GPT-3 2020. Turning point

  To understand it, learn

  • In-context learning
  • Few-shot versus zero-shot prompting
  • Why scale changed the interface

  Helpful first

  Scaling laws for language models.

• Chain of thought 2022. Turning point

  To understand it, learn

  • Chain-of-thought exemplars
  • Why intermediate steps help
  • Emergence and its measurement

  Helpful first

  Few-shot prompting.

• ChatGPT 2022. Milestone

  To understand it, learn

  • Capability versus product versus distribution
  • Dialogue-formatted training data
  • System prompts and conversation state

  Helpful first

  InstructGPT.

• GPT-4o 2024. Milestone

  To understand it, learn

  • Native versus cascaded multimodality
  • Audio as tokens
  • Why latency matters for voice

  Helpful first

  CLIP and multimodal encoders.

• o1 2024. Turning point

  To understand it, learn

  • Test-time compute as a scaling axis
  • Reinforcement learning on reasoning
  • Hidden reasoning tokens

  Helpful first

  Chain of thought and process supervision.

• GPT-5 2025. Milestone

  To understand it, learn

  • Routing between fast and reasoning models
  • Unified systems versus single models
  • Reading vendor benchmark claims

  Helpful first

  o-series reasoning models.

Terms to know

• Few-shot prompting
• In-context learning
• Chain of thought
• Self-consistency
• Zero-shot chain of thought
• Rule-based reward model

Turning points in this phase

• GPT-3.5GPT-3.5 series2022
• GPT-4GPT-4 Technical Report2023

Phase 4. Embeddings and RAG

17 milestones, 4 with study notes

Start with

• word2vec 2013. Milestone

  To understand it, learn

  • Skip-gram and CBOW objectives
  • Negative sampling
  • Cosine similarity between vectors

  Helpful first

  Why Bengio's model learned word vectors in the first place.

• BERT 2018. Turning point

  To understand it, learn

  • Masked language modelling
  • Bidirectional context
  • Why encoders suit understanding and retrieval

  Helpful first

  The Transformer's encoder half.

• RAG 2020. Milestone

  To understand it, learn

  • Dense retrieval with embeddings
  • Vector search
  • Conditioning generation on retrieved passages

  Helpful first

  Sentence embeddings and BERT encoders.

• CLIP 2021. Milestone

  To understand it, learn

  • Contrastive learning
  • A shared image-text embedding space
  • Zero-shot classification by text prompt

  Helpful first

  The Vision Transformer.

Terms to know

• Skip-gram
• Masked language model
• Next sentence prediction
• Dense retrieval
• Retrieval-augmented generation (RAG)

Phase 5. Fine-tuning

6 milestones, 1 with study notes

Start with

• QLoRA 2023. Notable

  To understand it, learn

  • Low-rank adapters
  • 4-bit NormalFloat quantization
  • Why the frozen base model dominates memory

  Helpful first

  LoRA and 8-bit quantization.

Terms to know

• Low-rank adaptation (LoRA)
• 4-bit NormalFloat (NF4)
• Ghost Attention (GAtt)

Turning points in this phase

• Llama 2Llama 2: Open Foundation and Fine-Tuned Chat Models2023

Phase 6. Local models and deployment

30 milestones, 3 with study notes

Start with

• FlashAttention 2022. Milestone

  To understand it, learn

  • Memory hierarchy on a GPU
  • Tiling to avoid writing the attention matrix
  • Exact versus approximate attention

  Helpful first

  Self-attention and why it is quadratic.

• QLoRA 2023. Notable

  To understand it, learn

  • Low-rank adapters
  • 4-bit NormalFloat quantization
  • Why the frozen base model dominates memory

  Helpful first

  LoRA and 8-bit quantization.

• DeepSeek-V3 2024. Milestone

  To understand it, learn

  • Mixture-of-experts: total versus active parameters
  • Multi-head latent attention
  • What a final-run training cost does and does not include

  Helpful first

  Sparse MoE routing and grouped-query attention.

Terms to know

• Multi-query attention
• Speculative decoding
• Grouped-query attention (GQA)
• 4-bit NormalFloat (NF4)

Turning points in this phase

• Grouped-query attentionGQA: Training Generalized Multi-Query Transformer Models2023

Phase 7. Agents

28 milestones, 2 with study notes

Start with

• ReAct 2022. Milestone

  To understand it, learn

  • The thought, action, observation loop
  • Tool calls as model output
  • Why grounding reduces hallucination

  Helpful first

  Chain-of-thought prompting.

• MCP 2024. Milestone

  To understand it, learn

  • Tools, resources and prompts as protocol primitives
  • Why a standard beats M times N integrations
  • Client and server roles

  Helpful first

  Function calling.

Phase 8. Training from scratch

32 milestones, 4 with study notes

Start with

• GPT-2 2019. Turning point

  To understand it, learn

  • Zero-shot task transfer
  • Byte-level BPE tokenization
  • Staged release of risky models

  Helpful first

  GPT-1 and next-token prediction.

• Chinchilla 2022. Turning point

  To understand it, learn

  • Compute-optimal allocation
  • Roughly 20 tokens per parameter
  • Why labs now overtrain small models

  Helpful first

  Kaplan's scaling laws.

• LLaMA 2023. Turning point

  To understand it, learn

  • Open weights versus open source
  • Overtraining for cheap inference
  • RoPE, SwiGLU and pre-norm in practice

  Helpful first

  Chinchilla scaling.

• DeepSeek-V3 2024. Milestone

  To understand it, learn

  • Mixture-of-experts: total versus active parameters
  • Multi-head latent attention
  • What a final-run training cost does and does not include

  Helpful first

  Sparse MoE routing and grouped-query attention.

Terms to know

• Adam
• Sparse mixture of experts
• Mixed precision training
• Scaling law
• Expert capacity
• Compute-optimal training
• Ghost Attention (GAtt)

Turning points in this phase

• Sparse MoEOutrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer2017
• C4C4: Colossal Clean Crawled Corpus2019
• Scaling lawsScaling Laws for Neural Language Models2020
• GShardGShard: Scaling Giant Models with Conditional Computation2020
• Llama 2Llama 2: Open Foundation and Fine-Tuned Chat Models2023