51 terms

Glossary

Each term is defined in plain words, and linked to the milestone that introduced it and the milestones that use it.

• 4-bit NormalFloat (NF4)
• Adam
• Attention
• Attention with linear biases (ALiBi)
• Byte-pair encoding (BPE)
• Chain of thought
• Compute-optimal training
• Context distillation
• Dense retrieval
• Emergent ability
• Encoder-decoder
• Entropy
• Expert capacity
• Few-shot prompting
• Gated recurrent unit (GRU)
• Ghost Attention (GAtt)
• Group relative policy optimization (GRPO)
• Grouped-query attention (GQA)
• Helpful, honest and harmless (HHH)
• In-context learning
• Instruction tuning
• Language model
• Layer normalisation
• Long short-term memory (LSTM)
• Low-rank adaptation (LoRA)
• Masked language model
• Mixed precision training
• Mixture of experts
• Multi-head attention
• Multi-query attention
• Next sentence prediction
• Positional encoding
• Pretraining and fine-tuning
• Reinforcement learning from human feedback (RLHF)
• Residual connection
• Retrieval-augmented generation (RAG)
• Reward model
• Rotary position embedding (RoPE)
• Rule-based reward model
• Scaling law
• Self-consistency
• Skip-gram
• Sliding-window attention
• Sparse attention
• Sparse mixture of experts
• Speculative decoding
• Statistical machine translation
• Test-time compute
• Text-to-text format
• Word embedding
• Zero-shot chain of thought