NLP Lineage

17 June 2021 · Language modelling

LoRA

LoRA: Low-Rank Adaptation of Large Language Models

E. Hu, Y. Shen, P. Wallis, et al. (8 authors)

Date
17 June 2021
What the date marks
arXiv submission
Kind
Training method
Organisation
Microsoft
Access
MIT
Other dates
ICLR 2022
Threads
In the guided tour
Chapter 9. Making it run anywhere
Learning phases
5. Fine-tuning

A milestone: 1 later milestone descends directly from it, and its descendants reach 2 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Fine-tuning updated every weight and produced a full-size copy per task.

The problem

A 175B model cannot be fine-tuned, stored or served per task by most organisations.

What it introduced

Freeze the base model and train a small low-rank update alongside it, often under 1% of parameters.

Why it matters

It turned fine-tuning from a datacentre operation into something you can do on one GPU.

What modern models inherit

Parameter-efficient fine-tuning, the default way people specialise open models.

What came next

QLoRA added quantization; adapters became the standard customisation route.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

LoRA has 18 documented ancestors across 5 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 9 turning points among them.

LoRA has 18 documented ancestors across 5 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

LoRA has 27 recorded ancestors, including indirect ones, across 5 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 9 turning points among them.

LoRA has 27 recorded ancestors, including indirect ones, across 5 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2021LoRAThis milestone.
  2. 2020GPT-3 Turning pointLoRA built directly on thisChapter 5. Scale becomes a science
  3. 2020Scaling laws Turning pointGPT-3 was influenced by thisChapter 5. Scale becomes a science
  4. 2019Sparse Transformer Turning pointGPT-3 built directly on thisChapter 4. Pretrain once, use everywhere
  5. 2019GPT-2 Turning pointGPT-3 built directly on thisChapter 4. Pretrain once, use everywhere
  6. 2019WebTextGPT-2 was influenced by thisChapter 4. Pretrain once, use everywhere
  7. 2018GPT-1GPT-2 built directly on thisChapter 4. Pretrain once, use everywhere
  8. 2017Transformer Turning pointLoRA built directly on thisChapter 3. Attention, then only attention
  9. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  10. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  11. 2015LM pretraining Turning pointGPT-1 was influenced by thisChapter 2. Words become vectors
  12. 2015BPE tokenizationGPT-2 built directly on thisChapter 2. Words become vectors
  13. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  14. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  15. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  16. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  17. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  18. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  19. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2021LoRAThis milestone.
  2. 2020GPT-3 Turning pointLoRA built directly on thisChapter 5. Scale becomes a science
  3. 2020Scaling laws Turning pointGPT-3 was influenced by thisChapter 5. Scale becomes a science
  4. 2019Sparse Transformer Turning pointGPT-3 built directly on thisChapter 4. Pretrain once, use everywhere
  5. 2019GPT-2 Turning pointGPT-3 built directly on thisChapter 4. Pretrain once, use everywhere
  6. 2019WebTextGPT-2 was influenced by thisChapter 4. Pretrain once, use everywhere
  7. 2019Transformer-XLSparse Transformer was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  8. 2018GPT-1GPT-2 built directly on thisChapter 4. Pretrain once, use everywhere
  9. 2018ELMoGPT-1 was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  10. 2018ULMFiTGPT-1 was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  11. 2017CoVeELMo was influenced by thisChapter 3. Attention, then only attention
  12. 2017Transformer Turning pointLoRA built directly on thisChapter 3. Attention, then only attention
  13. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  14. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  15. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  16. 2015LM pretraining Turning pointGPT-1 was influenced by thisChapter 2. Words become vectors
  17. 2015BPE tokenizationGPT-2 built directly on thisChapter 2. Words become vectors
  18. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  19. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  20. 2014GloVeELMo built directly on thisChapter 2. Words become vectors
  21. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  22. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  23. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  24. 2013word2vecELMo was influenced by thisChapter 2. Words become vectors
  25. 2011NLP from scratchELMo was influenced by thisChapter 1. Language as probability
  26. 2003Neural LMNLP from scratch was influenced by this
    word2vec was influenced by this    Chapter 1. Language as probability
  27. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  28. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

LoRA has 1 documented descendants across 1 chapter, reaching forward to QLoRA in 2023. Showing all of them.

LoRA has 1 documented descendants across 1 chapter, reaching forward to QLoRA in 2023. Showing all of them.

LoRA has 1 recorded descendants, including indirect ones, across 1 chapter, reaching forward to QLoRA in 2023. Showing all of them.

LoRA has 1 recorded descendants, including indirect ones, across 1 chapter, reaching forward to QLoRA in 2023. Showing all of them.

  1. 2021LoRAThis milestone.
  2. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  1. 2021LoRAThis milestone.
  2. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere

Direct links and their sources

Where it came from

Built directly on

What it led to

Built on directly by

Terms