Byte-pair encoding (BPE)

A way to split text into tokens. Start from single characters, then repeatedly merge the most frequent pair into a new token. Common words become one token, and rare words are split into smaller pieces, so any word can be written.