Entropy

A measure of how unpredictable a source of symbols is: the average number of bits needed to encode each symbol. For language, it sets a floor on how well any model can predict the next word.