What are common tokenization methods used in LLMs and how do they differ?

Common tokenization methods for LLMs include Byte Pair Encoding (BPE), WordPiece, and SentencePiece. BPE iteratively merges the most frequent pairs of symbols to create subword units, which balances vocabulary size and out-of-vocabulary (OOV) handling. WordPiece, used in BERT, operates similarly but is more focused on maximizing the likelihood of the training data. SentencePiece, applicable for various languages, treats input as a sequence of characters and uses a similar merging approach, effectively allowing for a more language-agnostic tokenization.