tokenizer

Definition of tokenizer

1. : the component that splits text into tokens before it goes into a model, and joins tokens back into text afterward

   • Different models use different tokenizers, so the same prompt can have different token counts.
   • A strange tokenizer quirk explains why some models struggled to count letters in a word.