AI foundations
Token
A unit of text processed or generated by a language model.
Definition
Models break text into tokens, which can be words, word fragments, punctuation or other symbols depending on the tokenizer.
Why it matters
Token volumes affect context limits, latency and cost, and differ between models and languages.
Related concepts
- Context window
The maximum information a model can process in one interaction.
- Latency
The time taken for a system to produce a result or complete a task.
- Cost per task
The full variable cost of completing a defined unit of work.