Inference & optimisation
Tokens per second
The rate at which a model generates output after generation starts.
Also known as: TPS
Definition
Tokens per second measures generation speed and is commonly reported separately from the initial time-to-first-token delay.
Why it matters
It helps compare user experience and infrastructure efficiency, but faster generation does not imply better answer quality.
Related concepts
- Throughput
The volume of requests or tokens a system can process over time.
- Time to first token
The delay before a generative model begins returning its response.
- Token
A unit of text processed or generated by a language model.