Inference & optimisation
Time to first token
The delay before a generative model begins returning its response.
Also known as: TTFT
Definition
Time to first token measures the interval from submitting a request until the first generated token reaches the user.
Why it matters
It strongly affects perceived responsiveness in copilots even when total task completion time is unchanged.
Related concepts
- Tokens per second
The rate at which a model generates output after generation starts.
- Prompt caching
Reusing computation for repeated prompt or context prefixes.
- Latency
The time taken for a system to produce a result or complete a task.