Inference & optimisation
Throughput
The volume of requests or tokens a system can process over time.
Definition
Throughput measures completed requests, tasks or generated tokens per unit of time under a stated workload.
Why it matters
A system may have acceptable single-user latency but fail under the concurrency of bank-wide adoption or peak operations.
Related concepts
- Tokens per second
The rate at which a model generates output after generation starts.
- Operational fitness
Whether a system works reliably within real service constraints.
- Latency
The time taken for a system to produce a result or complete a task.