Production operations
Latency
The time taken for a system to produce a result or complete a task.
Definition
Latency may include retrieval, model generation, tool calls, validation and human approval, and should be measured across the full workflow.
Why it matters
Average latency can hide damaging tail delays, so production decisions should consider percentiles and timeouts.
Related concepts
- Operational fitness
Whether a system works reliably within real service constraints.
- Cost per task
The full variable cost of completing a defined unit of work.
- Availability
The proportion of time a service can successfully respond.