Inference & optimisation
Inference
Running a trained model to produce a prediction, answer or action.
Definition
Inference is the execution phase in which a deployed model processes new inputs using its learned parameters.
Why it matters
Inference configuration, hosting, data flow and compute budget materially affect performance, cost and operational risk.
Related concepts
- Inference budget
The compute, token, time or attempt allowance given to solve a task.
- Throughput
The volume of requests or tokens a system can process over time.
- Latency
The time taken for a system to produce a result or complete a task.