Inference & optimisation
Inference budget
The compute, token, time or attempt allowance given to solve a task.
Also known as: test-time compute
Definition
An inference budget constrains how much reasoning, generation, tool use or repeated sampling a system may use for one request.
Why it matters
Benchmark rankings can change when systems receive different budgets, making cost-normalised comparisons essential for finance decisions.
Related concepts
- Reasoning model
A model optimised to spend additional computation solving complex tasks.
- Pass@k
Whether at least one of k attempts successfully completes a task.
- Cost per task
The full variable cost of completing a defined unit of work.