Inference & optimisation
Quantisation
Representing model weights with lower numerical precision to reduce compute.
Also known as: quantization
Definition
Quantisation compresses a model by storing and calculating parameters with fewer bits than the original training precision.
Why it matters
It can reduce hosting cost and improve speed, but accuracy and safety regressions must be tested on finance-specific edge cases.
Related concepts
- Small language model
A relatively compact language model optimised for focused or efficient deployment.
- Model parameters
The learned numerical values that determine a model's behaviour.
- Inference
Running a trained model to produce a prediction, answer or action.