Inference & optimisation
Prompt caching
Reusing computation for repeated prompt or context prefixes.
Definition
Prompt caching stores intermediate processing for stable instructions or documents so later requests avoid recomputing the same context.
Why it matters
It can cut cost and latency for repeated policy or document contexts, while cache isolation and invalidation require control.
Related concepts
- Data leakage
Sensitive information reaching an unauthorised model, user or output.
- Context window
The maximum information a model can process in one interaction.
- Latency
The time taken for a system to produce a result or complete a task.