Models & architecture
Attention
A mechanism that weights which parts of an input matter for each output.
Definition
Attention lets a model dynamically combine information from different token positions when creating contextual representations.
Why it matters
Long context does not guarantee reliable attention to every relevant fact, especially across lengthy financial and policy documents.
Related concepts
- Transformer
The neural-network architecture underlying most modern language models.
- Long-context model
A model designed to process unusually large amounts of input at once.
- Context window
The maximum information a model can process in one interaction.