Sparse attention – Definition & Meaning

Definition of sparse attention

: an attention mechanism that allows each query to attend to only a selected subset of tokens or positions instead of computing every possible pairwise connection
- The long-context model used sparse attention to avoid comparing every token with every other token.
- A block-sparse attention pattern reduced memory use while preserving access to important earlier passages.

Two meanings, one word

In everyday English

Sparse describes something thinly distributed or containing relatively few elements.

In AI

An attention mechanism that allows each query to attend to only a selected subset of tokens or positions instead of computing every possible pairwise connection.

Origin & history

first known use2019

Sparse forms of attention appeared in early Transformer research, and the 2019 Sparse Transformer work made the term prominent by replacing full quadratic self-attention with structured sparse patterns for long sequences. Many learned, routed, block-based, and training-free variants have since appeared.

Test yourself

Which of these is the meaning of sparse attention?

Related terms

attention a technique that lets a model weigh how relevant every part of the input is to every other… transformer a neural network design that processes a whole sequence at once and uses attention to work out… context window the maximum amount of text, measured in tokens, that a model can consider at one time… KV cache memory that stores intermediate attention results for text a model has already processed, so… inference the act of running a trained model to produce an output, such as answering a prompt or… latency the delay between sending a request to a model and getting a response, often measured as time…

Cite this entry

"Sparse attention." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/sparse-attention/. Accessed 6 Oct. 2026.