Memory wall – Definition & Meaning

Definition of memory wall

: a performance limit reached when a processor or AI accelerator can perform calculations faster than the memory system can supply, move, or hold the required data, making memory bandwidth, capacity, or data transfer the main bottleneck
- The serving cluster hit the memory wall because moving model weights and KV-cache data took longer than the matrix calculations.
- Long-context agents increased memory demand so quickly that adding more compute did little to improve throughput.

Origin & history

first known use1995

Computer architects William A. Wulf and Sally A. McKee popularized the phrase in their 1995 paper Hitting the Memory Wall, which described the widening gap between processor speed and memory performance. The term became central to AI infrastructure discussions as large models, high-bandwidth memory, and growing KV caches shifted many training and inference workloads from compute-bound to memory-bound.

Editorial references

Sources & further reading

4 sources

Hitting the memory wall: implications of the obvious Research Paper · ACM SIGARCH Computer Architecture News · 1995-03
AI and Memory Wall Research Article · IEEE Micro · 2024-03-25
The Physics of Long-Context Inference: Breaking the Memory Wall With NVIDIA Dynamo Conference Session · NVIDIA GTC · 2026-03
A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference Research Preprint · arXiv · 2026-07-29

Test yourself

Which of these is the meaning of memory wall?

Related terms

HBM a type of three-dimensionally stacked memory that uses a very wide connection to move data… KV cache memory that stores intermediate attention results for text a model has already processed, so… inference the act of running a trained model to produce an output, such as answering a prompt or… latency the delay between sending a request to a model and getting a response, often measured as time… throughput how much work an AI system can handle in a given time, often measured in tokens per second CXL memory pooling a server architecture that uses Compute Express Link to combine memory devices into a shared… unified memory a memory architecture or programming model that lets a CPU and one or more accelerators access… context window the maximum amount of text, measured in tokens, that a model can consider at one time…

Cite this entry

"Memory wall." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/memory-wall/. Accessed 20 Oct. 2026.