The Memory Wall and the Edge
By Dr. Jonathan Tapson, Chief Development Officer
How Sequora Rethinks AI Acceleration Around Data Movement
A funny thing happened in 2018, and the hardware industry has been reacting to it ever since.
Memory stocks soared. NVIDIA became the most valuable company in the world. Silicon fabrication lines are booked into the foreseeable future.
The cause was the widespread adoption of transformer models for language processing. CNNs swept through AI around 2014 without upending hardware. Transformers did, because they use compute and memory differently than any workload before them.
From “fetch once, use often” to “use once, throw away”
Until 2018, computer architecture rested on a simple idea: fetch a little data from memory, then use it a lot. Designers built elaborate cache hierarchies to keep the most-used data close to the processor.
You can see this in the field’s standard text, Hennessy and Patterson’s Computer Architecture. After the opening chapter on fundamentals, Chapter 2 is “Memory Hierarchy Design.”
Transformers broke that model. A transformer has a massive number of weights, and each inference uses each weight only once. The design problem collapsed into a single task: move as many weights as fast as possible to the processor, use them once, and throw them away.
Why memory, not math, is the bottleneck
Modern AI hardware is limited less by how fast it can compute than by how fast it can feed that computation. This is the memory wall: the widening gap between how fast processors compute and how fast, and how cheaply, memory can supply them.
The numbers are stark. A multiply-accumulate operation on-chip costs a fraction of a picojoule. Fetching its operands from off-chip DRAM can cost 100 to 1,000 times more. For neural network inference, data movement, not computation, dominates both energy use and runtime.
The wall shapes design on two fronts:
- Hardware. Architects spend most of their silicon area and power budget on moving data, not on arithmetic: large SRAM buffers, wide memory interfaces, and high-bandwidth memory stacks. These add cost, package complexity, and heat. On-chip SRAM is also expensive in area and leaks power continuously, so capacity is always scarce.
- Algorithms. Many of today’s most important models are memory-bound, meaning they move far more bytes than they compute on. Transformer text generation is the clearest example: every new token streams the full model weights, plus a growing key-value cache, through the processor for very little arithmetic. That’s why quantization, pruning, sparsity, and distillation have become essential. Their real purpose is to move fewer bytes.
The wall is highest at the edge
Edge devices can’t hide the memory wall behind expensive memory hardware. Batteries can’t afford the power cost of DRAM traffic, and always-on applications can’t tolerate its latency. Four constraints make the problem harder:
| Constraint | Why it matters at the edge |
|---|---|
| SRAM has stopped shrinking | Logic keeps getting smaller with each process node, but SRAM cells have nearly stopped scaling. SRAM was already the least dense memory, so on-chip capacity gets more expensive every generation. |
| Models keep growing | Model sizes double every 12 to 15 months, and customers expect cloud-level performance from edge devices. |
| Power, performance, and area rule | Many devices run on batteries. A 4 GB DRAM chip takes up at least 12 x 12 mm, too large for an eyeglass frame. And wearables must respond in real time. |
| Algorithms change faster than chips | A chip designed around a single algorithm can be obsolete before it reaches production. |
At BrainChip, we designed Sequora to treat data movement as the primary design constraint, not an afterthought. The goal is to avoid as much of the memory wall as possible, not just push it back with more bandwidth, more cache, or smarter scheduling.
Start with the right question
Sequora began with a deceptively simple question: what should you actually accelerate? Answering it carefully enables true hardware-software co-design, and a design that stays relevant as algorithms change.
1. Cover the full range of edge workloads. We selected algorithms spanning every domain the edge might need. We planned for 50. The list grew to almost 100.
2. Find where the energy goes. Because moving data costs more than computing, this is harder than it sounds. We developed a method to rank every operation by its true energy burden, including data movement.
3. Focus on the few operations that matter. About 5% of the hundreds of available operations consistently consumed 95% of the energy across a wide range of algorithms. Make those few operations efficient, and acceleration holds up even as algorithms evolve.
One metric that matters
We measure success with a single metric: tokens per billion weights per joule. Put simply, how many inference steps of a given model size can you run on a given amount of energy? It captures power and performance in one number.
That metric also exposes a hidden cost. Imagine an accelerator that runs one dense operation efficiently, hands the data to a host processor for a light operation, then takes it back for another dense operation. If those transfers cost more energy than the dense work saves, the host should have done everything.
So we co-designed Sequora with a compiler that breaks each algorithm apart and fuses the best chains of operations to run on the accelerator, minimizing round trips.
Minimalist hardware, flexible memory
The hardware is deliberately minimalist:
- A large NPU array of parallel neuromorphic processors, each with local SRAM
- An intelligent DMA processor that keeps the NPU array fed with high-speed memory transfers
- Memory-agnostic design that connects to whatever memory the system requires
Our software simulator lets us, and our customers, test combinations of SRAM, DRAM, flash, and other memory technologies against real workloads.
The results
| Measure | Result | Conditions |
|---|---|---|
| Energy efficiency | Over 20 tokens per billion weights per joule | Simulated in a 22 nm GlobalFoundries process with 1 MB of SRAM, with an error margin of ±20%; holds across workloads from keyword spotting to mixture-of-experts LLMs |
| Silicon area | Less than 4 mm² | Same 22 nm configuration with 1 MB of integrated SRAM; smaller than a conventional vector processing unit |
To our knowledge, no system with equivalent capability does better. And because area is dominated by local SRAM, Sequora scales to fit the application, small enough for space-constrained wearables.
The bottom line
The memory wall isn’t going away, and it’s highest at the edge. Sequora answers it by treating data movement as the design constraint that matters most. The result is a new option for on-device voice and language processing in small, power- and space-constrained form factors where it has never been possible before.
We can’t wait to see what the industry builds with it.
Bring your workload. See the difference.
- Model your use case: Talk to our team about running your workloads and memory configuration in the Sequora simulator.
- Go deeper: Sequora
- Stay connected: follow BrainChip for the next post in this series.
