New paper, open access on Zenodo: Mapping ML Workloads to Silicon (DOI 10.5281/zenodo.21082396).
Inference is a memory problem, not a compute problem
The headline number on an AI accelerator is TOPS — trillions of operations per second. It is also, for the workload that actually costs money, almost the wrong number to look at. Frontier-model training is compute-bound and embarrassingly parallel. But single-user, batch-1 LLM inference — the chat workload — spends most of its time reading the model and the KV-cache out of memory to produce each token. On a modern GPU that means the expensive tensor cores sit largely idle while the memory system is pinned. The binding constraint has moved from peak FLOPS to memory bandwidth and the energy of moving data.
A framework, then a design
The paper first builds a framework: it decomposes ML workloads — transformers, CNNs, GNNs, diffusion and state-space models — into compute primitives characterised by arithmetic intensity, memory-access pattern, and parallelism, and uses hierarchical roofline and utilisation modelling to show which architecture serves which workload, and why. Quantisation, it turns out, only sharpens the memory-bound problem rather than solving it.
That analysis motivates a concrete design — the Fused Memory-Compute Tile (FMCT): an SRAM-first, HBM-free accelerator with a three-mode phase-adaptive tile switched by compiler directive, a non-linear unit fused into the systolic datapath, compute-in-memory offered as a selectable mode rather than the whole architecture, and an inline KV-cache compression engine that extends context length without spending accelerator FLOPs. A validated component-level energy model shows roughly 4–7× batch-1 efficiency over HBM GPUs.
From analysis to licensable IP
Several of these building blocks are exactly the kind of contained, protocol-agnostic block that fits a soft-IP licensing model — including the KV-cache compression engine, which now appears in the Safety Soft-IP catalog as a functional-safety-instrumented block. If you are building inference silicon and want the memory-centric analysis behind it, the full paper is open access.