- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes MLP Cache, a post-hoc approach that approximates selected computations of frozen Transformer MLPs using compact local predictors. Shared graph-spectral coordinates support both hierarchical partitioning of the activation space and local prediction. A tree-based selection procedure balances coverage, approximation error, and predictor storage, while unsupported inputs fall back to the original MLP. Experiments on Qwen3.5-9B demonstrate approximately 24–28% coverage of MLP calls in cache-enabled layers on two coarse-perception benchmarks. The full-model implementation achieves a 1.10× batch-one BF16 decoding speedup on A100, although full-model prefill and the tested quantized configurations do not show corresponding wall-clock improvements.
Critical strengths
-
I find the proposed formulation interesting: it builds a post-hoc cache of local predictors for frozen MLPs and jointly selects region granularity and predictor capacity under memory and approximation-error budgets. While lookup-based computation and memory–compute trade-offs have been explored previously, this formulation, together with the use of shared spectral coordinates for both routing and prediction, appears to offer a distinctive contribution.
-
The method is carefully developed, with thoughtful technical choices addressing representation sharing, memory allocation, approximation error, and fallback behavior. The experiments examine several of these design choices and help explain their roles in the overall method. Together, these contributions make the work technically substantial and well supported.
Critical weaknesses
My main concern is the practical scope of the method. The full-model decoding speedup reported in the main paper is limited to batch size one, whereas serving systems often batch multiple requests. At larger batch sizes, cache hits and misses may coexist within a layer: the GPU still needs to execute the original MLP for misses, while the CPU processes hits, potentially introducing synchronization overhead. Moreover, batching improves the efficiency of the original MLP, so processing the tokens that would hit the cache together with the misses may incur only a modest additional cost.
It is therefore unclear whether the reported speedup extends to batched decoding. Clarifying how the CPU and GPU branches are scheduled and synchronized, and reporting end-to-end speedups across batch sizes, would help establish the method’s practical applicability.
Questions for authors
- Could you report full-model decoding latency and throughput across a small range of batch sizes greater than one, using the same model and cache configuration as the batch-one result? This would help establish whether the speedup persists under batched execution and identify the operating range where caching is beneficial.
- How are CPU cache-hit computations and GPU cache-miss computations scheduled during batched decoding? Does subsequent computation wait for both branches to complete, and how much does this synchronization contribute to latency? If available, a timing breakdown would also be appreciated, but is not necessary.
4 comments
Reviewer yYSy (2/3)
Q1b. Why the gain shrinks with batch size. As the reviewer points out, in a batch hits and misses coexist within a layer, and that is the mechanism. At batch one, a hit means that the layer's MLP does not run at all, so its weights, 302 MB per layer in BF16, are never read. In a batch, the MLP runs as soon as one request misses, and its weights are then read however many of the others hit. Only at large batch does removing the hit rows shorten the matrix product itself. To map the whole range, we extended the paper's single-MLP benchmark (Figure 7) to batch sizes from 1 to 4,096 and hit rates from 0 to 0.5, with each row hitting independently.
Table T11. Speedup of one cached MLP over the original MLP, by batch size and hit rate. Values above 1 mean that the cached MLP is faster.
| Batch | A100: 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | M5 Pro: 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 0.86 | 0.94 | 1.03 | 1.14 | 1.28 | 1.45 | 0.97 | 1.05 | 1.14 | 1.25 | 1.39 | 1.55 |
| 2 | 0.61 | 0.61 | 0.60 | 0.52 | 0.51 | 0.52 | 0.96 | 0.96 | 0.97 | 0.97 | 0.97 | 0.97 |
| 8 | 0.61 | 0.52 | 0.50 | 0.51 | 0.51 | 0.52 | 0.96 | 0.98 | 0.98 | 0.97 | 0.96 | 0.97 |
| 32 | 0.59 | 0.50 | 0.50 | 0.51 | 0.50 | 0.51 | 0.94 | 0.94 | 0.94 | 0.94 | 0.93 | 0.97 |
| 64 | 0.60 | 0.51 | 0.51 | 0.51 | 0.51 | 0.51 | 0.94 | 0.94 | 0.94 | 0.94 | 0.94 | 0.99 |
| 128 | 0.63 | 0.54 | 0.55 | 0.54 | 0.58 | 0.58 | 0.93 | 0.94 | 0.95 | 0.86 | 0.84 | 1.07 |
| 256 | 0.70 | 0.61 | 0.62 | 0.62 | 0.63 | 0.67 | 0.95 | 0.96 | 1.05 | 1.17 | 1.22 | 1.46 |
| 1,024 | 0.89 | 0.86 | 0.88 | 1.10 | 1.15 | 1.42 | 0.92 | 0.97 | 1.07 | 1.17 | 1.30 | 1.57 |
| 4,096 | 0.94 | 1.02 | 1.15 | 1.26 | 1.46 | 1.70 | 0.91 | 0.97 | 0.95 | 1.18 | 1.39 | 1.67 |
At batch 1 the cached MLP is faster from a hit rate of 0.1 on the M5 Pro and 0.2 on the A100. On the M5 Pro, batch 2 to 128 stays within 7% of the original MLP except at two points, and from batch 256 the cached MLP is 1.15–1.67× faster at hit rates of 0.3 and above. On the A100, batch 2 to 256 is slower at every hit rate, because handing rows to the CPU at every layer costs more than the removed rows save, and the gain returns from batch 1,024. A cached layer therefore pays at batch one and at large batch, and least in between.
Q1c. Caching only the layers that hit most often. A cached layer saves time when its hit rate times the time one hit saves exceeds its fixed cost, which it pays on every call for the Router, the lookup and merging the hit rows back. In a batch each hit saves less while the fixed cost stays, so layers with low hit rates stop paying first. Hit rates differ across layers by an order of magnitude, so the set of cached layers can follow the batch size, with no rebuild. We tested the simplest version: cache only the seven layers with the highest decode hit rate, a quarter of the 28, and run the original MLP in the rest (layers 4–8, 26 and 28 for the 1 GiB/layer cache; layers 4–6 and 27–30 for the 4 GiB/layer cache).
These are the second values in Table T3. On the M5 Pro with BF16 weights the seven layers keep decode faster up to batch 8 and within about 1% of the baseline at batch 16, where all 28 layers are 7–13% slower. On the A100 they halve the BF16 loss at batch 16, from 16% to 7%. At batch 1 and 2, all 28 layers gain more. The number of cached layers is thus chosen when the model is deployed and needs no rebuild: all 28 layers for a deployment that decodes one or two requests at a time, and only the seven layers with the highest decode hit rate for one that decodes larger batches.
Reviewer yYSy (3/3)
Q2a. How the two branches are scheduled. Execution is layer by layer: the next layer starts only after the current layer's output has been assembled from both branches, so later computation does wait for both. On the A100, the GPU computes the Router coordinates, looks up the region and runs the support check, which separates the hit rows from the miss rows. The CPU then reconstructs the hit rows while the GPU runs the original MLP on the misses. In the submitted runtime each cached layer sent its routing decision to the host and waited for it, so a decode step could not run as one CUDA graph. In the new runtime the whole decode step is one CUDA graph: the GPU publishes the hit rows to pinned host memory, a resident CPU thread reconstructs them, and a conditional graph node runs the original MLP on the misses in parallel. On the M5 Pro there is no CPU branch: the lookup, the reconstruction and the merge all run on the GPU inside the compute graph.
Q2b. What the synchronization costs. We timed each part on the A100, the platform with a CPU branch:
- Decode, submitted runtime: synchronization added 0.87 ms per token. Serving the hit rows on the host accounts for 0.62 ms; the rest comes from giving up the single decode graph, which raised GPU idle time from 1.6 to 4.3 ms per step.
- Decode, new runtime: 0.88 ms per token faster at the same hit rate, so nearly all of that cost is gone. The cached path that remains inside the graph costs 0.98 ms per token, against 3.63 ms of MLP time that the hits save.
- Prefill of a 4,096-row batch: the two branches run concurrently, and CPU reconstruction overlaps the miss MLP for 82–93% of its duration. The main cost is a fixed host path of 0.8–1.0 ms per layer, paid even when no row hits.
- Prefill of one request (227 rows on average): the miss MLP finishes before the CPU starts, so the branches overlap for only 3–14% of the reconstruction time.
On the M5 Pro, with no CPU branch, the whole cached path costs 2.2–2.5 ms per decode step over the 28 layers, and the hits save far more. What remains on the A100 comes from handing hit rows to the CPU. The M5 design fits an A100 as well, because the entries that hits use are few, 8 GiB for the whole benchmark and 35 MiB for decode, small enough to hold in GPU memory and reconstruct there. We leave that A100 runtime to future work.
The batched results give the operating range of the cache. With every layer cached, decode is faster up to batch 2 on the A100, and on the M5 Pro up to batch 4 with the 1 GiB/layer cache and batch 8 with the 4 GiB/layer cache. Caching only the layers that hit most often keeps M5 decode faster up to batch 8 and within about 1% of the baseline at batch 16. The revision states this range in the abstract and the limitations.
We hope the batched measurements and the scheduling breakdown answer your questions. In light of these results, we hope you might view our work even more favorably. Please feel free to let us know if you have any additional questions!
Thank you for the additional experiments and clarifications. These answer my questions about batched decoding and CPU/GPU scheduling. Although the practical gains are limited to certain settings, I think the method offers an interesting contribution. I will maintain my rating.
Reviewer yYSy (1/3)
We thank the reviewer for the positive assessment, and for finding the formulation and its shared spectral coordinates a distinctive contribution. In response to the questions on batched decoding and on how the CPU and GPU branches are scheduled, we rebuilt the runtime and measured full-model decode at batch 1 to 16 on two platforms. On an Apple M5 Pro, batch-one decode is now 1.20× faster with a cache that holds 1.2 GiB in memory, and decode stays faster up to batch 8 when only the layers that hit most often are cached.
What changed since the submission. We rebuilt the runtime: every cached layer used to send its routing decision to the host and wait, and now a whole decode step runs as one GPU graph (Q2a). We measured the full model on a second platform, an Apple M5 Pro, whose CPU and GPU share one memory, so the lookup and the reconstruction of hits run on the GPU itself. And we keep in memory only the cache entries that hits use, with the rest on SSD. We call the paper's two configurations the 1 GiB/layer and 4 GiB/layer caches, after their per-layer budgets.
Table T1. Request level on MME-Coarse at batch one, with the cache in all 28 cached layers. TTFT is the time to first token. Ratios are against the baseline measured with each run.
On the M5 Pro with BF16 weights, the 1 GiB/layer cache holds 1.1–1.2 GiB in memory and makes batch-one decode 1.20× faster, with a 0.56-point drop on MME-Coarse on the same runs (91.25% without the cache, 90.69% with it). The 4 GiB/layer cache holds about 8 GiB and gives 1.38×, at a cost of about 3 points. The submission reported 1.10× on the A100 with 96 GiB of host memory.
Q1a. Full-model decode at batch sizes above one. We thank the reviewer for suggesting this measurement. We measured decode at batch 1, 2, 4, 8 and 16 on MME-Coarse on both platforms, each configuration against an uncached baseline measured alongside it.
Table T3. Full-model decode by batch size on MME-Coarse. Baseline rows give decode throughput in tokens per second. The other rows give the change in throughput against the baseline (%), with the cache in all 28 cached layers / in only the seven layers with the highest decode hit rate (Q1c).
With the cache in all 28 layers (first value), BF16 decode on the M5 Pro is 19% and 35% faster at batch 1, stays faster up to batch 4 with the 1 GiB/layer cache and up to batch 8 with the 4 GiB/layer cache, and is 13% and 7% slower at batch 16. On the A100 it is 9% and 16% faster at batch 1, 1% and 6% faster at batch 2, and 4–16% slower beyond, because every cached layer still hands its hit rows to the CPU.