acceptodds
Under review as a conference paper at ICLR 2027

MLP Cache: A Post-Hoc Memory–Compute Trade-off for Transformer Inference

Abstract

MLPs impose substantial inference cost in Transformers by applying a full, high-capacity transformation to every token. Deployed workloads can revisit recurring local MLP input regions, there's opportunity for compute reuse beyond exact-match caching. We introduce MLP Cache, a post-hoc cache that stores compact local predictors of a frozen MLP without retraining the backbone. The central challenge is to allocate limited memory across region scale and predictor capacity: broader regions cover more inputs but may require larger predictors, finer regions are easier to approximate but require more entries. MLP Cache constructs a coarse-to-fine hierarchy using shared graph-spectral coordinates, whose signs define candidate regions and whose continuous values provide features for region-specific predictors. Tree dynamic programming selects region–predictor pairs to maximize workload coverage under memory and approximation-error budgets. On frozen Qwen3.5-9B, MLP Cache replaces 24–28% of MLP calls on MME-Coarse and MMStar-Coarse, with accuracy decreases of 1–2%. The cached model achieves a batch-one decoding wall-clock speedup on A100. Controlled MLP experiments show that cached prediction offers greater speedups for larger MLPs, the gains depends on the cost of uncached execution relative to cache lookup. For a 9B-shaped MLP, hit rates of 30% and 50% yield A100 prefill speedups of and , and M5 Pro prefill speedups of and .

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Cache, Computation Reuse, Transformer

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_yYSyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MLP Cache, a post-hoc approach that approximates selected computations of frozen Transformer MLPs using compact local predictors. Shared graph-spectral coordinates support both hierarchical partitioning of the activation space and local prediction. A tree-based selection procedure balances coverage, approximation error, and predictor storage, while unsupported inputs fall back to the original MLP. Experiments on Qwen3.5-9B demonstrate approximately 24–28% coverage of MLP calls in cache-enabled layers on two coarse-perception benchmarks. The full-model implementation achieves a 1.10× batch-one BF16 decoding speedup on A100, although full-model prefill and the tested quantized configurations do not show corresponding wall-clock improvements.

Critical strengths
  1. I find the proposed formulation interesting: it builds a post-hoc cache of local predictors for frozen MLPs and jointly selects region granularity and predictor capacity under memory and approximation-error budgets. While lookup-based computation and memory–compute trade-offs have been explored previously, this formulation, together with the use of shared spectral coordinates for both routing and prediction, appears to offer a distinctive contribution.

  2. The method is carefully developed, with thoughtful technical choices addressing representation sharing, memory allocation, approximation error, and fallback behavior. The experiments examine several of these design choices and help explain their roles in the overall method. Together, these contributions make the work technically substantial and well supported.

Critical weaknesses

My main concern is the practical scope of the method. The full-model decoding speedup reported in the main paper is limited to batch size one, whereas serving systems often batch multiple requests. At larger batch sizes, cache hits and misses may coexist within a layer: the GPU still needs to execute the original MLP for misses, while the CPU processes hits, potentially introducing synchronization overhead. Moreover, batching improves the efficiency of the original MLP, so processing the tokens that would hit the cache together with the misses may incur only a modest additional cost.

It is therefore unclear whether the reported speedup extends to batched decoding. Clarifying how the CPU and GPU branches are scheduled and synchronized, and reporting end-to-end speedups across batch sizes, would help establish the method’s practical applicability.

Questions for authors
  1. Could you report full-model decoding latency and throughput across a small range of batch sizes greater than one, using the same model and cache configuration as the batch-one result? This would help establish whether the speedup persists under batched execution and identify the operating range where caching is beneficial.
  2. How are CPU cache-hit computations and GPU cache-miss computations scheduled during batched decoding? Does subsequent computation wait for both branches to complete, and how much does this synchronization contribute to latency? If available, a timing breakdown would also be appreciated, but is not necessary.
4 comments
AuthorsOpenReview ↗

Reviewer yYSy (1/3)

We thank the reviewer for the positive assessment, and for finding the formulation and its shared spectral coordinates a distinctive contribution. In response to the questions on batched decoding and on how the CPU and GPU branches are scheduled, we rebuilt the runtime and measured full-model decode at batch 1 to 16 on two platforms. On an Apple M5 Pro, batch-one decode is now 1.20× faster with a cache that holds 1.2 GiB in memory, and decode stays faster up to batch 8 when only the layers that hit most often are cached.

What changed since the submission. We rebuilt the runtime: every cached layer used to send its routing decision to the host and wait, and now a whole decode step runs as one GPU graph (Q2a). We measured the full model on a second platform, an Apple M5 Pro, whose CPU and GPU share one memory, so the lookup and the reconstruction of hits run on the GPU itself. And we keep in memory only the cache entries that hits use, with the rest on SSD. We call the paper's two configurations the 1 GiB/layer and 4 GiB/layer caches, after their per-layer budgets.

Table T1. Request level on MME-Coarse at batch one, with the cache in all 28 cached layers. TTFT is the time to first token. Ratios are against the baseline measured with each run.

ConfigurationRequests/sTTFT (ms)End to end (ms)Decode, batch 1 (ms/token)
M5 Pro, BF163.63214.8275.460.60
M5 Pro, BF16 + 1 GiB3.78 (1.045×)213.6264.350.66 (1.20×)
M5 Pro, BF16 + 4 GiB3.93 (1.080×)210.2254.243.83 (1.38×)
M5 Pro, 4-bit4.30211.5232.721.18
M5 Pro, 4-bit + 1 GiB4.22 (0.978×)216.2236.720.54 (1.03×)
M5 Pro, 4-bit + 4 GiB4.39 (1.028×)208.4227.618.94 (1.12×)
A100, BF165.76122.3173.616.19
A100, BF16 + 1 GiB4.73 (0.822×)164.0211.314.85 (1.09×)
A100, BF16 + 4 GiB4.62 (0.802×)171.7216.514.02 (1.15×)
A100, GPTQ W4A165.68145.0176.09.41
A100, GPTQ + 1 GiB4.59 (0.807×)186.0218.19.76
A100, GPTQ + 4 GiB4.33 (0.762×)199.8231.19.44

On the M5 Pro with BF16 weights, the 1 GiB/layer cache holds 1.1–1.2 GiB in memory and makes batch-one decode 1.20× faster, with a 0.56-point drop on MME-Coarse on the same runs (91.25% without the cache, 90.69% with it). The 4 GiB/layer cache holds about 8 GiB and gives 1.38×, at a cost of about 3 points. The submission reported 1.10× on the A100 with 96 GiB of host memory.

Q1a. Full-model decode at batch sizes above one. We thank the reviewer for suggesting this measurement. We measured decode at batch 1, 2, 4, 8 and 16 on MME-Coarse on both platforms, each configuration against an uncached baseline measured alongside it.

Table T3. Full-model decode by batch size on MME-Coarse. Baseline rows give decode throughput in tokens per second. The other rows give the change in throughput against the baseline (%), with the cache in all 28 cached layers / in only the seven layers with the highest decode hit rate (Q1c).

ConfigurationBatch 1Batch 2Batch 4Batch 8Batch 16
M5 Pro, BF16 baseline (tokens/s)17.332.362.1114.6184.0
M5 Pro, BF16 + 1 GiB+19.3 / +9.3+10.7 / +6.0+3.6 / +5.2−3.8 / +2.5−13.0 / −1.1
M5 Pro, BF16 + 4 GiB+35.2 / +11.3+23.4 / +7.5+11.8 / +8.1+4.7 / +5.8−7.2 / +0.1
M5 Pro, 4-bit baseline (tokens/s)53.191.2148.5170.9204.4
M5 Pro, 4-bit + 1 GiB+1.9 / +4.2−4.0 / +0.6−7.5 / 0.0+1.7 / +3.4−3.0 / +1.3
M5 Pro, 4-bit + 4 GiB+9.4 / +5.9−1.4 / +0.9−5.5 / +1.7+5.7 / +4.8+1.1 / +1.7
A100, BF16 baseline (tokens/s)61.7122.4234.6431.5745.6
A100, BF16 + 1 GiB+9.1 / +1.9+0.9 / +0.2−8.0 / −2.1−7.3 / −6.4−15.7 / −7.1
A100, BF16 + 4 GiB+15.8 / +2.5+6.4 / +0.9−4.4 / −2.3−5.5 / −6.1−15.6 / −7.0
A100, GPTQ baseline (tokens/s)107.8201.8342.5492.3640.5
A100, GPTQ + 1 GiB−4.4 / −2.7−9.5 / −4.1−9.2 / −4.0−10.3 / −6.9−3.7 / −1.8
A100, GPTQ + 4 GiB−1.7 / −2.7−10.4 / −3.7−8.0 / −3.6−7.9 / −4.0−1.7 / −2.5

With the cache in all 28 layers (first value), BF16 decode on the M5 Pro is 19% and 35% faster at batch 1, stays faster up to batch 4 with the 1 GiB/layer cache and up to batch 8 with the 4 GiB/layer cache, and is 13% and 7% slower at batch 16. On the A100 it is 9% and 16% faster at batch 1, 1% and 6% faster at batch 2, and 4–16% slower beyond, because every cached layer still hands its hit rows to the CPU.

AuthorsOpenReview ↗

Reviewer yYSy (2/3)

Q1b. Why the gain shrinks with batch size. As the reviewer points out, in a batch hits and misses coexist within a layer, and that is the mechanism. At batch one, a hit means that the layer's MLP does not run at all, so its weights, 302 MB per layer in BF16, are never read. In a batch, the MLP runs as soon as one request misses, and its weights are then read however many of the others hit. Only at large batch does removing the hit rows shorten the matrix product itself. To map the whole range, we extended the paper's single-MLP benchmark (Figure 7) to batch sizes from 1 to 4,096 and hit rates from 0 to 0.5, with each row hitting independently.

Table T11. Speedup of one cached MLP over the original MLP, by batch size and hit rate. Values above 1 mean that the cached MLP is faster.

BatchA100: 00.10.20.30.40.5M5 Pro: 00.10.20.30.40.5
10.860.941.031.141.281.450.971.051.141.251.391.55
20.610.610.600.520.510.520.960.960.970.970.970.97
80.610.520.500.510.510.520.960.980.980.970.960.97
320.590.500.500.510.500.510.940.940.940.940.930.97
640.600.510.510.510.510.510.940.940.940.940.940.99
1280.630.540.550.540.580.580.930.940.950.860.841.07
2560.700.610.620.620.630.670.950.961.051.171.221.46
1,0240.890.860.881.101.151.420.920.971.071.171.301.57
4,0960.941.021.151.261.461.700.910.970.951.181.391.67

At batch 1 the cached MLP is faster from a hit rate of 0.1 on the M5 Pro and 0.2 on the A100. On the M5 Pro, batch 2 to 128 stays within 7% of the original MLP except at two points, and from batch 256 the cached MLP is 1.15–1.67× faster at hit rates of 0.3 and above. On the A100, batch 2 to 256 is slower at every hit rate, because handing rows to the CPU at every layer costs more than the removed rows save, and the gain returns from batch 1,024. A cached layer therefore pays at batch one and at large batch, and least in between.

Q1c. Caching only the layers that hit most often. A cached layer saves time when its hit rate times the time one hit saves exceeds its fixed cost, which it pays on every call for the Router, the lookup and merging the hit rows back. In a batch each hit saves less while the fixed cost stays, so layers with low hit rates stop paying first. Hit rates differ across layers by an order of magnitude, so the set of cached layers can follow the batch size, with no rebuild. We tested the simplest version: cache only the seven layers with the highest decode hit rate, a quarter of the 28, and run the original MLP in the rest (layers 4–8, 26 and 28 for the 1 GiB/layer cache; layers 4–6 and 27–30 for the 4 GiB/layer cache).

These are the second values in Table T3. On the M5 Pro with BF16 weights the seven layers keep decode faster up to batch 8 and within about 1% of the baseline at batch 16, where all 28 layers are 7–13% slower. On the A100 they halve the BF16 loss at batch 16, from 16% to 7%. At batch 1 and 2, all 28 layers gain more. The number of cached layers is thus chosen when the model is deployed and needs no rebuild: all 28 layers for a deployment that decodes one or two requests at a time, and only the seven layers with the highest decode hit rate for one that decodes larger batches.

AuthorsOpenReview ↗

Reviewer yYSy (3/3)

Q2a. How the two branches are scheduled. Execution is layer by layer: the next layer starts only after the current layer's output has been assembled from both branches, so later computation does wait for both. On the A100, the GPU computes the Router coordinates, looks up the region and runs the support check, which separates the hit rows from the miss rows. The CPU then reconstructs the hit rows while the GPU runs the original MLP on the misses. In the submitted runtime each cached layer sent its routing decision to the host and waited for it, so a decode step could not run as one CUDA graph. In the new runtime the whole decode step is one CUDA graph: the GPU publishes the hit rows to pinned host memory, a resident CPU thread reconstructs them, and a conditional graph node runs the original MLP on the misses in parallel. On the M5 Pro there is no CPU branch: the lookup, the reconstruction and the merge all run on the GPU inside the compute graph.

Q2b. What the synchronization costs. We timed each part on the A100, the platform with a CPU branch:

  • Decode, submitted runtime: synchronization added 0.87 ms per token. Serving the hit rows on the host accounts for 0.62 ms; the rest comes from giving up the single decode graph, which raised GPU idle time from 1.6 to 4.3 ms per step.
  • Decode, new runtime: 0.88 ms per token faster at the same hit rate, so nearly all of that cost is gone. The cached path that remains inside the graph costs 0.98 ms per token, against 3.63 ms of MLP time that the hits save.
  • Prefill of a 4,096-row batch: the two branches run concurrently, and CPU reconstruction overlaps the miss MLP for 82–93% of its duration. The main cost is a fixed host path of 0.8–1.0 ms per layer, paid even when no row hits.
  • Prefill of one request (227 rows on average): the miss MLP finishes before the CPU starts, so the branches overlap for only 3–14% of the reconstruction time.

On the M5 Pro, with no CPU branch, the whole cached path costs 2.2–2.5 ms per decode step over the 28 layers, and the hits save far more. What remains on the A100 comes from handing hit rows to the CPU. The M5 design fits an A100 as well, because the entries that hits use are few, 8 GiB for the whole benchmark and 35 MiB for decode, small enough to hold in GPU memory and reconstruct there. We leave that A100 runtime to future work.

The batched results give the operating range of the cache. With every layer cached, decode is faster up to batch 2 on the A100, and on the M5 Pro up to batch 4 with the 1 GiB/layer cache and batch 8 with the 4 GiB/layer cache. Caching only the layers that hit most often keeps M5 decode faster up to batch 8 and within about 1% of the baseline at batch 16. The revision states this range in the abstract and the limitations.

We hope the batched measurements and the scheduling breakdown answer your questions. In light of these results, we hope you might view our work even more favorably. Please feel free to let us know if you have any additional questions!

Thank you for the additional experiments and clarifications. These answer my questions about batched decoding and CPU/GPU scheduling. Although the practical gains are limited to certain settings, I think the method offers an interesting contribution. I will maintain my rating.

Reviewer_c2oqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MLP Cache, a post-hoc approach that replaces selected evaluations of a frozen Transformer MLP with compact local predictors. A shared router predicts graph-spectral coordinates, whose signs define a hierarchy of activation regions and whose continuous values serve as predictor features. The method jointly selects regions and predictor capacities under memory and approximation-error budgets. Inputs outside supported cached regions fall back to the original MLP. Experiments examine approximation quality, cache coverage, memory requirements, workload shift, and inference latency.

Critical strengths
  1. Reusing computation through local function approximation is an interesting formulation. The paper extends caching beyond exact matches by storing local approximations of a frozen MLP, enabling reuse across distinct but nearby activations.

  2. Experiments establish feasibility and characterize meaningful trade-offs. On Qwen3.5-9B, the higher-coverage configuration replaces approximately 24–28% of MLP calls in cache-enabled layers, with accuracy decreases of 0.93–2.22 percentage points on the two evaluated benchmarks.

Critical weaknesses
  1. The demonstrated benefits come at substantial cost. The higher-coverage configuration requires approximately 96 GiB of additional host memory and incurs accuracy degradation, while delivering only a 1.10× full-model decoding speedup. This limits its demonstrated practical appeal.

  2. The optimization objective is insufficiently aligned with latency. Maximizing coverage does not directly account for routing, reconstruction, communication, and synchronization costs. Full-model prefill becomes slower, and combining the cache with quantization yields no additional wall-clock improvement.

  3. The empirical evidence for general applicability remains limited. Evaluation primarily covers small visual-perception subsets. The substantial decline in late-layer answer-token coverage for longer responses, together with sensitivity to workload shift, leaves effectiveness on diverse tasks and extended generation unclear.

Questions for authors

How well does the method generalize beyond short-answer visual perception tasks? The substantial decline in late-layer answer-token coverage when visual evidence is requested raises questions about benefits during extended generation. Could the authors report end-to-end latency and task quality for longer responses or reasoning-oriented workloads, separating prefill and decoding coverage?

5 comments
AuthorsOpenReview ↗

Reviewer c2oq (1/5)

We thank the reviewer for the constructive review, and for seeing in local function approximation a way to extend caching beyond exact matches. In response to the questions on the cost of the speedup, on the alignment of the coverage objective with latency, and on the evidence beyond short perception answers, we rebuilt the runtime, measured where the time goes in every cached layer, and evaluated the full benchmarks and longer answers. At batch one on an Apple M5 Pro, a cache that holds 1.2 GiB in memory now makes decode 1.20× faster with a 0.56-point drop on MME-Coarse.

W1a. Memory and speed. We call the paper's two configurations the 1 GiB/layer and 4 GiB/layer caches, after their per-layer budgets. At batch one on an Apple M5 Pro, the 1 GiB/layer cache holds 1.1–1.2 GiB in memory and makes BF16 decode 1.20× faster and whole requests 4.5% faster on the same runs. The 4 GiB/layer cache holds about 8 GiB and gives 1.38× decode and 8.0% more requests, at a cost of about 3 points on MME-Coarse on that device. Two parts of our implementation had held the cache back, and we rebuilt both.

The runtime: the submitted runtime sent every cached layer's routing decision to the host and waited for it, which cost about 1 ms per layer in prefill and 0.87 ms per token in decode. The new runtimes no longer wait on the host within a step. On the M5 Pro the Router, the lookup and the rebuilding of hits all run inside the compute graph, and on the A100 a decode step is again one CUDA graph.

The memory: the submission kept every predictor in host memory, although all 240 MME-Coarse questions together touch only 35 MiB of decode entries. The M5 runtime keeps the full table on SSD and holds in memory only the entries that hits use. A hit whose entry is not yet loaded runs the original MLP and loads the entry for later.

All runs use Qwen3.5-9B on the inputs of Table 3, three passes each, one request at a time. On the M5 Pro the baseline and the cache alternate inside one process. MLX does not run GPTQ or SmoothQuant kernels, so the quantized model on the M5 Pro uses MLX's own 4-bit format.

Table T1. Request level on MME-Coarse, with the cache in all 28 cached layers. TTFT is the time to first token. Ratios are against the baseline measured with each run.

ConfigurationRequests/sTTFT (ms)End to end (ms)Decode, batch 1 (ms/token)
M5 Pro, BF163.63214.8275.460.60
M5 Pro, BF16 + 1 GiB3.78 (1.045×)213.6264.350.66 (1.20×)
M5 Pro, BF16 + 4 GiB3.93 (1.080×)210.2254.243.83 (1.38×)
M5 Pro, 4-bit4.30211.5232.721.18
M5 Pro, 4-bit + 1 GiB4.22 (0.978×)216.2236.720.54 (1.03×)
M5 Pro, 4-bit + 4 GiB4.39 (1.028×)208.4227.618.94 (1.12×)
A100, BF165.76122.3173.616.19
A100, BF16 + 1 GiB4.73 (0.822×)164.0211.314.85 (1.09×)
A100, BF16 + 4 GiB4.62 (0.802×)171.7216.514.02 (1.15×)
A100, GPTQ W4A165.68145.0176.09.41
A100, GPTQ + 1 GiB4.59 (0.807×)186.0218.19.76
A100, GPTQ + 4 GiB4.33 (0.762×)199.8231.19.44

On the M5 Pro with BF16 weights, decode is 1.20× and 1.38× faster and requests per second rise by 4.5% and 8.0%. MME-Coarse accuracy on these runs is 91.25% without a cache, 90.69% with the 1 GiB/layer cache and 88.19% with the 4 GiB/layer cache. The gain comes from the hits, not from the new runtime: with every lookup forced to miss, the cached path costs 2.2–2.5 ms per decode step. On the A100, batch-one decode is 1.09× and 1.15× faster against a baseline that now runs the whole decode step as one CUDA graph (16.19 ms per token; 18.10 in the submission). Whole requests are slower there because prefill takes 41–55 ms longer, a cost of the host path (W2c).

AuthorsOpenReview ↗

Reviewer c2oq (2/5)

W1a, continued: memory actually held. Table T1 was timed in steady state, after warm-up passes had loaded the entries in use: 1.1–1.2 GiB for the 1 GiB/layer cache and 8.1–8.5 GiB for the 4 GiB/layer cache. To check that the gain does not depend on having seen the questions, we replayed the cache traffic of MME-Coarse as a stream of new questions, in random image order, through a memory tier of fixed size that starts empty and evicts the least recently used entries (LRU).

Table T1b. Share of hits whose entry is already in the memory tier when a question starts, over the second half of a stream of unseen MME-Coarse questions (mean over 10 orders).

Memory tier (LRU)Decode hits in memoryPrefill hits in memoryAll hits in memory
2 GiB79%65%65%
4 GiB89%70%71%
8 GiB93%75%76%
Unlimited93%76%76%

On unseen questions a 4–8 GiB tier already holds 89–93% of the decode entries, and decode carries most of the M5 gain. Hits whose entry is not in memory fall back to the original MLP. The speedup therefore no longer needs 96 GiB of memory: a cache holding 1–8 GiB makes decode 1.20–1.38× faster where computing an MLP is slow relative to a lookup.

W1b. Accuracy. We reran both caches on all of MME (2,374 questions) and MMStar (1,500), with three seeds and paired bootstrap intervals (Table T4, under W3a). On coarse perception, the workload the caches were built for, the change is −0.3 and −0.8 points on MME-Coarse and −1.1 and −0.8 on MMStar-Coarse with the two caches, and every interval includes zero. The M5 runs of Table T1 give −0.6 points for the 1 GiB/layer cache and −3.1 for the 4 GiB/layer cache. The loss concentrates in the other categories, which are a workload shift: 2–7 points at the paper's settings, of which a runtime setting recovers 43–97% without a rebuild (W3a).

W2a. The objective is not aligned with latency. The reviewer is right that coverage does not account for routing, reconstruction, communication and synchronization, so we measured what it leaves out, per cached layer. A cached layer pays a fixed cost on every call, for the Router, the lookup and merging rows back, and saves the MLP computation of the rows that hit. It speeds the model up only when its hit rate times the time one hit saves exceeds that fixed cost. A control run in which every lookup is forced to miss measures both sides.

Table T2. Time per cached layer, in ms per decode token or per prefill request. Fixed cost is the forced-miss run minus the baseline. Saved by hits is the forced-miss run minus the normal cached run; a negative value means that serving the hits takes longer than computing them. Two values are the 1 GiB and 4 GiB/layer caches; prefill rows use the 4 GiB/layer cache. The A100 prefill row comes from a separate run on 16 requests.

SettingHit rate (%)MLP timeFixed costSaved by hitsFull-model change
M5 Pro, BF16, decode at batch 142.9 / 64.41.000.080.44 / 0.67decode 1.20× / 1.38×
M5 Pro, 4-bit, decode at batch 134.8 / 55.50.250.090.11 / 0.17decode 1.03× / 1.12×
A100, BF16, decode at batch 138.6 / 50.70.250.040.08 / 0.11decode 1.09× / 1.15×
M5 Pro, BF16, prefill of one request27.44.41−0.050.09TTFT −2.0%
M5 Pro, 4-bit, prefill of one request25.63.810.290.42TTFT −2.0%
A100, BF16, prefill of one request26.30.470.98−1.13prefill +59 ms

In decode at batch one, a hit skips the whole layer, weight reads included. On the M5 Pro a BF16 MLP layer takes 1.00 ms, mostly reading 302 MB of weights, against a fixed cost of 0.08 ms, so hits pay back several times over. On the A100 the layer takes 0.25 ms and the fixed cost is 0.04 ms, so hits still pay. In prefill a hit only removes rows from a small matrix product, so the saving is small on either device. Coverage therefore turns into latency only through this per-layer balance, and the revision adds this accounting next to the coverage objective. Each side has a lever that we have measured: larger models spend more time in each MLP (a single-MLP speedup of 1.23×, 1.45× and 1.63× for 4B-, 9B- and 27B-shaped MLPs at a 50% hit rate), and moving the decode lookup into the compute graph removed 0.88 ms per token on the A100.

AuthorsOpenReview ↗

Reviewer c2oq (3/5)

W2b. Aligning the cache with latency: which layers to cache. Hit rates differ across layers by an order of magnitude, so a layer is worth caching only when its hit rate times the saving per hit exceeds its fixed cost. Choosing the cached layers by this measured benefit aligns the cache with latency and needs no rebuild. Batch size shows why it matters: in a batch, a layer skips its weight reads only when every row hits, so a layer that hits a third of its rows rarely saves anything at batch 16 but still pays its fixed cost. We tested the simplest rule: cache only the seven layers with the highest decode hit rate, a quarter of the 28, and run the original MLP in the rest (layers 4–8, 26 and 28 for the 1 GiB/layer cache; layers 4–6 and 27–30 for the 4 GiB/layer cache).

Table T3. Full-model decode by batch size on MME-Coarse. Baseline rows give decode throughput in tokens per second. The other rows give the change in throughput against the baseline (%), with the cache in all 28 cached layers / in the seven layers with the highest decode hit rate.

ConfigurationBatch 1Batch 2Batch 4Batch 8Batch 16
M5 Pro, BF16 baseline (tokens/s)17.332.362.1114.6184.0
M5 Pro, BF16 + 1 GiB+19.3 / +9.3+10.7 / +6.0+3.6 / +5.2−3.8 / +2.5−13.0 / −1.1
M5 Pro, BF16 + 4 GiB+35.2 / +11.3+23.4 / +7.5+11.8 / +8.1+4.7 / +5.8−7.2 / +0.1
M5 Pro, 4-bit baseline (tokens/s)53.191.2148.5170.9204.4
M5 Pro, 4-bit + 1 GiB+1.9 / +4.2−4.0 / +0.6−7.5 / 0.0+1.7 / +3.4−3.0 / +1.3
M5 Pro, 4-bit + 4 GiB+9.4 / +5.9−1.4 / +0.9−5.5 / +1.7+5.7 / +4.8+1.1 / +1.7
A100, BF16 baseline (tokens/s)61.7122.4234.6431.5745.6
A100, BF16 + 1 GiB+9.1 / +1.9+0.9 / +0.2−8.0 / −2.1−7.3 / −6.4−15.7 / −7.1
A100, BF16 + 4 GiB+15.8 / +2.5+6.4 / +0.9−4.4 / −2.3−5.5 / −6.1−15.6 / −7.0
A100, GPTQ baseline (tokens/s)107.8201.8342.5492.3640.5
A100, GPTQ + 1 GiB−4.4 / −2.7−9.5 / −4.1−9.2 / −4.0−10.3 / −6.9−3.7 / −1.8
A100, GPTQ + 4 GiB−1.7 / −2.7−10.4 / −3.7−8.0 / −3.6−7.9 / −4.0−1.7 / −2.5

On the M5 Pro the seven layers keep BF16 decode faster up to batch 8 and within about 1% of the baseline at batch 16, where all 28 layers cost 7–13%. On the A100 they halve the BF16 loss at batch 16, from 16% to 7%. At batch 1 and 2, all 28 layers gain more. The number of cached layers is thus chosen when the model is deployed and needs no rebuild: all 28 layers for a deployment that decodes one or two requests at a time, and only the seven layers with the highest decode hit rate for one that decodes larger batches.

W2c. Full-model prefill became slower. Prefill is no longer slower where the lookup runs inside the compute graph: on the M5 Pro the cached path adds no cost in BF16 prefill and the hits save 0.09 ms per layer (Table T2), so the time to first token is 0.7–2% shorter. The A100 slowdown comes from routing each cached layer through the host on a GPU with separate memory: this costs 0.98 ms per layer even when nothing hits, while the MLP layer it replaces takes only 0.47 ms. Recording A100 prefill as CUDA graphs would not remove this, because every hit would still need a lookup on the host and a copy back. The remedy is to take the host out of the hit path, as the M5 runtime does. It fits an A100 as well, because the entries that hits use, 8 GiB for the benchmark and 35 MiB for decode, are small enough to hold in GPU memory. We leave this A100 prefill runtime to future work.

W2d. Quantization. A hit saves a weight read, and quantized weights are cheaper to read, so each hit saves less against the same fixed cost. On the M5 Pro with 4-bit weights the MLP takes 0.25 ms per layer against a fixed cost of 0.09 ms, so hits still pay: decode is 1.03× and 1.12× faster, and 4-bit weights with the 4 GiB/layer cache are the fastest configuration we measured on that device, at a cost of about 3 points on MME-Coarse. On the A100 the lookup still goes through the host, so the cache does not yet add to GPTQ there (9.41 ms per token alone, 9.76 and 9.44 ms with the two caches). The revised conclusion states this condition instead of a general claim.

Reviewer_31h3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MLP Cache, a post-hoc approximation framework for frozen Transformer MLPs. It constructs local predictors using shared graph-spectral coordinates, which serve both as regression features and as the basis for a hierarchical partition. A tree dynamic program, combined with a sweep over error and memory prices, selects region–predictor pairs. At inference time, supported inputs use cached predictors stored in host memory, while other inputs fall back to the original MLP. Experiments on Qwen3.5-9B report approximately 24–28% replacement of MLP calls in cache-enabled layers and a 1.10× batch-one BF16 decoding speedup, with accuracy losses and substantial additional host-memory usage.

Critical strengths

1.A useful formulation of selective approximation for frozen models. Jointly choosing region granularity and predictor capacity is a reasonable way to allocate a limited cache budget. Sharing predictor coordinates across regions reduces duplicated input-side parameters, while retaining the original MLP provides a fallback for unsupported inputs.

2.The experiments distinguish computational savings from actual latency savings. Table 3 reports both GPU activity and wall-clock measurements, including cases where caching slows execution. The controlled experiments and layerwise analysis provide useful evidence about CPU reconstruction, communication overhead, and uneven hit rates. This makes the limitations of computation reuse more informative than a FLOP-only evaluation.

Critical weaknesses

1.The practical advantage over existing deployment options is not established. The higher-coverage configuration uses 96.36 GiB of additional predictor storage to reduce BF16 decode latency from 18.1 to 16.4 ms/token, while increasing prefill latency from 414.1 to 466.0 ms and lowering accuracy on both primary benchmarks. In the same table, GPTQ alone achieves 11.2 ms/token with higher reported accuracy than this cached BF16 configuration. Adding the cache to GPTQ increases decode latency to 11.8 ms/token, while adding it to SmoothQuant provides no reported decode improvement. The controlled single-MLP speedups therefore do not establish a competitive full-model operating point. Request-level latency is particularly important because the primary workloads produce short answers, leaving limited decoding work over which to amortize the prefill overhead.

2.The evaluation does not sufficiently establish the scope or necessity of the proposed cache construction. The main full-model quality evaluation uses one backbone and two small coarse-perception subsets. Section 3.1 shows that generated-token hit rates in layers 16–31 fall from 49.09% to 6.17% when answers require supporting evidence, but corresponding quality and latency results are not provided. The advantage of the spectral construction also remains unclear: in Table 2, the deployed Non-bisecting Spectral partition does not consistently outperform simpler alternatives, and all partition comparisons retain spectral prediction features. A complete comparison with a simpler cache would help establish whether the additional construction complexity is justified.

Questions for authors

1.Deployment evidence: What is the total request latency, including prefill and generation, on the actual evaluation workloads? Please also report quality and latency for the answer-plus-evidence setting already studied in Figure 4, using the learned cache rather than prescribed hit rates.

2.Method comparison: How does the complete system compare with a simpler PCA- or k-means-based cache under matched total memory and held-out approximation-error budgets? Can the comparison separate the contribution of spectral prediction features from that of the partition hierarchy?

3.Calibration and generalization: Are predictor fitting, candidate-error estimation, support-width selection, and layer selection performed on separate data? How are these data separated from evaluation at the image/question level, and how well does the construction error budget predict error on held-out and shifted inputs?

4.Construction and resource costs: What are the total cache-construction time and peak memory requirements, including activation collection, the spectral solve, Router training, and predictor fitting? Please also specify the CPU configuration and account for Router and directory memory in addition to predictor payload.

8 comments
AuthorsOpenReview ↗

Reviewer 31h3 (1/6)

We thank the reviewer for the careful questions, and for recognizing the formulation of selective approximation for frozen models and experiments that separate computational savings from wall-clock savings. In response to the questions on the practical advantage, the scope of the evaluation and the separation of data, we rebuilt the runtime, measured whole requests on two platforms, and evaluated the full benchmarks with paired intervals. At batch one on an Apple M5 Pro, a cache that holds 1.2 GiB in memory now makes decode 1.20× faster and whole requests 4.5% faster, with a 0.56-point drop on MME-Coarse.

We also thank the reviewer for suggesting the comparison with simpler caches. As predictor features, spectral coordinates beat principal components in all 24 settings of our grid. With a local bandwidth for the kernel (Local-bandwidth Spectral), the spectral cache covers 1.1–3.9 points more than a cache built entirely from PCA, and 3.9 points more at layer 2 with 1 GiB. Spectral coordinates and principal components coincide only on Gaussian-like inputs, and the MLP inputs are not Gaussian.

W1a. Practical advantage over existing deployment options. There is now a competitive full-model operating point on real hardware. At batch one on an Apple M5 Pro, the paper's smaller configuration, which we call the 1 GiB/layer cache after its per-layer budget, holds 1.1–1.2 GiB in memory and makes BF16 decode 1.20× faster and whole requests 4.5% faster on the same runs. The larger one, the 4 GiB/layer cache, gives 1.38× decode and 8.0% more requests with about 8 GiB in memory, at a cost of about 3 points on MME-Coarse on that device. Two parts of our implementation had held the cache back, and we rebuilt both.

The runtime: the submitted runtime sent every cached layer's routing decision to the host and waited for it, which cost about 1 ms per layer in prefill and 0.87 ms per token in decode. The new runtimes no longer wait on the host within a step. On the M5 Pro the Router, the lookup and the rebuilding of hits all run inside the compute graph, and on the A100 a decode step is again one CUDA graph.

The memory: the submission kept every predictor in host memory, 96.36 GiB for the 4 GiB/layer cache, although all 240 MME-Coarse questions together touch only 35 MiB of decode entries. The M5 runtime keeps the full table on SSD and holds in memory only the entries that hits use. A hit whose entry is not yet loaded runs the original MLP and loads the entry for later.

All runs use Qwen3.5-9B on the inputs of Table 3, three passes each, one request at a time. On the M5 Pro the baseline and the cache alternate inside one process. MLX does not run GPTQ or SmoothQuant kernels, so the quantized model on the M5 Pro uses MLX's own 4-bit format.

Table T1. Request-level latency on MME-Coarse (240 questions), with the cache in all 28 cached layers. TTFT is the time to first token. Ratios are against the baseline measured with each run.

ConfigurationRequests/sTTFT (ms)Total latency (ms)Decode, batch 1 (ms/token)
M5 Pro, BF163.63214.8275.460.60
M5 Pro, BF16 + 1 GiB3.78 (1.045×)213.6264.350.66 (1.20×)
M5 Pro, BF16 + 4 GiB3.93 (1.080×)210.2254.243.83 (1.38×)
M5 Pro, 4-bit4.30211.5232.721.18
M5 Pro, 4-bit + 1 GiB4.22 (0.978×)216.2236.720.54 (1.03×)
M5 Pro, 4-bit + 4 GiB4.39 (1.028×)208.4227.618.94 (1.12×)
A100, BF165.76122.3173.616.19
A100, BF16 + 1 GiB4.73 (0.822×)164.0211.314.85 (1.09×)
A100, BF16 + 4 GiB4.62 (0.802×)171.7216.514.02 (1.15×)
A100, GPTQ W4A165.68145.0176.09.41
A100, GPTQ + 1 GiB4.59 (0.807×)186.0218.19.76
A100, GPTQ + 4 GiB4.33 (0.762×)199.8231.19.44

On the M5 Pro with BF16 weights, decode is 1.20× and 1.38× faster and requests per second rise by 4.5% and 8.0%. MME-Coarse accuracy on these runs is 91.25% without a cache, 90.69% with the 1 GiB/layer cache and 88.19% with the 4 GiB/layer cache. The gain comes from the hits, not from the new runtime: with every lookup forced to miss, the cached path costs 2.2–2.5 ms per decode step.

AuthorsOpenReview ↗

Reviewer 31h3 (2/6)

W1a, continued. On the comparison with quantization: with 4-bit weights a hit saves a cheaper weight read, yet decode on the M5 Pro is still 1.03× and 1.12× faster, and 4-bit weights with the 4 GiB/layer cache are the fastest configuration we measured on that device (18.94 ms per token against 21.18 ms for 4-bit alone, at 87.92% on MME-Coarse). On the A100, batch-one BF16 decode is 1.09× and 1.15× faster against a baseline that now runs the whole decode step as one CUDA graph. GPTQ alone is faster there (9.41 ms per token), and the cache does not yet add to it, because the lookup still goes through the host on that platform.

Memory actually held. Table T1 was timed in steady state, after warm-up passes had loaded the entries in use: 1.1–1.2 GiB for the 1 GiB/layer cache and 8.1–8.5 GiB for the 4 GiB/layer cache. To check that the gain does not depend on having seen the questions, we replayed the cache traffic of MME-Coarse as a stream of new questions, in random image order, through a memory tier of fixed size that starts empty and evicts the least recently used entries (LRU).

Table T1b. Share of hits whose entry is already in the memory tier when a question starts, over the second half of a stream of unseen MME-Coarse questions (mean over 10 orders).

Memory tier (LRU)Decode hits in memoryPrefill hits in memoryAll hits in memory
2 GiB79%65%65%
4 GiB89%70%71%
8 GiB93%75%76%
Unlimited93%76%76%

On unseen questions a 4–8 GiB tier already holds 89–93% of the decode entries, and decode carries most of the M5 gain. Hits whose entry is not in memory fall back to the original MLP.

Why the platform decides. A cached layer pays a fixed cost on every call, for the Router, the lookup and merging rows back, and saves the MLP computation of the rows that hit. It speeds the model up when its hit rate times the time one hit saves exceeds that fixed cost. A control run in which every lookup is forced to miss measures both sides.

Table T2. Time per cached layer, in ms per decode token or per prefill request. Fixed cost is the forced-miss run minus the baseline. Saved by hits is the forced-miss run minus the normal cached run; a negative value means that serving the hits takes longer than computing them. Two values are the 1 GiB and 4 GiB/layer caches; prefill rows use the 4 GiB/layer cache. The A100 prefill row comes from a separate run on 16 requests.

SettingHit rate (%)MLP timeFixed costSaved by hitsFull-model change
M5 Pro, BF16, decode at batch 142.9 / 64.41.000.080.44 / 0.67decode 1.20× / 1.38×
M5 Pro, 4-bit, decode at batch 134.8 / 55.50.250.090.11 / 0.17decode 1.03× / 1.12×
A100, BF16, decode at batch 138.6 / 50.70.250.040.08 / 0.11decode 1.09× / 1.15×
M5 Pro, BF16, prefill of one request27.44.41−0.050.09TTFT −2.0%
M5 Pro, 4-bit, prefill of one request25.63.810.290.42TTFT −2.0%
A100, BF16, prefill of one request26.30.470.98−1.13prefill +59 ms

In decode at batch one, a hit skips the whole layer, weight reads included. On the M5 Pro a BF16 MLP layer takes 1.00 ms, mostly reading 302 MB of weights, against a fixed cost of 0.08 ms, so hits pay back several times over. With 4-bit weights the layer takes 0.25 ms against the same fixed cost, so the gain shrinks but remains. On the A100 the BF16 layer also takes only 0.25 ms, so hits pay less. In prefill a hit only removes rows from a small matrix product, and on the A100 each cached layer also pays 0.98 ms on the host even when nothing hits (W1b).

Each term of the rule has a lever that we have measured: larger models spend more time in each MLP (a single-MLP speedup of 1.23×, 1.45× and 1.63× for 4B-, 9B- and 27B-shaped MLPs at a 50% hit rate), and moving the decode lookup into the compute graph removed 0.88 ms per token on the A100. The cache is therefore a competitive operating point for batch-one decoding where a weight read is slow relative to a lookup, and the revised conclusion states this condition instead of claiming that the cache complements quantization in general.

AuthorsOpenReview ↗

Reviewer 31h3 (3/6)

W1b / Q1a. Total request latency. We thank the reviewer for suggesting the total latency of a request, prefill and generation together (Table T1). On the M5 Pro with BF16 weights, total latency falls from 275 ms to 264 and 254 ms, and the time to first token falls as well, so the gain does not depend on amortizing a prefill overhead over a long answer. On the A100, decode is faster with the cache, but total latency rises from 174 ms to 211 and 217 ms, because prefill takes 41–55 ms longer. This cost comes from routing each cached layer through the host on a GPU with separate memory, and unified memory removes it. Recording A100 prefill as CUDA graphs would not remove it: every hit would still need a host lookup and a copy back. The remedy is to take the host out of the hit path, as the M5 runtime does: the entries that hits use, 8 GiB for the benchmark and 35 MiB for decode, are small enough to hold in GPU memory. We leave this runtime to future work.

Q1b. Answers with evidence, with the learned caches. Following the reviewer's suggestion, we measured quality and latency with the learned caches when answers must state their evidence, a workload shift: the caches were built from short answers, so the explanation tokens were never seen during construction. We ran the prompt of Figure 4 on MME-Coarse and MMStar-Coarse with the deployed caches, unchanged, with three seeds, at the paper's predictor-range width of 1.25 and at 0.9. This width scales the range of inputs each predictor accepts relative to the range it was fitted on; it is a runtime setting.

Table T5. A100, answers with evidence on MME-Coarse (240) and MMStar-Coarse (250). Accuracy change against BF16 in points, with the 95% paired interval; hit rates are prefill / decode over the 28 cached layers.

ConfigurationMME: accuracy changeMMStar: accuracy changeHit, MME (%)Hit, MMStar (%)
BF16 (accuracy)92.50%72.80%——
1 GiB, width 1.25−0.83 (−1.94, +0.14)−0.40 (−2.00, +1.07)7.8 / 2.76.5 / 2.4
1 GiB, width 0.9+0.14 (−0.69, +1.11)−0.53 (−1.73, +0.53)5.8 / 1.44.4 / 1.3
4 GiB, width 1.25−1.11 (−3.06, +0.69)−2.67 (−6.13, +0.67)25.3 / 13.021.7 / 12.5
4 GiB, width 0.9−0.69 (−2.50, +0.97)−2.27 (−4.40, −0.13)18.0 / 8.814.5 / 8.3

Answers now run 30–42 tokens instead of 3–4. Decode coverage falls from 36–56% to 2.4–13.0% at width 1.25, so most generated tokens use the original MLP, and accuracy stays within its 95% interval of BF16 in all four settings. On the M5 Pro, prefill is 2–6% shorter in these runs and decode runs at about baseline speed.

W2a. Scope of the evaluation. We reran both caches on all of MME (2,374 questions) and MMStar (1,500), with three seeds and paired bootstrap intervals, and separated coarse perception, the workload the caches were built for, from the other categories.

Table T4. Accuracy change against BF16 in percentage points, with the 95% paired interval, and the hit rate over the 28 cached layers. Coarse perception is shown at the paper's width (1.25); the other rows at both widths.

BenchmarkCache / layerBF16 (%)Width 1.25: changeHit (%)Width 0.9: changeHit (%)
MME-Coarse (240)1 GiB91.39−0.28 (−1.81, +1.39)8.50——
MME-Coarse (240)4 GiB91.39−0.83 (−2.92, +1.25)27.65——
MMStar-Coarse (250)1 GiB71.87−1.07 (−3.33, +1.07)7.39——
MMStar-Coarse (250)4 GiB71.87−0.80 (−4.13, +2.53)24.09——
MME, other categories (2,134)1 GiB84.46−2.30 (−3.08, −1.50)6.68−1.30 (−1.92, −0.69)4.82
MME, other categories (2,134)4 GiB84.46−2.17 (−3.00, −1.34)20.24−0.83 (−1.55, −0.11)14.04
MMStar, other categories (1,250)1 GiB65.44−1.97 (−3.25, −0.69)4.57−0.05 (−1.01, +0.93)3.06
MMStar, other categories (1,250)4 GiB65.44−6.88 (−8.69, −5.09)15.13−2.99 (−4.48, −1.55)9.70
Full MME (2,374)1 GiB85.16−2.09 (−2.84, −1.36)6.87−1.24 (−1.83, −0.65)5.02
Full MME (2,374)4 GiB85.16−2.04 (−2.81, −1.28)21.04−0.98 (−1.66, −0.31)14.71
Full MMStar (1,500)1 GiB66.51−1.82 (−2.96, −0.71)4.91−0.07 (−0.91, +0.80)3.32
Full MMStar (1,500)4 GiB66.51−5.87 (−7.49, −4.29)16.21−2.78 (−4.07, −1.47)10.51

On coarse perception every interval includes zero. The other categories are a workload shift: there the support check sends many unfamiliar inputs back to the original MLP, and the remaining hits cost 2–7 points at width 1.25, of which width 0.9 recovers 43–97%. All results are on Qwen3.5-9B: a second backbone would need its own caches, about 2,000 H100-hours each, which the rebuttal period did not allow.

Reviewer_AhbqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MLP Cache, a post-hoc method that replaces some MLP evaluations in a frozen Transformer with small local predictors stored in host memory. The motivation is that a deployed model serves a much narrower workload than it was trained for, so MLP inputs keep revisiting the same regions of activation space.

The method works per layer, in three steps:

  1. Regions. The authors compute graph-spectral coordinates from calibration activations and train a small MLP Router to predict them. The signs of these coordinates group tokens into a coarse-to-fine tree of regions.
  2. Predictors. For each region, they fit a small linear predictor that maps a few of the Router coordinates to the MLP output.
  3. Selection. A tree-based dynamic program chooses which regions to keep, and at what predictor size, to maximize the fraction of MLP calls served (hit rate) within an error budget and a memory budget. Layers where caching hurts the model output most are then disabled.

At inference, a token is a hit if it passes a support check, maps to a stored region, and falls within that predictor's fitted range. Hits are computed on the CPU from host memory, and misses run the original MLP on the GPU.

On Qwen3.5-9B with caches built from Cauldron VQA, the higher-coverage configuration replaces 27.67% and 24.09% of MLP calls in the 28 cached layers on MME-Coarse and MMStar-Coarse, with accuracy drops of 2.22 and 0.93 points, using 96.36 GiB of predictor storage. End to end on A100, batch-one decode improves from 18.1 to 16.4 ms/token (1.10×). Prefill wall-clock gets slower (414 to 466 ms), and the cache gives no wall-clock gain on GPTQ or SmoothQuant backbones. Controlled benchmarks on synthetic SwiGLU MLPs with prescribed hit rates show A100 prefill speedups of 1.24× and 1.60× at 30% and 50% hits for a 9B-shaped MLP. The paper also includes a per-layer analysis of where reuse happens (image, prompt and generated tokens), ablations on predictor representation and partitioning method, budget and calibration-size sweeps, and a workload-shift study.

Critical strengths
  1. It takes on a hard and mostly unexplored problem: approximate reuse of MLP computation in a frozen model. Almost all reuse in LLM inference today is exact (KV cache, prefix caching). Approximating the MLP itself without a large accuracy drop is hard, and most prior work that does it needs training (PKM, UltraMem, Engram) or replaces the operator wholesale (MADDNESS, LUTI-MLP). This paper shows that, for a narrow workload, about a quarter of MLP calls in the cached layers can be replaced for a 1–2 point accuracy drop without touching the backbone, and it gives a complete mechanism for doing so: a region hierarchy, small shared-feature predictors, and budgeted selection. This is the main reason I see value in the paper. Even with the shortcomings, I feel it opens up a direction the community should discuss.

  2. The analysis is honest and informative. The paper reports its negative results openly, including the prefill slowdown and the lack of gain on quantized backbones, and keeps cache coverage separate from realized latency savings. The per-layer analysis in Figure 4 is interesting on its own: image-token reuse peaks early, and late-layer reuse of generated tokens falls from 49.09% to 6.17% when the answer must include evidence, while prompt reuse in the same layers drops much less, from 45.80% to 31.67%. This makes the paper useful even where the method does not yet pay off.

Critical weaknesses
  1. The memory–compute trade does not yet pay off in a realistic serving setting.
    • Memory cost is high compared to the gain. The higher-coverage configuration uses 96.36 GiB of host memory, more than five times the BF16 weights of a 9B model, to get a 1.10× batch-one decode speedup at a 2.22 point accuracy drop on MME-Coarse. The smaller configuration uses 17.77 GiB, close to the model size, but replaces only 8.48% and 7.39% of calls and gives 1.04× decode (18.1 to 17.4 ms/token).
    • Prefill gets slower even though GPU work goes down. GPU time drops from 332.9 to 317.0 ms, but wall-clock rises from 414.1 to 466.0 ms. The same happens on GPTQ (435 to 488 ms) and SmoothQuant (397 to 470 ms), and decode on both quantized backbones shows no gain. The paper attributes this to hit rates varying from 10% to 70% across layers: low-coverage layers pay routing overhead, and high-coverage layers become limited by CPU reconstruction and transfer. This points to the execution design, not just the hit rate.
    • In the full model, the only regime with a wall-clock gain is batch-one decode. In that regime a hit skips the whole MLP, including its weight reads. Production systems serve with continuous batching over variable-length batches, usually with CUDA graphs. With more than one sequence in the batch, the MLP weights still have to be read whenever any sequence misses, so the main source of the decode saving largely goes away, while Router cost and per-layer host round trips remain. A data-dependent hit/miss path with a host round trip at every layer also looks hard to capture in a CUDA graph. None of this is evaluated.

The value of the method rests on the memory–compute trade being worthwhile for inference. The current evidence supports that reusable computation exists, but not yet that it can be turned into a favorable trade.

  1. It is hard to tell how much the proposed machinery contributes, because simple baselines are missing.
    • PCA performs about as well. In Table 2, PCA partitioning reaches 23.56% hit rate at 4 GiB against 23.38% for the Non-bisecting Spectral variant that is actually used (14.58% vs 15.77% at 1 GiB). The predictor features are held fixed to the Router coordinates in this table, so a full PCA variant was never evaluated: PCA for both the partition and the predictor features, with no Router at all. PCA is a single linear projection, exact, and needs no Router training, so it would also be cheaper at inference. The kernel bandwidth is set so that the mean anchor affinity is 0.8, which is a very wide kernel. In that regime I would expect the leading spectral coordinates to be close to the principal components, and it would be worth checking this directly on the real activations.
    • The evaluation is narrow.It covers one model, with a primary evaluation of 490 short-answer perception questions with calibration on closely related VQA data. The accuracy standard deviation is around 1 point, which is comparable to the reported drops. Figure 4 already shows that reuse depends strongly on the answer format, so it is unclear how the results carry over to text-only or long-form generation workloads.

Without these, it is difficult to attribute the reported coverage to the spectral hierarchy and the local predictors rather than to properties of the workload.

Questions for authors

Clarifications

  1. Execution cost in a serving setting. What speedup does MLP Cache give at batch sizes like 8, 32 and 64 under continuous batching, against a baseline that uses CUDA graphs? Can the cached path be graph-captured, given the per-layer GPU→CPU→GPU round trip?Figure 7 shows this breakdown for the controlled MLP; can you give the same for full-model prefill?

  2. Full PCA variant. What are the hit rate, accuracy and inference cost of a full PCA variant (PCA for both partition and predictor features, no Router)? How strongly do the leading spectral coordinates correlate with the leading principal components on the real activations?

  3. Construction cost. The method is post-hoc with respect to the backbone, but it still trains a Router per layer, solves a spectral problem, fits a candidate bank and sweeps the dynamic program. How much compute does construction take, and how long does a rebuild take after a workload shift?

Suggestions for future work

  1. MoE models. It would be interesting to see how MLP Cache behaves on MoE models. Each expert sees a narrower input distribution, which could help reuse, but caching per expert multiplies the memory cost across a large number of experts. I also wonder whether the MoE router's decision could serve as the top level of the region tree, so that a separate Router is not needed.

Minor presentation points:

  1. In Table 3, BF16, GPTQ and SmoothQuant with the same cache all report exactly 89.31 on MME-Coarse. Please confirm these numbers
14 comments
AuthorsOpenReview ↗

Reviewer Ahbq (1/12)

We sincerely thank the reviewer for the exceptionally thoughtful review. We particularly appreciate your recognition that approximate reuse of MLP computation in a frozen Transformer is a difficult and largely unexplored problem, and that our work provides a complete mechanism for addressing it. We share your emphasis on understanding when this memory–compute trade-off becomes practically beneficial. We distinguish the framework's scientific contribution from the maturity of its serving implementation. MLP Cache establishes a systematic way to materialize and selectively reuse local functions of a frozen model without retraining the backbone. The new experiments then characterize the operating conditions under which this reuse translates into actual latency savings.

We rebuilt the runtime and measured requests and batched decoding on two platforms. At batch one on an Apple M5 Pro, a cache that holds 1.2 GiB in memory now makes decoding 1.20× faster, and one that holds about 8 GiB makes it 1.38× faster.

The reviewer also conjectured that our wide-kernel spectral coordinates are close to principal components. The two coincide only on Gaussian inputs, and the MLP inputs are not Gaussian. Spectral coordinates beat principal components as predictor features in all 24 settings of our grid, and with a local bandwidth the spectral cache covers 1.1–3.9 points more than full PCA (3.9 at layer 2, 1 GiB).

W1a. Memory cost against the gain. We call the paper's two configurations the 4 GiB/layer and 1 GiB/layer caches, after their per-layer budgets. At batch one on an Apple M5 Pro the trade now reads differently: 1.2 GiB of memory gives 1.20× decode with a 0.56 pp drop, and about 8 GiB gives 1.38× at a cost of about 3 points on MME-Coarse. Three factors explain the improved operating point: a redesigned execution path, workload-aware cache residency, and the hardware-dependent balance between MLP evaluation and cache servicing.

  • The runtime. Every cached layer used to send its routing decision to the host and wait, which cost about 1 ms per layer in prefill and 0.87 ms per token in decode. The new runtimes no longer wait on the host inside a step: on the M5 Pro the Router, the lookup and the rebuilding of hits run inside the compute graph, and on the A100 a decode step is again one CUDA graph (Q1).
  • Where the cache is held. The submission kept every predictor in host memory, although hits concentrate on a small part of the table. The M5 runtime keeps the full table on SSD and holds in memory only the entries that hits use. A hit whose entry is not yet in memory runs the original MLP and loads the entry for later.
  • The platform. On the A100 a BF16 MLP layer takes 0.25 ms per decode token, so each hit saves little. On the M5 Pro, the device of the paper's controlled benchmark (Figure 7), the same layer takes 1.0 ms, nearly all of it reading 302 MB of weights, while a hit takes 39 µs.

All runs use Qwen3.5-9B on the inputs of Table 3, three passes each, one request at a time. On the M5 Pro the baseline and the cache alternate inside one process. MLX does not run GPTQ or SmoothQuant kernels, so the quantized model on the M5 Pro uses MLX's own 4-bit format.

Table T1. Request level on MME-Coarse, with the cache in all 28 cached layers. TTFT is the time to first token. Ratios are against the baseline measured with each run.

ConfigurationRequests/sTTFT (ms)End to end (ms)Decode, batch 1 (ms/token)
M5 Pro, BF163.63214.8275.460.60
M5 Pro, BF16 + 1 GiB3.78 (1.045×)213.6264.350.66 (1.20×)
M5 Pro, BF16 + 4 GiB3.93 (1.080×)210.2254.243.83 (1.38×)
M5 Pro, 4-bit4.30211.5232.721.18
M5 Pro, 4-bit + 1 GiB4.22 (0.978×)216.2236.720.54 (1.03×)
M5 Pro, 4-bit + 4 GiB4.39 (1.028×)208.4227.618.94 (1.12×)
A100, BF165.76122.3173.616.19
A100, BF16 + 1 GiB4.73 (0.822×)164.0211.314.85 (1.09×)
A100, BF16 + 4 GiB4.62 (0.802×)171.7216.514.02 (1.15×)
A100, GPTQ W4A165.68145.0176.09.41
A100, GPTQ + 1 GiB4.59 (0.807×)186.0218.19.76
A100, GPTQ + 4 GiB4.33 (0.762×)199.8231.19.44

On the M5 Pro with BF16 weights both caches are faster in decode and end to end: decode is 1.20× and 1.38× faster, and requests per second rise by 4.5% and 8.0%. The gain comes from the hits, not from the new runtime: with every lookup forced to miss, the cached path costs 2.2–2.5 ms per decode step. On the A100, batch-one BF16 decode is 1.09× and 1.15× faster against a baseline that now runs the whole decode step as one CUDA graph (16.19 ms per token; 18.10 in the submission). Whole requests are slower there because prefill takes 41–55 ms longer, a cost of the host path that W1c breaks down.

AuthorsOpenReview ↗

Reviewer Ahbq (2/12)

W1a, continued: memory actually held. Table T1 measures steady-state performance after warm-up. The 1 GiB/layer and 4 GiB/layer caches use only 1.1–1.2 and 8.1–8.5 GiB of resident predictor memory for this workload, compared with their total serialized predictor payloads of 17.77 and 96.36 GiB, respectively. Only 35 MiB of this serves decode. To check that the gain does not depend on having seen the questions, we replayed the cache traffic of MME-Coarse as a stream of new questions, in random image order, through a memory tier of fixed size that starts empty and evicts the least recently used entries (LRU).

Table T1b. Share of hits whose entry is already in the memory tier when a question starts, over the second half of a stream of unseen MME-Coarse questions (mean over 10 orders).

Memory tier (LRU)Decode hits in memoryPrefill hits in memoryAll hits in memory
2 GiB79%65%65%
4 GiB89%70%71%
8 GiB93%75%76%
Unlimited93%76%76%

Decode carries most of the M5 gain (16.5 of the 20.5 ms saved per request), and on unseen questions a 4–8 GiB tier already holds 89–93% of the decode entries. The predictors live in host memory or on SSD, never in GPU memory; beyond them, the Routers, the lookup directory and the support-check metadata take 1.8 GiB of GPU memory and 0.83 GiB of host memory. A deployment that caches only decode can drop the image-token predictors and keep 41.75 of the 96.36 GiB.

Accuracy. On the M5 runs of Table T1, MME-Coarse accuracy is 91.25% without a cache, 90.69% with the 1 GiB/layer cache and 88.19% with the 4 GiB/layer cache. On the A100, with three seeds and paired intervals, the change is −0.28 and −0.83 points and both intervals include zero (Table T4); the −2.22 points quoted in the review lie inside the new interval.

W1b. When the trade pays off. We thank the reviewer for identifying the gap between reusable computation and a favorable memory–compute trade. Our new measurements reveal a general operating principle: caching becomes beneficial when the hit-rate-weighted savings from avoided MLP computation exceed the fixed costs of routing, lookup, and output assembly. This balance, rather than hit rate alone, determines the speedup and depends on the operator size, hardware memory architecture, and batch size. We measure these costs directly using forced-miss controls (Table T2).

Table T2. Time per cached layer, in ms per decode token or per prefill request. Fixed cost is the forced-miss run minus the baseline. Saved by hits is the forced-miss run minus the normal cached run; a negative value means that serving the hits takes longer than computing them. Two values are the 1 GiB and 4 GiB/layer caches; prefill rows use the 4 GiB/layer cache.

SettingHit rate (%)MLP timeFixed costSaved by hitsFull-model change
M5 Pro, BF16, decode at batch 142.9 / 64.41.000.080.44 / 0.67decode 1.20× / 1.38×
M5 Pro, 4-bit, decode at batch 134.8 / 55.50.250.090.11 / 0.17decode 1.03× / 1.12×
A100, BF16, decode at batch 138.6 / 50.70.250.040.08 / 0.11decode 1.09× / 1.15×
M5 Pro, BF16, prefill of one request27.44.41−0.050.09TTFT −2.0%
M5 Pro, 4-bit, prefill of one request25.63.810.290.42TTFT −2.0%
A100, BF16, prefill of one request26.30.470.98−1.13prefill +59 ms

The rule reproduces every measured change in decode latency to within 0.3 ms per token on both platforms. On the M5 Pro a BF16 MLP layer takes 1.00 ms against a fixed cost of 0.08 ms, so hits pay back several times over; on the A100 the layer takes 0.25 ms, so hits still pay, but less. In prefill a hit only removes rows from a small matrix product, and on the A100 each cached layer also costs 0.98 ms on the host (W1c). Each term has a lever that we have measured: a partition that can be routed in one pass would add 1.4–7.0 points of hit rate (Table T6); larger models spend more time in each MLP (a single-MLP speedup of 1.23×, 1.45× and 1.63× for 4B-, 9B- and 27B-shaped MLPs at a 50% hit rate); and moving the decode lookup into the compute graph removed 0.88 ms per token on the A100. These results show that the memory–compute trade-off is not determined by cache coverage alone. It can be improved through better partitioning, larger operator savings, and lower cache-service overhead, providing multiple independent directions for optimizing the framework. The revision states the condition in the abstract, the introduction and the limitations.

AuthorsOpenReview ↗

Reviewer Ahbq (3/12)

W1c / Q1 (prefill breakdown). Prefill gets slower although GPU work goes down. We thank the reviewer for suspecting the execution design, and for suggesting the full-model counterpart of Figure 7. We traced full-model prefill on the A100 with host and device events on one clock, in the paper's setting, batches of 4,096 token rows, and for single requests (227 rows on average).

Table T9. A100 prefill with the 4 GiB/layer cache (ms): the baseline, the cached path with every lookup forced to miss, and the normal cached run. Tables T9 and T10 come from a separate run on 16 requests.

SettingBaselineForced missCached, hits servedFixed cost (forced miss − baseline)Serving the hits (cached − forced miss)
Batch of 4,096 rows414.0437.2455.9+23.2 (0.83 per layer)+18.7
One request (227 rows)91.5119.1150.6+27.6 (0.98 per layer)+31.5

Table T10. The cached MLP path in one 4,096-row prefill batch, by stage, summed over the 28 cached layers (ms). Host times come from the profiled pass, which lengthens them, so they show proportions.

StageHostGPU kernelsCopies
Router, both token types11.47.0—
Lookup and support check2.71.5—
Routing decision copied to the host, and the wait for it8.32.80.8
Original MLP on the miss rows (submit / run)15.7106.8—
CPU reconstruction of the hit rows, and their copy to the GPU84.0—14.3
Assembly of the layer output2.70.50.1
Waiting for the previous pinned buffer1.6——
Other host dispatch of the cached path (allocation, gather and scatter, checks)31.1——

The reviewer's reading is right: the slowdown comes from the execution path, in three parts. First, a fixed host cost per cached layer, paid even when nothing hits: with every lookup forced to miss, prefill still costs 0.83–0.98 ms more per layer, for dispatching the cached path, launching the Router, and copying its decision to the host and waiting for it. Second, copying the hit rows back cancels the kernel time they save: MLP kernel time falls by 12.5 ms per batch, which is the drop in GPU time that the paper reported, but copying the hit rows to the GPU adds 14.5 ms. Third, CPU reconstruction is hidden in a large batch, where it overlaps the original MLP on the misses for 82–86% of its time, and exposed in a single request, where the overlap falls to 3–14%.

This cost comes from routing each cached layer through the host on a GPU with separate memory, not from uneven hit rates across layers, and the revision replaces the paper's explanation with this breakdown. Recording A100 prefill as CUDA graphs would remove the dispatch, but every hit would still need a lookup on the host and a copy back. The remedy is to take the host out of the hit path. On the M5 Pro, where the GPU reads the cache directly, the cached path costs 0.08 ms per layer in decode and prefill is not slower. The same design fits an A100, because the entries that hits use, 8 GiB for the benchmark and 35 MiB for decode, are small enough to hold in GPU memory. We leave this A100 prefill runtime to future work.

W1d. Quantized backbones. A hit saves a weight read, and quantized weights are cheaper to read, so each hit saves less against the same fixed cost: on the M5 Pro an MLP layer takes 0.25 ms per decode token with 4-bit weights, against 1.00 ms with BF16. The cache and quantization still stack there at batch one: on top of 4-bit weights, the 4 GiB/layer cache gives 1.12× decode and 2.8% more requests, at a cost of about 3 points on MME-Coarse, and the 1 GiB/layer cache gives 1.03× decode with accuracy unchanged. On the A100 the lookup goes through the host, so the cache does not yet add to GPTQ there. The revised conclusion states this condition instead of a general claim.

Reviewer_2ZCPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MLP Cache replaces some calls of a frozen Transformer MLP with region-specific low-rank predictors: graph-spectral coordinates define both a region hierarchy and the predictor features, and a tree DP selects regions and ranks under memory and error budgets. On Qwen3.5-9B it replaces 24–28% of MLP calls in the enabled layers on MME-/MMStar-Coarse at −2.22/−0.93 accuracy points, giving 1.10× batch-one decode on A100 with 96.36 GiB of host memory.

Critical strengths
  1. Clean formulation, honest measurement. Post-hoc reuse is framed as choosing region scale and predictor capacity under budgets, and GPU time, wall-clock, prefill and decode are reported separately, negative results included.
  2. Useful analyses of layer-wise prompt/answer reuse, data-limited coverage, and why hit rate does not predict latency.
Critical weaknesses
  1. The paper's own Table 3 shows the trade does not pay. The only full-model gain is 1.10× batch-one decode for 96.36 GiB and −2.2 points, while prefill slows from 414 to 466 ms. GPTQ alone brings decode from 18.1 to 11.2 ms/token at about 0.3 points, and adding the cache on top of GPTQ makes both decode (11.2→11.8 ms) and prefill (435.4→488.3 ms) slower; end-to-end and batched decoding are not reported. Yet the conclusion says the cache "complements quantization".
  2. The evidence cannot resolve the accuracy cost or the novelty. 240/250 coarse-perception questions with three samples cannot resolve 1–2 points (2.22 points is about five questions); the abstract leads with synthetic-benchmark speedups that §3.4 says do not carry over to full-model prefill; and post-hoc contextual sparsity and visual-token pruning, which target the same redundancy, are neither discussed nor compared.
Questions for authors
  1. End-to-end latency on the evaluated workloads, and decode at batch 4 and 8?
  2. Accuracy on the full MME/MMStar with confidence intervals, and the uncached baselines for MMStar Math/Logical? Table 3 lists 89.31 on MME-Coarse with the cache for all three backbones — is this a transcription error?
4 comments
AuthorsOpenReview ↗

Reviewer 2ZCP (1/4)

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing the clean formulation, the honest measurement and the analyses of where reuse occurs. In response to the questions on whether the trade pays, on quantization and on the accuracy evidence, we rebuilt the runtime, measured whole requests on two platforms, and evaluated the full benchmarks with paired intervals. At batch one on an Apple M5 Pro, a cache that holds 1.2 GiB in memory now makes decode 1.20× faster and whole requests 4.5% faster, with a 0.56-point drop on MME-Coarse.

W1a. Whether the trade pays. It pays at batch one on an Apple M5 Pro. The paper's smaller configuration, which we call the 1 GiB/layer cache after its per-layer budget, holds 1.1–1.2 GiB in memory and makes BF16 decode 1.20× faster and whole requests 4.5% faster on the same runs. The larger one, the 4 GiB/layer cache, gives 1.38× decode and 8.0% more requests with about 8 GiB in memory, at a cost of about 3 points on MME-Coarse on that device. Two parts of our implementation had held the cache back, and we rebuilt both.

The runtime: the submitted runtime sent every cached layer's routing decision to the host and waited for it, which cost about 1 ms per layer in prefill and 0.87 ms per token in decode. The new runtimes no longer wait on the host within a step. On the M5 Pro the Router, the lookup and the rebuilding of hits all run inside the compute graph, and on the A100 a decode step is again one CUDA graph.

The memory: the submission kept every predictor in host memory, 96.36 GiB for the 4 GiB/layer cache, although all 240 MME-Coarse questions together touch only 35 MiB of decode entries. The M5 runtime keeps the full table on SSD and holds in memory only the entries that hits use. A hit whose entry is not yet loaded runs the original MLP and loads the entry for later.

All runs use Qwen3.5-9B on the inputs of Table 3, three passes each, one request at a time. On the M5 Pro the baseline and the cache alternate inside one process. MLX does not run GPTQ or SmoothQuant kernels, so the quantized model on the M5 Pro uses MLX's own 4-bit format.

Table T1. Request level on MME-Coarse, with the cache in all 28 cached layers. TTFT is the time to first token. Ratios are against the baseline measured with each run.

ConfigurationRequests/sTTFT (ms)End to end (ms)Decode, batch 1 (ms/token)
M5 Pro, BF163.63214.8275.460.60
M5 Pro, BF16 + 1 GiB3.78 (1.045×)213.6264.350.66 (1.20×)
M5 Pro, BF16 + 4 GiB3.93 (1.080×)210.2254.243.83 (1.38×)
M5 Pro, 4-bit4.30211.5232.721.18
M5 Pro, 4-bit + 1 GiB4.22 (0.978×)216.2236.720.54 (1.03×)
M5 Pro, 4-bit + 4 GiB4.39 (1.028×)208.4227.618.94 (1.12×)
A100, BF165.76122.3173.616.19
A100, BF16 + 1 GiB4.73 (0.822×)164.0211.314.85 (1.09×)
A100, BF16 + 4 GiB4.62 (0.802×)171.7216.514.02 (1.15×)
A100, GPTQ W4A165.68145.0176.09.41
A100, GPTQ + 1 GiB4.59 (0.807×)186.0218.19.76
A100, GPTQ + 4 GiB4.33 (0.762×)199.8231.19.44

On the M5 Pro with BF16 weights, decode is 1.20× and 1.38× faster and requests per second rise by 4.5% and 8.0%. MME-Coarse accuracy on these runs is 91.25% without a cache, 90.69% with the 1 GiB/layer cache and 88.19% with the 4 GiB/layer cache. The gain comes from the hits, not from the new runtime: with every lookup forced to miss, the cached path costs 2.2–2.5 ms per decode step. On the A100, batch-one decode is 1.09× and 1.15× faster against a baseline that now runs the whole decode step as one CUDA graph (16.19 ms per token; 18.10 in the submission). Whole requests are slower there because prefill takes 41–55 ms longer, a cost of routing each cached layer through the host on a GPU with separate memory.

AuthorsOpenReview ↗

Reviewer 2ZCP (2/4)

W1a, continued: memory actually held. Table T1 was timed in steady state, after warm-up passes had loaded the entries in use: 1.1–1.2 GiB for the 1 GiB/layer cache and 8.1–8.5 GiB for the 4 GiB/layer cache. To check that the gain does not depend on having seen the questions, we replayed the cache traffic of MME-Coarse as a stream of new questions, in random image order, through a memory tier of fixed size that starts empty and evicts the least recently used entries (LRU).

Table T1b. Share of hits whose entry is already in the memory tier when a question starts, over the second half of a stream of unseen MME-Coarse questions (mean over 10 orders).

Memory tier (LRU)Decode hits in memoryPrefill hits in memoryAll hits in memory
2 GiB79%65%65%
4 GiB89%70%71%
8 GiB93%75%76%
Unlimited93%76%76%

On unseen questions a 4–8 GiB tier already holds 89–93% of the decode entries, and decode carries most of the M5 gain (16.5 of the 20.5 ms saved per request). Hits whose entry is not in memory fall back to the original MLP.

Why the platform decides. A cached layer pays a fixed cost on every call, for the Router, the lookup and merging rows back, and saves the MLP computation of the rows that hit. It speeds the model up when its hit rate times the time one hit saves exceeds that fixed cost. A control run in which every lookup is forced to miss measures both sides.

Table T2. Time per cached layer, in ms per decode token or per prefill request. Fixed cost is the forced-miss run minus the baseline. Saved by hits is the forced-miss run minus the normal cached run; a negative value means that serving the hits takes longer than computing them. Two values are the 1 GiB and 4 GiB/layer caches; prefill rows use the 4 GiB/layer cache. The A100 prefill row comes from a separate run on 16 requests.

SettingHit rate (%)MLP timeFixed costSaved by hitsFull-model change
M5 Pro, BF16, decode at batch 142.9 / 64.41.000.080.44 / 0.67decode 1.20× / 1.38×
M5 Pro, 4-bit, decode at batch 134.8 / 55.50.250.090.11 / 0.17decode 1.03× / 1.12×
A100, BF16, decode at batch 138.6 / 50.70.250.040.08 / 0.11decode 1.09× / 1.15×
M5 Pro, BF16, prefill of one request27.44.41−0.050.09TTFT −2.0%
M5 Pro, 4-bit, prefill of one request25.63.810.290.42TTFT −2.0%
A100, BF16, prefill of one request26.30.470.98−1.13prefill +59 ms

In decode at batch one, a hit skips the whole layer, weight reads included. On the M5 Pro a BF16 MLP layer takes 1.00 ms, mostly reading 302 MB of weights, against a fixed cost of 0.08 ms, so hits pay back several times over. On the A100 the same layer takes 0.25 ms, so hits pay less. In prefill a hit only removes rows from a small matrix product, and on the A100 each cached layer also goes through the host, which costs 0.98 ms even when nothing hits. Recording A100 prefill as CUDA graphs would not remove this, because every hit would still need a lookup on the host and a copy back. The remedy is to take the host out of the hit path, as the M5 runtime does. It fits an A100 as well, because the entries that hits use, 8 GiB for the benchmark and 35 MiB for decode, are small enough to hold in GPU memory. We leave this A100 prefill runtime to future work.

Each term of the rule has a lever that we have measured. Larger models spend more time in each MLP: the single-MLP speedup at a 50% hit rate is 1.23×, 1.45× and 1.63× for MLPs shaped like those of 4B, 9B and 27B models. Moving the decode lookup into the compute graph removed 0.88 ms per token on the A100. The trade therefore pays where a weight read is slow relative to a lookup, and the revision states this condition in the abstract, the introduction and the limitations.

AuthorsOpenReview ↗

Reviewer 2ZCP (3/4)

W1b. Quantization. A hit saves a weight read, and quantized weights are cheaper to read, so each hit saves less against the same fixed cost (Table T2). On the M5 Pro an MLP layer takes 0.25 ms per decode token with 4-bit weights, against 1.00 ms with BF16, while the fixed cost stays at 0.08–0.09 ms. The cache and quantization still stack there at batch one: on top of 4-bit weights, the 4 GiB/layer cache gives 1.12× decode and 2.8% more requests, at a cost of about 3 points on MME-Coarse (Table T1). On the A100 the GPTQ layer is cheaper still and the lookup goes through the host, so the cache does not yet add to GPTQ there. The cache therefore adds to quantization where the lookup is cheap relative to the quantized MLP, and the revised conclusion states this condition instead of a general claim.

W2a. 240 or 250 questions cannot resolve a 1–2 point drop. Agreed. We reran both caches on all of MME (2,374 questions) and MMStar (1,500), with three seeds and paired bootstrap intervals. On coarse perception, the workload the caches were built for, the change is −0.3 to −1.1 points and all four intervals include zero; the M5 runs of Table T1 give −0.6 points for the 1 GiB/layer cache and −3.1 for the 4 GiB/layer cache. The other categories, such as math and logical reasoning, are a workload shift for a cache built from general VQA. There the support check sends many unfamiliar inputs back to the original MLP, lowering the hit rate by 21–38%, and the hits that remain cost 2–7 points at the paper's predictor-range width of 1.25. This width scales the range of inputs each predictor accepts relative to the range it was fitted on, and it is a runtime setting. Narrowing it to 0.9 recovers 43–97% of that loss without a rebuild.

Table T4. Accuracy change against BF16 in percentage points, with the 95% paired interval, and the hit rate over the 28 cached layers. Coarse perception is shown at the paper's width (1.25); the other rows at both widths.

BenchmarkCache / layerBF16 (%)Width 1.25: changeHit (%)Width 0.9: changeHit (%)
MME-Coarse (240)1 GiB91.39−0.28 (−1.81, +1.39)8.50——
MME-Coarse (240)4 GiB91.39−0.83 (−2.92, +1.25)27.65——
MMStar-Coarse (250)1 GiB71.87−1.07 (−3.33, +1.07)7.39——
MMStar-Coarse (250)4 GiB71.87−0.80 (−4.13, +2.53)24.09——
MME, other categories (2,134)1 GiB84.46−2.30 (−3.08, −1.50)6.68−1.30 (−1.92, −0.69)4.82
MME, other categories (2,134)4 GiB84.46−2.17 (−3.00, −1.34)20.24−0.83 (−1.55, −0.11)14.04
MMStar, other categories (1,250)1 GiB65.44−1.97 (−3.25, −0.69)4.57−0.05 (−1.01, +0.93)3.06
MMStar, other categories (1,250)4 GiB65.44−6.88 (−8.69, −5.09)15.13−2.99 (−4.48, −1.55)9.70
Full MME (2,374)1 GiB85.16−2.09 (−2.84, −1.36)6.87−1.24 (−1.83, −0.65)5.02
Full MME (2,374)4 GiB85.16−2.04 (−2.81, −1.28)21.04−0.98 (−1.66, −0.31)14.71
Full MMStar (1,500)1 GiB66.51−1.82 (−2.96, −0.71)4.91−0.07 (−0.91, +0.80)3.32
Full MMStar (1,500)4 GiB66.51−5.87 (−7.49, −4.29)16.21−2.78 (−4.07, −1.47)10.51

The support check is what limits the loss under shift. Replaying both caches on held-out inputs at five layers, tokens between width 0.9 and 1.25 carry 1.5–2.0× the error per token of tokens within 0.9, and tokens beyond 1.25, which the check sends back to the original MLP, would carry 5–7×. Without the check, the cache's error averaged over all tokens would be 2.0–3.1× its value on the construction set; with it, the error is 1.0–1.4× at the paper's width and 0.5–0.8× at width 0.9.

W2b. The abstract leads with synthetic speedups. Agreed. The revised abstract leads with the full-model results on both platforms and presents the single-MLP benchmark as a controlled analysis of when a hit pays.

W2c. Contextual sparsity and visual-token pruning. Both target the same redundancy, and we add both to related work. Contextual sparsity (Deja Vu, Liu et al., ICML 2023; TEAL, Liu et al., ICLR 2025) predicts which neurons to compute for each input; it still reads the selected weights, while a cache hit replaces the whole layer output with a stored low-rank function, so no weight is read. Visual-token pruning removes image tokens after an early layer (FastV, Chen et al., ECCV 2024) or freezes them where they change little (ShortV, Yuan et al., ICCV 2025). A cache approximates the output instead of discarding it, so it also covers tokens that pruning must keep, including the text tokens that carry all of our decode gain.

AuthorsOpenReview ↗

Reviewer 2ZCP (4/4)

Q1a. End-to-end latency on the evaluated workloads. We now time complete requests, one at a time (Table T1). On MME-Coarse on the M5 Pro, end-to-end latency with BF16 weights falls from 275 ms to 264 and 254 ms with the two caches; with 4-bit weights it goes from 233 ms to 237 and 228 ms. On the A100 it rises from 174 ms to 211 and 217 ms, because of the host path in prefill (W1a).

Q1b. Decode at batch 4 and 8. In a batch, a layer skips its weight reads only when every row hits, so the gain shrinks as the batch grows. We measured full-model decode at batch 1 to 16 on both platforms, each against a baseline measured alongside it. Because a layer that hits only some of its rows still pays its fixed cost, we also tested caching only the seven layers with the highest decode hit rate, a quarter of the 28, with the original MLP in the rest. This needs no rebuild.

Table T3. Full-model decode by batch size on MME-Coarse. Baseline rows give decode throughput in tokens per second. The other rows give the change in throughput against the baseline (%), with the cache in all 28 cached layers / in the seven layers with the highest decode hit rate.

ConfigurationBatch 1Batch 2Batch 4Batch 8Batch 16
M5 Pro, BF16 baseline (tokens/s)17.332.362.1114.6184.0
M5 Pro, BF16 + 1 GiB+19.3 / +9.3+10.7 / +6.0+3.6 / +5.2−3.8 / +2.5−13.0 / −1.1
M5 Pro, BF16 + 4 GiB+35.2 / +11.3+23.4 / +7.5+11.8 / +8.1+4.7 / +5.8−7.2 / +0.1
M5 Pro, 4-bit baseline (tokens/s)53.191.2148.5170.9204.4
M5 Pro, 4-bit + 1 GiB+1.9 / +4.2−4.0 / +0.6−7.5 / 0.0+1.7 / +3.4−3.0 / +1.3
M5 Pro, 4-bit + 4 GiB+9.4 / +5.9−1.4 / +0.9−5.5 / +1.7+5.7 / +4.8+1.1 / +1.7
A100, BF16 baseline (tokens/s)61.7122.4234.6431.5745.6
A100, BF16 + 1 GiB+9.1 / +1.9+0.9 / +0.2−8.0 / −2.1−7.3 / −6.4−15.7 / −7.1
A100, BF16 + 4 GiB+15.8 / +2.5+6.4 / +0.9−4.4 / −2.3−5.5 / −6.1−15.6 / −7.0
A100, GPTQ baseline (tokens/s)107.8201.8342.5492.3640.5
A100, GPTQ + 1 GiB−4.4 / −2.7−9.5 / −4.1−9.2 / −4.0−10.3 / −6.9−3.7 / −1.8
A100, GPTQ + 4 GiB−1.7 / −2.7−10.4 / −3.7−8.0 / −3.6−7.9 / −4.0−1.7 / −2.5

On the M5 Pro with BF16 weights, decode at batch 4 is 3.6% and 11.8% faster with all 28 layers and 5.2% and 8.1% faster with seven. At batch 8 it changes by −3.8% and +4.7% with all layers and by +2.5% and +5.8% with seven. On the A100 the cache gains only at batch 1 and 2, because every cached layer still hands its hit rows to the CPU.

Q2. Full benchmarks, MMStar Math and Logical baselines, and the three 89.31 entries. On the full benchmarks, the change at the paper's width is −2.0 to −2.1 points on MME and −1.8 and −5.9 on MMStar with the two caches, and at width 0.9 it is −1.0 to −1.2 and −0.1 to −2.8 (Table T4, in part 3/4 under W2a). Table T8 adds the baselines that Figure 8 left out.

Table T8. MMStar Math and Logical, 250 questions each: accuracy (%), mean ± SD over three sampling runs.

ConfigurationMathLogical
BF1664.00 ± 1.4466.27 ± 1.89
GPTQ W4A1660.53 ± 1.4063.47 ± 1.15
SmoothQuant W8A855.73 ± 0.2362.00 ± 0.40
BF16 + 4 GiB cache, width 1.25 (paper)55.47 ± 2.8457.07 ± 0.61
BF16 + 4 GiB cache, width 0.961.33 ± 1.6260.40 ± 1.74

At the paper's width the cache is 8.5 and 9.2 points below BF16 on Math and Logical. Width 0.9 recovers 5.9 and 3.3 points with no rebuild, and on Math the cache is then level with GPTQ. The three 89.31 entries are not a transcription error: they come from three independent runs, on BF16, GPTQ and SmoothQuant backbones, each with 643 of 720 question–seed pairs correct, and their standard deviations differ (1.05, 1.34 and 2.71 points).

We hope these results address your concerns. In light of them, we hope you might view our work more favorably. Please feel free to let us know if you have any additional questions!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank all the reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the strengths of MLP Cache, particularly the hard and mostly unexplored problem it takes on, its clean formulation under memory and error budgets, and its honest measurements, which separate coverage from wall-clock latency.

MLP Cache trades memory for computation in a frozen Transformer, without retraining it. Within a workload, the MLP layers see similar inputs again and again, and on such inputs a small local model approximates their output well. MLP Cache therefore divides the space of MLP inputs into regions and stores a compact predictor for the regions that the workload visits often, within a memory budget and an error budget. At inference, an input that falls in a stored region is answered by a lookup instead of the MLP computation, and every other input runs the original MLP unchanged, so the cache only answers where it was fitted.

Beyond the immediate speedup numbers, the central contribution of MLP Cache is to establish a new post-hoc memory–compute trade-off for frozen Transformers. Rather than caching exact computation results, we materialize compact local functions that can serve distinct activations within recurring regions. This introduces a new allocation problem: which parts of a frozen model's computation should be stored, at what granularity and predictor capacity, under limited memory and approximation-error budgets? We provide a complete framework to address this problem through hierarchical regions, local predictors, budgeted selection, and selective fallback. Importantly, the coordinate representation, predictor design, and serving backend remain modular components that can be optimized independently.

The reviewers raised important questions about the practical value of the cache, particularly whether it makes whole requests faster and at what memory cost, whether the spectral construction adds anything over simpler PCA- and k-means-based caches, and whether the accuracy holds on full benchmarks and under workload shift. We also appreciate the suggestions to break down where the time goes and to report the construction cost and the separation of construction and evaluation data. These points helped us strengthen the evaluation and make the scope of our contributions clearer.

In response, we conducted additional experiments and expanded several parts of the manuscript:

  1. Whole requests on two platforms: We rebuilt the runtime and measured whole requests on the A100 and on an Apple M5 Pro. At batch one on the M5 Pro, a cache that holds 1.2 GiB in memory makes decode 1.20× faster and whole requests 4.5% faster, with a 0.56 pp accuracy drop. A larger cache that holds about 8 GiB makes decode 1.38× faster, at a cost of about 3 points (Table T1).
  2. When the cache pays: A per-layer breakdown gives one rule: a cached layer saves time when its hit rate times the time one hit saves exceeds its fixed cost. The cache therefore pays where computing an MLP is slow relative to a lookup. With unified memory, as on the M5 Pro, the lookup runs on the GPU itself, and decode stays faster up to batch 8 with the larger cache. The same breakdown traces the prefill cost on the A100 to routing each cached layer through the host, which unified memory removes (Tables T2 and T3).
  3. Spectral coordinates against PCA and k-means: PCA sees only the covariance of the inputs, while spectral coordinates also follow their clusters and curved structure, so the two agree only on Gaussian-like data. The MLP inputs are not Gaussian, and in a grid of PCA-, k-means- and spectral-based caches, spectral coordinates beat principal components as predictor features in every setting. With a local kernel bandwidth, a standard refinement of spectral methods, the spectral cache also covers 1.1–3.9 points more than a cache built entirely from PCA (Tables T6 and T12).
  4. Accuracy on the full benchmarks: We reran both caches on all of MME and MMStar with paired intervals. On coarse perception, the workload the caches were built for, every accuracy change is within its 95% interval of zero. On the other categories, which are a workload shift, the cache sends unfamiliar inputs back to the original MLP, and tightening one runtime setting recovers 43–97% of the remaining loss without rebuilding the cache (Tables T4 and T5).
  5. Construction cost and data separation: We report the construction cost stage by stage, and an image-overlap audit of the construction set.

Together, these results establish a modular framework for post-hoc computation reuse and clarify when its memory–compute trade-off becomes beneficial. The revisions will be uploaded soon. We thank the reviewers for their constructive feedback.

AuthorsOpenReview ↗

Revised Manuscript Uploaded — Summary and Invitation for Discussion

We sincerely thank all reviewers for their thoughtful and constructive feedback. We have now uploaded our revised manuscript, incorporating the additional experiments, analyses, and clarifications discussed in our rebuttal. The major revisions are highlighted in blue.

To briefly summarize the key updates:

  1. Practical efficiency: We redesigned the runtime and evaluated full-model inference on both the A100 and Apple M5 Pro. On the M5 Pro, MLP Cache achieves 1.20–1.38× faster batch-one BF16 decoding and 4.5–8.0% higher end-to-end request throughput, while explicitly characterizing the accuracy–memory trade-off, batching behavior, and remaining A100 overheads.

  2. Methodology and novelty: We conducted extensive PCA, k-means, and spectral comparisons, separating partitioning from predictor representations. These analyses clarify the benefits of spectral coordinates while reinforcing the modularity of our framework.

  3. Robustness and reproducibility: We expanded evaluation to the full MME and MMStar benchmarks with paired confidence intervals, investigated longer responses and workload shifts, and added detailed analyses of memory residency, construction costs, and data overlap.

Beyond these empirical improvements, we have clarified our central contribution: MLP Cache establishes a post-hoc framework for trading memory for reusable computation in frozen Transformers, together with a systematic characterization of when this trade-off becomes beneficial.

We greatly appreciate the reviewers' contributions to strengthening this work. We warmly welcome further discussion and would be more than happy to address any remaining questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.