- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the loss of individual key-value associations when fast weight models such as linear attention, DeltaNet, and TTT compress the context into a bounded state. RECUE keeps a per-head cache of associations whose keys score highest on a regularized diagonal Mahalanobis distance, which the appendix relates to a diagonal approximation of ridge leverage. At each read, the query’s cosine similarity to the cached keys decides which cached reconstruction residuals matter. A ridge regression over the cached features turns those residuals into a correction to the fast weight (only the output projection in LaCT’s gated MLP memory), which is scaled by the gap between the top two similarities, used for the current read, and then discarded. Top-K RECUE solves the ridge problem on the K closest entries, while Full RECUE keeps the whole cache and reweights the residual targets so the inverse Gram matrix can be precomputed. Applied to official LaCT-760M and DeltaNet-1.3B checkpoints without training, it raises LaCT’s 8K single-needle RULER average from 70.27 to 94.13 and lifts the LongBench average by 0.68 and 0.65 points on the two models. Ablations cover the cache policy, correction magnitude, cue temperature, standard LM benchmarks, and a comparison with a LaCT model pretrained with a 4K sliding window.
Critical strengths
-
The method is training-free, works on official checkpoints, and the LaCT gains are large right where the compressed memory starts to fail. The 8K single-needle average goes from 70.27 to 94.13, MK-NIAH-1 at 8K from 48.6 to 72.4, and S-NIAH-2 at 16K from 37.6 to 58.4 (65.2 with Top-16), with no general regression in Table 8. Patching only the output projection of the gated MLP memory is a clean way to carry the idea from linear to nonlinear fast weights. It should apply to other TTT memories with a linear readout.
-
The ablations are quite interesting for inference-time patches. Table 5 matches correction magnitude across methods (RCR around 0.17) and still finds a wide spread (38.2 for Ridge Rehearsal, 62.4 for Attention Read, 82.6 for Full RECUE at fixed strength), so the gain is not just a larger perturbation. The UUID-protected setting in Table 4 separates keeping the target from the makeup of the rest of the cache. The result that self-recall error, the criterion LoLA uses, does no better than random is a useful data point for anyone designing hybrid caches.
Critical weaknesses
-
The rehearsal and rollback proposals hide what is actually computed. The patch is used for a single read and then dropped (Algorithm 2 simply adds u to o_base), so the output equals o_base + η Σ_i ρ_i(q) β_i(q) r_i with β(q) = (KK^T + λI)^{-1} K q. That is a ridge-regression read over cue-weighted cached residuals, so here a temporary correction to the memory and a direct read from the cache are the same computation. What separates RECUE from the direct-retrieval methods in Section 2 is residuals instead of values and ridge weights instead of softmax weights. Attention Read adds raw values, so it cannot tell us which of the two matters, and the paper needs a residual-valued attention read and a read rule closer to LoLA’s to support its positioning.
-
Outside synthetic recall the evidence is not strong at all. The LongBench gains are under 0.7 points on averages near 15, with no confidence intervals. About two-thirds of DeltaNet’s gain comes from MultiNews alone (11.10 to 17.07), where Attention Read and Top-16 also jump, which looks more like a shift in generation behavior than recovered associations. On DeltaNet RULER, Full RECUE is within 0.6 points of Attention Read on five of the six averages, and τ was chosen from sweeps on RULER and LongBench themselves, with no latency or memory numbers despite efficiency being the motivation.
Questions for authors
-
Can you report a residual-valued attention read (softmax over the same cosine cue, residuals r_i as values, same margin gate and 25% cap)? Can you also report a variant that commits the correction to the state instead of rolling it back?
-
In Table 8, DeltaNet’s LAMBADA perplexity drops from 12.27 to 8.61 and accuracy rises by 6.6 points, which is a large effect for a read-time patch. LAMBADA passages are short relative to the 512-token refresh interval, so how is the cache populated in this setting? Can you confirm that in teacher-forced scoring no continuation tokens reach the cache or the residuals before they are scored?
-
Table 4 reports RCR of 0.2568 and 0.2585 for SRE, which is above the 25% cap on the correction norm. Is RCR measured before the cap, and how often does the cap bind for each method? Do the matched-RCR comparisons in Table 5 hold for the corrections that are actually applied?
-
Eq. 11 defines μ and σ² over the prefill keys, but Algorithm 1 computes them over the retained cache plus the new chunk. Once the cache is full, half of that set was kept for being atypical. Does this make the notion of an underrepresented direction drift over long inputs, and how often is the needle actually in the cache at 8K and 16K under each policy?
-
What are the per-token latency and memory overhead relative to the base model, given that each head stores cue keys, the ridge map P, and residuals R for up to 2K entries? Appendix D treats a 2K cache and 2K of extra window as the same budget, which seems to undercount the cache. Does RECUE still help on a gated backbone such as Gated DeltaNet, where much of the loss is deliberate forgetting rather than interference?
7 comments
Responses to Reviewer VcDA (2/4)
W1 & W2 & Q1: Temporary correction vs. direct read (Continued)
Commit variant. For a single read, committing and rolling back give identical outputs; they differ only in subsequent reads, so this variant directly tests the rehearsal interference of Eq. 7. This variant is well-defined because ReCue’s correction solves the memory’s own regression objective, whereas an attention read has no natural state counterpart to commit. We commit the applied correction at each decoding step and update the cached residuals accordingly, with all other settings unchanged (8K, 500 samples per task).
- Rollback vs. commit.
| Model | Task | Baseline | Full rollback | Full commit | Top-16 rollback | Top-16 commit |
|---|---|---|---|---|---|---|
| LaCT | S-NIAH-3 | 34.00 | 85.20 | 46.20 | 84.60 | 41.00 |
| LaCT | MQ-NIAH | 29.65 | 37.60 | 33.80 | 38.25 | 40.70 |
| DeltaNet | S-NIAH-3 | 8.60 | 13.40 | 14.20 | 14.00 | 1.80 |
| DeltaNet | MQ-NIAH | 12.45 | 19.70 | 20.90 | 21.95 | 17.30 |
Committing is unstable. On LaCT S-NIAH-3, whose UUID answers require the most decoding steps, it reduces accuracy by 39.0 (Full) and 43.6 (Top-16) points, and on DeltaNet, Top-16 with commit falls below the baseline on S-NIAH-3 (1.80 vs. 8.60). Where committing helps, the gains are at most 2.45 points. Rollback prevents corrections from accumulating across reads and is never far below commit, so we retain it.
Efficiency. We measure time to first token (TTFT), per-token decoding latency (average over 127 steps), and peak allocated memory on an A100-SXM4-80GB GPU in BF16 with batch size 1.
Efficiency at 8K and 16K context.
| Model | Method | TTFT 8K (ms) | TTFT 16K (ms) | Decode 8K (ms/token) | Decode 16K (ms/token) | Peak 8K (GiB) | Peak 16K (GiB) |
|---|---|---|---|---|---|---|---|
| LaCT | Base | 309.79 | 590.83 | 45.77 | 46.47 | 2.862 | 3.186 |
| Attention Read | 379.06 | 722.37 | 54.40 | 55.67 | 3.214 | 3.623 | |
| Top-16 ReCue | 403.18 | 750.06 | 66.83 | 68.11 | 3.360 | 3.764 | |
| Full ReCue | 412.52 | 744.59 | 56.48 | 56.84 | 3.517 | 3.904 | |
| DeltaNet | Base | 140.90 | 262.98 | 34.83 | 34.55 | 3.023 | 3.476 |
| Attention Read | 214.89 | 391.24 | 42.74 | 41.74 | 3.117 | 3.570 | |
| Top-16 ReCue | 218.94 | 403.76 | 52.98 | 50.66 | 3.162 | 3.615 | |
| Full ReCue | 257.58 | 442.00 | 42.55 | 41.59 | 3.209 | 3.662 |
- Decoding: Relative to the base model, Full ReCue increases per-token latency by 23.4% and 22.3% on LaCT and by 22.2% and 20.4% on DeltaNet at 8K and 16K. Like Attention Read, its per-token correction costs per head, and it decodes at a speed comparable to Attention Read. Top-16 ReCue decodes more slowly, as it solves a small linear system at each step.
- Memory: Since the cache size is fixed, the additional memory for the cached keys, values, ridge map , and residuals does not grow substantially with context length (Full ReCue: 0.66 → 0.72 GiB on LaCT, 0.19 GiB on DeltaNet).
- TTFT: and are prepared at each cache refresh rather than at each decoding step, so their cost appears in TTFT instead of decoding latency.
Q2: LAMBADA Verification
We thank the reviewer for carefully checking the LAMBADA results for possible leakage. We confirm that no continuation token enters the cache or its residuals before it is scored.
Cache initialization. LaCT applies its fast weight before updating it at the end of each chunk, so tokens in an incomplete final chunk have never been written into it and are not cached. DeltaNet updates its fast weight at every token, so we also cache the remaining partial chunk at the end of prefill, independent of the 512-token refresh interval; we will make this explicit in Appendix E. Since all 5,153 LAMBADA contexts contain 59–228 tokens, each DeltaNet head retains every context association within its 512-entry budget, with residuals computed from the state after the context.
Causal scoring. Each target token is scored before it is supplied as input; previously scored target tokens enter the recurrent state, as required by teacher forcing. We verified this with an instrumented audit:
- Provenance: on all examples, cached positions in every layer and head come only from the context, and cache and residual payloads remain unchanged during target scoring.
- Future invariance: changing or appending future target tokens leaves all preceding logits exactly unchanged (334 prefix tests across 128 examples).
- Zero-correction control: setting the correction to zero while keeping the full cache and ridge pipeline reproduces native logits exactly on all examples.
Why the gain is large. Since the cache holds the entire context, ReCue can reinstate any context association. Accuracy improves by 8.16 points when the target word appears in the context, versus −0.53 points when it does not.
Responses to Reviewer VcDA (3/4)
Q3: RCR and the correction cap
We thank the reviewer for carefully checking the RCR values against the correction cap. RCR in Tables 4 and 5 was measured before the cap, which is why SRE exceeds 0.25. Since clipping changes only the reads where the cap binds, the gap between pre-cap and post-cap RCR comes entirely from these reads. RCR and cap-binding rates below are computed over decode steps × layers × heads.
Cap-binding rates in Table 4. The cap binds about twice as often under SRE as under the other policies, consistent with its larger pre-cap RCR. Mahalanobis binds about as often as Random, yet achieves much higher accuracy.
- Cap-binding rates of the cache-selection policies (Table 4 setting).
| Cache policy | Standard (500) | UUID-protected (445) |
|---|---|---|
| Random | 9.23% | 9.45% |
| SRE | 19.78% | 19.14% |
| Mahalanobis | 9.63% | 9.92% |
Matched RCR on applied corrections. We re-ran the Table 5 comparison with RCR matched on the corrections actually applied (LaCT, S-NIAH-3 8K, 2,048 cache entries, 25% cap). Fixed strengths were calibrated on a separate set of 100 samples and evaluated on 500 samples without re-tuning; post-cap RCR matches within 0.72% relative error.
- Matched-RCR comparison on applied corrections.
| Method | Fixed η | Accuracy | RCR (pre-cap) | RCR (post-cap) | Cap-binding rate |
|---|---|---|---|---|---|
| Ridge Rehearsal | 0.002280 | 36.40 | 0.08444 | 0.06186 | 5.48% |
| Attention Read | 0.002599 | 62.00 | 0.08524 | 0.06153 | 5.71% |
| Full ReCue | 0.006140 | 74.20 | 0.08451 | 0.06221 | 4.75% |
| Full ReCue + margin gate | – | 85.20 | 0.17260 | 0.06177 | 10.46% |
With applied corrections matched, the ranking is unchanged: Full ReCue with a fixed strength outperforms Attention Read by 12.2 points, and the margin gate adds another 11.0 points.
Q4: Cache statistics and needle retention
We thank the reviewer for this careful observation. Algorithm 1 computes and over the candidate pool (retained cache and new chunk) rather than over all prefill keys as in Eq. 11. To measure the resulting drift, we compare the final candidate-pool statistics with those of the full prefill on LaCT-760M (S-NIAH-3, 500 samples per length, 2,048 cache entries per head).
- Drift of the candidate-pool statistics.
| Context | Std. mean difference ↓ | Variance ratio | Top-2048 overlap ↑ |
|---|---|---|---|
| 8K | 0.0790 | 1.0631 | 92.18% |
| 16K | 0.1554 | 1.1545 | 89.95% |
The drift grows with context length, but the selection is largely preserved (overlap of about 90%).
We also measure needle retention on out-of-window samples (370 at 8K, 445 at 16K). Token retention is the fraction of answer UUID tokens retained, averaged over all 24 layers × 4 heads; full-UUID retention is the fraction of samples in which all UUID tokens remain together in at least one head’s cache.
- Needle retention and S-NIAH-3 accuracy.
| Context | Policy | Token retention | Full-UUID retention | Accuracy |
|---|---|---|---|---|
| 8K | Random | 33.22% | 0.00% | 15.20 |
| SRE | 41.49% | 99.19% | 19.00 | |
| Mahalanobis (Ours) | 87.23% | 100.00% | 85.20 | |
| Mahalanobis (all-past statistics) | 88.54% | 100.00% | 84.80 | |
| 16K | Random | 14.27% | 0.00% | 5.60 |
| SRE | 20.28% | 27.19% | 5.40 | |
| Mahalanobis (Ours) | 69.75% | 94.16% | 19.20 | |
| Mahalanobis (all-past statistics) | 71.54% | 96.85% | 22.20 |
Mahalanobis retains far more of the needle than Random and SRE and achieves the highest accuracy. Following the reviewer’s observation, accumulating exact statistics over all past keys, as defined in Eq. 11, requires only additional memory per head and no additional computation. It improves 16K accuracy from 19.20 to 22.20, consistent with the larger drift at longer contexts.
Responses to Reviewer VcDA (1/4)
We thank Reviewer VcDA for the detailed and constructive feedback. We appreciate that the reviewer finds ReCue training-free and effective on official checkpoints, a clean extension to nonlinear fast weights, and supported by informative ablations.
For weaknesses and questions, we address each point below.
W1 & W2 & Q1: Temporary correction vs. direct read
We thank the reviewer for the insightful question on how ReCue relates to direct retrieval and for suggesting clarifying experiments.
We agree that ReCue’s output can be written as obase+η∑iρi(q)βi(q)ri. However, any reinstatement can be interpreted as a read, since the current output decomposes as
o=q⊤(S+ΔS)=obase+q⊤ΔS,
which, like direct retrieval, adds a cache-derived term to obase. Therefore, what matters is whether the read is consistent with the geometry of the memory. In the test-time regression view [1], a fast weight memory is a linear regressor updated by gradient descent on a regression objective, so a consistent correction reads through a kernel linear in the query feature and accounts for correlations among keys through the Gram matrix, as ReCue’s ridge solve does. Softmax attention is instead a locally constant (Nadaraya–Watson) regressor whose nonnegative, normalized weights yield a convex combination of cached targets.
The residual-valued attention read the reviewer suggested is therefore the right experiment: it uses the same residual targets, cue, margin gate, 25% cap, and cache as ReCue, and changes only how the cache-derived term is computed.
On LaCT, Full ReCue outperforms both attention reads on all NIAH and LongBench averages, with gains of 9.3 and 7.2 points over the residual attention read on the 8K and 16K single-needle averages. On DeltaNet, the reads perform comparably on NIAH, while Full ReCue achieves a higher LongBench average (14.86 vs. 14.34), outperforming the residual attention read on 10 of 14 tasks.
MultiNews. On DeltaNet, Full ReCue and the residual attention read score nearly identically on MultiNews (17.07 vs. 17.16), so the gap between them does not come from MultiNews. Excluding MultiNews, Full ReCue still outperforms the residual attention read by a similar margin on both backbones (LaCT: 16.43 vs. 15.89; DeltaNet: 14.69 vs. 14.13).
Choice of τ. The gains do not depend on tuning τ: for τ ∈ {0.05, 0.1, 0.2}, Full ReCue achieves LongBench averages of 15.53–15.71 on LaCT and 14.80–14.88 on DeltaNet, above Attention Read (15.20 and 14.65), and improves or matches RULER over the base model at every setting (Tables 6–7).
[1] Wang et al., 2025. Test-Time Regression: A Unifying Framework for Designing Sequence Models with Associative Memory.