acceptodds
Under review as a conference paper at ICLR 2027

ReCue: Reinstatement through Query-Cued Rehearsal for Fast Weight Models

Abstract

Transformers retain past key–value pairs explicitly, enabling strong associative retrieval but requiring a growing KV cache. Fast weight models offer a more efficient alternative by compressing the history into a bounded state, but this compression can degrade individual key–value associations through interference. Rehearsing past associations may recover them, yet fast weight updates are non-local and can perturb other associations. We introduce ReCue: Reinstatement through Query-Cued Rehearsal, a training-free method that preserves associations along underrepresented key directions in a bounded cache and selectively rehearses them using the current query. ReCue jointly rehearses the cued associations through ridge regression, applies the resulting correction only to the current read, and immediately restores the original state. On pretrained LaCT and DeltaNet models, ReCue improves associative recall on RULER and improves average performance on LongBench over the base models and query-independent rehearsal. These results show that query-cued rehearsal can recover information lost through fast weight compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Associative Memory, Linear Attention, Test-Time Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VcDAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the loss of individual key-value associations when fast weight models such as linear attention, DeltaNet, and TTT compress the context into a bounded state. RECUE keeps a per-head cache of associations whose keys score highest on a regularized diagonal Mahalanobis distance, which the appendix relates to a diagonal approximation of ridge leverage. At each read, the query’s cosine similarity to the cached keys decides which cached reconstruction residuals matter. A ridge regression over the cached features turns those residuals into a correction to the fast weight (only the output projection in LaCT’s gated MLP memory), which is scaled by the gap between the top two similarities, used for the current read, and then discarded. Top-K RECUE solves the ridge problem on the K closest entries, while Full RECUE keeps the whole cache and reweights the residual targets so the inverse Gram matrix can be precomputed. Applied to official LaCT-760M and DeltaNet-1.3B checkpoints without training, it raises LaCT’s 8K single-needle RULER average from 70.27 to 94.13 and lifts the LongBench average by 0.68 and 0.65 points on the two models. Ablations cover the cache policy, correction magnitude, cue temperature, standard LM benchmarks, and a comparison with a LaCT model pretrained with a 4K sliding window.

Critical strengths
  1. The method is training-free, works on official checkpoints, and the LaCT gains are large right where the compressed memory starts to fail. The 8K single-needle average goes from 70.27 to 94.13, MK-NIAH-1 at 8K from 48.6 to 72.4, and S-NIAH-2 at 16K from 37.6 to 58.4 (65.2 with Top-16), with no general regression in Table 8. Patching only the output projection of the gated MLP memory is a clean way to carry the idea from linear to nonlinear fast weights. It should apply to other TTT memories with a linear readout.

  2. The ablations are quite interesting for inference-time patches. Table 5 matches correction magnitude across methods (RCR around 0.17) and still finds a wide spread (38.2 for Ridge Rehearsal, 62.4 for Attention Read, 82.6 for Full RECUE at fixed strength), so the gain is not just a larger perturbation. The UUID-protected setting in Table 4 separates keeping the target from the makeup of the rest of the cache. The result that self-recall error, the criterion LoLA uses, does no better than random is a useful data point for anyone designing hybrid caches.

Critical weaknesses
  1. The rehearsal and rollback proposals hide what is actually computed. The patch is used for a single read and then dropped (Algorithm 2 simply adds u to o_base), so the output equals o_base + η Σ_i ρ_i(q) β_i(q) r_i with β(q) = (KK^T + λI)^{-1} K q. That is a ridge-regression read over cue-weighted cached residuals, so here a temporary correction to the memory and a direct read from the cache are the same computation. What separates RECUE from the direct-retrieval methods in Section 2 is residuals instead of values and ridge weights instead of softmax weights. Attention Read adds raw values, so it cannot tell us which of the two matters, and the paper needs a residual-valued attention read and a read rule closer to LoLA’s to support its positioning.

  2. Outside synthetic recall the evidence is not strong at all. The LongBench gains are under 0.7 points on averages near 15, with no confidence intervals. About two-thirds of DeltaNet’s gain comes from MultiNews alone (11.10 to 17.07), where Attention Read and Top-16 also jump, which looks more like a shift in generation behavior than recovered associations. On DeltaNet RULER, Full RECUE is within 0.6 points of Attention Read on five of the six averages, and τ was chosen from sweeps on RULER and LongBench themselves, with no latency or memory numbers despite efficiency being the motivation.

Questions for authors
  1. Can you report a residual-valued attention read (softmax over the same cosine cue, residuals r_i as values, same margin gate and 25% cap)? Can you also report a variant that commits the correction to the state instead of rolling it back?

  2. In Table 8, DeltaNet’s LAMBADA perplexity drops from 12.27 to 8.61 and accuracy rises by 6.6 points, which is a large effect for a read-time patch. LAMBADA passages are short relative to the 512-token refresh interval, so how is the cache populated in this setting? Can you confirm that in teacher-forced scoring no continuation tokens reach the cache or the residuals before they are scored?

  3. Table 4 reports RCR of 0.2568 and 0.2585 for SRE, which is above the 25% cap on the correction norm. Is RCR measured before the cap, and how often does the cap bind for each method? Do the matched-RCR comparisons in Table 5 hold for the corrections that are actually applied?

  4. Eq. 11 defines μ and σ² over the prefill keys, but Algorithm 1 computes them over the retained cache plus the new chunk. Once the cache is full, half of that set was kept for being atypical. Does this make the notion of an underrepresented direction drift over long inputs, and how often is the needle actually in the cache at 8K and 16K under each policy?

  5. What are the per-token latency and memory overhead relative to the base model, given that each head stores cue keys, the ridge map P, and residuals R for up to 2K entries? Appendix D treats a 2K cache and 2K of extra window as the same budget, which seems to undercount the cache. Does RECUE still help on a gated backbone such as Gated DeltaNet, where much of the loss is deliberate forgetting rather than interference?

7 comments
AuthorsOpenReview ↗

Responses to Reviewer VcDA (1/4)

We thank Reviewer VcDA for the detailed and constructive feedback. We appreciate that the reviewer finds ReCue training-free and effective on official checkpoints, a clean extension to nonlinear fast weights, and supported by informative ablations.

For weaknesses and questions, we address each point below.


W1 & W2 & Q1: Temporary correction vs. direct read

We thank the reviewer for the insightful question on how ReCue relates to direct retrieval and for suggesting clarifying experiments.

We agree that ReCue’s output can be written as obase+η∑iρi(q)βi(q)rio_{\text{base}} + \eta\sum_i \rho_i(q)\beta_i(q) r_i. However, any reinstatement can be interpreted as a read, since the current output decomposes as

o=q⊤(S+ΔS)=obase+q⊤ΔS,o = q^\top(S + \Delta S) = o_{\text{base}} + q^\top \Delta S,

which, like direct retrieval, adds a cache-derived term to obaseo_{\text{base}}. Therefore, what matters is whether the read is consistent with the geometry of the memory. In the test-time regression view [1], a fast weight memory is a linear regressor updated by gradient descent on a regression objective, so a consistent correction reads through a kernel linear in the query feature and accounts for correlations among keys through the Gram matrix, as ReCue’s ridge solve does. Softmax attention is instead a locally constant (Nadaraya–Watson) regressor whose nonnegative, normalized weights yield a convex combination of cached targets.

The residual-valued attention read the reviewer suggested is therefore the right experiment: it uses the same residual targets, cue, margin gate, 25% cap, and cache as ReCue, and changes only how the cache-derived term is computed.

  • RULER NIAH averages and LongBench average.
BackboneMethodS-NIAH 4KS-NIAH 8KS-NIAH 16KMulti 4KMulti 8KMulti 16KLongBench
LaCTAttention Read89.0784.8050.5355.0043.4332.3215.20
Residual Attention88.8084.8751.2754.7043.6332.3215.14
Full ReCue90.2094.1358.4756.8248.9736.1515.71
DeltaNetAttention Read86.0747.2034.5323.6520.033.7714.65
Residual Attention85.5348.6034.4724.6721.454.3314.34
Full ReCue86.3347.3334.7324.9520.154.2814.86
  • LongBench per task.
TaskLaCT Residual Attn.LaCT Full ReCueDeltaNet Residual Attn.DeltaNet Full ReCue
NarrativeQA2.922.813.323.05
Qasper5.175.514.084.04
MultiFieldQA13.7914.5713.6614.84
HotpotQA6.056.634.155.05
2WikiMQA6.607.297.948.08
MuSiQue3.953.702.252.34
GovReport12.3714.067.296.48
QMSum14.9014.6811.0011.81
MultiNews5.406.3417.1617.07
TREC13.5013.5023.0025.00
TriviaQA29.7532.7126.7527.45
SAMSum28.6029.507.738.63
LCC32.6232.4738.5540.04
RepoBench-P36.3736.1633.9434.22
Avg.15.1415.7114.3414.86

On LaCT, Full ReCue outperforms both attention reads on all NIAH and LongBench averages, with gains of 9.3 and 7.2 points over the residual attention read on the 8K and 16K single-needle averages. On DeltaNet, the reads perform comparably on NIAH, while Full ReCue achieves a higher LongBench average (14.86 vs. 14.34), outperforming the residual attention read on 10 of 14 tasks.

MultiNews. On DeltaNet, Full ReCue and the residual attention read score nearly identically on MultiNews (17.07 vs. 17.16), so the gap between them does not come from MultiNews. Excluding MultiNews, Full ReCue still outperforms the residual attention read by a similar margin on both backbones (LaCT: 16.43 vs. 15.89; DeltaNet: 14.69 vs. 14.13).

Choice of τ. The gains do not depend on tuning τ: for τ ∈ {0.05, 0.1, 0.2}, Full ReCue achieves LongBench averages of 15.53–15.71 on LaCT and 14.80–14.88 on DeltaNet, above Attention Read (15.20 and 14.65), and improves or matches RULER over the base model at every setting (Tables 6–7).


[1] Wang et al., 2025. Test-Time Regression: A Unifying Framework for Designing Sequence Models with Associative Memory.

AuthorsOpenReview ↗

Responses to Reviewer VcDA (2/4)

W1 & W2 & Q1: Temporary correction vs. direct read (Continued)

Commit variant. For a single read, committing and rolling back give identical outputs; they differ only in subsequent reads, so this variant directly tests the rehearsal interference of Eq. 7. This variant is well-defined because ReCue’s correction solves the memory’s own regression objective, whereas an attention read has no natural state counterpart to commit. We commit the applied correction at each decoding step and update the cached residuals accordingly, with all other settings unchanged (8K, 500 samples per task).

  • Rollback vs. commit.
ModelTaskBaselineFull rollbackFull commitTop-16 rollbackTop-16 commit
LaCTS-NIAH-334.0085.2046.2084.6041.00
LaCTMQ-NIAH29.6537.6033.8038.2540.70
DeltaNetS-NIAH-38.6013.4014.2014.001.80
DeltaNetMQ-NIAH12.4519.7020.9021.9517.30

Committing is unstable. On LaCT S-NIAH-3, whose UUID answers require the most decoding steps, it reduces accuracy by 39.0 (Full) and 43.6 (Top-16) points, and on DeltaNet, Top-16 with commit falls below the baseline on S-NIAH-3 (1.80 vs. 8.60). Where committing helps, the gains are at most 2.45 points. Rollback prevents corrections from accumulating across reads and is never far below commit, so we retain it.

Efficiency. We measure time to first token (TTFT), per-token decoding latency (average over 127 steps), and peak allocated memory on an A100-SXM4-80GB GPU in BF16 with batch size 1.

Efficiency at 8K and 16K context.

ModelMethodTTFT 8K (ms)TTFT 16K (ms)Decode 8K (ms/token)Decode 16K (ms/token)Peak 8K (GiB)Peak 16K (GiB)
LaCTBase309.79590.8345.7746.472.8623.186
Attention Read379.06722.3754.4055.673.2143.623
Top-16 ReCue403.18750.0666.8368.113.3603.764
Full ReCue412.52744.5956.4856.843.5173.904
DeltaNetBase140.90262.9834.8334.553.0233.476
Attention Read214.89391.2442.7441.743.1173.570
Top-16 ReCue218.94403.7652.9850.663.1623.615
Full ReCue257.58442.0042.5541.593.2093.662
  • Decoding: Relative to the base model, Full ReCue increases per-token latency by 23.4% and 22.3% on LaCT and by 22.2% and 20.4% on DeltaNet at 8K and 16K. Like Attention Read, its per-token correction costs O(Bd)O(Bd) per head, and it decodes at a speed comparable to Attention Read. Top-16 ReCue decodes more slowly, as it solves a small linear system at each step.
  • Memory: Since the cache size is fixed, the additional memory for the cached keys, values, ridge map PP, and residuals RR does not grow substantially with context length (Full ReCue: 0.66 → 0.72 GiB on LaCT, 0.19 GiB on DeltaNet).
  • TTFT: PP and RR are prepared at each cache refresh rather than at each decoding step, so their cost appears in TTFT instead of decoding latency.

Q2: LAMBADA Verification

We thank the reviewer for carefully checking the LAMBADA results for possible leakage. We confirm that no continuation token enters the cache or its residuals before it is scored.

Cache initialization. LaCT applies its fast weight before updating it at the end of each chunk, so tokens in an incomplete final chunk have never been written into it and are not cached. DeltaNet updates its fast weight at every token, so we also cache the remaining partial chunk at the end of prefill, independent of the 512-token refresh interval; we will make this explicit in Appendix E. Since all 5,153 LAMBADA contexts contain 59–228 tokens, each DeltaNet head retains every context association within its 512-entry budget, with residuals computed from the state after the context.

Causal scoring. Each target token is scored before it is supplied as input; previously scored target tokens enter the recurrent state, as required by teacher forcing. We verified this with an instrumented audit:

  • Provenance: on all examples, cached positions in every layer and head come only from the context, and cache and residual payloads remain unchanged during target scoring.
  • Future invariance: changing or appending future target tokens leaves all preceding logits exactly unchanged (334 prefix tests across 128 examples).
  • Zero-correction control: setting the correction to zero while keeping the full cache and ridge pipeline reproduces native logits exactly on all examples.

Why the gain is large. Since the cache holds the entire context, ReCue can reinstate any context association. Accuracy improves by 8.16 points when the target word appears in the context, versus −0.53 points when it does not.

AuthorsOpenReview ↗

Responses to Reviewer VcDA (3/4)

Q3: RCR and the correction cap

We thank the reviewer for carefully checking the RCR values against the correction cap. RCR in Tables 4 and 5 was measured before the cap, which is why SRE exceeds 0.25. Since clipping changes only the reads where the cap binds, the gap between pre-cap and post-cap RCR comes entirely from these reads. RCR and cap-binding rates below are computed over decode steps × layers × heads.

Cap-binding rates in Table 4. The cap binds about twice as often under SRE as under the other policies, consistent with its larger pre-cap RCR. Mahalanobis binds about as often as Random, yet achieves much higher accuracy.

  • Cap-binding rates of the cache-selection policies (Table 4 setting).
Cache policyStandard (500)UUID-protected (445)
Random9.23%9.45%
SRE19.78%19.14%
Mahalanobis9.63%9.92%

Matched RCR on applied corrections. We re-ran the Table 5 comparison with RCR matched on the corrections actually applied (LaCT, S-NIAH-3 8K, 2,048 cache entries, 25% cap). Fixed strengths were calibrated on a separate set of 100 samples and evaluated on 500 samples without re-tuning; post-cap RCR matches within 0.72% relative error.

  • Matched-RCR comparison on applied corrections.
MethodFixed ηAccuracyRCR (pre-cap)RCR (post-cap)Cap-binding rate
Ridge Rehearsal0.00228036.400.084440.061865.48%
Attention Read0.00259962.000.085240.061535.71%
Full ReCue0.00614074.200.084510.062214.75%
Full ReCue + margin gate–85.200.172600.0617710.46%

With applied corrections matched, the ranking is unchanged: Full ReCue with a fixed strength outperforms Attention Read by 12.2 points, and the margin gate adds another 11.0 points.


Q4: Cache statistics and needle retention

We thank the reviewer for this careful observation. Algorithm 1 computes μ\mu and σ2\sigma^2 over the candidate pool (retained cache and new chunk) rather than over all prefill keys as in Eq. 11. To measure the resulting drift, we compare the final candidate-pool statistics with those of the full prefill on LaCT-760M (S-NIAH-3, 500 samples per length, 2,048 cache entries per head).

  • Drift of the candidate-pool statistics.
ContextStd. mean difference ↓Variance ratioTop-2048 overlap ↑
8K0.07901.063192.18%
16K0.15541.154589.95%

The drift grows with context length, but the selection is largely preserved (overlap of about 90%).

We also measure needle retention on out-of-window samples (370 at 8K, 445 at 16K). Token retention is the fraction of answer UUID tokens retained, averaged over all 24 layers × 4 heads; full-UUID retention is the fraction of samples in which all UUID tokens remain together in at least one head’s cache.

  • Needle retention and S-NIAH-3 accuracy.
ContextPolicyToken retentionFull-UUID retentionAccuracy
8KRandom33.22%0.00%15.20
SRE41.49%99.19%19.00
Mahalanobis (Ours)87.23%100.00%85.20
Mahalanobis (all-past statistics)88.54%100.00%84.80
16KRandom14.27%0.00%5.60
SRE20.28%27.19%5.40
Mahalanobis (Ours)69.75%94.16%19.20
Mahalanobis (all-past statistics)71.54%96.85%22.20

Mahalanobis retains far more of the needle than Random and SRE and achieves the highest accuracy. Following the reviewer’s observation, accumulating exact statistics over all past keys, as defined in Eq. 11, requires only O(dk)O(d_k) additional memory per head and no additional computation. It improves 16K accuracy from 19.20 to 22.20, consistent with the larger drift at longer contexts.

Reviewer_XJ6uOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReCue, a training-free method for improving associative recall in fast-weight models. It selects historical KV pairs using a Mahalanobis-based criterion and performs query-conditioned ridge regression to temporarily correct the recurrent memory readout. Experiments on LaCT and DeltaNet demonstrate improvements on RULER and LongBench.

Critical strengths

S1. The method is conceptually simple and requires no additional training. Query-conditioned residual correction provides an interesting alternative to directly attending to cached values.

S2. The retrieval improvements are substantial. On LaCT, the 8K single-needle average increases from 70.27 to 94.13, outperforming direct Attention Read (84.80). The authors also provide useful ablations for cache selection and query cueing.

Critical weaknesses

W1. Missing computational efficiency evaluation. The method introduces additional cached associations, regression-related matrices, and query-time computations, but does not report actual decoding latency, throughput, or peak memory. Without these measurements, its practical advantage over simpler attention-based retrieval remains unclear.

W2. Limited improvements on realistic tasks. On LongBench, Full ReCue only marginally outperforms Attention Read (15.71 vs. 15.20 on LaCT; 14.86 vs. 14.65 on DeltaNet). It is unclear whether these modest gains justify the additional computational complexity.

Questions for authors

Can the authors report inference latency, memory consumption, and throughput, including the additional matrices used by ridge rehearsal, and compare methods under equal total memory budgets?

Are the LongBench improvements over Attention Read statistically robust? How sensitive are the results to the approximate cache statistics and residual-refresh procedure used in the implementation?

3 comments
AuthorsOpenReview ↗

Responses to Reviewer XJ6u

We thank Reviewer XJ6u for the constructive feedback. We appreciate that the reviewer considers ReCue simple and training-free, an interesting alternative to directly attending to cached values.

For weaknesses and questions, we address each point below.


W1 & Q1: Latency, throughput, and memory cost

We thank the reviewer for emphasizing the importance of measuring actual efficiency and comparing methods under equal memory budgets. We measure time to first token (TTFT), per-token decoding latency (average over 127 steps), and peak allocated memory on an A100-SXM4-80GB GPU in BF16 with batch size 1, where throughput is the inverse of decoding latency.

Efficiency at 8K and 16K context.

ModelMethodTTFT 8K (ms)TTFT 16K (ms)Decode 8K (ms/token)Decode 16K (ms/token)Peak 8K (GiB)Peak 16K (GiB)
LaCTBase309.79590.8345.7746.472.8623.186
Attention Read379.06722.3754.4055.673.2143.623
Top-16 ReCue403.18750.0666.8368.113.3603.764
Full ReCue412.52744.5956.4856.843.5173.904
DeltaNetBase140.90262.9834.8334.553.0233.476
Attention Read214.89391.2442.7441.743.1173.570
Top-16 ReCue218.94403.7652.9850.663.1623.615
Full ReCue257.58442.0042.5541.593.2093.662
  • Decoding: Relative to the base model, Full ReCue increases per-token latency by 23.4% and 22.3% on LaCT and by 22.2% and 20.4% on DeltaNet at 8K and 16K. Like Attention Read, its per-token correction costs O(Bd)O(Bd) per head, and it decodes at a speed comparable to Attention Read. Top-16 ReCue decodes more slowly, as it solves a small linear system at each step.
  • Memory: Since the cache size is fixed, the additional memory for the cached keys, values, ridge map PP, and residuals RR does not grow substantially with context length.
  • TTFT: PP and RR are prepared at each cache refresh rather than at each decoding step, so their cost appears in TTFT instead of decoding latency.

Equal memory budgets. We reduce the ReCue caches so that their peak memory, including PP, RR, and the cached features, is within 2% of Attention Read’s, and evaluate S-NIAH-3 at 8K.

S-NIAH-3 8K accuracy at a matched memory budget.

ModelMethodCache entries per headAccuracyPeak memory (GiB)
LaCTAttention Read2,04863.03.214
Top-16 ReCue1,52080.63.242
Full ReCue1,20073.83.267
DeltaNetAttention Read51212.83.117
Top-16 ReCue51214.03.162
Full ReCue25612.63.114

On LaCT, both ReCue variants outperform Attention Read at a matched memory budget. On DeltaNet, the methods perform comparably.


W2 & Q2: LongBench gains, statistical robustness, and sensitivity

We thank the reviewer for raising the question of whether the LongBench gains are statistically robust and how sensitive they are to our implementation choices.

LongBench gains. Full ReCue improves the LongBench average over Attention Read by 0.51 points on LaCT and 0.21 points on DeltaNet, outperforming it on 11 of 14 tasks and 10 of 14 tasks, respectively. A paired bootstrap over all 3,350 examples gives the following results.

Paired bootstrap on LongBench.

ModelAttention ReadFull ReCueDifference95% CI
LaCT15.2015.71+0.51[0.22, 0.80]
DeltaNet14.6514.86+0.21[−0.19, 0.62]

The improvement is statistically significant on LaCT; on DeltaNet, the difference is positive but not significant.

Sensitivity to cache statistics. The implementation computes cache statistics over the retained cache and the new chunk rather than over all past keys. We compare it with exact statistics accumulated over all past keys, which costs only O(dk)O(d_k) additional memory per head.

Effect of cache statistics on LaCT S-NIAH-3.

ContextSelection overlapAccuracy (retained cache + new chunk)Accuracy (all past keys)
8K92.18%85.2084.80
16K89.95%19.2022.20

The selected caches largely overlap. Exact statistics leave 8K accuracy unchanged within two samples and improve 16K accuracy by 3.0 points.

Sensitivity to residual refresh. For LaCT, cache refreshes are aligned with fast weight updates at chunk boundaries, so residuals are always computed from the current state. For DeltaNet, which updates its state at every token, residuals are refreshed every 512 tokens. Recomputing them at every token has little effect.

Effect of the residual refresh interval on DeltaNet (8K).

TaskEvery 512 tokensEvery tokenDifference
S-NIAH-313.4013.80+0.40
MQ-NIAH19.7019.700.00
AuthorsOpenReview ↗

Follow-up to Reviewer XJ6u

Dear Reviewer XJ6u,

We are grateful for the thought and effort you have put into our submission!

We have uploaded a revised manuscript that incorporates your feedback: the efficiency analysis of latency and memory overhead (Section 5.4 and Appendix F).

Given the approaching deadline, we would appreciate any further feedback or remaining questions you may have.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer XJ6u,

As the discussion period closes in a few hours, we wanted to gently follow up on whether our responses and revisions have addressed your concerns. A brief confirmation would be greatly appreciated.

Thank you again for your time!

Reviewer_JVESOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces RECUE, a training-free inference-time method for mitigating associative interference in fast-weight models, including linear attention, DeltaNet, and TTT. During prefill, RECUE maintains a bounded per-head cache of key–value associations, selected using a regularized, centered diagonal Mahalanobis score motivated by an approximation to ridge leverage. During decoding, query-relevant associations are retrieved via cosine similarity, and their reconstruction residuals are jointly rehearsed through ridge regression, using either a top-K subset or the full cache with exponentially weighted residual targets. The resulting correction is applied only to the current read, scaled by a gate based on the top-two cosine-similarity margin, and then rolled back without modifying the underlying fast-weight state. Experiments on pretrained LaCT (760M) and DeltaNet (1.3B) show substantial improvements on RULER needle-in-a-haystack tasks, with LaCT's average single-needle accuracy at 8K increasing from 70.27 to 94.13. The method also achieves modest gains on LongBench while largely preserving general language modeling performance.

Critical strengths
  • S1. Well-motivated problem formulation and a simple, training-free mechanism. The paper provides a clear motivation for addressing associative interference in compressed fast-weight memories and formalizes the non-local effects of naive rehearsal (Eq. 7). RECUE addresses this issue through query-conditioned rehearsal, jointly optimizing the correction via ridge regression and applying it temporarily to the current read. The approach requires no additional training and can be applied to both linear (DeltaNet) and nonlinear (LaCT) fast-weight memories by modifying only the linear readout. The related work is well organized, positioning RECUE relative to hybrid explicit-memory methods, regression-based memories, and approaches to scaling recurrent memory.
  • S2. Consistent empirical improvements and informative ablations. RECUE improves associative recall on RULER across both backbones, with particularly notable gains at longer context lengths and on multi-association tasks. It also achieves modest improvements on LongBench while largely preserving performance on general language modeling benchmarks.
Critical weaknesses
  • W1. Insufficient justification and evaluation of the cache-selection criterion. Cache composition is critical, as Random and SRE selection even underperform the base model in Table 4 (5.6/5.4 vs. 8.2). However, the rationale for prioritizing underrepresented keys is unclear, given that Eq. 5 suggests stronger interference in dense key directions, while ridge leverage (App. A) measures influence rather than forgetting. Furthermore, the ablation is limited to a single NIAH setting, potentially favoring novelty-based selection. The authors should clarify this motivation, evaluate settings with repeated or updated associations (e.g., RULER VT, CWE/FWE), and discuss alternative selection strategies such as recency, full leverage, or redundancy-aware selection.
  • W2. Limited comparison with hybrid architectures. RECUE complements fast-weight memory with explicit KV memory and is primarily positioned against sparse-cache methods (e.g., LoLA, HOLA). However, related hybrid architectures combine recurrent memory with softmax attention across layers (e.g., Jamba [Lieber et al., 2024], Samba [Ren et al., 2025], Qwen3-Next) or within layers (e.g., Hymba [Dong et al., 2025], hybrid quadratic-linear transformers [Irie et al., 2025]). Their relationship to RECUE is not discussed, and, beyond the sliding-window attention already present in LaCT, the evaluation lacks hybrid baselines with full attention, cache-size sensitivity analysis, and measurements of memory or computational overhead. Discussing these trade-offs and whether RECUE is intended as an alternative or a complement to hybrid architectures would clarify its practical positioning.
Questions for authors
  • Q1. Hyperparameter sensitivity. As a training-free method, RECUE may be sensitive to its hyperparameters, yet only the cue temperature is ablated. How sensitive is performance to the correction cap, regularization strengths, and cache refresh interval? How frequently is clipping activated, and why are different cache sizes used for LaCT and DeltaNet (2048 vs. 512)? Can the same configuration generalize across tasks and backbones?
  • Q2. Repeated or updated key–value associations. When a key has multiple or updated values, ridge regression may average conflicting residuals. At the same time, the similarity-margin gate may suppress corrections for equally matching keys. How does RECUE handle such cases, particularly when the most recent value should take precedence?
5 comments
AuthorsOpenReview ↗

Responses to Reviewer JVES (1/3)

We thank Reviewer JVES for the thoughtful and constructive review. We are glad that the reviewer found the formulation of rehearsal interference well motivated and the mechanism simple and applicable to both linear and nonlinear fast weight memories without training.

For weaknesses and questions, we address each point below.


W1: Justification of Cache Selection

We thank the reviewer for the careful reading of the cache-selection design. We clarify the motivation and provide additional experiments below.

Why underrepresented directions. Eq. 5 indeed implies larger readout interference along dense key directions. However, such associations are poor candidates for rehearsal. Consider a query qq targeting a cached association. By Eq. 7, rehearsing association tt changes the readout at qq by St⊤q−St−1⊤q=βt(kt⊤q)(vt−v^t)S_t^\top q - S_{t-1}^\top q = \beta_t (k_t^\top q)(v_t - \hat v_t), so the correction at qq mixes the residuals of all cached associations whose keys are similar to qq. When the target lies along a dense direction, many cached keys are similar to qq, and the correction becomes a mixture of residuals rather than restoring the target. When the target lies along an underrepresented direction, the correction at qq is dominated by the target’s own residual, so rehearsal can reinstate it locally.

Leverage as reinstatement locality. Ridge leverage measures how much a data point’s fitted value is determined by its own target rather than by the other points, and thus explains the locality of reinstatement. For ridge rehearsal over the cache, the correction at the cached keys is HRHR, with H=KK⊤(KK⊤+λI)−1H = KK^\top(KK^\top+\lambda I)^{-1}. The diagonal HiiH_{ii} is the leverage of association ii, i.e., how much of its own residual is restored, while the off-diagonal entries are the crosstalk from other cached residuals. Our selection score efficiently approximates this leverage. As shown below, Mahalanobis-selected caches achieve higher mean self-fit HiiH_{ii} and lower crosstalk ∑j≠i∣Hij∣\sum_{j \ne i} |H_{ij}| than both random and SRE caches on LaCT (S-NIAH-3 16K).

ComparisonSelf-fit ↑Crosstalk ↓
Mahalanobis (Ours) vs. Random+11.8%−13.6%
Mahalanobis (Ours) vs. SRE+16.5%−22.1%

Selecting by forgetting is not sufficient. Self-Recall Error (SRE) directly selects the most forgotten associations ∥vi−v^i∥2\|v_i - \hat v_i\|^2, yet in Table 4 it underperforms even the base model while producing the largest correction ratio. Thus, the most degraded associations are not necessarily those that rehearsal can restore locally, consistent with the larger locality gap against SRE above.

Is the advantage specific to needle novelty? The UUID-protected setting in Table 4 already partly addresses this concern: when the target association is forced into the cache for all policies, Mahalanobis still achieves the highest recall (29.21 vs. 13.03 / 9.44). Its advantage thus stems from the composition of the remaining cache, not merely from retaining an outlier needle.

Additional experiments. We extend the cache-selection ablation (LaCT, Full ReCue) to tasks with repeated or aggregated associations, and add recency and full ridge leverage (Full Mahalanobis) as alternative policies.

PolicyS-NIAH-3 16KVT 16KCWE 16KFWE 16K
Base8.201.007.7634.80
Random5.601.727.6238.47
Recency10.402.526.2438.33
SRE5.400.528.2840.73
Full Mahalanobis25.600.766.9238.98
Mahalanobis (Ours)19.200.928.7039.93

On retrieval (S-NIAH-3), leverage-based policies substantially outperform all others, and full leverage further improves recall, supporting the leverage interpretation above. On aggregation tasks (CWE, FWE), where frequent associations matter, our criterion remains comparable to or better than the alternatives, indicating that prioritizing underrepresented directions does not harm such settings. On VT, the base model is near floor and all policies remain close to it.

AuthorsOpenReview ↗

Responses to Reviewer JVES (2/3)

W2: Relation to Hybrid Architectures

We thank the reviewer for raising the connection to hybrid architectures, which helps clarify the practical positioning of ReCue.

Positioning. ReCue targets the bounded-memory regime: given additional memory, it reinforces the fast weight memory rather than extending an attention path. In App. D, pretraining LaCT with a 4K instead of a 2K sliding window eliminates out-of-window recall, as the larger window reduces reliance on the fast weight memory. ReCue on the 2K model instead substantially improves out-of-window recall.

Full-attention hybrids. Since their global attention layers already retain every key–value pair, ReCue leaves performance nearly unchanged.

ModelS-NIAH 4KS-NIAH 8KS-NIAH 16KLongBench
Hymba-1.5B92.8789.8021.9323.11
+ Full ReCue92.8089.5322.0022.92
Qwen3.5-0.8B100.00100.00100.0035.58
+ Full ReCue100.00100.00100.0035.46

ReCue is therefore an alternative for bounded-memory models rather than a complement to full-attention hybrids: under a fixed memory budget, it makes better use of the memory than a larger attention window (App. D).

Cache-size sensitivity. We vary the per-head cache size BB.

ModelMetricBase256512102420484096
LaCTS-NIAH 8K70.2760.9366.4085.8094.1392.80
Multi-NIAH 8K35.3723.0734.5545.4348.9747.53
S-NIAH 16K47.6742.3345.5349.1358.4770.33
Multi-NIAH 16K25.3714.0316.4223.9536.1540.63
LongBench15.0315.1315.2915.3315.7115.58
DeltaNetS-NIAH 8K42.8047.2047.3347.2046.4045.40
Multi-NIAH 8K12.3218.6320.1519.9518.4216.40
S-NIAH 16K34.2734.4734.7334.4734.6034.40
Multi-NIAH 16K2.783.924.283.873.633.20
LongBench14.2114.8614.8614.8514.6714.59

On LaCT, accuracy increases with BB, saturating at 2048 at 8K, while 16K benefits from larger caches; too small a cache falls below the base model, as poorly composed caches yield harmful corrections. DeltaNet saturates by BB = 512. Since the ridge correction has at most dkd_k degrees of freedom (tr(H)≤dk\mathrm{tr}(H) \le d_k), the useful cache size scales with the key dimension: 2048 for LaCT (dkd_k = 384) and 512 for DeltaNet (dkd_k = 128), which, given DeltaNet’s 4× more heads, cache the same number of associations per layer.

Memory and computational overhead. We measure time to first token (TTFT), per-token decoding latency (average over 127 steps), and peak allocated memory on an A100-SXM4-80GB GPU in BF16 with batch size 1.

Efficiency at 8K and 16K context.

ModelMethodTTFT 8K (ms)TTFT 16K (ms)Decode 8K (ms/token)Decode 16K (ms/token)Peak 8K (GiB)Peak 16K (GiB)
LaCTBase309.79590.8345.7746.472.8623.186
Attention Read379.06722.3754.4055.673.2143.623
Top-16 ReCue403.18750.0666.8368.113.3603.764
Full ReCue412.52744.5956.4856.843.5173.904
DeltaNetBase140.90262.9834.8334.553.0233.476
Attention Read214.89391.2442.7441.743.1173.570
Top-16 ReCue218.94403.7652.9850.663.1623.615
Full ReCue257.58442.0042.5541.593.2093.662
  • Decoding: Relative to the base model, Full ReCue increases per-token latency by 23.4% and 22.3% on LaCT and by 22.2% and 20.4% on DeltaNet at 8K and 16K. Like Attention Read, its per-token correction costs O(Bd)O(Bd) per head, and it decodes at a comparable speed (within −0.4% to +3.8%). Top-16 ReCue decodes more slowly, as it solves a small linear system at each step.
  • Memory: Since the cache size is fixed, the additional memory for the cached keys, values, ridge map PP, and residuals RR does not grow substantially with context length (Full ReCue: 0.66 → 0.72 GiB on LaCT, 0.19 GiB on DeltaNet).
  • TTFT: PP and RR are prepared at each cache refresh rather than at each decoding step, so their cost appears in TTFT instead of decoding latency.
AuthorsOpenReview ↗

Responses to Reviewer JVES (3/3)

Q1: Hyperparameter sensitivity

We thank the reviewer for raising the question of hyperparameter sensitivity, which is particularly important for a training-free method. Following the settings of Table 5, we vary each hyperparameter on LaCT (S-NIAH-3 8K) while fixing the others at their defaults. The base model achieves 34.00, and the default setting differs slightly across runs (85.20–85.80).

Correction cap α\alpha

α\alpha12.5%25%50%none
Accuracy80.8085.8085.2080.80
Clipping19.22%10.49%5.58%–

Ridge regularization λrr\lambda_{\text{rr}}

λrr\lambda_{\text{rr}}0.10.313
Accuracy85.6085.4085.4084.60
Clipping18.05%14.38%10.47%7.18%

Cache-selection regularization λsel\lambda_{\text{sel}}

λsel\lambda_{\text{sel}}0.00010.0010.010.1
Accuracy85.6085.6084.8083.20
Clipping10.50%10.53%10.56%10.49%

Refresh interval

Interval20484096
Accuracy85.8085.60
Clipping10.49%10.49%

Sensitivity. ReCue is robust to these hyperparameters: all settings substantially outperform the base model, and accuracy remains stable across a wide range of each.

Generalization. We use a single configuration per backbone across all tasks (RULER, LongBench, and lm-evaluation-harness). Only the cache size needs to be set per backbone, in proportion to its key dimension (roughly 4–5× dkd_k; 2048 for LaCT with dkd_k = 384 and 512 for DeltaNet with dkd_k = 128).


Q2: Repeated or updated associations

We thank the reviewer for raising the case of updated associations. To examine it, we construct MV-NIAH-Latest, where each new value updates the previous one and the model is asked for the most recent value. The table below reports overall accuracy and accuracy by the distance of the most recent value from the end of the context (≤2K corresponds to LaCT’s sliding window).

Model8K Avg.8K ≤2K8K >2K16K Avg.16K ≤2K16K >2K
DeltaNet35.6048.630.000.400.930.00
+ Full ReCue50.4063.6614.182.606.050.00
LaCT62.0080.0512.6950.6086.0523.86
+ Full ReCue32.6037.9817.9155.6071.1643.86
+ C + W53.4066.9416.4257.8081.8639.65
+ C + W + T67.4080.6031.3467.0088.8450.53

Analysis. On DeltaNet, Full ReCue improves accuracy at both distances. On LaCT, it improves accuracy when the most recent value lies beyond 2K, but degrades it when the value lies within 2K, lowering the 8K average. We trace this to two causes.

  1. Unseen recent tokens. Under LaCT’s apply-then-update rule, tokens in the current, incomplete chunk are visible only to the sliding window, not to the fast weights or the cache. If the most recent value lies there, ReCue reinstates an outdated value that conflicts with local attention.
  2. Indistinguishable keys. Without positional encoding, keys of the same association are nearly identical, so the query cue cannot prefer the latest value even when both are cached.

DeltaNet updates its state at every token and caches the prefill tail, so the latest value is always written into the state and overwrites previous ones through the delta rule.

Extensions. Each cause suggests a training-free extension.

  • (C) Final-chunk caching. We include the final, incomplete chunk in the cache.
  • (W) Window-aware margin. We add keys in the sliding window as competitors in the margin gate (Eq. 23), η=[c(1)−max⁡(c(2),cW)]01\eta = [c_{(1)} - \max(c_{(2)}, c_W)]_0^1, where cWc_W is the highest cosine similarity between the query and a key in the window.
  • (T) Recency weighting. We weight each cached entry by recency in the query cue, ρi(q)←ρi(q)⋅2−(T−ti)/2048\rho_i(q) \leftarrow \rho_i(q) \cdot 2^{-(T - t_i)/2048}, where tit_i is the entry’s position and TT is the last context position.

C + W partially recovers accuracy within 2K, and adding T fully recovers it while further improving accuracy beyond 2K, so that ReCue exceeds the base model at both lengths.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.