acceptodds
Under review as a conference paper at ICLR 2027

Can first-order memory repair compressed attention?

Abstract

Compressing attention by sharing key and value (KV) heads reduces KV-cache storage, but can hurt predictions. We address this loss of predictive capacity through a repair mechanism called **First-order Memory** (FoM). Specifically, FoM adds a compact recurrent state that summarizes causal features with storage independent of context length. We then study how well FoM repairs compressed attention, and assess whether it can also outperform stateless repair. Across a controlled byte-level study (one UTF-8 byte per token) and studies using Pythia-class models (up to 1B), FoM recovers prediction quality, often even outdoing comparably sized stateless MLP repair. In a held-out 410M MQA study, FoM lowers mean cross-entropy by 0.02235 nats per token over the selected MLP across five new seeds; this strong advantage is, however, not universal, and in some cases a strong MLP nearly matches mean performance. Overall, the experiments are encouraging and suggest that FoM can provide valuable repair, possibly at an additional execution cost.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
language models, model compression, efficient inference, KV cache, multi-query attention, grouped-query attention, first-order memory, recurrent memory, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AiNLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper adds a recurrent concatenation-based residual module to recover prediction quality after reducing KV-cache storage through KV-head sharing.

Critical strengths

The experiments provide preliminary evidence that recurrent EMA may help recover some prediction quality lost through KV-head sharing.

Critical weaknesses
  1. Poor presentation. The paper is hard to follow, particularly the relationship between KV-head sharing, cache compression, and the proposed repair module. The motivation for the architecture and training procedure is not clearly explained.

  2. Limited insight and theoretical support. Adding recurrent state after attention offers limited new insight. The claimed prediction-repair benefit is not rigorously proved: the propositions establish basic state properties, not recovery of prediction quality. The distinction from existing recurrent repair methods is unclear.

  3. Missing component-wise ablations and limited generalization. The paper does not provide a controlled module-by-module ablation of the repair architecture. The paper does not convincingly show that the gains generalize beyond data similar to the distillation dataset.

  4. Unconvincing practical benefit. The repair adds computation, parameters, and a persistent state that must be stored. The quality gains over MLP repair are small, while Table 8 reports lower throughput than the MLP at the same tested batch size under an 8 GiB allocation limit.

Questions for authors
  1. What insights motivate the chosen state update, repair width, and layer placement, what evidence justifies these choices, and which ablations isolate their individual effects?

  2. Do FoM's gains generalize to data substantially different from the distillation dataset without further adaptation?

2 comments
Reviewer_AiNLOpenReview ↗

Response

I maintain my original evaluation.

AuthorsOpenReview ↗

Clarification of our empirical contribution

Dear Reviewer AiNL,

We thank you for your review and the time invested in checking our work. ICLR’s reviewer guidelines explicitly recognize empirical findings as a contribution and don’t necessarily require state-of-the-art results. We see our paper as a contribution of this kind. We test whether recurrent memory can recover prediction quality after KV-head sharing, and we report where it helps and where it doesn’t. We think these findings are useful even though they don’t establish a general recovery guarantee or a consistent practical advantage.

The cache reduction comes from KV-head sharing. First-order Memory (FoM) keeps that layout and adds a repair state whose size doesn’t grow with context length. We agree that the propositions don’t prove prediction recovery; that claim rests on the experiments. In the held-out 410M MQA study, FoM lowers mean cross-entropy by 0.02235 nats per token relative to the selected MLP, and it beats the MLP in all five seeds (Table 4).

We used the EMA update because it gives a simple causal summary with bounded state. The width and placement were chosen empirically. Table 6 includes a tuned MLP and a control trained with state reset. Stateful FoM beats the reset-trained control on mean cross-entropy across context lengths in all five seeds, though the tuned MLP nearly closes the gap in mean performance. These comparisons provide evidence about state carry, but we agree they don’t isolate every architectural choice or show that our width and placement are the best ones.

Without further adaptation, the generalization result is negative. On PG19, the frozen FoM model does worse than the MLP. After adaptation on PG19, the ordering reverses (Table 5). So we don’t claim that the advantage transfers reliably across distributions, and we see this as a limitation for future work to address.

We also agree that Table 8 shows lower throughput than the MLP at the same batch size. The larger batch size relative to GQA-4 comes from head sharing. What FoM offers is recovered prediction quality in some settings while keeping the compressed KV layout. Whether that’s worth the extra execution cost is still an open question.

We’d ask that the positive results and these limitations be weighed together when assessing our paper’s empirical contribution.

Reviewer_KwYvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether a fixed-size recurrent state can recover some of the predictive quality lost when multi-head attention is compressed into MQA or GQA. The proposed First-order Memory (FoM) adds a correction to the attention output using features derived from the normalized attention input and the compressed attention output. Its main variant maintains an exponential moving average with a fixed decay, while the byte-level study uses a GRU. The additional state does not grow with context length and does not require restoring the per-token KV records removed by head sharing.

The authors compare FoM with unrepaired compressed models and approximately parameter-matched MLP repairs in a controlled byte-level model and Pythia models ranging from 160M to 1B parameters. They also examine state resets, transfer to PG19, and KV-quantization simulations. In the 410M MQA study with five new adaptation seeds, FoM reduces mean cross-entropy by 0.02235 nats/token relative to the selected GELU MLP. However, in the more extensively tuned 410M KV2 comparison, FoM and a tanh MLP have similar mean performance, with FoM's advantage concentrated at 2048 tokens. Systems measurements show that FoM retains the cache savings from head sharing while adding execution overhead.

Critical strengths

The fixed-size state provides a clear proof of concept for repairing compressed attention, supported by several controlled comparisons.

FoM preserves the compressed KV layout and improves predictive quality over unrepaired compressed models. Parameter-matched MLPs, evaluation-time resets, and reset-trained controls further examine the roles of additional parameters and state carry. The authors also report cases where a stronger MLP nearly matches FoM, transfer reverses its advantage, and deployment becomes slower. These results help identify the method's limitations and scope.

Critical weaknesses
  1. The incremental novelty, rationale for adopting FoM, and intended use cases remain insufficiently established. The main mechanism adds an EMA or GRU state and a readout correction after compressed attention. Recurrent memory and EMA are established components; focusing repair on losses from head sharing is a specific change in problem setting relative to the other compression-repair approaches discussed in the paper. The causality, boundedness, and unrolling-equivalence analyses largely follow from the chosen recurrence and do not guarantee recovery of predictive quality. The contribution therefore rests mainly on demonstrating that recurrent repair provides a stable and meaningful additional benefit over simpler repair methods for head-sharing losses. This has not yet been sufficiently established.

    The 0.02235 nats/token improvement over the selected GELU MLP in Table 4 is consistent across five new adaptation seeds and constitutes positive evidence. However, that MLP was searched over only two learning rates and uses a different activation from FoM. In the more extensively searched KV2 tanh-MLP comparison in Table 6, the mean CE difference is only approximately 0.0039 nats/token. FoM performs worse at 512 and 1024 tokens, benefits mainly at 2048 tokens, and loses its advantage under frozen transfer to PG19. These mean CE differences of 0.02235 and approximately 0.0039 correspond to relative PPL reductions of approximately 2.21% and 0.39%, respectively, when PPL is computed by exponentiating mean CE. These results support improvements in particular configurations but do not yet establish when the recurrent module is worth adopting or whether its advantage over a well-tuned MLP is robust.

  2. The experiments do not establish that improvements in language-modeling metrics translate into practical task gains, and the benefits have not been linked to execution costs under matched conditions. Quality evaluation primarily measures text cross-entropy, without dedicated commonsense reasoning, copying or associative retrieval, or multi-document comprehension tasks. Language-modeling tests up to 2048 tokens do not directly establish better recovery of long-range information. All Pythia adaptation uses 512-token sequences, so advantages at longer evaluation lengths also involve robustness beyond the adaptation length. Validation on practical tasks within the models' supported capabilities and context ranges is needed. Evaluation such as LongBench on backbones that support longer contexts could further clarify practical relevance.

    The existing efficiency measurements make the cost–benefit question particularly important. In the single-seed 410M deployment setting in Table 8, both FoM and the MLP accommodate 140 sequences. FoM achieves cached-decoding throughput of 6,230 tokens/s versus 7,734 for the MLP, and end-to-end generation throughput of 4,610 versus 5,252 tokens/s—approximately 19.4% and 12.2% lower, respectively. The MLP also achieves the batch-capacity improvement over GQA. Thus, this experiment does not show an additional capacity benefit from recurrence, while its execution overhead needs to be justified by corresponding quality gains. However, the quality comparisons and timing measurements do not use matched prompt and generation lengths, leaving the task gains obtained for additional cost at the same operating point unclear. The authors acknowledge these limitations, but a clear and compelling use case for FoM remains to be demonstrated.

Questions for authors
  1. Novelty and use cases: Beyond applying established recurrent mechanisms to repair head-sharing losses, what is FoM's most important new contribution? Under which compression levels, quality requirements, or resource constraints is it preferable to a well-tuned, parameter-matched MLP? Please identify concrete scenarios and supporting evidence from the available results.

  2. Strong baselines and robustness: In the main 410M MQA comparison, would the 0.02235 nats/token advantage remain if the MLP received an activation and learning-rate search comparable to that used in the KV2 study? Could controls with matched adaptation and evaluation lengths further distinguish the benefits of recurrent repair from robustness to a change in length distribution?

  3. Practical task benefits: Could representative commonsense reasoning, copying, or associative retrieval tasks first be evaluated within the current models' supported context ranges to determine whether the small CE/PPL improvements correspond to practical capability gains? For longer-document applications, evaluation on a backbone with suitable context support, using tasks such as those in LongBench, would be a possible extension. The concern is representative capability evidence; changing the backbone or using a particular benchmark is not an independent condition for acceptance.

  4. Joint quality and efficiency evaluation: Could task quality, prefill and decoding latency, throughput, additional parameter and state storage, and total peak GPU memory be reported jointly on the same backbone, hardware, precision, prompt/generation lengths, and batch size? FoM and the well-tuned MLP should use the same KV layout, with GQA serving separately as a quality and capacity reference. Coverage across lengths and batch sizes is needed to assess whether the execution overhead is justified by corresponding quality benefits. If further measurements are unavailable, please clarify the range of use supported by the existing evidence.

2 comments
AuthorsOpenReview ↗

Clarification of our empirical contribution

Dear Reviewer KwYv,

We thank you for your review and for the time invested in checking our work. ICLR’s reviewer guidelines explicitly recognize empirical findings as a contribution and don’t necessarily require state-of-the-art results. We see our paper as a contribution of this kind. It studies repair after KV-head sharing using a recurrent state whose size does not grow with context length. EMA and recurrent memory are established components. The propositions describe properties of the repair state. The evidence for improved prediction quality comes mainly from the experiments.

The held-out 410M MQA study shows a mean cross-entropy reduction of 0.02235 nats per token over the selected GELU MLP across five seeds (Table 4). You correctly noted that this MLP was searched over only two learning rates. We cannot determine from the available results whether the advantage would remain after a broader activation and learning-rate search in the same MQA setting. The fixed-KV2 comparison does not settle this question because its configuration differs.

The 410M fixed-KV2 study provides a concrete setting where FoM improves on the more extensively tuned tanh MLP. At 2048 tokens, Table 6 reports CE of 4.2582 for FoM and 4.2932 for the MLP, with FoM better in all five seeds. The MLP performs better at 512 and 1024 tokens. The mean difference across lengths is only about 0.0039. Stateful FoM also improves on the reset-trained control in all five seeds when CE is averaged across the tested context lengths. This provides evidence that state carry can help, while leaving its advantage over the wider MLP dependent on context length.

Your concern about adaptation length is valid. All Pythia adaptation uses 512-token sequences. Evaluation at 1024 and 2048 tokens remains within the pretrained context range but extends beyond the adaptation length. Both repairs face the same change in length. However, the experiments do not separate compression recovery from differences in robustness to that change. Controls with adaptation at the longer evaluation lengths would be needed to address this question.

We also agree that next-token cross-entropy does not establish improvements on dedicated reasoning or retrieval tasks. Those capabilities were not directly evaluated. The PG19 experiment further limits the generalization claim. Frozen FoM performs worse than the MLP, although adaptation on PG19 reverses the ordering (Table 5). We therefore cannot claim reliable transfer without adaptation or infer practical task gains from the reported CE improvements.

Table 8 also supports your assessment of execution cost. FoM and the MLP both accommodate the same tested batch of 140 sequences, but FoM has lower throughput. The larger batch relative to GQA-4 comes from head sharing. Quality was not measured at the same prompt and generation lengths used in Table 8. We therefore cannot use the 2048-token quality advantage to claim that the measured execution overhead is justified. A joint comparison under matched conditions remains necessary.

We ask that these experimental findings be weighed alongside the further work needed to establish practical value.

I would like to express my sincere gratitude to the authors for their response, as well as for the results and contributions of this work. However, these responses address only a small portion of my concerns, and the paper has not been revised in detail based on the reviewers’ feedback. Overall, I am maintaining my current score. I hope the authors will continue to refine this work in their future research so that it may eventually meet the standards of ICLR and make a meaningful contribution to your field. Finally, I would like to thank the authors once again for their efforts.

Reviewer_sBWDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates recurrent repair after converting pretrained MHA checkpoints to shared-KV attention. FoM uses EMA or GRU states to generate residual corrections while preserving the compressed KV layout. Experiments on a byte-level model and Pythia checkpoints show improved cross-entropy over unrepaired models, with configuration-dependent advantages over parameter-matched MLPs and additional execution cost.

Critical strengths

The held-out MQA comparison demonstrates consistent repair gains without restoring KV heads, and the reset-trained control provides evidence that carrying state contributes within the proposed architecture.

Critical weaknesses

The most prominent advantage of MQA has not yet been separated from the influence of the MLP tuning strategy. Section 5.1 fixes the MQA MLP to GELU and searches only two learning rates. The later fixed-KV2 comparison uses a broader search and selects a tanh MLP that nearly matches FoM’s mean CE. This does not establish that the MQA advantage would disappear, but a comparably broad validation search is needed to determine how much benefit remains over a strong stateless repair.

Against the tuned fixed-KV2 MLP, FoM improves quality at 2048 tokens but loses at 512 and 1024, leaving only a small mean difference (Table 6). In Table 8, both methods support batch size 140, while FoM’s cached-decoding throughput is 6,230 versus 7,734 tokens/s. Since these timings use a different context configuration from the clearest quality gain, they do not establish a compelling joint quality–cost advantage. Matched measurements at the context where FoM helps would clarify whether the gain justifies its overhead.

Using fixed-size recurrent states to compensate for compression losses has precedents in LESS and KVLinC, although they target different compression settings. FoM’s novelty therefore lies mainly in adapting this approach to KV-head sharing and studying its effectiveness. However, the practical benefit remains unclear: in the fixed-KV2 comparison, a well-tuned MLP nearly matches FoM’s mean cross-entropy while achieving higher decoding throughput (Tables 6 and 8). Its competitiveness with alternative compression approaches is also untested. These results limit the evidence for a meaningful advance beyond existing repair strategies.

Questions for authors

please see above.

2 comments

No rebuttal has been provided, so I am maintaining my original rating of 2.

Dear Reviewer sBWD,

Thank you for your review and your follow up. Our contribution is mainly an experimental study of recurrent repair after KV-head sharing. Section 7 acknowledges the related approaches in LESS and KVLinC. In the held-out 410M MQA study, FoM lowers mean cross-entropy by 0.02235 nats per token compared with the selected GELU MLP across five seeds (Table 4). We do not yet know whether this gain would remain after trying more activation functions and learning rates for the MLP. The later KV2 study uses a different setup, so it cannot answer that question.

In the KV2 study, FoM does better than the more thoroughly tuned tanh MLP at 2048 tokens in all five seeds. Cross-entropy is 4.2582 for FoM and 4.2932 for the MLP (Table 6). The MLP does better at 512 and 1024 tokens, leaving FoM with an average advantage of only about 0.0039 across lengths. FoM also improves on a control trained with its incoming memory state forced to zero before every token update. This comparison averages cross-entropy across context lengths and favors FoM in all five seeds. It provides evidence that carrying memory between tokens can help.

With a batch of 140 sequences, FoM is slower than the MLP (Table 8). Both fit more sequences than GQA-4 because they use fewer KV heads. Quality and speed were tested at different lengths, so the tradeoff remains unclear. We also have not established superiority over alternative compression methods. We ask that the observed gains and the cases where a simpler MLP performs well be considered together with these practical limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.