- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper adds a recurrent concatenation-based residual module to recover prediction quality after reducing KV-cache storage through KV-head sharing.
Critical strengths
The experiments provide preliminary evidence that recurrent EMA may help recover some prediction quality lost through KV-head sharing.
Critical weaknesses
-
Poor presentation. The paper is hard to follow, particularly the relationship between KV-head sharing, cache compression, and the proposed repair module. The motivation for the architecture and training procedure is not clearly explained.
-
Limited insight and theoretical support. Adding recurrent state after attention offers limited new insight. The claimed prediction-repair benefit is not rigorously proved: the propositions establish basic state properties, not recovery of prediction quality. The distinction from existing recurrent repair methods is unclear.
-
Missing component-wise ablations and limited generalization. The paper does not provide a controlled module-by-module ablation of the repair architecture. The paper does not convincingly show that the gains generalize beyond data similar to the distillation dataset.
-
Unconvincing practical benefit. The repair adds computation, parameters, and a persistent state that must be stored. The quality gains over MLP repair are small, while Table 8 reports lower throughput than the MLP at the same tested batch size under an 8 GiB allocation limit.
Questions for authors
-
What insights motivate the chosen state update, repair width, and layer placement, what evidence justifies these choices, and which ablations isolate their individual effects?
-
Do FoM's gains generalize to data substantially different from the distillation dataset without further adaptation?
2 comments
Clarification of our empirical contribution
Dear Reviewer AiNL,
We thank you for your review and the time invested in checking our work. ICLR’s reviewer guidelines explicitly recognize empirical findings as a contribution and don’t necessarily require state-of-the-art results. We see our paper as a contribution of this kind. We test whether recurrent memory can recover prediction quality after KV-head sharing, and we report where it helps and where it doesn’t. We think these findings are useful even though they don’t establish a general recovery guarantee or a consistent practical advantage.
The cache reduction comes from KV-head sharing. First-order Memory (FoM) keeps that layout and adds a repair state whose size doesn’t grow with context length. We agree that the propositions don’t prove prediction recovery; that claim rests on the experiments. In the held-out 410M MQA study, FoM lowers mean cross-entropy by 0.02235 nats per token relative to the selected MLP, and it beats the MLP in all five seeds (Table 4).
We used the EMA update because it gives a simple causal summary with bounded state. The width and placement were chosen empirically. Table 6 includes a tuned MLP and a control trained with state reset. Stateful FoM beats the reset-trained control on mean cross-entropy across context lengths in all five seeds, though the tuned MLP nearly closes the gap in mean performance. These comparisons provide evidence about state carry, but we agree they don’t isolate every architectural choice or show that our width and placement are the best ones.
Without further adaptation, the generalization result is negative. On PG19, the frozen FoM model does worse than the MLP. After adaptation on PG19, the ordering reverses (Table 5). So we don’t claim that the advantage transfers reliably across distributions, and we see this as a limitation for future work to address.
We also agree that Table 8 shows lower throughput than the MLP at the same batch size. The larger batch size relative to GQA-4 comes from head sharing. What FoM offers is recovered prediction quality in some settings while keeping the compressed KV layout. Whether that’s worth the extra execution cost is still an open question.
We’d ask that the positive results and these limitations be weighed together when assessing our paper’s empirical contribution.
Response
I maintain my original evaluation.