- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies normalized recurrent memories (like sLSTM or Forgetting Attention), whose output is a weighted average of past writes, with weights determined by products of retention gates generated by parameters shared across time. The authors ask how much a gate-parameter update can reduce prediction loss while keeping the change in these weights within a fixed KL or L1 budget.
Their main result is that "local sufficiency", meaning the shared gate family matches the best first-order improvement of freely adjustable weights, does not guarantee a useful finite improvement: on a synthetic constructed two-sequence examples, second-order drift in the gates compounds over time, consumes the budget, and makes the best gain vanish as sequences grow.
The paper is mainly theory and the main text only present a 20-seed synthetic experiment with a small frozen gate network, where quadratic features beat random ones; there are no real-data or real-model experiments. I am, however, not sure that this small experiment part is well constructed enough to support the claim of the paper, mainly because the paper is really hard to follow and highly technical, and my understanding of it is only partial.
Finally, the paper is 86 pages long with appendices A through N, combined with an extremely compressed main text, make it hard to verify claims within a review. I was not able to check the mathematical derivations.
Critical strengths
-
A clear and potentially important conceptual point.
- The paper shows that first-order sufficiency of a shared gate parameterization does not guarantee a useful finite improvement under a fixed sensitivity budget, because small second-order gate errors compound multiplicatively along the sequence and consume the budget as length grows. This separation between "pointing in the right direction" and "achieving a non-vanishing improvement at a fixed budget" I interesting, and I think not made explicit in prior work on recurrent memory, and it identifies a length-dependent failure mode that a purely local (natural gradient style) analysis would miss. This is the main reason the paper could merit acceptance.
-
A constructive solution, though of limited generality.
- Adding a single squared-difference feature to the gate input cancels the drift and restores a length-independent gain from the same baseline gates, with two-sided rate bounds (Theorem 3.1), and the general results link drift order to rates. However, the single-feature repair is shown to be minimal only for this specific construction, and following the main text, it is not obvious how a general solution could be implemented. The strength therefore lies mainly in the principle (supplying the higher-order terms of the inverse gate map) rather than in a fix that can be applied as-is to practical architectures.
Critical weaknesses
-
The main text is very hard to follow, and the problem is not properly motivated.
-
The introduction does not explain why the studied problem matters. It opens with the allocation formula and a specific gate parameterization before stating what question is being asked, and it never describes a practical setting in which one would update retention gates under an allocation-KL or operator-norm budget. The motivation for this budget (that it controls how much the update changes the output sensitivity to perturbations of stored values) has to be reverse-engineered from eq (4) in Section 2. Key concepts are introduced in a single dense sentence or only through equations: "local sufficiency" is defined in one line of the introduction, and terms such as "same-baseline repair", "finite entropic tilt", "response ratio", and "local task efficiency" appear with little intuition. The core mechanism, small second-order gate errors that compound multiplicatively along the sequence and consume the budget, is simple and could be explained in a few sentences or a figure, but it is never stated plainly; the reader must extract it from the theorems and appendices. The flow of ideas is not pedagogical: results are stated with many technical qualifiers, and the notation load is very large for a 9 page main text. The length of the appendices (86 pages in total) is not a problem in itself, but the main text should allow a reader to understand the problem, why it matters, the intuition behind the results, and their scope without consulting them. In its current form, it does not, which makes the paper inaccessible to much of the ICLR audience and makes it difficult to assess the significance and correctness of the contribution. I would encourage the authors to restructure the introduction around (i) a concrete motivating setting, (ii) an intuitive explanation of the compounding-drift mechanism, and (iii) a plain-language statement of the main result before the formal notation.
-
This weaknesses should also be read as a weakness of my own review. Indeed, even if I am familiar with memory recurrent architectures and trust region optimization (which seems closely related), I cannot ensure a deep understanding of what the paper presented. I will reflect this in my confidence score.
-
-
Limited evidence of relevance beyond a constructed example, and weak empirical support.
- The central separation is proven on a highly specific construction, and in a closely related specialization a shared bias already suffices, which raises the question of how general the phenomenon is. The experiment does not close this. It is fully synthetic and uses a single sequence length, so it does not directly test the predicted length dependence. It also compares theory-derived quadratic features against random sine features, which seems like a weak baseline. There are no experiments on an actual normalized-memory architecture or on real data.
Questions for authors
I do not have specific clarification questions whose answers would materially change my assessment. My main concerns relate to the presentation and motivation of the work, in particular the accessibility of the main text and the lack of a clear explanation of why the studied problem matters in practice, as well as the scope of the empirical validation. I believe addressing these points would require a substantial restructuring of the paper rather than clarifications that could be provided within the rebuttal period. I want to stress that I find the underlying idea interesting, and I would encourage the authors to pursue a revised version that states the motivation and the compounding-drift intuition upfront, and that tests the predicted length dependence on a realistic normalized-memory architecture.
3 comments
I thank the authors for the response. You engaged very late in the discussion period and it will be very difficult to give you feedbacks before the end of the discussion period. I will do my best to be on time.
Thank you very much for your understanding and for taking the time to consider our response despite the limited time remaining.
We sincerely apologize for responding so late in the discussion period and for leaving you with so little time to review our revision. We greatly appreciate your willingness to provide feedback and completely understand if the remaining time does not allow for a detailed response.
Thank you for explaining why the original presentation made the contribution hard to assess. We have rebuilt the main text around the motivating intervention, mechanism, and four-step proof outline, and added length studies and a pretrained-model experiment. The main text has 10 pages; the PDF has 51, down from 86. References use revised numbering.
1. Motivation and a reading path through the main text.
We adapt retention to a task while keeping stored representations fixed. We seek better predictions while limiting how much the update changes the influence of each past input. Figure 1 follows one input through its write strength, later forget gates, and normalized memory weight. Section 2.2 explains the budget: KL between updated and initial weights bounds the change in sensitivity to separately perturbed stored inputs, with the evaluated gates held fixed. This gives a trust region on memory weights. Section 2.3 defines local sufficiency as matching the best initial loss-reduction rate of directly adjustable weights at equal local Fisher cost.
2. The mechanism, the role of the bias, and what the proof establishes.
Each memory weight contains a product of later gates, so errors in their logarithms add over time. In the construction, this moves weight between earlier and later copies of the same value. That movement consumes KL budget without changing the current output. A locally useful direction can therefore become increasingly costly as more copies are retained.
Section 3.1 and Figure 2 explain why one shared bias does not remove this effect. For one alternating signal, a bias can cancel the drift. Two signals with different magnitudes need different corrections. Their squared consecutive-value differences are constant within each signal but differ between them; one square coefficient and the bias supply both corrections.
Theorem 3.1 gives the separation: with uniform initial weights and a fixed small positive budget, the best loss reduction in the slope-and-bias model decreases as T−1/2, whereas the square feature restores a positive improvement independent of history length T. Both start from the same memory. Section 3.2 explains four proof steps: separate useful reweighting from temporal drift; use exact identities to control all feasible coefficients; identify the unavoidable quadratic drift; and construct an exact cancellation. The bound covers all feasible coefficients.
The extension to nonuniform weights is now in Section 4 (Theorem 4.1, formerly H.2). It uses the spread of retained positions under stated gate-response and tail conditions. Section 5 derives gate-input corrections and explains their scope for nonlinear networks and changing representations. The single-feature exact repair remains specific to the construction.
3. Length dependence, structured controls, and a pretrained model.
The length study freezes forty trained gate networks, in two groups of twenty. One update direction is chosen at length 64 and retained at six lengths from 16 to 512. Linear adds ξ, a linear combination of eleven fixed gate features, to the sigmoid input; the inverse quadratic correction (Inv2) adds ξ+f0ξ2/2, where f0 is the initial gate. Each method's scale is selected separately using independent KL-calibration sequences. Inv2 gives larger test loss reductions for all forty networks at every length; all observed network–method–length mean test costs meet the 10−3 cap. At length 512 its advantage is about 0.0032 in each group, from initial squared loss 0.03125. Section 6.2, Figure 4, and Table 6 connect these gains to accumulated errors and actual KL.
Section 6.3 adds unweighted squares and cross-products to the equal-width feature comparison. They outperform random features when fitted with a smaller budget before calibration, but the ranking reverses when only previously fitted coefficients are scaled. This makes the fitting procedure part of the empirical conclusion.
Appendix F evaluates a released 760M-parameter Forgetting Transformer (FoX) on PG-19 books, adapting final-block gates with the pretrained model fixed. Here ξ is affine in each head's fixed input. In a fixed-direction scale sweep, log-gate error decreases approximately as the step squared for Linear and cubed for Inv2. Separate fits at context 8192 reduce next-token loss; Inv2 improves over Linear by 2.41×10−5 nats/model token. Const2 replaces f0 by its per-head mean over fitting inputs; Perm2 uses another head's f0. Both attain lower loss (Tables 7–8). None of the four fits reaches the convergence criterion. At context 16384, keeping the fitted coefficients and scales gives book-averaged KL above the cap; length-specific recalibration brings all four means below it. These experiments test context transfer and local cancellation; the sharp asymptotic rate is proved for the construction.