acceptodds
Under review as a conference paper at ICLR 2027

Recurrent Memory Repair: When Local Sufficiency Fails at Finite Budgets

Abstract

We study how shared retention gates can improve prediction while limiting changes in the influence of past writes. For normalized recurrent memory with fixed candidate values and write strengths, we constrain changes in their allocation weights by Kullback–Leibler (KL) divergence or the operator norm of the sensitivity change. We construct a sigmoid-gate family with a shared bias whose best first-order loss decrease at the baseline matches that of freely adjustable allocations at equal Fisher cost. Yet we prove that its best loss decrease vanishes as sequences grow at a fixed sufficiently small positive budget. Adding the squared difference between consecutive candidates to the sigmoid input with one new shared coefficient yields a loss decrease bounded away from zero with length, at the same budget and from the same pre-update gates and allocation. For this construction, the fixed-task improvement survives all separately bounded candidate-value perturbations within a positive radius independent of length and budget, with pre- and post-update gate values held fixed. General results characterize how gate changes accumulate to consume the budget and which gate features realize prescribed allocation changes. In a 20-seed synthetic study with a frozen trained gate-generating network, quadratic augmentation attains larger calibration loss decreases in every seed than equally sized random augmentation at matched local task efficiency and mean KL cap.

Primary area:
learning theory
Keywords:
Recurrent memory, Gating mechanisms, Parameter sharing, Information geometry, Sensitivity-constrained optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kxmAOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies normalized recurrent memories (like sLSTM or Forgetting Attention), whose output is a weighted average of past writes, with weights determined by products of retention gates generated by parameters shared across time. The authors ask how much a gate-parameter update can reduce prediction loss while keeping the change in these weights within a fixed KL or L1 budget.

Their main result is that "local sufficiency", meaning the shared gate family matches the best first-order improvement of freely adjustable weights, does not guarantee a useful finite improvement: on a synthetic constructed two-sequence examples, second-order drift in the gates compounds over time, consumes the budget, and makes the best gain vanish as sequences grow.

The paper is mainly theory and the main text only present a 20-seed synthetic experiment with a small frozen gate network, where quadratic features beat random ones; there are no real-data or real-model experiments. I am, however, not sure that this small experiment part is well constructed enough to support the claim of the paper, mainly because the paper is really hard to follow and highly technical, and my understanding of it is only partial.

Finally, the paper is 86 pages long with appendices A through N, combined with an extremely compressed main text, make it hard to verify claims within a review. I was not able to check the mathematical derivations.

Critical strengths
  • A clear and potentially important conceptual point.

    • The paper shows that first-order sufficiency of a shared gate parameterization does not guarantee a useful finite improvement under a fixed sensitivity budget, because small second-order gate errors compound multiplicatively along the sequence and consume the budget as length grows. This separation between "pointing in the right direction" and "achieving a non-vanishing improvement at a fixed budget" I interesting, and I think not made explicit in prior work on recurrent memory, and it identifies a length-dependent failure mode that a purely local (natural gradient style) analysis would miss. This is the main reason the paper could merit acceptance.
  • A constructive solution, though of limited generality.

    • Adding a single squared-difference feature to the gate input cancels the drift and restores a length-independent gain from the same baseline gates, with two-sided rate bounds (Theorem 3.1), and the general results link drift order to rates. However, the single-feature repair is shown to be minimal only for this specific construction, and following the main text, it is not obvious how a general solution could be implemented. The strength therefore lies mainly in the principle (supplying the higher-order terms of the inverse gate map) rather than in a fix that can be applied as-is to practical architectures.
Critical weaknesses
  • The main text is very hard to follow, and the problem is not properly motivated.

    • The introduction does not explain why the studied problem matters. It opens with the allocation formula and a specific gate parameterization before stating what question is being asked, and it never describes a practical setting in which one would update retention gates under an allocation-KL or operator-norm budget. The motivation for this budget (that it controls how much the update changes the output sensitivity to perturbations of stored values) has to be reverse-engineered from eq (4) in Section 2. Key concepts are introduced in a single dense sentence or only through equations: "local sufficiency" is defined in one line of the introduction, and terms such as "same-baseline repair", "finite entropic tilt", "response ratio", and "local task efficiency" appear with little intuition. The core mechanism, small second-order gate errors that compound multiplicatively along the sequence and consume the budget, is simple and could be explained in a few sentences or a figure, but it is never stated plainly; the reader must extract it from the theorems and appendices. The flow of ideas is not pedagogical: results are stated with many technical qualifiers, and the notation load is very large for a 9 page main text. The length of the appendices (86 pages in total) is not a problem in itself, but the main text should allow a reader to understand the problem, why it matters, the intuition behind the results, and their scope without consulting them. In its current form, it does not, which makes the paper inaccessible to much of the ICLR audience and makes it difficult to assess the significance and correctness of the contribution. I would encourage the authors to restructure the introduction around (i) a concrete motivating setting, (ii) an intuitive explanation of the compounding-drift mechanism, and (iii) a plain-language statement of the main result before the formal notation.

    • This weaknesses should also be read as a weakness of my own review. Indeed, even if I am familiar with memory recurrent architectures and trust region optimization (which seems closely related), I cannot ensure a deep understanding of what the paper presented. I will reflect this in my confidence score.

  • Limited evidence of relevance beyond a constructed example, and weak empirical support.

    • The central separation is proven on a highly specific construction, and in a closely related specialization a shared bias already suffices, which raises the question of how general the phenomenon is. The experiment does not close this. It is fully synthetic and uses a single sequence length, so it does not directly test the predicted length dependence. It also compares theory-derived quadratic features against random sine features, which seems like a weak baseline. There are no experiments on an actual normalized-memory architecture or on real data.
Questions for authors

I do not have specific clarification questions whose answers would materially change my assessment. My main concerns relate to the presentation and motivation of the work, in particular the accessibility of the main text and the lack of a clear explanation of why the studied problem matters in practice, as well as the scope of the empirical validation. I believe addressing these points would require a substantial restructuring of the paper rather than clarifications that could be provided within the rebuttal period. I want to stress that I find the underlying idea interesting, and I would encourage the authors to pursue a revised version that states the motivation and the compounding-drift intuition upfront, and that tests the predicted length dependence on a realistic normalized-memory architecture.

3 comments

Thank you for explaining why the original presentation made the contribution hard to assess. We have rebuilt the main text around the motivating intervention, mechanism, and four-step proof outline, and added length studies and a pretrained-model experiment. The main text has 10 pages; the PDF has 51, down from 86. References use revised numbering.

1. Motivation and a reading path through the main text.

We adapt retention to a task while keeping stored representations fixed. We seek better predictions while limiting how much the update changes the influence of each past input. Figure 1 follows one input through its write strength, later forget gates, and normalized memory weight. Section 2.2 explains the budget: KL between updated and initial weights bounds the change in sensitivity to separately perturbed stored inputs, with the evaluated gates held fixed. This gives a trust region on memory weights. Section 2.3 defines local sufficiency as matching the best initial loss-reduction rate of directly adjustable weights at equal local Fisher cost.

2. The mechanism, the role of the bias, and what the proof establishes.

Each memory weight contains a product of later gates, so errors in their logarithms add over time. In the construction, this moves weight between earlier and later copies of the same value. That movement consumes KL budget without changing the current output. A locally useful direction can therefore become increasingly costly as more copies are retained.

Section 3.1 and Figure 2 explain why one shared bias does not remove this effect. For one alternating signal, a bias can cancel the drift. Two signals with different magnitudes need different corrections. Their squared consecutive-value differences are constant within each signal but differ between them; one square coefficient and the bias supply both corrections.

Theorem 3.1 gives the separation: with uniform initial weights and a fixed small positive budget, the best loss reduction in the slope-and-bias model decreases as T−1/2T^{-1/2}, whereas the square feature restores a positive improvement independent of history length TT. Both start from the same memory. Section 3.2 explains four proof steps: separate useful reweighting from temporal drift; use exact identities to control all feasible coefficients; identify the unavoidable quadratic drift; and construct an exact cancellation. The bound covers all feasible coefficients.

The extension to nonuniform weights is now in Section 4 (Theorem 4.1, formerly H.2). It uses the spread of retained positions under stated gate-response and tail conditions. Section 5 derives gate-input corrections and explains their scope for nonlinear networks and changing representations. The single-feature exact repair remains specific to the construction.

3. Length dependence, structured controls, and a pretrained model.

The length study freezes forty trained gate networks, in two groups of twenty. One update direction is chosen at length 64 and retained at six lengths from 16 to 512. Linear adds ξ\xi, a linear combination of eleven fixed gate features, to the sigmoid input; the inverse quadratic correction (Inv2) adds ξ+f0ξ2/2\xi+f^0\xi^2/2, where f0f^0 is the initial gate. Each method's scale is selected separately using independent KL-calibration sequences. Inv2 gives larger test loss reductions for all forty networks at every length; all observed network–method–length mean test costs meet the 10−310^{-3} cap. At length 512 its advantage is about 0.00320.0032 in each group, from initial squared loss 0.031250.03125. Section 6.2, Figure 4, and Table 6 connect these gains to accumulated errors and actual KL.

Section 6.3 adds unweighted squares and cross-products to the equal-width feature comparison. They outperform random features when fitted with a smaller budget before calibration, but the ranking reverses when only previously fitted coefficients are scaled. This makes the fitting procedure part of the empirical conclusion.

Appendix F evaluates a released 760M-parameter Forgetting Transformer (FoX) on PG-19 books, adapting final-block gates with the pretrained model fixed. Here ξ\xi is affine in each head's fixed input. In a fixed-direction scale sweep, log-gate error decreases approximately as the step squared for Linear and cubed for Inv2. Separate fits at context 8192 reduce next-token loss; Inv2 improves over Linear by 2.41×10−52.41\times10^{-5} nats/model token. Const2 replaces f0f^0 by its per-head mean over fitting inputs; Perm2 uses another head's f0f^0. Both attain lower loss (Tables 7–8). None of the four fits reaches the convergence criterion. At context 16384, keeping the fitted coefficients and scales gives book-averaged KL above the cap; length-specific recalibration brings all four means below it. These experiments test context transfer and local cancellation; the sharp asymptotic rate is proved for the construction.

I thank the authors for the response. You engaged very late in the discussion period and it will be very difficult to give you feedbacks before the end of the discussion period. I will do my best to be on time.

Thank you very much for your understanding and for taking the time to consider our response despite the limited time remaining.

We sincerely apologize for responding so late in the discussion period and for leaving you with so little time to review our revision. We greatly appreciate your willingness to provide feedback and completely understand if the remaining time does not allow for a detailed response.

Reviewer_tXsYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper theoretically reveals a phenomenon in normalized recurrent memory models. Local first order sufficiency does not guarantee non-vanishing predictive improvements for long sequences under a limited allocation budget. It shows that this is caused by cumulative drift in shared gating and feature space constraints, and proposes fixing this decay by introducing quadratic features of differences between neighboring candidate values, finally providing statistical validation in controlled synthetic experiments.

Critical strengths
  1. Distinguish 'local first-order sufficiency' and 'effective response to long sequences under a limited budget', rigorously translating gate design into implementability issues of feature space and rank, and reveal the mechanism of cumulative drift in shared gating.
  2. Strictly controls parameter scale (14 coefficients) and local task efficiency (around 1), conducts paired tests and bootstrap confidence interval analysis over 20 independent random seed clusters, and honestly reports perturbation radius boundaries and test-set KL budget overruns.
Critical weaknesses
  1. The synthetic experiments mainly compare squared difference features (Quadratic) with equal width random features (Random), lacking ablation comparisons with other mathematically motivated structured nonlinear features.
  2. Experiments are only conducted on frozen gate generators and synthetic tasks, and the learned updates exceed the preset KL budget on most test seeds (15/20 and 18/20), with no evidence that this mechanism can directly transfer to and improve mainstream sequence model performance on real long sequence tasks.
Questions for authors

see weakness

1 comment

Thank you for identifying the missing controls and test-cost overruns. The added calibration keeps all observed synthetic controller-mean test costs within the cap. Structured quadratics outperform random features when fitted with the calibration margin, but the ranking reverses when only the fitted update is scaled. A pretrained language-model experiment also shows a small next-token loss improvement from the inverse-map correction. References below are to the revised manuscript.

1. Held-out cost: separate fitting, scale selection, and evaluation.

The submission's unchanged-coefficient test exceeded the 10−310^{-3} mean-KL cap in 15/20 Quadratic and 18/20 Random controllers. We retain those results in Appendix E.11. The new protocol uses 16 fitting sequences, 64 independent sequences to select an update multiplier by KL alone, and 256 fresh test sequences. Test loss never selects the multiplier.

We evaluate two procedures against the same final cap τ=10−3\tau=10^{-3}. Post-fit scaling fits at τ\tau and scales to calibration mean KL at most 0.8τ0.8\tau. Budget-aware fitting fits at 0.8τ0.8\tau before the same scale selection. Both also check fitting cost. Every observed controller-mean test cost satisfies τ\tau in both procedures (Appendix E.7). We designed the second procedure after examining the first and used fresh calibration and test draws.

2. Structured controls: distinguish quadratic structure from inverse weighting.

The submission's “Quadratic” features are the revision's inverse-weighted quadratics. We add unweighted squares and cross-products (d12,2d1d2,d22)(d_1^2,2d_1d_2,d_2^2), where d=zt−zt−1d=z_t-z_{t-1} is the difference between consecutive stored vectors. Inverse-weighted features multiply these terms by f0/[2(1−f0)2]f^0/[2(1-f^0)^2], where f0f^0 is the initial forget gate. This comparison tests whether that weighting helps beyond the quadratic structure itself.

Each feature set adds three columns to the same eleven base features: fourteen fitted coefficients, the same initial memory, and the same fitting procedure. Their first-order loss-reduction rates at equal local Fisher cost are matched. We retain the original squared-error task of shifting each sequence's initial memory output by a fixed vector.

Group A reuses the submission's twenty controllers; Group B contains twenty additional controllers. The table gives unweighted-minus-random test loss reduction in units of 10−410^{-4}; positive values favor unweighted quadratics. Brackets are paired 95% bootstrap intervals over controllers.

ProcedureGroup AGroup B
Post-fit scaling-12.175 [-15.926, -7.968]-10.785 [-15.071, -5.841]
Budget-aware fitting4.984 [2.072, 7.159]6.891 [6.116, 7.606]

Both outcomes matter: quadratic structure helps under the second procedure, and its advantage depends on fitting and scale selection. Inverse-weighted-minus-unweighted intervals include zero in both groups under both procedures (Section 6.3, Tables 1 and 5).

3. Real model and data: gate adaptation in a pretrained Forgetting Transformer (FoX).

We adapt the final-block forget gates of a released 760M-parameter FoX on PG-19 book text, keeping pretrained parameters fixed. Unlike the independently fitted quadratic-feature coefficients above, these corrections fit an affine increment ξ\xi from each head's fixed input to its sigmoid input. Linear adds ξ\xi; Inv2 adds ξ+f0ξ2/2\xi+f^0\xi^2/2, where f0f^0 is the initial gate. Two controls replace f0f^0 by its fitting-data mean for that head (Const2) or another head's initial gate (Perm2). All four fit the same 36,888 coefficients.

After separate fitting and independent KL calibration, evaluation covers 762 windows from 96 held-out books at context length 8192. Negative log-likelihood (NLL) is measured per scored model token and averaged equally across books. KL averages over heads and scored queries before the book average. All four updates lower baseline NLL. Inv2 improves over Linear by 2.41×10−52.41\times10^{-5} nats/token, with book-paired 95% interval [1.88,2.97]×10−5[1.88,2.97]\times10^{-5}. Const2 and Perm2 attain still lower NLL. All four fits stop before meeting the convergence criterion, so this is a comparison of attained updates (Tables 7–8).

Inv2's book-averaged attention-weight KL is 0.840τ0.840\tau; 21/96 individual book means exceed τ\tau. The cap controls the aggregate mean. On the common 95-book subset, keeping the same update at length 16384 raises its mean KL to 2.344τ2.344\tau; length-specific KL recalibration brings it to 0.835τ0.835\tau (Appendix F.5).

Separately, shrinking a fixed fitted direction gives log-gate residual orders 2.00 for Linear and 3.00 for Inv2, consistent with cancellation of the quadratic error. Thus the added experiments connect local cancellation to real-model prediction while showing that it does not by itself determine the best finite-update task performance.

Reviewer_eVb2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is a theoretical study of normalised recurrent memory, in particular the cumulative-normalised update of sLSTM, and of Forgetting Attention at a fixed query. The main result (Theorem 3.1) states that recurrent memory model can have locally optimal update directions but become increasingly unable to improve its predictions as sequence length grows, even under a fixed positive update budget.

Critical strengths
  • It is a rich theoretical study which established that:
    • A clean, two-sided separation with an identified mechanism.
    • The repair is the right control, and the constructed controls show that drift cancellation matters.
Critical weaknesses
  • Unclear generality to practical recurrent architectures.
  • The empirical results do not directly verify the proposed failure mechanism.
Questions for authors
  1. Can the authors characterize when the finite-budget obstruction persists for more general fixed-width, input-dependent sigmoid-gate networks?
  2. Can the authors measure cumulative log-retention drift in their learned-controller experiments and quantify how much of the allocation-KL budget it consumes?
  3. What motivates the allocation-KL constraint as an operational budget for training recurrent models, and how does it relate to commonly used optimization or stability constraints? Furthermore, to what extent do the predicted failure and repair persist when retention gates and candidate values are jointly input-dependent, rather than holding candidate values and the gate-generating network fixed?
1 comment

Thank you for the questions. We now quantify the KL cost of accumulated errors and distinguish local network corrections from global conditions for finite-budget failure. References use revised numbering.

1. When does the obstruction extend to nonlinear gate networks?

Fixed width alone does not imply an obstruction: a network's own curvature can supply the correction. Section 5.2 and Appendix C.14 give a local test. For a C3C^3 network, let u(θ)u(\theta) collect sigmoid inputs on fixed evaluation data, with gates f0f^0, Jacobian JJ, and Hessian H\mathcal H evaluated at the baseline. For a desired first-order log-gate change vv, choose a parameter direction a1a_1 satisfying Ja1=v/(1−f0)Ja_1=v/(1-f^0). A second-order parameter correction cancels the quadratic log-gate error along this direction exactly when f0v⊙22(1−f0)⊙2−12H[a1,a1]∈range⁡J.\frac{f^0v^{\odot2}}{2(1-f^0)^{\odot2}}-\frac12\mathcal H[a_1,a_1]\in\operatorname{range}J. Products and divisions are componentwise. The first term is the required correction; the second is supplied by network curvature.

A finite-budget obstruction requires unavoidable drift throughout the low-cost parameter region, including distant parameters. For repeated inputs with repeating log-gate changes, Theorem B.1 assumes a global drift lower bound in terms of task gain and a matching attainable path, then derives the finite-budget rate. Theorem 4.1 (originally H.2) extends the rate to nonuniform weights through their temporal spread: the standard deviation of copy positions under the initial weights. Spreads must remain comparable across sequences and bounded away from zero. A uniform exponential-moment bound controls centered positions divided by their spread. Under these conditions, growing spread makes the best feasible gain vanish at a fixed small budget.

2. How much KL does the accumulated error consume?

The correction reduces actual KL at a common step. The study uses forty frozen, trained controllers, twenty each in Groups A and B (Section 6.2, Appendix E.10). Shared coefficients form an increment ξj\xi_j from eleven fixed features. Linear adds it to each sigmoid input. The inverse-map correction, Inv2, adds ξj+fj0ξj2/2\xi_j+f_j^0\xi_j^2/2 and cancels the quadratic log-gate error near zero. Here fj0f_j^0 and fjf_j are initial and updated gates. Both use the same first-order direction, chosen at length 64 and retained at lengths 16–512.

Let p0p^0 and pp be initial and updated memory weights. Define

R_i=\sum_{j>i}\left[\log(f_j/f_j^0)-(1-f_j^0)\xi_j\right].$$ The normalized weights $q_i\propto p_i^0e^{A_i}$ give the first-order reference; $p_i\propto p_i^0e^{A_i+R_i}$. We measure the residual variance and use the exact identity $$D_{\rm KL}(p\|p^0)=D_{\rm KL}(q\|p^0)+D_{\rm KL}(p\|q)+(\mathbb E_p-\mathbb E_q)A.$$ Here $\mathbb E_p A=\sum_i p_iA_i$. The mismatch $D_{\rm KL}(p\|q)$ is only part of the excess budget; the last term is signed. At length 512 in Group A, Linear's mean actual KL is $7.54\times10^{-4}$ versus reference KL $3.65\times10^{-5}$. Their difference is $7.17\times10^{-4}$; the residual variance $\operatorname{Var}_q(R)$ is $1.44\times10^{-3}$ (Table 6). At a common multiplier, Inv2 reduces mismatch and actual KL in all 240 controller–length pairs, while improving loss in 39. When each method's multiplier is selected separately using independent KL-calibration sequences, Inv2 gives larger test improvements for all forty controllers at all six lengths. All 480 observed controller–method–length mean test costs meet $10^{-3}$. This separates fixed-step realization from task benefit after scale selection. **3. Why this budget, and what changes with joint adaptation?** We consider adapting retention while controlling changes in each stored representation's influence. For $m=\sum_i p_i z_i$, hold the evaluated old and new gate sequences fixed and perturb each $z_i$ by a vector of norm at most $\epsilon$. A KL cap $\tau$ bounds the change in this perturbation response by $\epsilon\sqrt{2\tau}$ (Section 2.2). This is a trust region on memory-weight sensitivity; its local quadratic form is the memory-weight Fisher metric. Parameter-norm and predictive-KL constraints control different objects. The guarantee is conditional on the evaluated gates, rather than on the full input-to-output feedback dynamics. Input-dependent gates and values are compatible with this analysis when their realized inputs and values remain fixed during gate adaptation. Jointly updating the values can bypass the obstruction: adding one vector to every stored value shifts the output at unchanged weights. Even fixed value-network parameters need not fix realized values under recurrent feedback. Section 5.2 and Appendix C.15 make this boundary explicit; the pretrained-model intervention in Appendix F fixes upstream representations and final-layer queries, keys, and values while adapting the forget gates.

Comments on the paper

1 comment

Revision overview

We thank the reviewers for identifying where the motivation, scope, and empirical evidence needed to be clearer. We have reorganized the main text and added experiments addressing those concerns.

The central result is unchanged: a shared gate family can match the best first-order loss reduction for a task, yet its best finite improvement can vanish as history length grows at a fixed small positive memory-weight KL budget. In the explicit sigmoid construction, one squared-difference feature restores length-independent improvement from the same initial memory. The theorem covers every feasible coefficient choice in that family.

A clearer main text. Sections 1–3 now introduce the adaptation setting and sensitivity budget, explain how log-gate errors accumulate, and give a four-step proof outline. Section 4 brings the temporal-spread result into the main text (Theorem 4.1, originally H.2). Section 5.2 distinguishes nonlinear gate-network corrections from the additional freedom of changing stored values.

Direct measurements and stronger controls. Section 6 measures cumulative log-gate errors and their contribution to actual KL in trained gate controllers across six history lengths. Independent cost calibration brings all observed controller-mean test costs within the cap. The added unweighted quadratic features beat random features when fitted at a lower budget before calibration; scaling only after fitting reverses the ranking. Both outcomes are reported.

A pretrained-model test. Appendix F evaluates final-block gate adaptation in a 760M-parameter Forgetting Transformer on PG-19 books. A finite-scale sweep measures error orders consistent with local cancellation. In separate fits, the inverse-map correction gives a small next-token loss improvement over a linear gate update, while two other quadratic controls attain lower loss. The comparison reports actual costs and finite-computation outcomes.

The added experiments are post-review follow-ups. Each reply below addresses one review independently; references use revised numbering unless marked original.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.