- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
When a trained recurrent model is replaced, its old state may not hold what the new model would compute from the same history. The paper asks how much auxiliary memory must be kept while streaming, before the replacement is known. For time invariant linear models it proves that the minimum equals the reachable dimension of the state sensitivity system minus what the anchor already holds. A controllable scalar input model never needs more than kd extra coordinates for order k migration, and a single rank one update family already forces this worst case. Tolerance versions give conditional singular value and bit bounds. Experiments compare coefficient memories, interpolation, and a Laguerre history bank on a byte level language model and a planted identification task.
Critical strengths
-
The rank one result is crisp. Arbitrary update directions and one well chosen rank one family share the same worst case local cost. This is close to what low rank adapters write into linear recurrent models, and the proofs are elementary enough to check.
-
The paper separates its question types and states its non claims. Exact order, later revealed directions, tolerance, and implementation cost each get their own statement, and the appendix lists what is not established.
Critical weaknesses
-
The characterized minimum does not matter at practical tolerances. The four state example needs eight coordinates for exact order two migration and one coordinate at the 1e-4 error level. The constructive section still targets exact closure, and no causal optimum for the tolerance setting is identified. The gap a practitioner cares about is opened and left open.
-
The paper's own constructions lose to its strongest baseline. The analytic Laguerre bank matches full replay's prediction error at the 1600 byte budget and streams several times faster, while the degree three coefficients stay four orders behind. The language experiment shows no meaningful predictive gain from migration. The bound is sharp, but no construction yet matters at a practical tolerance.
Questions for authors
-
Is the tolerance optimum causally attainable within any constant factor for the rank one family, or is that a separate problem. If it is separate, say so in the limitations rather than leaving the gap open.
-
Can the width study be extended to one realistically trained linear state space model under a rank one update to its transition, or is that outside what this paper can show. If it is outside, note it as future work.
-
The abstract says cubic interpolation improves mean state fidelity over degree three coefficients. The table shows lower error in three of five seeds with standard deviations about seventy percent of the means. Does the claim survive a paired per seed test.
-
In the coupled task, first order sensitivity costs more storage than linear stitching and performs far worse. What explains this ordering, and does stitching stay competitive when the slow poles are split.
-
How much of the history bank advantage comes from its target specific shared decoder. An ablation with a task agnostic decoder would isolate the bank itself.
3 comments
Thanks for the response and the added results. The paired test and abstract revision (Q3), the split-pole stitching check (Q4), and the clarification that the analytic decoder uses only (A, b) and the bank pole rather than C or labels (Q5) address my questions. My main concerns remain: there is still no causal guarantee at a fixed tolerance, the exact-order minimum differs from what practical accuracy requires, and the classical Laguerre bank still outperforms the proposed constructions. Given the clarifications and the correctness of the results, I am raising my score from 1 to 2.
Thank you for the follow-up and for examining the additional results. We appreciate your confirmation that the responses to Q3–Q5 address those questions. The corresponding changes are now in the revised manuscript and marked in blue.
The abstract and introduction no longer make a comparative claim that interpolation improves on coefficient retention. They focus on the causal-memory characterization and the memory-width findings. Section 5.2 and Appendix P.5 report the paired comparison in full, including the confidence interval, for the paired -test, and for the sign-flip test. The observed mean difference is not presented as an established advantage.
Appendix P.3 and Table 24 include the split-pole stitching and first-order results from our response, together with their calibration and evaluation protocol. Section 5.3 and Appendix N.6 clarify that the analytic decoder uses the replacement dynamics, injection, and basis pole, but neither the readout nor prediction labels.
We have also made the remaining scope explicit. Section 6 identifies the width study on a naturally trained linear recurrent model as future work. The stronger predictive performance of the Laguerre bank remains in the results, alongside comparisons at matched output and full-state tolerances.
These edits leave the matching local-information theorem and its assumptions unchanged. Thank you for helping us make the scope and interpretation more precise!
Thank you for the detailed questions. We agree that the required accuracy matters when interpreting a memory bound.
1. Causal optimality at fixed tolerance. We have not proved a constant-factor causal guarantee at fixed tolerance, even for the rank-one family. Theorem 1 gives the causal minimum for local order. Theorems 4–5 give the fixed-horizon, finite-query optimum through conditional singular values. Updating a horizon-specific singular basis causally may require more memory. Appendix N.6 certifies a particular history bank but does not compare it within a constant factor of every causal encoder. Section 6 already excludes finite-tolerance causal optimality. We will make the rank-one case explicit there. The four-state example shows why the distinction matters: eight coordinates are needed for local order, whereas one suffices at fixed-horizon tolerance 10−4.
2. A realistically trained linear SSM. We have not run the proposed width study on such a model. Our identification experiment estimates its family from independent transitions, but uses full calibration states and a planted slow block. Splitting the poles removes the repeated-pole simplification while retaining that structure. The language experiment also shows no meaningful predictive gain. Extending the width study to a naturally trained LRU with transition-only adaptation and frozen input features is a useful next step, and we will identify it as future work. LRU [1] and CompreSSM [2] provide relevant learned-model settings, but neither substitutes for that experiment. Our present contribution is the information characterization under the stated assumptions.
3. Paired comparison of interpolation and coefficients. The paired test does not establish an advantage. We recomputed the contrast using one observation per trained model, rather than treating migration events as independent. For cubic minus degree-three state NRMSE, the mean difference is −2.19×10−4. The descriptive paired-t 95% interval is [−9.92×10−4,5.54×10−4], with two-sided p=0.475. A sign-flip test gives p=0.75 under its symmetry assumption. Cubic has lower error in three of five seeds, as Figure 10 shows. We will revise the abstract to say: “Cubic interpolation had a lower observed mean state error, but the paired comparison was inconclusive and neither method produced a meaningful predictive gain.”
4. Stitching versus first-order memory. First-order memory retains derivatives at the anchor and omits higher-order terms. Stitching fits a regression for the actual finite replacement using full calibration states. It can therefore exploit correlations among histories in the calibration distribution, at the cost of fitting a shared model-pair map. The local-order theorem and the worst-history lower bound do not predict which of these approximations will work better on that distribution.
We followed your suggestion and evaluated stitching on the five previously fitted split-pole systems. We used the original independent calibration protocol and the same held-out histories, without retraining the recurrent models. Mean task NMSE is 1.59×10−6 for stitching and 7.22×10−5 for first order. Stitching remains better than first order, but is worse than degree three at 3.54×10−9 and the 64-coordinate bank at 3.08×10−10. Its maximum seed output NRMSE is 2.04×10−3, so it fails the 10−3 requirement across all five seeds. This follow-up uses the same synthetic fitted systems as the submitted study.
5. Dependence of the decoder on the task. The analytic decoder already reconstructs the full state without using the task readout. Appendix N.6 defines LM(A,b)=[ϕ0(A)b,…,ϕM−1(A)b]. Its construction uses the replacement dynamics A,b and the bank pole. It uses neither C, prediction labels, nor a fitted output objective. The same summary and decoder can be used with another readout without refitting. The width needed for accurate prediction may change with C, which is why the paper also reports full-state error. This is the classical Laguerre construction [3].
The decoder must still be given the replacement model. The same retained history can produce different target states under different replacements, and the decoder must distinguish them. We include the resulting shared storage and construction work in the cost accounting. The history bank is stronger than coefficient retention for prediction in these experiments. That finding does not alter the matching bound on the information that must be retained before a replacement is known.
[1] Orvieto et al., “Resurrecting Recurrent Neural Networks for Long Sequences,” ICML 2023.
[2] Chahine et al., “The Curious Case of In-Training Compression of State Space Models,” ICLR 2026.
[3] Wahlberg, “System Identification Using Laguerre Models,” IEEE TAC 1991.