acceptodds
Under review as a conference paper at ICLR 2027

Information Limits of Replay-Free Recurrent State Migration

Abstract

When a recurrent model is replaced, its old state may not determine what the new model would compute from the same history. We characterize the information that must be retained before the replacement is known. For a time-invariant affine linear family, the minimum continuous causal memory for order- local migration equals the reachable dimension of a lifted coefficient system minus the information in the anchor state. A controllable -state model with scalar input needs at most additional real coordinates, even when arbitrary matrix replacement directions are revealed only after streaming. A single rank-one direction can already require all . Conditional singular values and auxiliary-bit bounds distinguish this local information law from recovery at a fixed tolerance. We also give invariant-subspace and future-output constructions. Controlled experiments compare the resulting representations. In a small language model, equal-memory cubic interpolation improves mean state fidelity over third-order coefficients but provides no meaningful predictive gain. An analytic history basis preserves useful adaptation more economically in a coupled identification task. With repeated slow poles, eight history coordinates meet normalized output error across five fitted seeds despite substantial full-state error. Splitting the poles raises the smallest tested successful width to 64. These results separate necessary historical information from the cost and accuracy of a chosen representation.

Primary area:
learning theory
Keywords:
replay-free state migration, recurrent neural networks, state-space models, causal memory, memory lower bounds, streaming algorithms, reachability, observability, approximation theory, model reduction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qLocOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

When a trained recurrent model is replaced, its old state may not hold what the new model would compute from the same history. The paper asks how much auxiliary memory must be kept while streaming, before the replacement is known. For time invariant linear models it proves that the minimum equals the reachable dimension of the state sensitivity system minus what the anchor already holds. A controllable scalar input model never needs more than kd extra coordinates for order k migration, and a single rank one update family already forces this worst case. Tolerance versions give conditional singular value and bit bounds. Experiments compare coefficient memories, interpolation, and a Laguerre history bank on a byte level language model and a planted identification task.

Critical strengths
  1. The rank one result is crisp. Arbitrary update directions and one well chosen rank one family share the same worst case local cost. This is close to what low rank adapters write into linear recurrent models, and the proofs are elementary enough to check.

  2. The paper separates its question types and states its non claims. Exact order, later revealed directions, tolerance, and implementation cost each get their own statement, and the appendix lists what is not established.

Critical weaknesses
  1. The characterized minimum does not matter at practical tolerances. The four state example needs eight coordinates for exact order two migration and one coordinate at the 1e-4 error level. The constructive section still targets exact closure, and no causal optimum for the tolerance setting is identified. The gap a practitioner cares about is opened and left open.

  2. The paper's own constructions lose to its strongest baseline. The analytic Laguerre bank matches full replay's prediction error at the 1600 byte budget and streams several times faster, while the degree three coefficients stay four orders behind. The language experiment shows no meaningful predictive gain from migration. The bound is sharp, but no construction yet matters at a practical tolerance.

Questions for authors
  1. Is the tolerance optimum causally attainable within any constant factor for the rank one family, or is that a separate problem. If it is separate, say so in the limitations rather than leaving the gap open.

  2. Can the width study be extended to one realistically trained linear state space model under a rank one update to its transition, or is that outside what this paper can show. If it is outside, note it as future work.

  3. The abstract says cubic interpolation improves mean state fidelity over degree three coefficients. The table shows lower error in three of five seeds with standard deviations about seventy percent of the means. Does the claim survive a paired per seed test.

  4. In the coupled task, first order sensitivity costs more storage than linear stitching and performs far worse. What explains this ordering, and does stitching stay competitive when the slow poles are split.

  5. How much of the history bank advantage comes from its target specific shared decoder. An ablation with a task agnostic decoder would isolate the bank itself.

3 comments

Thank you for the detailed questions. We agree that the required accuracy matters when interpreting a memory bound.

1. Causal optimality at fixed tolerance. We have not proved a constant-factor causal guarantee at fixed tolerance, even for the rank-one family. Theorem 1 gives the causal minimum for local order. Theorems 4–5 give the fixed-horizon, finite-query optimum through conditional singular values. Updating a horizon-specific singular basis causally may require more memory. Appendix N.6 certifies a particular history bank but does not compare it within a constant factor of every causal encoder. Section 6 already excludes finite-tolerance causal optimality. We will make the rank-one case explicit there. The four-state example shows why the distinction matters: eight coordinates are needed for local order, whereas one suffices at fixed-horizon tolerance 10−410^{-4}.

2. A realistically trained linear SSM. We have not run the proposed width study on such a model. Our identification experiment estimates its family from independent transitions, but uses full calibration states and a planted slow block. Splitting the poles removes the repeated-pole simplification while retaining that structure. The language experiment also shows no meaningful predictive gain. Extending the width study to a naturally trained LRU with transition-only adaptation and frozen input features is a useful next step, and we will identify it as future work. LRU [1] and CompreSSM [2] provide relevant learned-model settings, but neither substitutes for that experiment. Our present contribution is the information characterization under the stated assumptions.

3. Paired comparison of interpolation and coefficients. The paired test does not establish an advantage. We recomputed the contrast using one observation per trained model, rather than treating migration events as independent. For cubic minus degree-three state NRMSE, the mean difference is −2.19×10−4-2.19\times10^{-4}. The descriptive paired-tt 95% interval is [−9.92×10−4,5.54×10−4][-9.92\times10^{-4},5.54\times10^{-4}], with two-sided p=0.475p=0.475. A sign-flip test gives p=0.75p=0.75 under its symmetry assumption. Cubic has lower error in three of five seeds, as Figure 10 shows. We will revise the abstract to say: “Cubic interpolation had a lower observed mean state error, but the paired comparison was inconclusive and neither method produced a meaningful predictive gain.”

4. Stitching versus first-order memory. First-order memory retains derivatives at the anchor and omits higher-order terms. Stitching fits a regression for the actual finite replacement using full calibration states. It can therefore exploit correlations among histories in the calibration distribution, at the cost of fitting a shared model-pair map. The local-order theorem and the worst-history lower bound do not predict which of these approximations will work better on that distribution.

We followed your suggestion and evaluated stitching on the five previously fitted split-pole systems. We used the original independent calibration protocol and the same held-out histories, without retraining the recurrent models. Mean task NMSE is 1.59×10−61.59\times10^{-6} for stitching and 7.22×10−57.22\times10^{-5} for first order. Stitching remains better than first order, but is worse than degree three at 3.54×10−93.54\times10^{-9} and the 64-coordinate bank at 3.08×10−103.08\times10^{-10}. Its maximum seed output NRMSE is 2.04×10−32.04\times10^{-3}, so it fails the 10−310^{-3} requirement across all five seeds. This follow-up uses the same synthetic fitted systems as the submitted study.

5. Dependence of the decoder on the task. The analytic decoder already reconstructs the full state without using the task readout. Appendix N.6 defines LM(A,b)=[ϕ0(A)b,…,ϕM−1(A)b]L_M(A,b)=[\phi_0(A)b,\ldots,\phi_{M-1}(A)b]. Its construction uses the replacement dynamics A,bA,b and the bank pole. It uses neither CC, prediction labels, nor a fitted output objective. The same summary and decoder can be used with another readout without refitting. The width needed for accurate prediction may change with CC, which is why the paper also reports full-state error. This is the classical Laguerre construction [3].

The decoder must still be given the replacement model. The same retained history can produce different target states under different replacements, and the decoder must distinguish them. We include the resulting shared storage and construction work in the cost accounting. The history bank is stronger than coefficient retention for prediction in these experiments. That finding does not alter the matching bound on the information that must be retained before a replacement is known.

[1] Orvieto et al., “Resurrecting Recurrent Neural Networks for Long Sequences,” ICML 2023.

[2] Chahine et al., “The Curious Case of In-Training Compression of State Space Models,” ICLR 2026.

[3] Wahlberg, “System Identification Using Laguerre Models,” IEEE TAC 1991.

Thanks for the response and the added results. The paired test and abstract revision (Q3), the split-pole stitching check (Q4), and the clarification that the analytic decoder uses only (A, b) and the bank pole rather than C or labels (Q5) address my questions.  My main concerns remain: there is still no causal guarantee at a fixed tolerance, the exact-order minimum differs from what practical accuracy requires, and the classical Laguerre bank still outperforms the proposed constructions. Given the clarifications and the correctness of the results, I am raising my score from 1 to 2.

Thank you for the follow-up and for examining the additional results. We appreciate your confirmation that the responses to Q3–Q5 address those questions. The corresponding changes are now in the revised manuscript and marked in blue.

The abstract and introduction no longer make a comparative claim that interpolation improves on coefficient retention. They focus on the causal-memory characterization and the memory-width findings. Section 5.2 and Appendix P.5 report the paired comparison in full, including the confidence interval, p=0.475p=0.475 for the paired tt-test, and p=0.75p=0.75 for the sign-flip test. The observed mean difference is not presented as an established advantage.

Appendix P.3 and Table 24 include the split-pole stitching and first-order results from our response, together with their calibration and evaluation protocol. Section 5.3 and Appendix N.6 clarify that the analytic decoder uses the replacement dynamics, injection, and basis pole, but neither the readout nor prediction labels.

We have also made the remaining scope explicit. Section 6 identifies the width study on a naturally trained linear recurrent model as future work. The stronger predictive performance of the Laguerre bank remains in the results, alongside comparisons at matched output and full-state tolerances.

These edits leave the matching local-information theorem and its assumptions unchanged. Thank you for helping us make the scope and interpretation more precise!

Reviewer_yXfQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how to reconstruct the state of a new model after updating the recurrent model without replaying the historical inputs. Its core contribution is to precisely characterize the minimum auxiliary memory required for local state transitions and to prove that low-rank updates do not necessarily reduce the memory requirement. The authors construct online memory based on parameter expansion and reachability space analysis, and distinguish between theoretical lower bounds of information, actual storage costs, and predictive fidelity. The verification includes theoretical proofs, numerical tests, language model and system identification experiments. The results show that more accurate state recovery does not necessarily improve prediction, while an appropriate historical basis can achieve near-complete replay prediction accuracy with less memory. Overall, the paper provides an information-theoretic framework for understanding state transitions without replay, but the conclusion is mainly limited to linear models and structured experiments, and has not yet been verified for the applicability to any neural network update.

Critical strengths

1.Provide precise and achievable lower bounds of information for state transitions without replay. The paper characterizes the minimum continuous causal memory as the reachable dimensions beyond the old states in the system with an improvement coefficient, and proves the sharp worst-case bound under unknown replacement directions; a rank-one direction can reach this bound. This goes beyond the design of a single migration algorithm, revealing which historical information cannot be remedied by state mapping, and why low-rank updates do not necessarily reduce memory requirements. This theoretical contribution with a clear scope of application is the primary reason for accepting it.

2.Through controlled comparison, reveal the differences between information requirements, representation costs, and prediction fidelity. Experiments matching memory budgets show that more accurate state reconstruction does not necessarily improve prediction, while the historical basis can achieve prediction accuracy close to full replay in structured identification tasks. The pairing experiments of pole structure and historical width further illustrate that the actual memory requirements depend on the system structure and the target output. These results provide explanatory empirical support for the theory and practical judgment basis for the selection of migration methods.

Critical weaknesses
  1. There is still a significant gap between theoretical information boundaries and actual deployable migrations. The core optimality conclusions, which are based on continuous real coordinates, exact arithmetic, and asymptotic local errors, have not established the necessary conditions for guaranteeing good performance of direction-independent constructions under finite precision, nor have they addressed the causal optimality in general finite tolerances. Therefore, this boundary cannot be directly translated into storage and precision budgets of actual systems, weakening the guiding power of the theoretical results for deployment decisions, but not negating its mathematical contributions.

  2. There is limited evidence for actual predicted benefits, and it relies on strong structural assumptions. Language experiments do not show meaningful predictive improvements; coupled identification experiments rely on complete state calibration and artificially set slow invariant structures, and the strongest analytical historical libraries also come from classical methods. Therefore, the existing experiments are not sufficient to prove that the proposed coefficient migration method has practical advantages in more general model updates, limiting the empirical contributions and external validity of the paper.

Questions for authors
  1. Under what conditions can a direction-independent kd-coordinate construction be stably implemented with finite precision? The small-scale numerical checks in the paper focus on fixed model families rather than general direction-independent constructions. Is there existing condition number analysis or numerical evidence that defines when such a construction becomes practically viable? This would influence my assessment of whether the core information bounds can be translated into implementable methods.

  2. Do existing experiments clearly demonstrate scenarios where coefficient memory offers distinct advantages over analytical history libraries? In the experiment involving splitting slow poles, third-order coefficients achieved very low task error; if we compare both methods under the same prediction tolerance and account for per-stream storage, shared decoder, and maintenance costs, what is the resulting comparison? This would affect my judgment regarding the independent practical value of the proposed method.

3 comments

Thank you for the careful review. We address the numerical question first, then compare the representations at the same prediction tolerance.

Q1. Finite precision. You are right that Appendix P.4 concerns a fixed family. It does not establish stability of the direction-independent construction at large dimensions. The latter uses the common denominator det⁡(I−zA0)k+1\det(I-zA_0)^{k+1}. Reducing the number of real coordinates does not ensure that those coordinates can be maintained accurately.

The fixed-family construction makes one source of error amplification explicit. Let WW be an orthonormal basis of the reachable lift, H=π0WH=\pi_0W, and NN an orthonormal basis of ker⁡H\ker H. The anchor-preserving transformation is J=[H;N⊤]J=[H;N^\top]. With HH of full row rank and a nontrivial auxiliary space, the eigenvalues of J⊤JJ^\top J are the squared nonzero singular values of HH, together with ones. Thus κ2(J)=max⁡{1,σmax⁡(H)}min⁡{1,σmin⁡(H)}.\kappa_2(J)=\frac{\max\{1,\sigma_{\max}(H)\}}{\min\{1,\sigma_{\min}(H)\}}. A small σmin⁡(H)\sigma_{\min}(H) can make reconstruction inaccurate even though HH has full rank. Computing WW also requires adequate numerical separation of the reachable subspace. These quantities must be assessed for the realization being used, including the universal construction.

Accumulated roundoff can be bounded directly. For zt=Fzt−1+gxtz_t=Fz_{t-1}+gx_t with exact zero initialization and step residual bounded by η\eta, the query error is at most η∑j<t∥DΔFj∥\eta\sum_{j<t}\|D_\Delta F^j\|, plus decoder error. If ∥Fj∥≤Kρj\|F^j\|\le K\rho^j with ρ<1\rho<1, this is at most η∥DΔ∥K/(1−ρ)\eta\|D_\Delta\|K/(1-\rho), plus decoder error. Adding Taylor truncation error gives a sufficient precision requirement for a particular system. Our theorems do not bound these constants uniformly over anchors and replacement families. Stable eigenvalues alone therefore do not settle the numerical question.

For the two-state fixed family, the condition numbers are 1.124–1.166 and the maximum discrepancy from the full polynomial response is 4.79×10−84.79\times10^{-8} in float32, before Taylor truncation error. The universal two-state construction has condition numbers 15.6–38.4. These tests do not establish reliability at large dimensions. Recent finite-precision RNN analysis [1] also distinguishes expressivity from robustness, but gives no stability guarantee for our encoder. A larger representation may be needed to meet a specified tolerance. We have not established a well-conditioned near-minimal universal realization.

Q2. Equal prediction tolerance. For output accuracy, the split-pole experiment favors the history bank on per-stream memory. We compare the maximum seed-level output NRMSE across five fitted seeds, relative to replay over the same 32-position windows.

At tolerance 10−310^{-3}, degree-three coefficients achieve 1.19×10−41.19\times10^{-4} with 1600 extra bytes per stream. A 32-coordinate bank achieves 2.36×10−42.36\times10^{-4} with 256 bytes. Both meet the tolerance. At 10−410^{-4}, degree three fails the all-five-seed requirement, while the 64-coordinate bank reaches 1.86×10−71.86\times10^{-7} with 512 bytes. The small task NMSE for degree three does not change this comparison. Task NMSE and output discrepancy from replay measure different errors.

For a small number of streams, the shared decoder storage can reverse the memory comparison. Under Appendix N.10's accounting, four 128-by-32 float64 decoders occupy 131072 bytes. The coefficient reductions occupy 9728. At tolerance 10−310^{-3}, total auxiliary storage is therefore 131072+256N131072+256N bytes for the bank and 9728+1600N9728+1600N for coefficients, excluding common model weights and active states. Coefficients use less total storage below the crossover. The bank becomes smaller at 91 simultaneously retained streams.

The bank requires O(M)O(M) work per input and O(dM)O(dM) per migration. The coefficient implementation also maintains an anchor and reduced responses. We did not time individual widths in the split-pole study, so those results do not determine a runtime crossover.

Coefficients do use less per-stream memory in the tested full-state comparison. At full-state NRMSE tolerance 10−410^{-4}, degree three reaches 5.04×10−55.04\times10^{-5} with 1600 bytes. The smallest successful tested bank has 800 coordinates and uses 6400 bytes. This comparison concerns reconstruction of the entire state, not prediction, and covers only the bank widths we tested.

The experiments thus give different answers for full-state reconstruction and output fidelity. Classical Laguerre memory [2] is the better predictive representation here. The information theorem applies regardless of which representation is used. Its empirical evaluation does not yet cover unconstrained updates to naturally trained neural models.

[1] Dankowiakowski and Ronca, “Metric Automata Theory: A Unifying Theory of RNNs,” NeurIPS 2025.

[2] Wahlberg, “System Identification Using Laguerre Models,” IEEE TAC 1991.

Thank you for the review. We have added the numerical derivation and matched-tolerance comparisons to the manuscript, with the changes in blue.

Q1. Finite precision. Appendix P.4 now derives the conditioning of the anchor-preserving coordinates. Let WW span the reachable lift orthonormally, H=π0WH=\pi_0W, and let NN span ker⁡H\ker H orthonormally. For J=[H;N⊤]J=[H;N^\top] with nonzero auxiliary dimension, J⊤JJ^\top J acts as the identity on ker⁡H\ker H and as H⊤HH^\top H on its orthogonal complement. Hence κ2(J)=max⁡{1,σmax⁡(H)}min⁡{1,σmin⁡(H)}.\kappa_2(J)=\frac{\max\{1,\sigma_{\max}(H)\}}{\min\{1,\sigma_{\min}(H)\}}. Here σmin⁡(H)\sigma_{\min}(H) is the smallest positive singular value. Full row rank alone does not prevent amplification of roundoff.

Appendix N.11 gives the separate accumulated-error calculation. For zt=Fzt−1+gxtz_t=Fz_{t-1}+gx_t with exact zero initialization and computed-step defects bounded by η\eta, the decoded error relative to the exact polynomial response is at most η∑j<t∥DΔFj∥+ηD\eta\sum_{j<t}\|D_\Delta F^j\|+\eta_D. If ∥Fj∥≤KFρFj\|F^j\|\le K_F\rho_F^j with ρF<1\rho_F<1, this is at most η∥DΔ∥KF/(1−ρF)+ηD\eta\|D_\Delta\|K_F/(1-\rho_F)+\eta_D. Taylor truncation and any numerical subspace-projection error must also fit the accuracy budget.

The revised Section 5.5 directs readers to these conditions. They give a sufficient check for a specified realization rather than a stability guarantee uniform over anchors or replacement families. We preserve the distinction between the fixed-family numerical example and the direction-independent construction. A smaller real-coordinate count need not be the better finite-precision implementation.

Q2. Coefficients and history banks at the same accuracy. Section 5.4 and Appendix P.3 now give the requested comparison. The criterion is the maximum seed-level output NRMSE across the five split-pole models, evaluated against replay over the same 32-position windows.

At output tolerance 10−310^{-3}, degree-three coefficients reach 1.19×10−41.19\times10^{-4} with 1600 extra bytes per stream. The width-32 bank reaches 2.36×10−42.36\times10^{-4} with 256 bytes. Both meet the tolerance. Shared storage changes the total: four width-32 full-state decoders occupy 131072 bytes, compared with 9728 for the coefficient reductions. The totals are therefore 131072+256N131072+256N and 9728+1600N9728+1600N bytes. For this four-target cache, coefficients use less total auxiliary memory through 90 simultaneously retained streams. The bank becomes smaller at 91. Common model weights and active states are excluded from both counts.

At output tolerance 10−410^{-4}, degree three does not meet the all-five-seed criterion. The width-64 bank does, with error 1.86×10−71.86\times10^{-7} and 512 per-stream bytes. This comparison remains explicit in the main text. The small task NMSE of degree three is not a substitute for the requested output fidelity.

For full-state reconstruction at tolerance 10−410^{-4}, degree three qualifies at 5.04×10−55.04\times10^{-5} with 1600 bytes. The smallest successful tested history bank has 800 coordinates and uses 6400 bytes. This is a fourfold per-stream storage advantage for coefficients on that reconstruction objective. It does not establish an advantage for prediction or over every possible history basis.

Appendix P.3 also gives maintenance and decoding costs. The bank requires O(M)O(M) work per input and O(dM)O(dM) per full-state query. Coefficient memory also maintains an anchor and reduced responses. Width-specific runtimes were not measured, so the storage crossover is not presented as a runtime crossover. Appendix N.10 separately accounts for measured serving, setup, and cached-decoder costs.

The revision thus answers the comparison in terms of the requested output, tolerance, and number of streams. The information theorem remains independent of which representation is preferable in a particular deployment. Thank you

Thank you for the detailed responses and revisions. The conditioning and roundoff analyses, together with the matched-tolerance comparisons accounting for shared decoder storage, help clarify my questions about numerical feasibility and the practical trade-offs between representations. Robustness of the direction-independent construction at larger scales and applicability beyond the structured experimental settings remain open concerns. I maintain my original rating of 3 (Weak acceptance).

Reviewer_f5ArOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the information required to migrate the hidden state of a recurrent model after its parameters are replaced, without replaying the full input history. The central problem is to determine what auxiliary information must be retained before the replacement parameters are known.

For time-invariant affine linear recurrent systems, the authors derive an exact characterization of the minimum continuous causal memory required for order-k local state migration. The result is expressed through the reachable dimension of a lifted coefficient system, after accounting for information already available in the anchor state. The paper further establishes a sharp worst-case bound of kd additional real coordinates for controllable scalar-input systems, including the case where replacement directions are unspecified during streaming.

The theoretical analysis is extended to fixed-horizon approximation tolerances, auxiliary-bit lower bounds, structured coefficient memories, and output-oriented state preservation.

Experiments on controlled linear systems, a small recurrent language model, and a synthetic system-identification task evaluate state reconstruction, predictive fidelity, memory requirements, and computational costs.

Overall, the paper makes a meaningful theoretical contribution to understanding the information limitations of replay-free recurrent state migration. My primary reservations concern the practical applicability and numerical realization of the theoretical results.

Critical strengths
  1. A theoretically meaningful characterization of the minimum information required for state migration.

The main strength is the exact causal memory characterization in Theorem 1, together with the sharp worst-case result in Theorem 2 and Corollary 3.

The paper goes beyond providing a sufficient memory construction by identifying the minimum number of additional continuous coordinates required under its assumptions. The combination of a reachability-based lower bound and a constructive causal upper bound is particularly valuable.

The result that a single rank-one replacement direction can already require kd additional coordinates is also informative. It establishes that low update rank does not necessarily imply low historical information requirements.

These results provide useful conceptual insight into the limitations of state-only migration and constitute the strongest reason for considering the paper for acceptance.

  1. Careful separation of information requirements, approximation accuracy, and implementation costs.

The paper distinguishes exact local information requirements from finite-tolerance recovery, numerical conditioning, and practical storage costs.

The conditional singular-value characterization and finite-bit analysis clarify why exact local dimension is not necessarily an appropriate measure of storage requirements at a given numerical tolerance.

The empirical section also evaluates several competing representations, including checkpoint replay, stitching, sensitivity-based methods, polynomial coefficients, and history banks. Importantly, the authors acknowledge cases where accurate state reconstruction does not translate into meaningful predictive improvements.

The detailed assumptions, experimental protocols, and explicit limitations strengthen the technical credibility of the work.

Critical weaknesses
  1. The strongest theoretical results have limited demonstrated applicability to practical recurrent model updates.

The central results assume a restricted family of linear recurrent dynamics with fixed input injections, while the sharp direction-independent memory bound relies on exact arithmetic and continuous real-valued histories.

These assumptions are mathematically well defined, but many practical recurrent models involve parameter-dependent input transformations, nonlinear gating mechanisms, or changes to upstream representations. Such changes may violate the assumptions underlying the analysis.

Furthermore, the universal direction-independent construction can be numerically ill-conditioned, particularly near repeated poles. Although Appendix P.4 provides a useful small-dimensional realization, the paper does not establish whether the theoretically minimal representation can be maintained reliably at larger state dimensions under realistic finite-precision constraints.

This limitation does not invalidate the theoretical results, but it leaves the practical significance of the strongest memory bounds uncertain.

  1. The experiments provide limited evidence of practical predictive benefits beyond favorable controlled settings.

In the recurrent language-model experiment, polynomial and interpolation-based methods improve state reconstruction accuracy, but the predictive benefits are negligible. The architecture contains a direct input-to-logit path, which weakens the dependence of predictions on the migrated recurrent state.

The coupled identification experiment demonstrates substantially greater predictive benefits, including near-replay accuracy using an analytic history bank. However, this setting assumes a planted slow invariant subspace, a known injection and readout, and full-state calibration observations.

These conditions support controlled investigation but may not reflect the difficulty of identifying and preserving migration-relevant information in naturally trained recurrent models.

Additionally, the reported per-stream memory reductions do not fully capture target-specific shared decoder storage and setup costs. Although these costs are disclosed, a clearer end-to-end analysis would help establish the conditions under which replay-free migration is practically advantageous.

Consequently, the experiments support the mathematical distinctions developed in the paper more convincingly than they establish a broadly applicable migration method.

Questions for authors

Q1. Numerical realization of the theoretical minimum.

Can the authors provide additional evidence or analysis concerning the numerical conditioning of the minimum-dimensional causal representation at larger state dimensions? What determines whether the direction-independent kd-coordinate construction remains numerically viable in finite precision, and how should practitioners choose between that construction and a larger but potentially better-conditioned representation?

Q2. Practical relevance to naturally trained recurrent models.

Can the authors clarify which classes of modern recurrent architectures and parameter updates satisfy the fixed-injection assumptions sufficiently well for the theoretical results to be useful? Beyond the deliberately structured synthetic task, is there evidence that replay-free migration can produce meaningful downstream predictive improvements in a recurrent model whose predictions depend strongly on its history state?

Q3. End-to-end memory and computation trade-offs.

How does the practical advantage of the analytic history representation change when shared decoder storage, decoder construction, memory maintenance, and migration costs are considered jointly? Under what number of concurrent streams or replacement queries does it become preferable to checkpoint replay or calibrated state stitching?

3 comments

Thank you for the careful review. We agree that the numerical realization and its costs need a fuller explanation.

Q1. Numerical realization. We have not established that the direction-independent construction is reliable at large dimensions. Appendix P.4 studies a small fixed family. Appendix N.11 tests the direction-independent construction at two states, with coordinate condition numbers 15.6, 27.5, and 38.4 for orders 1–3. The minimum counts real coordinates. It does not specify how many bits are needed to store them accurately.

For a particular realization, the relevant quantity is how much its dynamics and decoder amplify arithmetic error. Write zt=Fzt−1+gxtz_t=Fz_{t-1}+gx_t, with query decoder DΔD_\Delta. If initialization is exact, each computed step has additive residual of norm at most η\eta, and decoder evaluation contributes at most ηD\eta_D, then ∥h^t−ht(k)∥≤η∑j=0t−1∥DΔFj∥+ηD.\|\widehat h_t-h_t^{(k)}\|\leq\eta\sum_{j=0}^{t-1}\|D_\Delta F^j\|+\eta_D. Adding Taylor truncation error gives a sufficient accuracy check for that realization. Stable eigenvalues alone do not make the sum small. Non-normal transients, poor scaling, and an ill-conditioned anchor-preserving transformation can amplify errors. The bound on η\eta must account for coefficient rounding and the input range.

We would therefore compare the accumulated-error bound, the numerical separation of the reachable subspace, and the storage and maintenance costs. A larger, well-scaled coefficient array or orthogonal history basis is preferable when the minimum-dimensional companion realization amplifies roundoff excessively.

Q2. Relevant models and predictive benefit. A concrete case is transition-only adaptation of a linear recurrent block with its input projection and upstream features frozen, as can be imposed on an LRU [1]. The reachable-space theorem permits vector inputs. The kdkd corollary requires a controllable scalar-input system and does not apply unchanged to a usual vector-input block. Changes to upstream features, input injections, or nonlinear recurrent gates require a different analysis. Section 4 allows input-dependent dynamics under its own closure assumptions, but does not give the exact minimum for that larger class.

Zhao et al. [2] study sufficient linear filters for partially observable control, and CompreSSM [3] studies control-theoretic reduction of learned SSMs. These works motivate studying restricted linear memory. Their results do not establish reconstruction after an unknown replacement or relax the assumptions of our theorem.

Our experiments have not established a predictive gain on a naturally trained, history-dependent neural model. The language experiment shows negligible predictive benefit, and its direct skip weakens dependence on the migrated state. The identification task has no skip. Its mean task NMSE is 0.134 without adaptation, 0.375 with stale reuse, and 2.15×10−52.15\times10^{-5} with degree-three migration. It nevertheless uses full-state calibration and a planted slow subspace. Splitting the slow poles tests one aspect of that structure but leaves those restrictions in place, as Section 6 explains.

Q3. Total cost. For NN streams sharing a target set, write setup work as SjS_j, shared auxiliary bytes as HjH_j, and per-token work, per-query work, and per-stream bytes as cj,qj,mjc_j,q_j,m_j. With TT inputs and QQ queries per stream, compare Sj+N(Tcj+Qqj)S_j+N(Tc_j+Qq_j) and Hj+NmjH_j+Nm_j among methods that meet the required accuracy.

On the measured hardware, for 4096 tokens, seven queries, and four targets, Appendix N.10 gives 4.833+8.776N4.833+8.776N ms for the width-200 bank, 30.495N30.495N for full replay, 9.777N9.777N for checkpoint-256, and 136.456+7.065N136.456+7.065N for stitching. These estimates add separately measured setup and serving medians. Concurrent execution was not measured, and these sums are not medians of total latency. For this schedule, the bank costs less than replay from one stream, less than checkpoint-256 from five streams, and less than stitching through 76 streams. Stitching becomes cheaper once its calibration cost is spread over more streams.

For resident memory, NN counts histories retained at the same time. The bank uses 819200+1600N819200+1600N auxiliary bytes. At prefix 4096, replay uses 16384N16384N, so the bank becomes smaller at 56 streams. Checkpoint-256 uses 2048N2048N, giving a crossover at 1829 streams. Stitching retains no extra stream history. Checkpoint replay and stitching are less accurate here. Their lower cost is useful only when the application can tolerate the extra error.

[1] Orvieto et al., “Resurrecting Recurrent Neural Networks for Long Sequences,” ICML 2023.

[2] Zhao et al., “Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning,” ICML 2026.

[3] Chahine et al., “The Curious Case of In-Training Compression of State Space Models,” ICLR 2026.

Thank you for the careful review. We have incorporated the analysis from our response into the revised manuscript. Changes are marked in blue, with the main additions in Appendices N.10, N.11, and P.4.

Q1. Numerical realization. Appendix N.11 now derives an error bound for a chosen implementation. For the exact recurrence zt=Fzt−1+gxtz_t=Fz_{t-1}+gx_t, define the computed-step defect by rt=z^t−Fz^t−1−gxtr_t=\widehat z_t-F\widehat z_{t-1}-gx_t. With exact zero initialization, ∥rt∥≤η\|r_t\|\le\eta, and decoder evaluation error at most ηD\eta_D, the error relative to the exact degree-kk response satisfies ∥h^t−ht(k)∥≤η∑j=0t−1∥DΔFj∥+ηD.\|\widehat h_t-h_t^{(k)}\|\le\eta\sum_{j=0}^{t-1}\|D_\Delta F^j\|+\eta_D. A truncation bound then gives a sufficient absolute-error budget for the target state. The defect includes coefficient rounding and arithmetic error over the relevant input and state ranges. Numerical projection onto an approximate reachable subspace adds a separate representation error.

Appendix P.4 also derives the condition number of the anchor-preserving coordinate transformation. Together, these calculations identify what must be controlled: coordinate conditioning, dynamical amplification, decoder norm, and the numerical separation of the reachable subspace. They explain when a larger, better-scaled representation is preferable to the minimum-dimensional one.

Section 5.5 points to this analysis. The existing small fixed-family and direction-independent examples remain distinguished. We do not extrapolate their condition numbers to a large universal realization.

Q2. Applicability and predictive relevance. The introduction now identifies transition-only adaptation of a linear recurrent block, such as an LRU, with its input projection and upstream features frozen. It also distinguishes the general vector-input characterization from the sharp kdkd corollary, which requires a controllable scalar-input anchor. Changes to upstream features, input injections, or nonlinear recurrent gates need a different analysis.

The coupled task shows why preserving the state can matter: mean task NMSE is 0.134 without adaptation, 0.375 with stale reuse, and 2.15×10−52.15\times10^{-5} with degree-three migration. The analytic bank is stronger at the same per-stream budget. Sections 5.2–5.4 retain the experimental qualifications, including the direct skip in the language model and the observed calibration states and planted structure in the identification task. Section 6 identifies the width study on a naturally trained linear recurrent model as future work. The revision does not present an additional natural-data experiment.

Q3. Total cost. Appendix N.10 now combines setup, maintenance, migration, and shared storage. For NN streams, TT inputs and QQ queries per stream, it gives Sj+N(Tcj+Qqj)S_j+N(Tc_j+Qq_j) work and Hj+NmjH_j+Nm_j auxiliary bytes. Setup and shared storage are Sj,HjS_j,H_j. The remaining terms account for per-input work, queries, and per-stream memory. Replay work is summed over the actual query prefixes, and each new target adds its decoder-construction or calibration cost.

For the measured four-target, 4096-input, seven-query schedule, the estimates are 4.833+8.776N4.833+8.776N ms for the width-200 bank, 30.495N30.495N for replay, 9.777N9.777N for checkpoint-256, and 136.456+7.065N136.456+7.065N for stitching. The bank is cheaper than replay from one stream, checkpoint-256 from five, and stitching through 76. Beyond that, stitching amortizes its calibration cost. These estimates add separate setup and serving medians. They are not measurements of concurrent execution.

At prefix 4096, the bank uses 819200+1600N819200+1600N auxiliary bytes, versus 16384N16384N for replay and 2048N2048N for checkpoint-256. It becomes smaller at 56 and 1829 simultaneously retained streams, respectively. Processed-stream count determines amortized work, while simultaneously retained histories determine memory. Checkpoint replay and stitching have larger errors here, so their lower cost is useful when the application can tolerate those errors.

These additions make the implementation choices explicit without changing the matching causal-memory theorem or its assumptions.

The authors have adequately addressed all of my concerns. I therefore maintain my positive rating of 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up response

We thank all three reviewers for the careful reading and constructive discussion. We have incorporated the additional analyses, comparisons, and clarifications into the revised manuscript, with changes marked in blue. With the cutoff approaching, we summarize the main updates below for your convenience.

Reviewer f5Ar, and numerical realization, applicability, and cost. We appreciate f5Ar’s confirmation that all concerns have been adequately addressed and their continued positive assessment. The revision adds arithmetic-error and conditioning analyses (Appendices N.11 and P.4), together with accounting for setup, per-input maintenance, migration, and shared storage (Appendix N.10). The introduction also makes the intended transition-only adaptation setting explicit, with the input projection and upstream features fixed.

Reviewer yXfQ, and finite precision and comparisons at matched accuracy. The numerical analyses above also address yXfQ’s finite-precision question. Section 5.4 and Appendix P.3 now compare representations at the same requested accuracy. The table reports extra storage per stream in the split-pole study. Qualification requires the pooled event-window NRMSE to meet the tolerance in each of the five fitted seeds.

Reconstruction targetNRMSE toleranceDegree-three coefficientsSmallest qualifying tested Laguerre bank
Output10−310^{-3}1,600 bytes32 coordinates / 256 bytes
Output10−410^{-4}Above tolerance64 coordinates / 512 bytes
Full state10−410^{-4}1,600 bytes800 coordinates / 6,400 bytes

For full-state recovery at 10−410^{-4}, degree three uses one-quarter of the per-stream storage of the smallest qualifying bank in the tested grid. Shared storage also matters. With four cached full-state decoders, total auxiliary storage at output tolerance 10−310^{-3} is 9,728+1,600N9{,}728+1{,}600N bytes for coefficients and 131,072+256N131{,}072+256N bytes for the width-32 bank. Coefficients use less through 90 simultaneously retained streams, and the bank uses less from 91 onward. These totals exclude common model weights and active states. The comparison therefore depends on the reconstruction target, accuracy requirement, and number of streams sharing the decoders.

Reviewer qLoc, and additional controls and interpretation. We appreciate qLoc’s reconsideration, acknowledgment of the results’ correctness, and confirmation that the responses to Q3–Q5 address those questions. The corresponding changes are now in the paper:

  • Paired analysis and abstract (Q3). Section 5.2 and Appendix P.5 report the paired comparison using the five trained models as the comparison units. The abstract and introduction focus on the information characterization and memory-width findings.
  • Split-pole stitching (Q4). Appendix P.3 includes stitching and first-order comparisons on the same frozen fitted systems, held-out streams, and query schedule.
  • Decoder information (Q5). Section 5.3 and Appendix N.6 specify that the analytic full-state decoder uses the replacement dynamics, input injection, and bank pole, without the task readout or prediction labels.

The remaining theoretical question concerns causal memory optimality at a prescribed tolerance. Section 4 provides causal truncation and residual error bounds under its stated assumptions. Section 3 characterizes the information needed at a fixed horizon and tolerance for a finite query set. The sharp causal optimality theorem concerns local approximation order. Section 6 states the distinction explicitly and identifies optimal causal memory at fixed tolerance as an open extension.

The central contribution. For a fixed time-invariant affine linear family, the minimum continuous causal auxiliary memory for order-kk local migration equals the reachable coefficient dimension minus the information already present in the anchor. For a controllable dd-state scalar-input anchor with fixed injection, kdkd additional real coordinates suffice even when arbitrary matrix update directions are revealed only after streaming. A single suitably chosen rank-one direction can already require all kdkd coordinates. This matching characterization explains why low update rank alone need not reduce the historical information required for local migration.

We thank the reviewers again! Their questions have helped us give a more precise account of the information required for migration and a more useful comparison of the available implementations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.