acceptodds
Under review as a conference paper at ICLR 2027

What to Preserve in Recursive Computation: A Local Predictive Sufficiency Principle

Abstract

Recursive computation repeatedly compresses or reuses intermediate states, creating a simple tension: information that must remain useful across longer recursive paths is also exposed to more opportunities for loss before reaching the final prediction. Existing reconstruction or local-prediction objectives provide tractable supervision, but do not ensure that the retained information remains sufficient for subsequent recursive computation. We identify local predictive sufficiency with recursive predictive closure: controlling local predictive deficiencies at individual interfaces controls the resulting discrepancy at the root. We then turn this principle into a tractable training procedure. Starting from a variational characterization, we derive finite predictive tests and an empirical predictive deficiency that measures predictive value retained across compression. Its predictive sensitivities define margin-relaxed half-space constraints on parameter updates, and we project the host optimizer's proposed update onto their intersection only when predictive preservation would otherwise be violated. Across temporal graphs, language memory, vision-language-action control, and recursive self-improvement, the method matches or improves the corresponding host models under matched compression budgets, with larger gains under heavier recursive or memory demands, while better preserving predictive information across successive transformations. Crucially, the same task-agnostic training intervention applies unchanged across all four diverse settings: it constrains the host optimizer without changing the task objective, architecture, or test-time computation. These results establish predictive preservation at recursive interfaces as a general training principle for recursive compression.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Recursive Computation, Predictive Compression, Constrained Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mhxPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a training-time principle called "local predictive sufficiency" for recursive state compression across four domains: temporal graphs, language memory, vision-language-action (VLA) control, and recursive self-improvement (SRI). Starting from conditional mutual information on a rooted tree, the authors approximate the local predictive loss via finite random sketches and ridge-profiled covariance statistics (essentially a regularized CCA / linear readout gap), and then constrain the host optimizer's parameter update using half-space gradient projection (similar to GEM/A-GEM in continual learning). While regularizing intermediate state compression exclusively at training time without adding test-time overhead is a neat idea, the paper fails to clearly communicate its core mechanism. Instead, it buries a relatively standard continual-learning-style gradient constraint + local linear probe beneath 64 pages of heavy measure-theoretic packaging and a sprawling, superficial 4-domain evaluation. Overall, the presentation creates an unnecessary reading burden without going deep into the actual bottlenecks of any single domain. I lean toward Weak Reject.

Critical strengths

1.Training-time intervention without inference overhead: Focusing on constraining parameter updates during training so that the recursive compressor retains predictive features—while keeping the inference pass and memory budget identical to the host model—is a practical and clean design choice.

2.Conceptual motivation: Recognizing that endpoint (root) supervision attenuates over deep recursive paths and attempting to regularize intermediate compression interfaces locally is well-motivated.

Critical weaknesses

1.Excessive Mathematical Packaging Over a Standard Mechanism (Lack of Conceptual Clarity): Stripping away the heavy formalism (standard Borel spaces, kernel disintegrations, and variational KL characterizations in Section 2 and Appendix F), the actual algorithm in Section 3 is quite simple: it measures the drop in ridge-regression / CCA linear predictability of sketched future features (PvP_v) before and after compression (D^B=J^bal(U)−J^bal(ZΓ)\hat{D}B = \hat{J}{\text{bal}}(U) - \hat{J}_{\text{bal}}(Z^\Gamma)), and applies a GEM/A-GEM-style half-space projection (Htd≤btH_t d \le b_t) to prevent the task gradient from increasing this local loss. This is essentially a continual-learning-style gradient constraint (or auxiliary KL/prediction regularization) applied to intermediate representations. Rather than clarifying why a linear sketch readout in a 32- or 64-dimensional random subspace suffices to preserve complex recursive semantics, the paper spends puluhan pages on measure-theoretic derivations that are disconnected from the heavily approximated empirical estimator. This over-packaging obscures the essence of the method and severely hurts readability.

2.Inconsistent Instantiations Across Domains ("Unified Principle" vs. Ad-hoc Fine-tuning): Although the paper claims to validate a single unified optimization principle (Proposition 2, half-space projection) and uses Table 2 to argue that constrained projection is vastly superior to an Additive Predictive-Gradient Update, the actual implementations in Appendix E tell a different story:

In Language Memory (CCM) (Appendix E.2, p. 38 & Table 21, p. 44), the authors omit the projection mechanism altogether and fall back to "direct aggregate specialization" with λ=0.00594\lambda = 0.00594—which is simply adding the auxiliary gradient with a scalar weight, directly contradicting the ablation claim in Table 2. In SRI (Meta$^n$) (Appendix E.4), the method turns into offline training of a tiny 4-slot attention gating module on frozen archives using CCA scores. In VLA, the host backbone is completely frozen and only a small Γ\Gamma module is fine-tuned for a few thousand steps. stitching four very different ad-hoc fine-tuning recipes together under one theoretical umbrella makes the paper prioritize breadth over unified depth rather than a solid, unified algorithmic contribution. Superficial Domain Coverage that Misses Domain-Specific Bottlenecks: In rushing to show cross-domain generality across four disparate tasks, the paper fails to engage with the fundamental challenges of each domain:

Temporal Graphs: The core bottleneck in deeper temporal GNNs (like TGN) is not merely 1D state compression loss, but the loss of high-order structural and topological expressiveness (e.g., structural neighborhood identity, joint spatio-temporal correlation). Simply constraining a node's 1D state vector to linearly predict an adjacent event pair (Yv,Ypa(v))(Y_v, Y_{\text{pa}(v)}) does nothing to resolve structural expressiveness limits. This is evident in Table 1(a): on UCI Depth 3, adding this constraint onto TGN still trails DyGFormer (which explicitly models structural/patch interactions), and on Depth 4 the gain over vanilla TGN (94.07 →\to 94.89 AP) is marginal and matches DyGFormer within noise.

Sequence / Memory Modeling: In recurrent or linear-attention-style state compression, the fundamental challenge is managing fixed-capacity memory collisions (i.e., selective forgetting/erasure of obsolete keys, as addressed by Delta-rule / TTT mechanisms) and capturing long-range dependencies where information remains silent for many steps before being recalled. The paper's Markov tree assumption (M1) and adjacent-pair target Sv=(Yv,Ypa(v))S_v = (Y_v, Y_{\text{pa}(v)}) are inherently myopic: if an early piece of information is not useful for the immediate next step Ypa(v)Y_{\text{pa}(v)} but is critical 10 steps later, local sufficiency with respect to SvS_v provides no protection against discarding it.

3.Weak and Outdated Baselines / Limited Experimental Depth: Because the paper spreads itself thin across four areas, the evaluation in each area is narrow. In Language Memory, the method is only tested as a tiny residual adapter on top of a single host (CCM-merge) with minimal perplexity differences on MSC (e.g., 7.96 vs. 7.96 / 8.16 vs. 8.14 on Gemma), without comparing against stronger modern context-compression or linear/recurrent memory architectures. In VLA, the evaluation is restricted to 20 rollouts on a single setting (LIBERO-Mem T3), where completing just 1 or 2 more repetitions out of 20 rollouts swings the strict SR from 1.7% to 11.7%.

Questions for authors

1.Why does Table 2 on UCI claim that Additive Predictive-Gradient Update fails to improve over task-only training (92.64 vs. 92.62 AP), yet in the Language Memory experiments (Appendix E.2 & Table 21) you drop the feasibility projection and use an additive predictive gradient (λ=0.00594\lambda = 0.00594) anyway?

2.How does the adjacent predictive pair Sv=(Yv,Ypa(v))S_v = (Y_v, Y_{\text{pa}(v)}) preserve information that has a "silent period" (i.e., conditionally independent of the immediate parent event Ypa(v)Y_{\text{pa}(v)}, violating assumption M1, but required at the root YρY_\rho)?

3.In temporal graphs, how can a local predictive constraint on TGN's aggregated vector address the structural bottleneck (e.g., failure to distinguish non-isomorphic temporal neighborhoods), which is the primary reason message-passing TGNs underperform structural models like DyGFormer?

2 comments
AuthorsOpenReview ↗

Response Regarding the Review

We thank the reviewer for identifying two useful issues: finite-sketch justification and the distinction between our predictive-preservation principle and host-specific optimizers. We acknowledge that our presentation could improve.

1. Finite sketches and the theory–estimator connection.

For a properly normalized isotropic random sketch R satisfying E[RRT]=IE[RR^T]=I, we obtain

E[(RTe)T(RTe)]=eTE[RRT]e=eTeE[(R^Te)^T(R^Te)] = e^T E[RR^T]e = e^Te.

This concerns squared predictive error in expectation, not exact recovery of conditional mutual information by the ridge/covariance estimator. Appendix F.3 therefore derives an additional conditional approximation bound: Corollary F.1 separates finite-test coverage, linear-readout, and sampling residuals, while F.3.5 derives the gradient of the implemented empirical objective. The theory–estimator connection is explicitly analyzed, not absent. Our responses to other reviewers provide quantitative toy-model comparisons with analytically accessible information-theoretic quantities and finite-step update audits. Please consult those results.

2. Factual discrepancies and selective interpretation.

The CCM silent-period objection presumes exclusively adjacent-event prediction. Yet CCM recursively updates and reuses compressed memory, and Appendix E.2 explicitly includes both nearby future observations and the eventual response target in intermediate supervision. A counterexample against next-event-only prediction does not describe our implementation. The CCM aggregate predictive-gradient update is also disclosed in Appendix E.2 and Table 21. We could have explained this specialization more clearly; describing a documented choice as something we "quietly abandon" insinuates concealment without evidence. A UCI-specific projection ablation does not establish that all hosts require the same optimizer.

The characterization of our baselines as "weak and outdated" overlooks CTAN (ICML 2024), CCM (ICLR 2024), Metaⁿ (2026; CO-Bench, AAAI 2026), and MemoryVLA (ICLR 2026; LIBERO-Mem, AAAI 2026). These are relevant, contemporary host architectures and benchmarks, not an outdated evaluation.More comparisons may be useful; the existing evaluation is not therefore obsolete. The review highlights DyGFormer's UCI Depth-3 advantage without adequately acknowledging our higher reported UCI Depth-4 AUC and stronger Wikipedia results. Likewise, it foregrounds minimal MSC PPL differences while underrepresenting the depth-dependent DailyDialog findings. Selecting unfavorable comparisons while leaving out relevant countervailing results can mislead readers, regardless of intent.

3. Research scope, professional standards, and Confidence 4.

Our work formulates and analyzes a general information-theoretic principle for predictive preservation under recursive compression, then studies its realization in heterogeneous hosts. It does not claim to solve the central architectural bottleneck of each domain or to win four SOTA competitions. DyGFormer is a purpose-built structural architecture; cross-architecture rankings provide context but cannot replace matched-host tests of predictive preservation. Nor does identifying familiar CCA, ridge, and projection components establish that the preceding theoretical propositions lack novelty. Calling the mathematical analysis "packaging" is not a substitute for identifying a redundant theorem or incorrect proof; detailed derivations are placed in the appendix.

We accept that the manuscript can be difficult to follow, and we respect the right to recommend rejection, including a rating of 2. But terms such as "vibe-engineered," "rushing to show cross-domain generality," and "quietly abandon" do not establish technical flaws. They insinuate questionable motives or research practices without substantiating them. If a theorem is incorrect, identify the mathematical flaw; if experiments are insufficient, specify the missing evidence; if exposition is difficult, say so. None justifies replacing careful verification with disparaging speculation about the authors' research.

We therefore question how these categorical characterizations can be reconciled with Confidence 4, which indicates substantial direct expertise. The CCM architecture, predictive target, derivations, and protocols were available to check. Why should unverified assumptions be presented with such certainty? A reviewer need not know every host beforehand; that makes checking the original references more important, not optional. We acknowledge our presentation issues and expect comparable care in representing our research.

OpenReview is a lasting public record; selective characterizations affect future readers, not just this decision. We do not ask for a score change; we ask that the assessment be grounded in the paper actually written. Scientific criticism demands evidence, professional reviewing demands fairness, and Confidence 4 demands responsibility.

AuthorsOpenReview ↗

Responses to the Three Questions

Q1. Additive Predictive-Gradient Update.

Please refer to our response to Reviewer 7WAH, W5 (1/2), where we have already clarified this implementation choice and provided the corresponding projection analysis.

Q2. Silent Periods and Predictive Sufficiency.

The key distinction is between information being temporarily unexpressed in an intermediate observation and information being absent from the predictive boundary.

Under M1, the predictive boundaries satisfy Yv⊥Yρ∣(Ypa(v),C)Y_v \perp Y_\rho \mid (Y_{\mathrm{pa}(v)},C). By the data-processing inequality,

I(Yv;Yρ∣C)≤I(Yv;Ypa(v)∣C).I(Y_v;Y_\rho\mid C) \leq I(Y_v;Y_{\mathrm{pa}(v)}\mid C).

Consequently, if an earlier predictive event carries no information about its parent boundary, it cannot suddenly become informative about the root through the same Markov chain. Conversely, information relevant to the root must retain a predictive dependence through the intervening boundaries, even when it is not expressed in an intermediate observation.

A construction in which this information disappears from the parent boundary yet reappears at the root necessarily introduces a dependency outside M1. The adjacent-pair sufficiency principle preserves information along the stipulated predictive dependencies; it does not assume that information can disappear and subsequently reappear without an information-carrying path.

Q3. Temporal Graph Structural Expressivity.

Our method addresses predictive information loss during recursive compression, rather than the structural expressivity of the underlying graph architecture. We do not claim to resolve TGN's inability to distinguish certain non-isomorphic temporal neighborhoods. Instead, our results demonstrate that preserving predictive information improves performance within the same TGN architecture, without modifying its structural modeling capacity. This mechanism is complementary to the architectural improvements introduced by models such as DyGFormer.

This concludes our rebuttal to your review. We have no further points to address.

Reviewer_DvRMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what information should be retained when intermediate states are repeatedly compressed and reused. The authors prove that local sufficiency implies target sufficiency at the root node, and they also derive a bound on the total variation of the root’s predictive distribution.

Critical strengths
  1. this theory provides a clear criterion for determining information retention. Propositions 1 and 3 establish a link between conditional mutual information (at the local level) and the predictive distribution at the root node.
  2. the paper is well-organized and easy to follow.
Critical weaknesses
  1. proposition 3 provides bounds for local information gaps under ideal scenarios; however, the projection control actually employed in the paper can only constrain the first-order variation of the aggregated agent statistics.

  2. appendix F.4 presents bounds for a finite time horizon, yet equations (49) and (50) contain several unquantified error terms.

  3. the language memory experiments incorporate an additional residual module that utilizes aggregated prediction gradients, rather than employing the conflict projection operation featured in the main algorithm.

Questions for authors
  1. Is it possible to select a controllable recursive task, for which the true conditional mutual information can be computed, to illustrate the correspondences among M1–M3?

  2. Could the main text clearly distinguish between the three implementation approaches, projection, additive prediction gradients, and discrete retention, while also restricting the cross-domain conclusions to scenarios involving shared objectives?

7 comments
AuthorsOpenReview ↗

W1 (1/2): Predictive Preservation and Approximation Accuracy

Reviewer's Concerns (CW1–CW2). The reviewer raises two important concerns: (i) the quantitative connection between the ideal local information gaps underlying Proposition 3 and the first-order variation of aggregated predictive statistics controlled during optimization; and (ii) the unquantified approximation and finite-horizon residuals in Eqs. (49)–(50).

We address these concerns through controlled linear-Gaussian recursive experiments, approximation-error measurements, and a direct audit of the original optimization trajectories (continued in Part 2).

1. Local-to-root predictive preservation.

We construct recursive Gaussian systems at depths 8 and 16, where true local conditional mutual information (CMI) and root predictive distributions are analytically computable. We compare matched Task-only, Auxiliary (additive predictive gradients), and Projection (predictive-feasibility projection) training across five seeds.

Table R1. True predictive deficiency and root-level stability (mean ± SD, five seeds).

DepthMethodLocal CMI ↓Root CMI ↓Root TV ↓
D8Task-only0.5016 ± 0.06830.0128 ± 0.00270.0512 ± 0.0056
D8Auxiliary0.4133 ± 0.05360.0093 ± 0.00200.0437 ± 0.0047
D8Projection0.4435 ± 0.05360.0108 ± 0.00200.0472 ± 0.0045
D16Task-only1.5583 ± 0.42480.0213 ± 0.00840.0658 ± 0.0136
D16Auxiliary1.3029 ± 0.32520.0162 ± 0.00620.0573 ± 0.0115
D16Projection1.3466 ± 0.38730.0174 ± 0.00710.0593 ± 0.0127

Local CMI sums the interface deficiencies; Root CMI and Root TV are evaluated using the complete compressed trajectory. Auxiliary reduces Local CMI by 17.6%/16.4% (D8/D16), Root CMI by 27.0%/23.9%, and Root TV by 14.7%/13.0% (percentages from unrounded means). Projection likewise improves all three quantities at both depths. Analytically sufficient compressors attain numerically zero deficiencies, while perturbed and learned compressors satisfy Proposition 3's root-stability bound.

These results directly connect local predictive preservation with root-level stability under both Auxiliary and Projection updates. We will detail the roles of the different optimization implementations in our subsequent response.

2. Empirical approximation accuracy.

We quantify three numerical approximation components.

Table R2. Approximation accuracy.

ComponentD8 MAERelative errorD16 MAERelative error
Gaussian CMI first-order calibration0.0012744.05%0.0027825.87%
Bayes-to-Ridge readout0.00002150.147%0.00002470.104%
Pooled estimation (B=2048B=2048)0.0023044.33%0.0014132.60%

Relative errors are normalized by the mean magnitudes of their respective reference quantities: exact Gaussian CMI, Bayes-optimal predictive-value difference, and the population pooled statistic. The first-order trace approximation of the exact Gaussian log-determinant CMI achieves relative errors of 4.05%–5.87%, Ridge profiling achieves 0.104%–0.147%, and pooled finite-sample estimation achieves 2.60%–4.33%.

Together, Tables R1–R2 provide direct mechanism evidence and quantify the principal approximation errors relevant to CW1. The finite-horizon audit addressing CW2 follows in W1 (2/2).

AuthorsOpenReview ↗

W1 (2/2): Finite-Horizon Error Analysis and Manuscript Revision

Reviewer's Concern (CW2). The reviewer highlights the need to quantify the residual terms connecting first-order predictive control with finite-horizon guarantees in Eqs. (49)–(50). Following W1 (1/2), we examine the accuracy of first-order control and the accumulated effects of actual parameter updates along the original training trajectories.

3. Finite-horizon error analysis.

We audited 10 original Projection trajectories (five seeds per depth), comprising 5,040 actual parameter updates, without retraining. All selected checkpoints matched the originals exactly.

Let Ft=D^Bt(Γt)F_t=\widehat D_{\mathcal B_t}(\Gamma_t) denote the empirical pooled predictive statistic. Its cumulative change admits the decomposition

FT−F0=∑t=0T−1(Pt+Et+Rt+Wt)=P+E+R+W,F_T-F_0=\sum_{t=0}^{T-1}(P_t+E_t+R_t+W_t)=P+E+R+W,

where PP is the first-order contribution, EE the frozen-reference gradient correction, RR the finite-step Taylor remainder, and WW the training-window refresh effect.

Table R3. First-order accuracy and realized finite-horizon drift (mean ± SD, five seeds per depth).

MetricD8 ProjectionD16 Projection
First-order directional agreement ↑93.95 ± 1.38%96.40 ± 1.15%
Normalized Taylor remainder ↓9.64 ± 2.85%10.63 ± 1.15%
Relative surrogate reduction ↑91.67 ± 1.32%94.31 ± 1.36%
Cumulative surrogate change (FT−F0F_T-F_0)−0.883548 ± 0.055575−0.987703 ± 0.088245
First-order contribution (PP)−1.688960 ± 0.213805−1.566427 ± 0.154800
Frozen-reference correction (EE)+0.428829 ± 0.062448+0.314766 ± 0.033428
Taylor remainder (RR)+0.186148 ± 0.095495+0.217674 ± 0.048329
Window refresh (WW)+0.190435 ± 0.067915+0.046283 ± 0.028624

Directional agreement measures whether the implemented first-order prediction PtP_t matches the sign of the realized fixed-window change Pt+Et+RtP_t+E_t+R_t. The normalized Taylor remainder is ∑t∣Rt∣/∑t∣Pt+Et∣\sum_t|R_t|/\sum_t|P_t+E_t|. Both percentages are computed per trajectory and summarized across seeds. The four cumulative contributions are signed quantities evaluated using the original training windows (B=128B=128).

Across the original training updates, first-order sensitivities correctly predict the direction of realized fixed-window changes in 93.95% (D8) and 96.40% (D16) of steps. The normalized Taylor remainders are 9.64% and 10.63%, demonstrating accurate local first-order prediction under finite parameter updates.

Over the complete training trajectories, the empirical predictive statistic decreases by 91.67% (D8) and 94.31% (D16) on average, with reductions observed in all ten runs. The measured frozen-reference, Taylor, and window-refresh contributions reconstruct the cumulative changes to approximately 10−1610^{-16} numerical precision.

These results provide direct quantitative evidence connecting first-order predictive control with realized finite-horizon behavior.

Summary and Manuscript Revision.

Together, these experiments address the reviewer's concerns:

  • CW1: Tables R1–R2 establish the quantitative relationship between true local predictive deficiency, root-level stability, and the empirical predictive statistic.
  • CW2: Tables R2–R3 quantify approximation accuracy, first-order directional reliability, and accumulated finite-step corrections along actual optimization trajectories.

We will introduce a dedicated section, Controlled Validation of Local-to-Root Predictive Preservation, incorporating Tables R1–R3, the analytical Gaussian construction, approximation accuracy, and finite-horizon trajectory analysis. We will also refine Appendix F.4, incorporating the measured four-term update decomposition into the finite-horizon discussion and clarifying its connection to the pooled-to-local calibration and root-stability bounds in Eqs. (49)–(50).

We sincerely thank the reviewer for these thoughtful and constructive comments. Their careful assessment helped us recognize the importance of a dedicated controlled experiment and motivated a detailed examination of the original optimization trajectories. These additions have substantially strengthened the experimental foundation, theoretical precision, and overall quality of our manuscript.

AuthorsOpenReview ↗

W2 (1/2): Correspondence of Structural Assumptions to Practical Recursive Systems

W2 (1/2): Correspondence of Structural Assumptions to Practical Recursive Systems (Q1)

We sincerely thank the reviewer for these constructive questions. We clarify how assumptions M1–M3 correspond to practical recursive systems, followed by the rationale for our different implementations in W2 (2/2).

1. Correspondence of M1–M3 to Practical Recursive Systems (Q1).

The three assumptions in Appendix F.1 characterize the information dependencies and propagation mechanisms of recursive predictive computation:

  • M1 (Predictive-boundary Markov structure). Conditional on the retained context CC, the joint distribution of predictive events factorizes through adjacent parent–child predictive boundaries. Intuitively, when intermediate boundaries contain the predictive information needed for subsequent stages and screen off additional dependencies from earlier events or other branches, this Markov factorization holds.
  • M2 (Recursive host aggregation). Each stage forms its state from incoming compressed representations and local information, then passes a compressed representation to subsequent stages. Distinct branches satisfy the stipulated conditional-independence conditions, and earlier information influences later computation through these recursive interfaces rather than bypassing them.
  • M3 (Complete-interface screening). I(Sv;RvΓ∣Uv,C)=0I(S_v;R_v^\Gamma\mid U_v,C)=0 requires that incoming compressed messages provide no additional information about adjacent predictive events once the complete pre-compression state and retained context are given.

TGN. A temporal-graph query recursively expands through sampled temporal neighborhoods. Information from distant neighbors must pass through intermediate aggregation interfaces before influencing the final prediction. When these predictive boundaries retain the information required for subsequent prediction and screen off additional historical dependencies, they satisfy the conditional Markov structure in M1. Each layer aggregates and propagates compressed representations, corresponding to M2. For M3, the current pre-compression interface must retain the information in incoming child messages relevant to adjacent prediction; when this screening condition holds, those messages provide no additional predictive information once the complete interface state is known.

CCM. Dialogue history is progressively compressed into memory states reused across subsequent turns. When intermediate predictive boundaries contain the information needed to connect earlier and later predictions, earlier history introduces no additional conditional dependencies beyond these boundaries, corresponding to the sequential case of M1. Combining previous compressed memory with current-turn information to form the next memory state realizes M2. For M3, the complete memory-update interface can explicitly include the incoming compressed memory, making that input informationally redundant once the complete record is given.

Thus, TGN and CCM provide tree-structured and sequential realizations of recursive predictive computation. M1 concerns the conditional distribution of predictive events, M2 specifies recursive information propagation, and M3 characterizes complete-interface screening. These correspondences explain the applicability of the assumptions without treating recursive computation alone as proof of the probabilistic conditions.

Reviewer_7WAHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper the authors propose a principle for minimizing information loss in recursive compression processes (e.g. recurrent context compression in an LLM). They posit that maintaining local predictive sufficiency, the ability of each intermediate compressed representation to maintain the predictive information that the non-compressed state had about adjacent observed outcomes, is beneficial to maintaining end-task performance despite the recursive compression process, and not well captured in current training objectives (reconstruction, straightforward target/end state prediction). They motivate the approach by showing that local predictive sufficiency maintains final target predictability through recursive predictive closure in theory, and they propose a computationally tractable approach to estimate the empirical predictive deficiency in order to optimize for local predictive sufficiency.

To achieve the optimization: they approximate the deficiency as how much worse a linear probe predicts a fixed set of features of the nearby events from the compressed representation versus the uncompressed one, take its gradient with respect to the compression fn parameters to define constraints, and project the original optimizer’s proposed update onto those constraints when it would increase the deficiency beyond a fixed threshold.

They evaluate the approach on temporal graphs (TGN on Wikipedia and UCI), language memory (CCM-merge on DailyDialog and MSC with LLaMA-7B and Gemma-4), vision-language-action control (MemoryVLA on LIBERO-Mem T3), and recursive self-improvement (Meta$^n$ on CO-Bench), demonstrating that incorporating local predictive sufficiency as a constraint during optimization using the proposed formulation improves end-task performance, especially in deeper recursive chains.

Critical strengths
  • The authors propose an interesting and well-motivated idea for improving the accuracy of recursively compressed representations, which are commonly used in many real-world applications including LLM agents and robotic control.
  • The paper provides a theoretical grounding for the proposed approach, as well as empirical results across varied applications in said domains.
Critical weaknesses
  • There are not convincing experiments demonstrating that “controlling local predictive deficiencies controls the discrepancy at the root” empirically, i.e. that the approach works because of the given motivation. Specifically, it’s not clear how accurately the approximation tracks actual predictive deficiency. It’s sufficient to reduce loss, but is it because it’s actually modeling δv\delta_v, or because it’s a useful auxiliary signal that correlates with the task for some other reason?
  • The experimental results only compare to a controlled baseline, where the only thing varied is the specific proposed method, for one task (UCI). For the others, the reported “Ours” result also includes more training, more parameters, etc. than the baseline. There should be a reasonable equivalent to the “task-only training” row for all tasks for the argument that the approach generalizes to be compelling. (This compounds with the point from Q1 as well, below.)
  • The ablation table (Table 2) only reports over a single seed, but the differences between approaches in the table vary by much less than the std as reported over 5 seeds on the same task as far as I can tell (reported in Table 8 in the appendix). I’m not sure the ablation is convincing because of this.
  • The writing of the paper is dense with lots of jargon merged from across a few fields without clear explanation/definition. Terms that are being introduced in the paper need to be defined. Without this, the paper is quite hard to follow for a human.
    • For example, consider the sentence “The root target YρY_ρ is the global relevance variable, since it is ultimately the prediction that compression should preserve”. I understand what a relevance variable is in the context of information bottleneck and the paper. But what do we mean here by “global” relevance variable, global with respect to what? the following clause “since it is ultimately the prediction that compression should preserve” does nothing to help me understand this since I already know that YρY_ρ is the final target (aka the root target) and therefore the thing we want to be able to predict despite compression. It’s redundant with the definition of YρY_ρ from the prev sentence.
    • Please clearly define root-target sufficiency in the main paper. Prop. 1 refers to root-target sufficiency without it being concretely defined (predictive sufficiency is defined in Def. 2.1, but not exactly what is being subbed in. I think part of the difficulty here is that as someone who doesn’t think about actual graph-structured recursive processes much (in LLMs the unrolled computation is pretty typically a chain) the switching between final target (great, makes sense) and root target (wait, is the first or the last state the root here?) without a little more definitions/explanation was hard to follow.
    • For human readers, a figure depicting the overall intuition (tied to the concrete calculations used to actualize it) would be helpful. I’m thinking a figure that corresponds to section 2.2 (predictive preservation under recursive structure) would be great. And/or, it would be very helpful for Figure 3, or perhaps a part of Figure 3 focused on the basic problem setups and concepts from 2.2, to come earlier in the paper. Fig. 1. at first appears to be helpful for understanding the proposed learning approach, but when I try to read it I realize it provides no illustrative value over the text of the paper. First, it could be better anchored in the rest of the paper. As far as I can tell, the (1), (2) labels on the figure don’t correspond to anything numbered in the text or in Fig 3, while the arrows do correspond to below sections, but with no lettered, numbered, or otherwise connection. The caption is basically useless as it’s totally redundant with skimming the text that’s already there in the figure. Rather than just listing the components that go into making it tractable, describing succinctly how they interact would be more useful. Most importantly, the figure is supposed to be depicting a path of transformations of the optimization problem through various assumptions/approximations, but many variables change across each expression, signs flip, and the transformations (arrows) are not labeled with e.g. the substitutions, so it’s really not telling me what’s going on clearly at all. The first expression doesn’t even occur in the main paper, it’s defined in the appendix.
  • The paper is missing some older related work from the RNN literature that seems highly relevant. For example, Trinh et al. (ICML 2018) proposed using local auxiliary losses at intermediate RNN states to similarly minimize information loss over long chains, and Rae et al. (Compressive Transformer, ICLR 2020) proposes recursively compressing long contexts in LMs, comparing different reconstruction losses (i.e. what compression should preserve.) These seem like quite similar prior work and should be discussed, and potentially even used as concrete baselines. It would also be nice to see the basic motivation/framing of the work anchored in e.g. Shalizi & Crutchfield (2001), who define causal states as the minimal sufficient statistic to predict future observations given previous time steps.
Questions for authors
  • It seems like the projection described in Sec 3 as a core component of the proposed approach wasn’t used for the language memory task (buried in experimental details in the appendix.) The ablation on UCI suggests that the aggregate update doesn’t work particularly well compared to the projection. Why was that approach used for just this experiment? (This detail, and an explanation as to why this setting was selected for that dataset alone, should be included in the main text of the paper.)
  • Please define root-target sufficiency explicitly, distinguishing the joint version (Prop. 1) from the per-interface version (Sec. 3.4).
  • How many seeds/runs for SRI, and what is the variance? This is using nondeterministic API models, so I would like to see variance over multiple. Also, do the findings from Table 2 hold under more random seeds?
  • The proposed optimization method seems a bit gnarly and not that computationally efficient (2x training cost). The resulting improvements are not huge. As a result, I don’t envision this approach getting widely incorporated into implementations of recursive compression. Is there a cheaper or lower-hyperparam variant that retains most of the gain, and how sensitive are results to hyperparameters (beyond κ\kappa)? This might be future work, but it also might be clarified through improving some of the existing results (e.g. ablations).
10 comments
AuthorsOpenReview ↗

W1 (1/2): Mechanism Validation

We thank the reviewer for asking whether local predictive preservation translates into root-level information retention. We added a controlled linear-Gaussian recursive experiment that permits direct evaluation of true conditional mutual information (CMI) and root predictive distributions. This provides a numerical test of Propositions 1 and 3, complementing our experiments on practical recursive systems.

Experimental setting. We study depths 8 and 16 under the structural assumptions M1–M3. At each interface, a 10-dimensional local observation and up to two 4-dimensional child messages (zero-padded when absent) form an 18-dimensional input. A shared linear map compresses it to a 4-dimensional message for recursive propagation. The system uses a bounded-width tree. We compute Gaussian information-theoretic quantities from the known joint distribution, while a root-level classification readout provides Test CE and accuracy. We compare Task-only, Auxiliary (additive predictive gradients), and Projection (predictive-feasibility projection) under matched model capacity and training budgets, using five independent seeds and 2,048 held-out test samples per seed.

Table R1. Local information loss and root-level prediction (Moderate setting; mean ± SD over five seeds).

DepthMethodLocal CMI ↓Root CMI ↓Root TV ↓Test CE ↓Acc. (%) ↑
D8Task-only0.5016 ± 0.06830.0128 ± 0.00270.0512 ± 0.00560.3342 ± 0.014784.52 ± 0.75
D8Auxiliary0.4133 ± 0.05360.0093 ± 0.00200.0437 ± 0.00470.3323 ± 0.015084.72 ± 0.77
D8Projection0.4435 ± 0.05360.0108 ± 0.00200.0472 ± 0.00450.3334 ± 0.015784.74 ± 0.80
D16Task-only1.5583 ± 0.42480.0213 ± 0.00840.0658 ± 0.01360.3340 ± 0.012884.86 ± 0.77
D16Auxiliary1.3029 ± 0.32520.0162 ± 0.00620.0573 ± 0.01150.3326 ± 0.013184.97 ± 0.76
D16Projection1.3466 ± 0.38730.0174 ± 0.00710.0593 ± 0.01270.3329 ± 0.011984.96 ± 0.82

Local CMI is the sum of predictive deficiencies across non-root interfaces, ∑v≠ρδv\sum_{v\ne\rho}\delta_v. Root CMI, I(Yρ;U∣ZΓ,C)I(Y_\rho;U\mid Z^\Gamma,C), measures how much additional information about the root target remains available in the original states after conditioning on the compressed trajectory. A smaller Root CMI therefore indicates better preservation of root-target predictive information. Root TV measures the corresponding discrepancy between root predictive distributions.

Proposition 1 (Recursive Predictive Closure). Under M1–M3,

\Longrightarrow I(Y_\rho;U\mid Z^\Gamma,C)=0.$$ Local CMI measures the information about adjacent predictive events lost at each compression interface. Root CMI captures the predictive information about the final target that remains accessible from the original states given the compressed representations. Both are computed analytically from Gaussian conditional distributions. Analytically sufficient compressors yield numerically zero local deficiencies and root CMI. For learned compressors, Auxiliary reduces Local CMI by **17.6% (D8)** and **16.4% (D16)**, accompanied by **27.0%** and **23.9%** lower Root CMI. Projection also reduces both quantities at each depth. The Local CMI reductions are consistent across all five paired seeds. **Proposition 3 (Root Predictive Stability).** The accumulated local deficiency bounds root predictive discrepancy: $$\mathbb E\left[ \operatorname{TV}\left( P(Y_\rho\mid U,C), P(Y_\rho\mid Z^\Gamma,C) \right) \right] \leq \sqrt{\frac12\sum_{v\ne\rho}\delta_v}.$$ Direct evaluation shows that Auxiliary reduces Root TV by **14.7% (D8)** and **13.0% (D16)**; Projection also improves it at both depths. Analytically sufficient compressors, controlled perturbations, and learned compressors satisfy the theoretical bound. Both methods also lower mean Test CE and modestly improve accuracy. The directly computed CMI and TV results provide evidence that the observed gains are consistent with the proposed predictive-preservation mechanism. **Optimization instantiations.** Both Auxiliary and Projection improve local predictive preservation and root-level prediction in this controlled experiment. The results suggest that the mechanism can be realized through different update rules. We will discuss the choice between additive gradients and feasibility projection, together with the relevant empirical ablations, in our separate response to **Q1**.
AuthorsOpenReview ↗

W1 (2/2):Approximation Accuracy and Manuscript Revision

The reviewer also asks how faithfully the empirical predictive deficiency reflects the theoretical criterion. In particular, the approximation should be validated to establish whether its improvements reflect reduced predictive deficiency. Since exact CMI is generally intractable in high-dimensional systems, we use a controlled linear-Gaussian recursive model, where the relevant information-theoretic quantities can be evaluated analytically, to audit the successive approximation steps.

Appendix F.3, Eq. (22), describes the approximation chain:

\longrightarrow \Delta_v^{\mathrm{bal}} \longrightarrow \widehat D_{v,B}.$$ These steps proceed from the ideal local deficiency to a finite predictive-test quantity, its balanced predictive-value form, and finally the empirical estimate. Corollary F.1 provides the calibration conditions and residual terms connecting the resulting population criterion to the ideal local deficiency. We audit three components: the Gaussian CMI expansion, the Bayes-to-Ridge predictive-value approximation, and pooled empirical estimation. **Table R2. Numerical Accuracy of the Predictive-Deficiency Approximation.** | Component | Metric | D8 | Relative error | D16 | Relative error | |---|---|---:|---:|---:|---:| | **Gaussian CMI expansion** | First-order MAE | 0.001274 | 4.05% | 0.002782 | 5.87% | | | Mean second-order remainder bound | 0.001354 | — | 0.003032 | — | | **Bayes-to-Ridge approximation** | Predictive-value difference MAE | 0.0000215 | 0.147% | 0.0000247 | 0.104% | | | Linear-predictor error budget ($\epsilon_{\mathrm{lin}}$) | 0.0001266 | — | 0.0001274 | — | | **Pooled estimation ($B=2048$)** | Empirical MAE | 0.002304 | 4.33% | 0.001413 | 2.60% | Relative errors are calculated by dividing each MAE by the mean magnitude of its corresponding reference quantity. The reference quantities are the exact Gaussian CMI, Bayes-optimal predictive-value difference, and population pooled predictive-value difference, respectively. The second-order bounds and error budgets are reported separately from measured errors. **Gaussian CMI expansion.** In the Gaussian model, the exact CMI is calculated from the log-determinant of conditional covariance matrices. Its first-order trace approximation has relative errors of **4.05% (D8)** and **5.87% (D16)**. The observed remainders are within the corresponding second-order bounds. This provides an analytical check of the CMI approximation. **Bayes-to-Ridge approximation.** The MAE between Bayes-optimal and Ridge-based predictive-value differences is $2.15\times10^{-5}$ at D8 and $2.47\times10^{-5}$ at D16, corresponding to relative errors of **0.147%** and **0.104%**, respectively. Both measured errors are below the corresponding linear-predictor error budgets $\epsilon_{\mathrm{lin}}$. **Pooled empirical estimation.** At $B=2048$, the population pooled predictive-value differences are $0.053251$ (D8) and $0.054404$ (D16). The corresponding empirical MAEs are $0.002304$ and $0.001413$, yielding relative errors of **4.33%** and **2.60%**. This quantifies the finite-sample estimation accuracy of the pooled statistic. Together, these audits quantify three distinct approximation errors: **4–6%** for the Gaussian first-order expansion, approximately **0.1%** for the Bayes-to-Ridge approximation, and **2.6–4.3%** for pooled estimation at $B=2048$. These measurements, together with Corollary F.1, support the use of the empirical predictive-deficiency objective in the audited setting. **Manuscript revision.** We will add an experimental subsection, *Analytical Verification in Linear-Gaussian Recursive Systems*, specifying the data-generating process, recursive topology, compressor, and evaluation protocol, with an explicit explanation of assumptions M1–M3. We will include Tables R1–R2 and depth-wise Local CMI profiles for D8 and D16. This addition will make the relationship between local predictive preservation, root-level prediction, and approximation accuracy clearer and easier to verify. **Connection to the reviewer's comments.** Together, W1 (1/2) and W1 (2/2) address the **first item under Critical Weaknesses (CW1)**: the lack of direct empirical evidence linking local predictive deficiencies to root-level discrepancies, and the need to validate the accuracy of the proposed approximation. W1 (1/2) provides the direct mechanism evidence through analytically computed CMI and root predictive distributions, while W1 (2/2) quantifies the approximation errors involved in the practical objective. We thank the reviewer for prompting this additional analysis.
AuthorsOpenReview ↗

W2: Controlled Comparisons Across Applications

Our UCI experiments already provide comprehensive experimental evidence and controlled analyses. For the other three applications, we conduct targeted comparisons to assess whether the improvements stem from predictive preservation rather than additional parameters, training budgets, or factors unrelated to our core contribution.

DailyDialog (Language Memory). We add two controls using the same CCM-Merge host and trainable components: Task-only, without predictive-preservation supervision, and Single-event, using single-event rather than joint-event supervision. All methods are evaluated on the same pooled cohort over three independent seeds.

History depthCCM-MergeOursTask-onlySingle-event
L=4L=45.985.86 ± 0.035.90 ± 0.085.92 ± 0.08
L=6L=65.855.71 ± 0.055.74 ± 0.105.75 ± 0.10
L=8L=85.805.65 ± 0.075.68 ± 0.115.68 ± 0.12
L=11L=115.725.55 ± 0.125.59 ± 0.145.60 ± 0.14
L=13L=135.635.38 ± 0.125.40 ± 0.145.41 ± 0.15

PPL ↓\downarrow; mean ±\pm SD over three seeds for the trainable configurations. Ours achieves the lowest mean PPL across all depths, supporting the contribution of joint predictive preservation beyond additional training or single-event supervision.

Vision-Language-Action (VLA). We additionally compare the baseline, Task-only training, and our full method under the same evaluation protocol:

MethodT3 SR (%) ↑\uparrowT3 SGC (%) ↑\uparrow
Baseline0.0017.50
Task-only5.0021.10
Ours15.0028.30

Due to the substantial runtime and computational demands of VLA experiments, these additional controls were evaluated with one seed; we intend to extend them to three seeds in the revised manuscript.

Self-Recursive Improvement (SRI). SRI operates through discrete historical-trace selection rather than trainable compression, so a conventional Task-only optimization baseline is not directly applicable. Instead, Metan\mathrm{Meta}^{n} and Ours use matched backbones, initializations, candidate-generation opportunities, and recursive-search budgets. The comparison therefore focuses on the contribution of the retention strategy without granting additional model capacity or search opportunities.

Summary. These complementary controls strengthen the evidence that the improvements arise from our predictive-preservation design rather than merely additional capacity or computation, directly addressing Weakness 2 (W2).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We revised the manuscript to test the link between local predictive preservation and root prediction, strengthen the experimental controls, and clarify how the same criterion is implemented across hosts. Our replies to the individual reviewers provide further details.

  • Clarified the three implementations in the Abstract and Sections 1 and 3.3. TGN and MemoryVLA use predictive feasibility projection, CCM uses an additive predictive-gradient update, and Metaⁿ uses discrete predictive retention. Appendices E.2 and E.8 explain the choice and calibration of the CCM update. [7WAH, DvRM, mhxP]
  • Clarified root-target sufficiency for the full compressed trajectory and for a single interface in Sections 2.2 and 3.4. Section 3.1, Appendix F.1, and Figure 9 explain how information passes through the recursion and how the empirical predictive deficiency is constructed. [7WAH, DvRM, mhxP]
  • Added controlled linear-Gaussian experiments at depths 8 and 16 over five seeds. Section 4.1 and Tables 1–2 report analytical local CMI, root CMI and TV, and approximation errors. Appendices C.5 and D.5 give the construction under (M1)–(M3) and the full statistics. [7WAH, DvRM]
  • Expanded the finite-horizon analysis in Appendix F.4 to account for recomputing the reference state. Table 28 quantifies the first-order term, reference-state correction, Taylor remainder, and window refresh on 10 recorded projection trajectories. [DvRM]
  • Updated the UCI depth-3 ablation from one seed to three. Table 4 reports the mean AP and AUC, and Table 26 gives sample standard deviations for all 12 variants. [7WAH]
  • Added three-seed Task-only and Single-event controls for DailyDialog in Table 13. Table 14 reports a separate test-only evaluation with 95% confidence intervals. Table 3 and Appendix C.2 identify the pooled fixed-target protocol used for the main DailyDialog results. [7WAH, DvRM]
  • Added a single-seed Task-only control on LIBERO-Mem T3 in Appendix D.3, evaluated by strict task success. Table 16 reports SR and SGC for the main three-seed MemoryVLA comparison. [7WAH, mhxP]
  • Clarified the SRI protocol in Appendix C.4, including the single paired search per backbone and the three repeated evaluations of the resulting frozen strategies. [7WAH]
  • Added a matched TGN runtime profile in Appendix E.6. Over 40 batches on an A800, vanilla TGN took 25.1 s and TGN with optimized projection took 37.2 s (1.48×). [7WAH]
  • Expanded Appendix B to discuss Trinh et al.'s auxiliary losses, Compressive Transformer, and Shalizi and Crutchfield's causal-state formulation. [7WAH]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.