acceptodds
Under review as a conference paper at ICLR 2027

Sequential Updates Do More Than Remember: Computation Over History in Test-Time Learning

Abstract

Test-time learning is usually viewed as writing information into a fast state. We show that sequential test-time updates can do more: because each update acts on a state transformed by earlier updates, their dynamics can compute jointly and order-sensitively over history. We trace this historical computation through four stages: generation, visibility, behavioral use, and acquisition. For a broad class of smooth sequential updates with -scaled history dependence, an interaction involving distinct histories cannot appear before order , and its leading order-sensitive structure is determined by update chronology. In an exactly solvable DeepLinear specialization, these interactions survive in the predictor but attenuate rapidly with support; nuisance-cancelling trajectory contrasts expose rather than amplify them. The same chronology-sensitive structure persists in nonlinear learners and across two sizes of the official pretrained Test-Time Training MLP (TTT-MLP). Task supervision then recruits causal pairwise composition through the evolving fast state in pretrained TTT-MLP; separately, ControlledTTT reveals functional contributions at support two, three, and four. Finally, dense training trajectories show that deeper historical support reaches stable behavioral usefulness later, consistent with a conditional small-step timescale mechanism. Together, these results identify sequential test-time adaptation as a mechanism not only for storing history, but for computing over it.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time learning, test-time training, fast weights, sequential updates, historical computation, higher-order interactions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UfKPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyzes how state-dependent sequential updates generate joint, order-sensitive computation over history. It establishes a small-step historical-support bound, characterizes chronology-sensitive contrasts, and derives exact survival and attenuation results for a deep linear learner. Experiments extend the analysis to nonlinear learners and controlled interfaces of pretrained TTT-MLP checkpoints. Task-supervised recruitment and state interventions demonstrate behavioral use, while ControlledTTT experiments identify contributions involving up to four histories. Training trajectories show later acquisition of higher-support contributions, accompanied by a conditional frozen-feature timescale analysis.

Critical strengths
  • The theoretical analysis distinguishes the generation, survival, and visibility of history interactions, with explicit treatment of cancellation. The exact predictor-level results provide useful insight into what sequential updates can represent.
  • Reset and donor-state interventions, Dynamic-Stale controls, and repeated runs provide meaningful evidence that task-trained learners use history-dependent fast states. These experiments strengthen the paper beyond a purely algebraic analysis.
Critical weaknesses
  • The connection between the formal mechanism and behavioral gains remains incomplete. The theoretical coefficients and finite-input functional-support ablations are distinct objects; the reported interventions establish fast-state use but do not sufficiently identify which predicted chronology-sensitive interactions drive the observed accuracy gains. This limits the explanatory reach of the combined contribution.
  • The explanation of acquisition timing remains suggestive. Frozen-mode squared-loss gradient descent differs substantially from end-to-end AdamW training, and Appendix U.7 finds no consistent next-step optimizer ordering. The observed hierarchy therefore lacks direct evidence tying it to the proposed small-step mechanism.
  • The demonstrated behavioral scope is narrow: structural pretrained probes use controlled interfaces, recruitment requires task-specific fine-tuning, and higher-support results rely on a separate synthetic learner. Although these settings are informative, they leave the significance of the identified mechanism for ordinary pretrained inference insufficiently established.
Questions for authors
  1. Appendix U defines support components using a scalar correct-class margin. After ablation, is accuracy recomputed by decomposing all class logits and taking argmax, or inferred from the scalar margin? Please clarify, since a positive margin against the mean incorrect logit does not guarantee a correct multiclass prediction.

  2. Which part of the support-three and support-four behavioral benefit can be linked specifically to sequential recomputation, given that Stale evaluation can also contain higher-support terms? Do the relevant ablation costs persist with length-preserving counterfactuals that control token positions and write batching?

  3. Are the D1/D2 thresholds used to select R* evaluated on development examples disjoint from the reported test sets? Since the intervention summaries cover the seven recruited models, do reset damage and donor-follow effects also persist at fixed-budget final checkpoints across all eight runs?

  4. Does varying the inner step size change acquisition timing in the direction predicted by the frozen-mode analysis, after accounting for learned gates and overall learning progress? Given the unresolved AdamW ordering in Appendix U.7, this would help assess whether the proposed mechanism explains the observed timing hierarchy.

  5. How much of the pretrained chronology-sensitive signal survives when learned projections and native write batching are retained? The controlled probes and task-specific recruitment provide complementary evidence; please clarify what evidence connects the measured signal to behavior before task-specific fine-tuning.

Reviewer_SMn6OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
1
Read review
Paper summary

I believe this work studies how non-commutativity of update rules -- i.e. the dependence on the future induced by the previous states -- plays a role in learning dynamics during test-time-updates of fast weights. In full honesty I found the work extremely difficult to understand.

Critical strengths

The topic of how computations are built up sequentially during test-time-updates is extremely interesting, but I must be honest and say that it is difficult to understand the precise contribution of this work.

Critical weaknesses

This paper is simply very difficult to understand. The information density is very high and in my opinion insufficient work was done in making it readable to a human reader.

For example, some concrete issues:

  1. I note that the word "history" here seems to be used in a non-standard manner; it appears to refer to the instantaneous tokens that update a given fast state at a particular timestep along the sequence (p110). More naturally I think a "history" would usually refer to the union of all of these timesteps, this difference in nomenclature needs to be spelled out.
  2. What then is "\eta-scaled history dependence" on line 71? This is again not standard terminology -- does this means that the influence of a given token decays with powers of \eta as we move forward in time? (I think this turns out to be the condition argued on line 126, but I'm not certain if this is precisely what is meant.)
  3. On line 135, I am a little bit confused about whether we really want \Phi to be close to the identity: from line 111, it seems that the update takes the form θi=Φi(θi−1;yi)\theta_i = \Phi_i (\theta_{i-1};y_i) so if \Phi is close to the identity then it would immediately wipe out the entire past history. Perhaps the authors mean that this \Phi nearly leaves invariant θi\theta_i, i.e. that Φi=θi−1+ηXi\Phi_{i} = \theta_{i-1} + \eta X_i? (is this just a typo? I haven't traced through how this later affects the argument)
  4. On line 147, I don't think that the function f is defined before its use and thus its very hard to understand what this means. I also really don't know what "DeepLinear" means here, which is used repeatedly without definition: is this using a linear layer for the fast-state? (Or, as the name suggests, the product of many linear layers?) Knowing exactly what f is would help here. I really could not understand what this crucial terminology means for the paper.
  5. In Theorem 3.1 and the surrounding text, what is a "factor"? What does it mean for a "factor-only interaction" to be a "coordinate artifact" -- there is no notion of coordinate-invariance in this work, how can one say something is a coordinate artifact?
  6. And so on...

I have picked specific examples, but this confusion continues, and I confess that I really could not understand the main points of the paper.

Questions for authors

It seems to me that while significant analysis was done, an LLM may have been heavily used to draft the bulk of this paper. There is not necessarily anything wrong with this, but in such a situation its actually important for significant human input to go into rewriting the paper to make it accessible for a human reader. In my eyes the work needs far more of this human input, and at this point the work is simply completely inaccessible.

Thus my primary suggestion is to rewrite it in a way so that it is easier to understand (this is possibly too large of a task for the scope of this rebuttal).

Meta-suggestions that might be helpful here is:

  1. It seems this work might involve a few concepts used in perhaps slightly non-standard ways (e.g. "factor", "history", etc.) are defined carefully and explained (and motivated!) for a human reader before use.
  2. Similarly as some of the theorems are quite technical, it would be helpful to give "informal" explanations of the technical theorems, explaining how they relate to the main bulk of the exposition.

I would be happy to look at the paper again once this rewriting is done, but I do believe that the work needed is significant.

3 comments

Dear Reviewer,

Thank you again for your earlier feedback. We have substantially revised the manuscript to address the concerns you raised, and provided detailed responses explaining the changes.

As the discussion period is approaching its end, we would greatly appreciate it if you could take another look at the revised manuscript and let us know whether your concerns have been addressed, or if any substantive issues remain.

Thank you for your time and consideration.

Thank you very much for your explanation of my specific points and for your significant rewriting of the paper. I found it significantly more readable this time around, though I still believe that it could be clearer still.

Having now been able to absorb the content in a bit more detail, I do think that there is rather a large gap between the microscopic description of the theory and the more interesting experiments about model behavior in Section 5, i.e. I am not sure that I understand precisely how these are connected. I understand that it is quite difficult to make this connection in general, but I do encourage the authors to emphasize how the technical proofs relate to these results at a level that is more than qualitative, as well as devoting yet more time to the exposition.

I am raising my score to a 2 for weak reject in recognition of the effort that went into reorganizing the paper, but I still cannot recommend this for publication.

Thank you for taking the time to revisit our revised manuscript and for reconsidering your assessment. We sincerely appreciate your constructive engagement and recognition of the improvements made throughout the discussion.

Reviewer_yK87OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper attempts to show analytically and through practical experiments that sequential updates in TTT models do not just store memory in the weights, but that previous updates also affect the computations done on later incoming inputs. The paper studies this through several stages: how these historical interactions are created, whether they survive in the model output and can be detected, whether the model actually uses them for behavior, and when higher-order interactions become useful during training.

Critical strengths

[+] Analytical formulation for sequential computation over inputs - The paper gives a mathematical formulation for how sequential updates can create interactions between different historical inputs. [+] Attempts to validate the analytical findings beyond the simple linear model - The paper does not stop at the DeepLinear analysis, but also checks whether the same type of historical interactions can be observed in nonlinear models and in pretrained TTT-MLP models. This helps show that the theoretical effect is not limited to the simplified linear setting.

Critical weaknesses

[-] The paper is very hard to follow - I found the paper very hard to follow. Different sections use a lot of jargon and mathematical notation without enough introduction or motivation. In many places, ideas are presented in a very complicated and formal way, while a small concrete example could explain the intended message much better, for example in Section 2. The paper does present an overall pipeline, but the connection between the different stages is often difficult to follow, especially when the models, experiments, and definitions change between sections. [-] Unclear impact and significance - The main idea that sequential state-dependent updates can create interactions between previous inputs feels quite intuitive and the paper itself notes that the underlying algebra of sequential composition is classical. The paper provides a formal mathematical analysis of this behavior, but it is unclear what practical insight comes from it. The higher-order behavioral results are mainly shown on very specific synthetic tasks, which makes it hard to understand whether the same behavior is important in realistic TTT models. Overall, it is unclear how the findings can be used to improve model design, training, or inference. [-] The experiments move from realistic models to increasingly synthetic custom settings - The paper starts by showing analytical results on a simple custom linear model, and then progresses to pretrained TTT-MLP, which makes the results more interesting. However, when studying the behavioral use of higher-order interactions, especially support-3 and support-4, it moves to the synthetic setting. The reason for this transition is not clearly explained. It is unclear why similar behavioral experiments are not performed on the pretrained TTT-MLP, and this makes it difficult to understand how much these findings generalize to real TTT models. [-] Not sufficiently self-contained and experimental setups are not clearly introduced - In several sections, the paper presents results or statements without enough explanation of the experiment behind them. For example, Section 4 discusses “Nonlinear updates” and gives specific numbers without clearly introducing the setup. Sections 5.1 and 5.2 introduce D1, D2, grounding, intermediate states, and accuracy results without first giving a simple concrete example of the task. More generally, experiment results are sometimes presented before clearly defining the training protocol, dataset, task, model setup, and evaluation. If these details are in the appendix, the main text should at least summarize them and clearly point to the relevant appendix section. [-] Very narrow related work - The related-work section is very short and mostly mentions relevant directions without discussing them in enough depth. As a result, it is unclear how this work is positioned relative to existing literature and what exactly is new compared to prior views of TTT and sequential adaptation.

Questions for authors

see weaknesses

2 comments

I thank the authors for their detailed response and revision. I have skimmed through the revised manuscript and read the responses and discussion points. My sense is that the manuscript is in a better state, yet could still use a serious pass of clarifications. Aside from that, I still find the work not yet ready for publication, since it lacks a connection between the derived theory of interactions between writes and the actual model behaviour. Based on that, I maintain my reject recommendation (though lifting it from 1 to 2).

Dear Reviewer,

Thank you for taking the time to revisit our revised manuscript and responses. We sincerely appreciate your acknowledgment of the improvements and your willingness to reconsider your assessment.

Although we understand that some scientific concerns remain, we respect your perspective and appreciate your constructive engagement throughout the discussion.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Statement: Scientific Contributions and Assessment

We sincerely thank the Area Chair and all three reviewers for their time and consideration. We appreciate the reviewers' engagement with our responses and revised manuscript, particularly their acknowledgment of the substantial improvements made during the discussion.

As the discussion concludes, we would like to emphasize the scientific contributions of this work and their significance.

Our central contribution is a theoretical understanding of what sequential test-time learning can compute.

Existing perspectives often describe test-time updates in terms of accumulating or adapting information. Our work identifies a richer computational mechanism: successive updates can generate structured interactions among previously stored targets through the evolving model state.

We establish a series of rigorous results that characterize this mechanism, including the additive boundary of sequential updates, the distinction between interactions generated during updates and those introduced at readout, and exact predictor-level results concerning initialization and interaction visibility.

Together, these findings provide a mathematical account of how sequential computation can create and preserve interactions that cannot be understood through a purely additive view of stored information. They offer a new perspective on the representational capabilities of test-time learning, rather than merely proposing another method for improving downstream performance.

The theoretical analysis is complemented by experimental evidence across multiple settings.

Beyond the analytical framework, we investigate the relevant mechanisms through controlled nonlinear experiments and pretrained-model interventions. These experiments provide complementary perspectives on the role of the evolving fast state and connect the theoretical investigation to practical learning systems.

The mathematical results and behavioral experiments address different levels of the same scientific question. A complete quantitative mapping from individual theoretical coefficients to downstream behavior would be a valuable further development, but it is not the central claim of this work. Our contribution is to identify, characterize, and investigate the underlying computational mechanism.

We believe the scientific contribution deserves independent consideration.

We recognize the importance of clear presentation and have substantially rewritten the manuscript to improve its accessibility, organization, and precision. We appreciate that these improvements have been acknowledged during the discussion.

Importantly, the central theoretical results and experimental findings were already present in the original submission. The revisions have clarified the contribution rather than changed its scientific substance.

Throughout the discussion, no specific mathematical error has been identified that invalidates our principal theoretical findings. We therefore believe it is important to distinguish concerns about exposition or the scope of experimental interpretation from the validity, originality, and significance of the theoretical contribution itself.

We respectfully ask the Area Chair to evaluate the work in light of its full scientific contribution, including the established theoretical results, their conceptual implications, and the complementary experimental evidence, rather than relying primarily on the initial numerical assessments.

We remain confident that this research makes an original and substantive contribution to understanding sequential computation in test-time learning, and we hope it will receive favorable consideration for acceptance.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.