- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper analyzes how state-dependent sequential updates generate joint, order-sensitive computation over history. It establishes a small-step historical-support bound, characterizes chronology-sensitive contrasts, and derives exact survival and attenuation results for a deep linear learner. Experiments extend the analysis to nonlinear learners and controlled interfaces of pretrained TTT-MLP checkpoints. Task-supervised recruitment and state interventions demonstrate behavioral use, while ControlledTTT experiments identify contributions involving up to four histories. Training trajectories show later acquisition of higher-support contributions, accompanied by a conditional frozen-feature timescale analysis.
Critical strengths
- The theoretical analysis distinguishes the generation, survival, and visibility of history interactions, with explicit treatment of cancellation. The exact predictor-level results provide useful insight into what sequential updates can represent.
- Reset and donor-state interventions, Dynamic-Stale controls, and repeated runs provide meaningful evidence that task-trained learners use history-dependent fast states. These experiments strengthen the paper beyond a purely algebraic analysis.
Critical weaknesses
- The connection between the formal mechanism and behavioral gains remains incomplete. The theoretical coefficients and finite-input functional-support ablations are distinct objects; the reported interventions establish fast-state use but do not sufficiently identify which predicted chronology-sensitive interactions drive the observed accuracy gains. This limits the explanatory reach of the combined contribution.
- The explanation of acquisition timing remains suggestive. Frozen-mode squared-loss gradient descent differs substantially from end-to-end AdamW training, and Appendix U.7 finds no consistent next-step optimizer ordering. The observed hierarchy therefore lacks direct evidence tying it to the proposed small-step mechanism.
- The demonstrated behavioral scope is narrow: structural pretrained probes use controlled interfaces, recruitment requires task-specific fine-tuning, and higher-support results rely on a separate synthetic learner. Although these settings are informative, they leave the significance of the identified mechanism for ordinary pretrained inference insufficiently established.
Questions for authors
-
Appendix U defines support components using a scalar correct-class margin. After ablation, is accuracy recomputed by decomposing all class logits and taking argmax, or inferred from the scalar margin? Please clarify, since a positive margin against the mean incorrect logit does not guarantee a correct multiclass prediction.
-
Which part of the support-three and support-four behavioral benefit can be linked specifically to sequential recomputation, given that Stale evaluation can also contain higher-support terms? Do the relevant ablation costs persist with length-preserving counterfactuals that control token positions and write batching?
-
Are the D1/D2 thresholds used to select R* evaluated on development examples disjoint from the reported test sets? Since the intervention summaries cover the seven recruited models, do reset damage and donor-follow effects also persist at fixed-budget final checkpoints across all eight runs?
-
Does varying the inner step size change acquisition timing in the direction predicted by the frozen-mode analysis, after accounting for learned gates and overall learning progress? Given the unresolved AdamW ordering in Appendix U.7, this would help assess whether the proposed mechanism explains the observed timing hierarchy.
-
How much of the pretrained chronology-sensitive signal survives when learned projections and native write batching are retained? The controlled probes and task-specific recruitment provide complementary evidence; please clarify what evidence connects the measured signal to behavior before task-specific fine-tuning.
Dear Reviewer,
Thank you again for your earlier feedback. We have substantially revised the manuscript to address the concerns you raised, and provided detailed responses explaining the changes.
As the discussion period is approaching its end, we would greatly appreciate it if you could take another look at the revised manuscript and let us know whether your concerns have been addressed, or if any substantive issues remain.
Thank you for your time and consideration.