- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the online learning and retrieval properties of TTT layers and Delta-style memories. It establishes regret and retrieval guarantees for RLS-based methods, identifies limitations of constant-step Delta updates, and experimentally evaluates their performance under stable and drifting settings.
Critical strengths
Connecting TTT with online learning is a natural and promising direction. I find the formulation of TTT from an online learning perspective is interesting and potentially valuable for developing a deeper theoretical understanding of TTT mechanisms.
Critical weaknesses
However, I have several concerns regarding the paper's formulation and contributions:
-
The paper lacks sufficient background and a clear problem formulation. I suggest introducing the problem setup and existing update rules (e.g., Delta-rule memories) early in the paper.
-
Theorems 1–3 appear incremental, with relatively straightforward proofs. It remains unclear whether the TTT setting introduces new theoretical challenges or requires techniques beyond standard online learning analysis.
-
The experiments focus primarily on RLS and Delta-style updates, leaving the evaluation rather narrow. I encourage comparisons with a broader range of fast-weight methods [1, 2]. Additionally, some appendix experiments, such as FD-sketched RLS, seem peripheral to the paper's main contributions and add limited empirical value.
[1] Schlag, I., Irie, K., and Schmidhuber, J. Linear Transformers Are Secretly Fast Weight Programmers. ICML 2021.
[2] Feng, G., Luo, S., Hua, K., et al. In-Place Test-Time Training. ICLR 2026
Questions for authors
See Weaknesses.
Discussion deadline approaching
With the discussion deadline approaching today, we thank Reviewer fFsA for your response and kindly invite reviewers iMNC and FFAw to share your feedback on our rebuttal. We would like to know if any concern remains, or if any of you would consider updating your assessment.