- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This is a paper which studies the stability of test-time-training memories, explaining some conditions which provide diagnostics of whether (a) the updates to the memory are divergent, and (b) the memory is collapsing to a 1d subspace. Unfortunately the work is extremely difficult to read and I find it hard to suggest acceptance.
Critical strengths
This is a very interesting thing to study: stability of RNNs is something that has a long history in the literature, and the non-linear generalization that is important for TTT memories of the type discussed here is really of intrinsic interest. Unfortunately I do not feel that this problem is well addressed in this work.
Critical weaknesses
The primary issue is simply that the information-density of the work and the poor exposition make it extremely hard to read it and to understand what has been done. It seems clear this was mostly LLM written. I usually take the stance that there is nothing wrong with using an LLM to draft the work, but when doing this one must carefully study and edit the output text to make sure it is suitable for a human reader. I do not believe this was done to the extent needed in this case.
A few specific examples for concreteness:
- Proposition 1 on line 215 derives a stability condition. One wants to compare this stability condition to the previous literature, in particular that of Tang et. al 2026, which did a similar analysis. Yet this comparison is actually done way back on line 118, before Proposition 1 has even appeared in the text or any of the symbols on line 118 have been defined. It is very hard to follow the logical flow.
- Are L R, \eta_{-}, \alpha_{+} defined before they are used on line 244? I can't seem to find them.
- On line 255, the word "self-referential" is used to describe the instability, as a discovery -- but isn't every non-linear instability of this sort? For example if one performed a more standard thing, e.g. forced an an RNN with a nonlinear activation outside of its stability region, would the same effect happen? (More to the point, is this section fundamentally different from things that happen for an RNN? It might be -- but precisely how? etc.)
- On line 257, there is the line "This places the published family." I really don't understand what this sentence means at all.
This continues, though I will stop giving examples.
After several re-readings I do believe I understood most of the main points of the work, but this required considerable effort and I am not 100% sure.
Questions for authors
I really would like to suggest that the authors restructure the exposition of the results to make it easier to follow and digest the scientific results.
Some suggestions are:
- At the moment the logical flow is difficult to follow: as usual for mostly LLM writing, the text is extremely information-dense, and minutiae appear to be given the same emphasis as important and novel results. In my opinion this has to be carefully rewritten with significant human input.
- It is important to carefully contrast the results found with the earlier literature; this has to be done in a way that puts the results in a proper context (e.g. by first reviewing any pertinent RNN-stability literature on the same problem, then explaining how the nonlinearities here and the novel form of the update change the problem, etc., comparing the specific formulas found with the closest specific results in the earlier literature, etc.) Again I believe this requires significant human input.
- Finally, I suggest a more explicit separation between theory and experiment: i.e. at the moment theoretical arguments such as those around line 366 are scattered within the experimental section. It would be cleaner if the theory section provided a clear hypothesis which the experimental section cleanly tested.
I look forward to reading an improved version but I should stress that really significant rewriting would be necessary for me to improve my score. I do believe that the scientific problem is very interesting and encourage the authors to work on the exposition as needed.
Response to Reviewer UXw2 : rewrite plan and the RNN question
We thank the reviewer for the precise and detailed feedback. We are very glad that you find the topic interesting. We are also sincerely sorry that you found the paper too dense and not clear enough. We are preparing a revision that improves the clarity without radically changing the content and conclusions. We will upload it in the coming days. The structure changes, and we add new measurements. We will do our best to be clear and rigorous, and to change the structure so that the important elements are easier to extract.
As you suggest, the main text will keep only three results:
The rest moves to the appendix. The theory section will state hypotheses, and each experiment will test one of them.
The RNN question
In an RNN, the state is the vector h_t that summarises the sequence read so far, and the transition is h_{t+1} = σ(W h_t + U x_t). The weights W are fixed during the forward pass. With σ = tanh, the state stays bounded no matter what. If the network is forced outside its stability region, it oscillates or becomes chaotic, but its forward pass does not blow up. What blows up are the gradients of backpropagation (Pascanu et al., 2013). With σ linear or ReLU, the state can grow, but at most exponentially, at a rate set by W.
In the memory we study, the state is the weight matrix W_t of a small two-layer network. The transition is a gradient step on the inner loss: W_{t+1} = (1 − α) W_t − θ ∇L(W_t). For two layers and a quadratic loss, this gradient grows like the cube of the weights (it is exactly cubic when σ is linear). The Jacobian of the transition is (1 − α) I − θ H(W_t), where H is the curvature of the inner loss, and this curvature grows like ‖W_t‖². The local growth rate is therefore set by the state itself, not by fixed weights. Two consequences follow. The fixed point is stable only within a finite basin. Outside the basin, the growth is faster than exponential, and the state becomes infinite within a single forward pass. For example, gradient descent on x⁴/4 reads x ← x(1 − θx²). The origin is stable, the basin is |x| < √(2/θ), and far outside the basin log|x| roughly triples at each step.
Part of this analysis is classical. The local stability region of our Proposition 1 is that of LMS with momentum and leakage. Exceeding a local threshold is not divergence (Pascanu et al., 2013; Cohen et al., 2021). What is specific to this memory is that the curvature depends on the state. A threshold on the step θ therefore holds only at the state where it was computed. When the state grows, the curvature grows, and the same step becomes too large. No step fixed in advance can therefore guarantee the whole trajectory.
The closest comparison is the LMS filter and its normalised version (Nagumo and Noda, 1967). LMS learns a weight vector by gradient descent on an input x. Its step is stable as long as it stays below 2/‖x‖². This threshold falls as the input grows. Normalised LMS therefore divides the step by ‖x‖², and the threshold no longer depends on the input. Our memory has two layers, and each layer has its own input. The first layer receives the key k. Its norm ‖k‖ is fixed by key normalisation, as in normalised LMS. The second layer receives h = σ(W₁ k). Its norm grows with the state, and nothing normalises it. Bounding the size of each write plays that role for the second layer. This is the content of Lemma 1. It is also why orthogonalisation alone is not enough: it does not bound the size. The rewritten version will place our results relative to this literature before presenting them.
The examples
We also thank you for the precise examples. The comparison with Tang et al. now comes after Proposition 1: their condition λ_max(H) < 2 is the special case η = α = 0 of ours. The symbols L, R, η₋ and α₊ belonged to Corollary 2; the rule is now stated without them, its step margin leaves the main text, and the Corollary stays in the appendix with its definitions. The word "self-referential" is removed. The sentence "This places the published family" meant that this explains why ATLAS trains stably while the raw Titans-style rule diverges above a step threshold. It is rewritten that way.