Stability of deep test-time-training memories : divergence and silent collapse

Test-time-training (TTT) memories, introduced by Titans and ATLAS, store the past in the weights of a small neural network that keeps learning as the model runs. Each input triggers a gradient step on the memory's own prediction error. The memory updates itself on the fly at a fixed cost. This made long-term memory practical at scale and started a fast-growing line of work. The mechanism is delicate. The memory is a network continuously rewritten by the data it must retain, and its update is repeatedly reported to be unstable. No condition is known for when the update stays bounded. None is known for whether the trained memory stays healthy. For a memory that is itself a deep nonlinear network, neither has been established. We study both questions on two independent platforms and find two mechanisms. First, divergence. An update changes the network so that the next update is larger, and the state blows up. We give a condition that a practitioner can check and that prevents it: bound the size of each write, and keep a positive forgetting rate. Orthogonalising the write direction without bounding its size does not prevent it. Second, collapse. The memory stays numerically stable but loses its diversity. Nearly every write ends up pointing the same way. Collapse is learned during training, and the prediction loss barely moves while it happens. We give the tools to catch both failures: the stability condition above, and a paired retention probe with a rank diagnostic that read memory diversity directly. Whether collapse costs accuracy at scale is open. What we establish is that the prediction loss will not tell you it happened.