- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the inner-loop optimization of nonlinear fast-weight memories in test-time training (TTT). The authors observe that, in a LaCT-based model, fixed-base parallel TTT outperforms sequential TTT on single-needle retrieval tasks. Using a fixed-K/V reconstruction experiment, they argue that nonlinear memories are harder to optimize than linear ones within a single pass. To address this, they propose DeltaTTT, which replaces joint optimization of a two-layer MLP memory with layer-wise learning: each layer regresses the value with its own delta-rule update. This lets both layers use DeltaNet's chunkwise parallel algorithm while keeping exact token-wise updates and a nonlinear readout. On 0.5B models trained on 92.5B tokens with DeltaNet and LaCT backbones, DeltaTTT achieves lower training loss and better single-needle retrieval scores than the recurrent baselines, with higher training throughput than both LaCT variants.
Critical strengths
A simple and efficient method with consistent gains on both backbones. DeltaTTT turns the two-layer memory into two stacked delta-rule regressions, so it can directly reuse DeltaNet's chunkwise parallel algorithm while keeping exact, state-dependent per-token updates and a nonlinear readout. Its training throughput is 1.37× that of full attention, close to DeltaNet's 1.42×. On both backbones, it achieves lower training loss and better single-needle retrieval than the corresponding recurrent baselines; on the LaCT backbone, against a DeltaNet that also uses a learnable initial state, its single-needle score is 50.47 vs. 40.52. Nonlinear TTT has long faced a trade-off between speed and update granularity, and this method is simple to implement and easy to adopt. Even though the layerwise-optimization explanation has not been verified yet, the method's practical value supports acceptance.
Critical weaknesses
-
The motivating finding is not robust, and the core hypothesis is not directly verified. The paper starts from Table 1, where parallel TTT beats serial TTT, but this comparison uses the average over S1–S3. Broken down by task (the LaCT rows of Table 3), the advantage comes almost entirely from S1, where parallel scores 90.85 and serial 44.85. On S2 and S3, serial is actually better (28.10 vs. 15.80 and 11.80 vs. 6.95), and the two have almost the same training loss. S1 also uses repeated noise as the haystack, which makes it the easiest of the three tasks. The paper attributes this result to nonlinear memories being hard to optimize in a single pass, relying mainly on Figure 2, but the learning rates in that experiment were tuned on the 16-epoch results and then fixed (Appendix B). More importantly, DeltaTTT itself is not part of this controlled experiment, so the claim that layerwise learning is easier to optimize in a single pass is never tested directly. Since the chain from finding to hypothesis to method is the main thread of the paper, this gap weakens its central claim about optimization difficulty.
-
The sources of the gains are not controlled, which makes it hard to attribute them to layerwise optimization. DeltaTTT differs from the baselines in several ways at once. Each head has two 64×64 states, A and B, which is twice the state of DeltaNet, yet the paper has no state-matched baseline. Compared with LaCT, the memory also changes from SwiGLU to a two-layer SiLU MLP, and the update granularity changes from once every 512 tokens to every token. Any of these factors could explain a substantial part of the improvement, so the current experiments do not support the conclusion that the gains come from layerwise optimization.
Questions for authors
- On the DeltaNet backbone, the DeltaNet baseline does not use a learnable initial state, yet on the LaCT backbone this alone raises DeltaNet's single-needle score from 25.00 to 40.52. Could you report DeltaNet and GDN results on the DeltaNet backbone with the same learnable initialization as DeltaTTT?
- In Table 5(a), the "selected" Pre variant reports 1.962162 / 54.32, which matches neither DeltaTTT result in Table 3 (1.950079 / 48.78 on the DeltaNet backbone and 1.952344 / 50.47 on the LaCT backbone). Table 8 also reports 1.95 / 48.78 for the same initialization, and Appendix C states that the main-table models use Pre. Which run and configuration does Table 5(a) correspond to? Which backbone is used in Table 5(b)?
- Why does the same DeltaNet memory drop from 44.38 to 25.00 in single-needle score when placed in the LaCT backbone? Were the baselines tuned separately for each backbone? Delta MLP reaches only 4.88, and GDN has a higher training loss than DeltaNet. Were these baselines sufficiently tuned?
- Apart from both layers sharing the same target v, what is the substantive difference between DeltaTTT and the Delta MLP of Irie et al. (2021)?
- What were the measurement conditions for Table 4, including hardware, kernel implementation, batch size, and which backbone the DeltaTTT model uses? LaCT uses 512-token chunks here. What is its throughput at its default chunk size of 2048?