- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies how to parallelize the training of TTT (test-time training) layers with nonlinear MLP memories. With an MLP memory, each token’s gradient depends on all earlier updates, so existing methods such as TTT-MLP and LaCT evaluate all gradients in a block at the block’s starting weights, trading accuracy for speed. The authors propose ParaTTT, which first estimates the hidden-layer gradients for all tokens and then refines these estimates iteratively. Given the estimates, the weight updates reduce to linear recurrences, so each refinement needs only three chunkwise-parallel passes, all built on existing kernels for gated linear attention and the gated delta rule. The exact sequential recurrence is a fixed point of this iteration, and the paper proves geometric convergence for a two-layer memory under sufficient conditions. In practice, the method uses two iterations, together with a delta-rule update for the hidden layer and a centered readout. On 340M and 1.3B models trained on SlimPajama, ParaTTT achieves lower perplexity than Gated DeltaNet and Mamba-2, better results on several recall tasks, and 4.9× faster training than TTT-MLP.
Critical strengths
The parallelization approach is novel:Instead of iterating on the full weight state, ParaTTT iterates on an estimate of the hidden-layer gradient at each token. Given this estimate, the nonlinear recurrence becomes a set of linear recurrences. The method can then directly reuse the chunkwise kernels of gated linear attention and the gated delta rule, and the fixed point of the iteration is the exact sequential recurrence. DEER and ParaRNN apply Newton iterations to the full state. ParaTTT, in contrast, needs neither a stored trajectory of d×d states nor Jacobian matrices. This gives a new angle on parallelizing nonlinear TTT.
Critical weaknesses
-
In principle, more iterations should bring the model closer to the exact recurrence and improve results, but the experiments show the opposite. The paper is framed as an exact parallelization of online gradient descent, but training and prefill actually use only two iterations. When the number of iterations at evaluation increases, the model outputs do move closer to the exact recurrence (Figure 4(b)), but the results get worse. For the 1.3B model, going from 2 to 10 iterations drops SWDE from 62.8 to 44.4. At 3 iterations it is already 58.2, below GDN’s 59.0. LAMBADA perplexity rises from 9.00 to 10.38. Training with 5 iterations does not help consistently either: recall improves, but LAMBADA and 32K NIAH get worse. The authors attribute this to the model adapting to the finite number of iterations used in training. But this means the trained model is the two-iteration approximation, not exact online gradient descent. The main text only discusses the slow decline at 340M. The much larger drop at 1.3B appears only in Table 3 in the appendix. In addition, the recall tasks decode only a few answer tokens, and the state is mostly built by the two-iteration prefill. So good recall with exact recurrent decoding does not show that the mismatch between prefill and decoding is benign. The theory does not support the experiments either. The convergence theorem requires keys with norm at most 1 and a small ϵ. The actual models satisfy neither condition, as the authors acknowledge, yet the introduction still says the iteration provably converges to the exact recurrence.
-
The comparison of cost and benefit against linear recurrences is insufficient, ParaTTT trains at about one third of GDN’s throughput, uses about 2.5 times the memory and has twice the state. Yet the paper has no GDN baseline matched in state size or throughput. Its advantage over GDN also shrinks clearly at 1.3B: the SWDE lead falls from 9.4 points at 340M to 3.8. On long context the order even reverses. ParaTTT is below GDN at all four LongBench lengths, and also lower on NIAH at 16K and 32K. It is therefore unclear whether ParaTTT beats simpler linear recurrences at equal cost, and that is the core of its claim to add expressivity without giving up parallelism.
Questions for authors
- The drop from extra iterations is much larger at 1.3B than at 340M: going from 2 to 10 iterations, SWDE falls by 18.4 and 3.2 points, respectively. What causes this? Could you evaluate long-form generation, where most of the sequence is produced by exact recurrent decoding, to measure the effect of the mismatch between prefill and decoding?
- The LaCT baseline in the main table has its sliding-window attention removed. LaCT uses 2048-token chunks, so without the window, tokens within a chunk see almost nothing of the current chunk, and the model barely works . This row therefore does not show that ParaTTT is better than LaCT. Could you compare the two under matched settings, for example by adding the same sliding-window attention to both, or by giving LaCT smaller chunks so that it does not depend on the window?
- The figures in the main text are small, and text is cut off or overlaps in several places. In Figure 3(c), the y-axis label is cut off, the value 112.1 overlaps the title, and the x-axis labels for LaCT and Mamba-2 run together. In Figure 4(a), the y-axis label is cut off and several annotations overlap the curves or the legend. In Figure 2(c,d), values outside the plotted range are crowded at the left edge and hard to read.
8 comments
Response Part 2
W2: cost and benefit relative to linear recurrences. Much of the recent progress in recurrent architectures has come from efficient parallel primitives for linear recurrences, such as Mamba-2 and Gated DeltaNet. Work on TTT-MLP, Titans and M$^2$RNN takes a complementary direction: introducing richer nonlinear memories to support computations that these linear updates struggle with. That additional expressivity brings a computational challenge, because the same parallel algorithms no longer apply directly. ParaTTT brings these directions together by building its nonlinear TTT iterations from existing parallel recurrence primitives.
The comparison with Gated DeltaNet asks what this nonlinear memory adds at a common model size and training-data budget. This is also the approach used in Titans (Behrouz et al., 2025) [2] and M$^2$RNN (Mishra et al., 2026) [3]: their main language-model comparisons with Gated DeltaNet do not match total training compute, but compare models at common sizes and data budgets and report throughput separately (Sections 4.1 and 5.2, respectively). Our language-model study follows this approach. We report the additional compute and fast-weight state in Section 4.2 and Appendix G.1.
Our state-tracking experiments illustrate why moving beyond linear updates can be useful. ParaTTT learns parity and the shorter permutation task substantially better across seeds. The accuracies below compare ParaTTT with Gated DeltaNet after 4K training steps, showing the means and ranges over three seeds (Appendix F.1, Table 3):
| Task (length) | Gated DeltaNet (%) | ParaTTT (%) |
|---|---|---|
| Parity (256) | 63.3 (59.7–67.2) | 97.6 (96.1–99.3) |
| Permutation (64) | 38.1 (34.4–43.4) | 62.7 (55.7–69.0) |
Giving Gated DeltaNet four times as many steps raises its permutation accuracy to 47.4%, still below ParaTTT's 62.7% at the original budget. Extra training therefore does not recover the observed state-tracking benefit. The longest permutation task remains unsolved by both models. On MQAR, the models solve the same tested pair counts, so the benefit does not extend to higher associative-recall capacity in that test. This also does not rule out a contribution from ParaTTT's additional fast-weight state.
This motivates our focus on making nonlinear memory efficient. TTT-MLP and LaCT are the most direct baselines for that algorithmic contribution, since they are existing methods for using nonlinear TTT memories. ParaTTT trains 4.9 times faster than TTT-MLP at 340M while achieving lower perplexity and higher recall scores (Tables 1 and 11). The matched-error experiment further shows that it evaluates the same inner MLP more efficiently (Figure 5(a)), and the full LaCT comparison under Q2 shows better perplexity and most recall results with the same attention branch. These comparisons establish the improvement over incumbent nonlinear TTT methods.
The new language-model experiments also clarify where the gains over Gated DeltaNet occur. Training with randomly sampled iteration counts improves long-context retrieval, while the final phase improves SWDE. The following 1.3B results illustrate these gains with prefill and exact recurrent decoding for generated answers; NIAH and LongBench are measured at 32K (Tables 1 and 5):
| Model | LAMBADA ppl. | SWDE | FDA | NIAH | LongBench |
|---|---|---|---|---|---|
| Gated DeltaNet | 10.56 | 59.0 | 27.0 | 17.5 | 17.7 |
| Original ParaTTT | 9.00 | 62.8 | 28.1 | 14.7 | 17.3 |
| Final phase at | 8.77 | 67.8 | 26.0 | 14.8 | 17.8 |
| Sampled | 9.39 | 63.6 | 25.5 | 20.2 | 18.0 |
Q1: why is the degradation larger at 1.3B? One possibility is that the longer 1.3B training run allows more adaptation to a fixed N. The improvement from varying N during training is consistent with this explanation, but model size and training duration change together, so we cannot isolate the cause of the scale difference. W1(a,b) reports how varying during training reduces both the iteration sensitivity and the gap during long exact recurrent continuations.
Response Part 3
Q2: LaCT under matched settings. We followed your suggestion to give both models the same sliding-window attention branch. This preserves LaCT's intended design: large chunks improve hardware utilization, while window attention supplies local context within each chunk (Zhang et al., 2026 [4]). We therefore retain its published 2048-token chunk size and compare the 340M models with the same 2048-token attention window (Appendix F.2, Table 6). We include both the published LaCT setup and a variant with the same attention Q/K/V normalization used by ParaTTT (“+ norm” below). ParaTTT is ahead of both LaCT variants on perplexity, most recall tasks, LongBench and NIAH, as these results illustrate:
| Metric | LaCT (published) | LaCT + norm | ParaTTT + norm |
|---|---|---|---|
| WikiText ppl. | 25.45 | 24.99 | 24.67 |
| SWDE (%) | 53.2 | 58.4 | 59.8 |
| FDA (%) | 26.8 | 28.4 | 29.9 |
| DROP (%) | 13.7 | 15.1 | 14.6 |
| LongBench, 32K | 8.6 | 10.7 | 14.5 |
| NIAH, 32K (%) | 0.0 | 1.4 | 6.0 |
DROP is the exception.
Q3: figure readability. Thank you for pointing this out. The revised figures incorporate the preprint's fixes to the cut-off labels and overlapping annotations in Figures 3(c,d), 4(c) and 5(a), corresponding to Figures 2(c,d), 3(c) and 4(a) in the submission.
[1] https://proceedings.mlr.press/v267/sun25h.html
Response Part 1
We thank Reviewer AJoJ for the detailed feedback. Your concerns motivated the new training recipes, direct measurements of the prefill–decode mismatch, and comparison with the full LaCT hybrid. We address each point below. Please also see our general response for a summary of the revisions.
W1(a): degradation with more iterations. Decoding uses exact updates, but starts from the memory state produced by approximate prefill. Increasing N changes the prompt computation and that starting state, while leaving the decoding update unchanged. Since the model's weights were optimized through the N=2 computation, making prefill more accurate can still hurt its predictions.
We now distinguish finite-iteration training and prefill from exact recurrent decoding in the abstract and introduction, and discuss the larger degradation at 1.3B in Section 4.1.
To address this mismatch, we tested a final training phase at N=5 and randomly sampling N per layer throughout training. Both make performance less sensitive to the evaluation-time iteration count, as the changes from N=2 to N=10 show below for the 1.3B models (Table 4):
The 340M result you cite uses a fixed N=5 throughout training. The new sampled-N model instead varies the iteration count during training; at 1.3B, it improves NIAH and LongBench at every tested length while retaining N=2 prefill (Tables 1 and 5).
W1(b): short answers do not establish that the mismatch is harmless. Our models are pretrained base language models, so we evaluate long continuations using held-out perplexity, following standard practice in work such as TTT-MLP (Sun et al., 2025) [1]. After N=2 prefill, we process the continuation token by token with the exact recurrence and compare it with N=2 prefill of the same tokens. Both paths receive the reference tokens, which isolates the effect of switching to exact decoding over long continuations.
Both training changes substantially reduce the perplexity gap during exact recurrent evaluation. On a 16K continuation after 16K prefill, sampling N during training reduces the relative gap from 10.4% to 1.7%. The table below also includes exact processing of a whole 4K document; all values are relative perplexity increases over N=2 prefill at 1.3B (Appendix F.4, Table 8):
W1(c): differences between the theory and experiments. You are correct that the original theorem did not cover the experimental model. We have extended the proof to include the hidden-layer delta rule and centering (Theorem 2 and Corollary 1, Appendix B). Section 3.3 now also states the remaining differences explicitly.
The proof assumes keys with norm at most one, a soft clip and sufficient contraction conditions. The experiments use RMS-normalized keys and a hard projection, and the measured gates and learning rates fall outside the contraction conditions (Appendix F.5). We therefore present the theorem as a sufficient guarantee, and assess the trained models' agreement empirically: Figure 5(b,c) covers short sequences, followed by the longer-document tests in Appendix F.4.