acceptodds
Under review as a conference paper at ICLR 2027

ParaTTT: Parallelizing Test-Time Training Layers with Deep Memories

Abstract

Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Moreover, their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Starting from a guess of the hidden-layer gradients, each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The fixed point is the exact online gradient descent recurrence, while decoding requires no refinement iterations. We prove geometric convergence for a two-layer nonlinear memory under sufficient contraction conditions. In practice, two iterations suffice. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory 32× faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains 4.9× faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.

Primary area:
foundation or frontier models, including LLMs
Keywords:
TTT, nonlinear recurrences, sequence mixers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_AJoJOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies how to parallelize the training of TTT (test-time training) layers with nonlinear MLP memories. With an MLP memory, each token’s gradient depends on all earlier updates, so existing methods such as TTT-MLP and LaCT evaluate all gradients in a block at the block’s starting weights, trading accuracy for speed. The authors propose ParaTTT, which first estimates the hidden-layer gradients for all tokens and then refines these estimates iteratively. Given the estimates, the weight updates reduce to linear recurrences, so each refinement needs only three chunkwise-parallel passes, all built on existing kernels for gated linear attention and the gated delta rule. The exact sequential recurrence is a fixed point of this iteration, and the paper proves geometric convergence for a two-layer memory under sufficient conditions. In practice, the method uses two iterations, together with a delta-rule update for the hidden layer and a centered readout. On 340M and 1.3B models trained on SlimPajama, ParaTTT achieves lower perplexity than Gated DeltaNet and Mamba-2, better results on several recall tasks, and 4.9× faster training than TTT-MLP.

Critical strengths

The parallelization approach is novel:Instead of iterating on the full weight state, ParaTTT iterates on an estimate of the hidden-layer gradient at each token. Given this estimate, the nonlinear recurrence becomes a set of linear recurrences. The method can then directly reuse the chunkwise kernels of gated linear attention and the gated delta rule, and the fixed point of the iteration is the exact sequential recurrence. DEER and ParaRNN apply Newton iterations to the full state. ParaTTT, in contrast, needs neither a stored trajectory of d×d states nor Jacobian matrices. This gives a new angle on parallelizing nonlinear TTT.

Critical weaknesses
  1. In principle, more iterations should bring the model closer to the exact recurrence and improve results, but the experiments show the opposite. The paper is framed as an exact parallelization of online gradient descent, but training and prefill actually use only two iterations. When the number of iterations at evaluation increases, the model outputs do move closer to the exact recurrence (Figure 4(b)), but the results get worse. For the 1.3B model, going from 2 to 10 iterations drops SWDE from 62.8 to 44.4. At 3 iterations it is already 58.2, below GDN’s 59.0. LAMBADA perplexity rises from 9.00 to 10.38. Training with 5 iterations does not help consistently either: recall improves, but LAMBADA and 32K NIAH get worse. The authors attribute this to the model adapting to the finite number of iterations used in training. But this means the trained model is the two-iteration approximation, not exact online gradient descent. The main text only discusses the slow decline at 340M. The much larger drop at 1.3B appears only in Table 3 in the appendix. In addition, the recall tasks decode only a few answer tokens, and the state is mostly built by the two-iteration prefill. So good recall with exact recurrent decoding does not show that the mismatch between prefill and decoding is benign. The theory does not support the experiments either. The convergence theorem requires keys with norm at most 1 and a small ϵ. The actual models satisfy neither condition, as the authors acknowledge, yet the introduction still says the iteration provably converges to the exact recurrence.

  2. The comparison of cost and benefit against linear recurrences is insufficient, ParaTTT trains at about one third of GDN’s throughput, uses about 2.5 times the memory and has twice the state. Yet the paper has no GDN baseline matched in state size or throughput. Its advantage over GDN also shrinks clearly at 1.3B: the SWDE lead falls from 9.4 points at 340M to 3.8. On long context the order even reverses. ParaTTT is below GDN at all four LongBench lengths, and also lower on NIAH at 16K and 32K. It is therefore unclear whether ParaTTT beats simpler linear recurrences at equal cost, and that is the core of its claim to add expressivity without giving up parallelism.

Questions for authors
  1. The drop from extra iterations is much larger at 1.3B than at 340M: going from 2 to 10 iterations, SWDE falls by 18.4 and 3.2 points, respectively. What causes this? Could you evaluate long-form generation, where most of the sequence is produced by exact recurrent decoding, to measure the effect of the mismatch between prefill and decoding?
  2. The LaCT baseline in the main table has its sliding-window attention removed. LaCT uses 2048-token chunks, so without the window, tokens within a chunk see almost nothing of the current chunk, and the model barely works . This row therefore does not show that ParaTTT is better than LaCT. Could you compare the two under matched settings, for example by adding the same sliding-window attention to both, or by giving LaCT smaller chunks so that it does not depend on the window?
  3. The figures in the main text are small, and text is cut off or overlaps in several places. In Figure 3(c), the y-axis label is cut off, the value 112.1 overlaps the title, and the x-axis labels for LaCT and Mamba-2 run together. In Figure 4(a), the y-axis label is cut off and several annotations overlap the curves or the legend. In Figure 2(c,d), values outside the plotted range are crowded at the left edge and hard to read.
8 comments
AuthorsOpenReview ↗

Response Part 1

We thank Reviewer AJoJ for the detailed feedback. Your concerns motivated the new training recipes, direct measurements of the prefill–decode mismatch, and comparison with the full LaCT hybrid. We address each point below. Please also see our general response for a summary of the revisions.

W1(a): degradation with more iterations. Decoding uses exact updates, but starts from the memory state produced by approximate prefill. Increasing NN changes the prompt computation and that starting state, while leaving the decoding update unchanged. Since the model's weights were optimized through the N=2N=2 computation, making prefill more accurate can still hurt its predictions.

We now distinguish finite-iteration training and prefill from exact recurrent decoding in the abstract and introduction, and discuss the larger degradation at 1.3B in Section 4.1.

To address this mismatch, we tested a final training phase at N=5N=5 and randomly sampling NN per layer throughout training. Both make performance less sensitive to the evaluation-time iteration count, as the changes from N=2N=2 to N=10N=10 show below for the 1.3B models (Table 4):

TrainingLAMBADA ppl.SWDE (%)
Original (N=2N=2)9.00→10.389.00 \to 10.3862.8→44.462.8 \to 44.4
Final phase at N=5N=58.77→9.568.77 \to 9.5667.8→62.667.8 \to 62.6
Sampled NN9.39→9.479.39 \to 9.4763.6→61.963.6 \to 61.9

The 340M result you cite uses a fixed N=5N=5 throughout training. The new sampled-NN model instead varies the iteration count during training; at 1.3B, it improves NIAH and LongBench at every tested length while retaining N=2N=2 prefill (Tables 1 and 5).

W1(b): short answers do not establish that the mismatch is harmless. Our models are pretrained base language models, so we evaluate long continuations using held-out perplexity, following standard practice in work such as TTT-MLP (Sun et al., 2025) [1]. After N=2N=2 prefill, we process the continuation token by token with the exact recurrence and compare it with N=2N=2 prefill of the same tokens. Both paths receive the reference tokens, which isolates the effect of switching to exact decoding over long continuations.

Both training changes substantially reduce the perplexity gap during exact recurrent evaluation. On a 16K continuation after 16K prefill, sampling NN during training reduces the relative gap from 10.4% to 1.7%. The table below also includes exact processing of a whole 4K document; all values are relative perplexity increases over N=2N=2 prefill at 1.3B (Appendix F.4, Table 8):

TrainingExact 4K document (%)16K + 16K (%)
Original (N=2N=2)5.110.4
Final phase at N=5N=51.93.8
Sampled NN1.11.7

W1(c): differences between the theory and experiments. You are correct that the original theorem did not cover the experimental model. We have extended the proof to include the hidden-layer delta rule and centering (Theorem 2 and Corollary 1, Appendix B). Section 3.3 now also states the remaining differences explicitly.

The proof assumes keys with norm at most one, a soft clip and sufficient contraction conditions. The experiments use RMS-normalized keys and a hard projection, and the measured gates and learning rates fall outside the contraction conditions (Appendix F.5). We therefore present the theorem as a sufficient guarantee, and assess the trained models' agreement empirically: Figure 5(b,c) covers short sequences, followed by the longer-document tests in Appendix F.4.

AuthorsOpenReview ↗

Response Part 2

W2: cost and benefit relative to linear recurrences. Much of the recent progress in recurrent architectures has come from efficient parallel primitives for linear recurrences, such as Mamba-2 and Gated DeltaNet. Work on TTT-MLP, Titans and M$^2$RNN takes a complementary direction: introducing richer nonlinear memories to support computations that these linear updates struggle with. That additional expressivity brings a computational challenge, because the same parallel algorithms no longer apply directly. ParaTTT brings these directions together by building its nonlinear TTT iterations from existing parallel recurrence primitives.

The comparison with Gated DeltaNet asks what this nonlinear memory adds at a common model size and training-data budget. This is also the approach used in Titans (Behrouz et al., 2025) [2] and M$^2$RNN (Mishra et al., 2026) [3]: their main language-model comparisons with Gated DeltaNet do not match total training compute, but compare models at common sizes and data budgets and report throughput separately (Sections 4.1 and 5.2, respectively). Our language-model study follows this approach. We report the additional compute and fast-weight state in Section 4.2 and Appendix G.1.

Our state-tracking experiments illustrate why moving beyond linear updates can be useful. ParaTTT learns parity and the shorter permutation task substantially better across seeds. The accuracies below compare N=2N=2 ParaTTT with Gated DeltaNet after 4K training steps, showing the means and ranges over three seeds (Appendix F.1, Table 3):

Task (length)Gated DeltaNet (%)ParaTTT (%)
Parity (256)63.3 (59.7–67.2)97.6 (96.1–99.3)
Permutation (64)38.1 (34.4–43.4)62.7 (55.7–69.0)

Giving Gated DeltaNet four times as many steps raises its permutation accuracy to 47.4%, still below ParaTTT's 62.7% at the original budget. Extra training therefore does not recover the observed state-tracking benefit. The longest permutation task remains unsolved by both models. On MQAR, the models solve the same tested pair counts, so the benefit does not extend to higher associative-recall capacity in that test. This also does not rule out a contribution from ParaTTT's additional fast-weight state.

This motivates our focus on making nonlinear memory efficient. TTT-MLP and LaCT are the most direct baselines for that algorithmic contribution, since they are existing methods for using nonlinear TTT memories. ParaTTT trains 4.9 times faster than TTT-MLP at 340M while achieving lower perplexity and higher recall scores (Tables 1 and 11). The matched-error experiment further shows that it evaluates the same inner MLP more efficiently (Figure 5(a)), and the full LaCT comparison under Q2 shows better perplexity and most recall results with the same attention branch. These comparisons establish the improvement over incumbent nonlinear TTT methods.

The new language-model experiments also clarify where the gains over Gated DeltaNet occur. Training with randomly sampled iteration counts improves long-context retrieval, while the final N=5N=5 phase improves SWDE. The following 1.3B results illustrate these gains with N=2N=2 prefill and exact recurrent decoding for generated answers; NIAH and LongBench are measured at 32K (Tables 1 and 5):

ModelLAMBADA ppl.SWDEFDANIAHLongBench
Gated DeltaNet10.5659.027.017.517.7
Original ParaTTT9.0062.828.114.717.3
Final phase at N=5N=58.7767.826.014.817.8
Sampled NN9.3963.625.520.218.0

Q1: why is the degradation larger at 1.3B? One possibility is that the longer 1.3B training run allows more adaptation to a fixed N. The improvement from varying N during training is consistent with this explanation, but model size and training duration change together, so we cannot isolate the cause of the scale difference. W1(a,b) reports how varying NN during training reduces both the iteration sensitivity and the gap during long exact recurrent continuations.

AuthorsOpenReview ↗

Response Part 3

Q2: LaCT under matched settings. We followed your suggestion to give both models the same sliding-window attention branch. This preserves LaCT's intended design: large chunks improve hardware utilization, while window attention supplies local context within each chunk (Zhang et al., 2026 [4]). We therefore retain its published 2048-token chunk size and compare the 340M models with the same 2048-token attention window (Appendix F.2, Table 6). We include both the published LaCT setup and a variant with the same attention Q/K/V normalization used by ParaTTT (“+ norm” below). ParaTTT is ahead of both LaCT variants on perplexity, most recall tasks, LongBench and NIAH, as these results illustrate:

MetricLaCT (published)LaCT + normParaTTT + norm
WikiText ppl.25.4524.9924.67
SWDE (%)53.258.459.8
FDA (%)26.828.429.9
DROP (%)13.715.114.6
LongBench, 32K8.610.714.5
NIAH, 32K (%)0.01.46.0

DROP is the exception.

Q3: figure readability. Thank you for pointing this out. The revised figures incorporate the preprint's fixes to the cut-off labels and overlapping annotations in Figures 3(c,d), 4(c) and 5(a), corresponding to Figures 2(c,d), 3(c) and 4(a) in the submission.


[1] https://proceedings.mlr.press/v267/sun25h.html

[2] https://proceedings.neurips.cc/paper_files/paper/2025/file/a4ca07aa108036f80cbb5b82285fd4b1-Paper-Conference.pdf

[3] https://arxiv.org/abs/2603.14360

[4] https://arxiv.org/abs/2505.23884

Reviewer_b3EFOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ParaTTT, a method for parallelizing test-time training with deep nonlinear memories. The key idea is to approximate sequential memory updates through iterative gradient refinement with a fixed point iteration scheme. The authors provide convergence analysis for two-layer memories and show that just two refinement iterations work well empirically, achieving substantial speedups over sequential TTT.

Critical strengths
  1. Parallelizing nonlinear TTT is challenging, and the proposed approach could enable more expressive and efficient memory architectures.
  2. The reported speedups are substantial, and the observation that only two iterations are needed in practice is encouraging.
  3. The convergence analysis provides useful insight into why iterative refinement can approximate sequential updates.
Critical weaknesses
  1. Weak LaCT baseline. The main comparison uses LaCT without sliding-window attention (SWA), which significantly hurts its performance. Since SWA is an important component of LaCT, this makes the comparison less convincing.
  2. Limited scalability evidence. The effectiveness of using only two iterations is mainly demonstrated at 4K context length. It is unclear whether this remains sufficient for longer sequences or deeper memories, especially since the convergence analysis only covers two-layer memories.
  3. Unclear quality-efficiency tradeoff. ParaTTT does not show a clear performance advantage over simpler architectures such as Gated DeltaNet. It is therefore unclear whether the additional complexity is worthwhile in practice.
Questions for authors
  1. Why is LaCT without SWA used as the main baseline? How does ParaTTT compare against the full LaCT architecture under matched compute budgets?
  2. How does the required number of refinement iterations scale with sequence length and memory depth? Does (N=2) remain sufficient at longer contexts, such as 16K or 32K?
1 comment

We thank Reviewer b3EF for the positive assessment of our work! We have added the full hybrid comparison and examined the iteration count at longer contexts. Please also see our general response for a summary of the revisions.

W1 and Q1: comparison with the full LaCT hybrid. We used LaCT without SWA in the main comparison to isolate the contribution of the TTT recurrence from that of attention. Nonetheless, following your suggestion, we now also compare the full hybrids at 340M with the same 2048-token sliding-window attention branch (Appendix F.2, Table 6). We include both the published LaCT setup and a variant with the same attention Q/K/V normalization used by ParaTTT (“+ norm” below).

ParaTTT is ahead of both LaCT variants on perplexity, most recall tasks, and LongBench and NIAH at every tested length, as the selected results below illustrate:

MetricLaCT (published)LaCT + normParaTTT + norm
WikiText ppl.25.4524.9924.67
SWDE (%)53.258.459.8
FDA (%)26.828.429.9
DROP (%)13.715.114.6
LongBench, 32K8.610.714.5
NIAH, 32K (%)0.01.46.0

W2 and Q2: longer contexts and deeper memories. Under the theorem's assumptions, the contraction bound is independent of sequence length. The random-inner-MLP experiment in Appendix I also shows weak length dependence over the tested range.

For the trained models, we evaluated a variant that randomly samples NN per layer during training. It improves long-context retrieval while retaining N=2N=2 prefill, exceeding Gated DeltaNet on NIAH and LongBench at every tested length through 32K. The 4K and 32K results at 1.3B are below (Tables 1 and 5 give all lengths):

ModelNIAH 4KNIAH 32KLongBench 4KLongBench 32K
Gated DeltaNet50.817.517.417.7
Original ParaTTT53.814.716.417.3
Sampled NN58.820.218.818.0

These results show that N=2N=2 remains effective for retrieval beyond the 4K training length. Agreement with the exact recurrence is lower on longer documents, and increasing NN partly closes that gap (Appendix F.4, Table 9).

The algorithm and fixed-point result support deeper memories, but our convergence-rate analysis and experiments cover one hidden layer. Measuring how the required iteration count scales with depth remains future work.

W3: is the extra cost over Gated DeltaNet worthwhile? The added cost supports a richer nonlinear memory: Gated DeltaNet does not close the state-tracking gap even with four times as many training steps (Appendix F.1). Like Titans (Behrouz et al., 2025) [1] and M$^2$RNN (Mishra et al., 2026) [2], our main Gated DeltaNet comparison matches model size and data rather than total training compute. ParaTTT's contribution is to make nonlinear TTT more efficient: at 340M, it trains 4.9 times faster than TTT-MLP with better perplexity and recall (Tables 1 and 11). The comparison under W1 also shows better perplexity and most recall results than the full LaCT hybrid.


[1] https://proceedings.neurips.cc/paper_files/paper/2025/file/a4ca07aa108036f80cbb5b82285fd4b1-Paper-Conference.pdf

[2] https://arxiv.org/abs/2603.14360


Reviewer_HVP5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes ParaTTT, which updates a nonlinear recurrent memory (an MLP) in parallel over the sequence, similar to linear recurrent memories (Mamba-2, DeltaNet, GDN, etc.). Instead of iterating on the fast weights, it iterates on the per-token hidden-layer gradients: given a guess for these, every layer's update becomes a linear recurrence that existing chunkwise kernels can compute, and the gradients are then recomputed from the result. The exact online gradient descent recurrence is a fixed point of this iteration, and without damping it is reached after at most T iterations for T tokens. In practice the models use N = 2 iterations. PTTT is faster and better than TTT-MLP with block approximation (4.9x faster training at 340M, 32x faster at matched error for a single layer), and at 340M and 1.3B it improves perplexity and recall over GDN and Mamba-2, at about 3x lower training throughput than GDN.

Critical strengths
  1. Iterating on the gradients instead of the weights is a good idea. The fast weights are a d x d matrix per token, while the hidden-layer gradients are a d-vector per token, so the unknowns are d times smaller. Once the gradients are fixed, the hidden layer is a GLA/Mamba-2-style scan and the readout is a gated delta rule, so the method reuses existing kernels and never stores the full weight trajectory. Decoding is still one exact gradient step per token. This makes a nonlinear TTT memory trainable at a cost much closer to linear recurrent network than block-based TTT-MLP.

  2. Good results for a pure recurrent model, with useful ablations. At 340M, PTTT beats TTT-MLP with block approximation on the same kind of memory while training 4.9x faster. It has the best recurrent-model perplexity at both scales, with the largest gain on LAMBADA, and beats GDN on three of four recall tasks at both scales. Zero-shot commonsense accuracy is roughly unchanged.

Critical weaknesses
  1. The trained models rely on the N = 2 approximation, not on exact online gradient descent, so the "exact recurrence" framing is overstated. N sets how accurately the per-token TTT updates are computed; exact online GD is the limit. The exactness result (Prop. 2) needs no damping and N >= T, but the models are trained with damping lambda = 0.8 and N = 2, so even the first token's gradient is never exact, and top-1 agreement with the exact recurrence is about 94% (Fig. 4b). More importantly, computing the same updates more accurately at evaluation makes the models worse. At 1.3B, going from N = 2 to N = 10 drops SWDE from 62.8 to 44.4 and raises LAMBADA ppl from 9.00 to 10.38, and LAMBADA is actually best at N = 1 (Table 3).

  2. Gains over GDN are uneven and not compute-matched. At N = 2 a PTTT layer costs 4.7x a GDN layer, 340M training throughput is about 3x lower (37.9k vs 112.1k tok/s), and peak memory is about 2.5x (Table 7). At 1.3B the perplexity gain over GDN is small (Wiki 15.21 vs 15.78)

Questions for authors
  1. Hybrid models. All PTTT models are pure recurrent. Most practical recurrent LMs mix in attention, e.g. Qwen3-Next uses 3 GDN layers per attention layer, and LaCT uses a sliding-window branch. In a hybrid, attention handles much of the recall where PTTT gains the most, while each PTTT layer still costs about 4.7x a GDN layer. Does PTTT still beat GDN in a hybrid setup at similar cost?
  2. More iterations hurting. The paper explains the drop with more iterations as the model adapting to its training N. At 1.3B the drop is large (Table 3: SWDE 62.8 at N = 2, 44.4 at N = 10). Does the model get even worse when the prompt is processed exactly, which is what N = 2 approximates? Decoding uses exact updates, so does this hurt on longer outputs, beyond the short-answer recall tasks?
  3. Equal-cost comparison. Can you compare against GDN at equal training compute or wall-clock (e.g. GDN trained about 3x longer), and report seed variance at 340M?
  4. The 32x speedup. The 32x in Fig. 4a compares PTTT at N = 2 with TTT-MLP at a block size of about 5, chosen to match PTTT's error. TTT-MLP is usually trained with b = 16. What are its speed and error at b = 16? And App. G says TTT-MLP's fast kernel only exists for head dimension 64 and b = 16, with a 2.2x slower fallback otherwise. Which version is timed at b of about 5?
  5. Learned parameters. The theorem needs eta/(1-alpha) to be small (below about 0.05). What values of the inner learning rate eta and decay alpha do the trained models actually use, e.g. their typical range per layer? This would show whether the theorem applies to the trained models.
  6. Inference cost. What are decode speed and state size compared with GDN?
2 comments
AuthorsOpenReview ↗

Response Part 1

We thank Reviewer HVP5 for the constructive review and positive outlook on our work! Your questions led us to clarify the approximation, test exact recurrent evaluation more directly, and expand the efficiency measurements. Please also see our general response for a summary of the revisions.

W1: exactness of the trained model. The abstract, introduction and conclusion now distinguish the exact fixed point and recurrent decoding update from the finite-iteration computation used for training and prefill. Figure 1 also identifies both TTT-MLP's blocks and ParaTTT's iterations as approximations to the sequential trajectory.

Decoding uses exact updates, but starts from the memory state produced by approximate prefill. Increasing NN changes the prompt computation and that starting state, while leaving the decoding update unchanged. Since the model's weights were optimized through the N=2N=2 computation, making prefill more accurate can still hurt its predictions.

To address this mismatch, we tested a final training phase at N=5N=5 and randomly sampling NN per layer throughout training. Both reduce sensitivity to the evaluation-time iteration count, as the changes from N=2N=2 to N=10N=10 show below at 1.3B (Section 4.1, Table 4):

TrainingLAMBADA ppl.SWDE (%)
Original (N=2N=2)9.00→10.389.00 \to 10.3862.8→44.462.8 \to 44.4
Final phase at N=5N=58.77→9.568.77 \to 9.5667.8→62.667.8 \to 62.6
Sampled NN9.39→9.479.39 \to 9.4763.6→61.963.6 \to 61.9

Q2 below reports exact prompt processing and long recurrent continuations.

W2 and Q3: uneven gains and equal compute. Mamba-2 and Gated DeltaNet benefit from efficient parallel primitives. Nonlinear memories such as TTT-MLP, Titans and M$^2$RNN seek greater expressivity, but are harder to parallelize. ParaTTT builds on those existing primitives to make nonlinear TTT more efficient.

Our comparison with Gated DeltaNet asks what nonlinear memory adds at a common model size and data budget. Titans (Behrouz et al., 2025) [1] and M$^2$RNN (Mishra et al., 2026) [2] take the same approach: their main language-model comparisons with Gated DeltaNet do not match total training compute, and report throughput separately (Sections 4.1 and 5.2, respectively). We report this cost in Section 4.2.

The state-tracking experiments illustrate the benefit. After 4K training steps, N=2N=2 ParaTTT and Gated DeltaNet achieve the following accuracies (Appendix F.1, Table 3):

Task (length)Gated DeltaNet (%)ParaTTT (%)
Parity (256)63.397.6
Permutation (64)38.162.7

Giving Gated DeltaNet four times as many steps raises its permutation accuracy to 47.4%, still below ParaTTT's 62.7% at the original budget.

For our algorithmic contribution, the most direct baselines are the incumbent nonlinear TTT methods, TTT-MLP and LaCT. At 340M, ParaTTT trains 4.9 times faster than TTT-MLP with lower perplexity and higher recall scores (Tables 1 and 11). With the same attention branch, it also improves on the full LaCT hybrid in perplexity and most recall tasks (Table 6).

Q1: do the gains persist with attention? Compute constraints prevented us from training interleaved ParaTTT–attention hybrids. Nonetheless, M$^2$RNN (Mishra et al., 2026) [2] provides encouraging evidence: interleaving nonlinear recurrence with attention improves WikiText perplexity over the corresponding Gated DeltaNet hybrid by about 0.5 points at the 7B MoE scale (Section 5.3). These results give us reason to expect similar benefits from ParaTTT; we leave evaluating these hybrids and their cost to future work.

Q2: exact prompt processing and long recurrent continuations. Following your suggestion, we tested exact prompt processing and long teacher-forced recurrent continuations after N=2N=2 prefill. We compared the original model with the two variants introduced under W1: a final training phase at N=5N=5, and randomly sampled NN per layer. Both substantially reduce the perplexity gap with exact recurrent evaluation, with random sampling giving the smallest gap. At 1.3B, the relative increases over N=2N=2 prefill are below for whole 4K documents and a 16K recurrent continuation after 16K prefill (Appendix F.4, Table 8):

TrainingExact 4K document (%)16K + 16K (%)
Original (N=2N=2)5.110.4
Final phase at N=5N=51.93.8
Sampled NN1.11.7
AuthorsOpenReview ↗

Response Part 2

Q4: matched error and TTT-MLP's default block size. We added the requested comparison to Appendix I, Table 13. It holds the inner MLP fixed at head width 64 and compares forward evaluation on 4K tokens. At its default block size, TTT-MLP takes about nine times as long as ParaTTT with two iterations and has more than twice the error:

ConfigurationRelative error (%)Time (ms)
ParaTTT, N=2N=21.32.4
TTT-MLP, b=16b=163.122.2
TTT-MLP, b=4b=41.286.6

The 32$\times$ matched-error result interpolates between measured block sizes, using the faster TTT-MLP implementation at each size. In this batch-1 forward benchmark, the compiled scan with CUDA graphs is faster at small blocks because padding hurts the fused kernel. The 2.2$\times$ slower fallback reported in Appendix G concerns the separate batch-8 forward-and-backward head-dimension sweep; Appendix I details both implementations.

Q5: learned learning rates and decays. We measured them on held-out text and added layer summaries for both model sizes (Appendix F.5, Table 10). The trained models use learning-rate-to-forgetting ratios above the regime covered by the theorem's sufficient bound, as the median values across their 24 layers show below (Table 10 gives the full ranges):

Quantity340M1.3B
Decay α\alpha0.960.96
Readout rate η(2)\eta^{(2)}0.140.11
Hidden rate η(1)\eta^{(1)}0.350.23
η(2)/(1−α)\eta^{(2)}/(1-\alpha)3.94.1
η(1)/(1−α)\eta^{(1)}/(1-\alpha)9.07.5

Although the theorem's guarantee does not apply to these trained models, its conditions are sufficient but not necessary for convergence. Empirically, we find that increasing the iteration count improves their agreement with the exact recurrence, both on short sequences (Figure 5(b,c)) and over whole documents from 4K to 32K (Appendix F.4, Table 9).

Q6: decode speed and state size. We added measurements at both model sizes (Appendix G.1, Figure 7). At batch size 1, ParaTTT's decode latency stays nearly constant with context length and is about 10% above Gated DeltaNet's. Its fast-weight state is also constant in context length: two 128×128128\times128 matrices per head, versus one for Gated DeltaNet.


[1] https://proceedings.neurips.cc/paper_files/paper/2025/file/a4ca07aa108036f80cbb5b82285fd4b1-Paper-Conference.pdf

[2] https://arxiv.org/abs/2603.14360


Reviewer_NeptOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ParaTTT makes nonlinear test-time-training memories easier to parallelize and improves the tradeoff between model quality and training speed. Current linear recurrent models like Mamba2 are efficient but have limited memory expressivity. Previous work TTT uses a neural network to replace the linear memory, but slows down the training.

ParaTTT estimates the gradient trajectory during iterations rather than the much larger weight trajectory to update the hidden layers. The eval is on 340M and 1.3B models with up to 100B tokens. ParaTTT generally improves over linear baselines on lm perplexity and task accuracy, and the training is faster than TTT but slower than linear baselines.

Critical strengths
  • The efficiency of working gradient rather than weight is a sounding improvement.
  • The efficiency and accuracy tradeoff is an important topic for linear models.
Critical weaknesses
  • 1-hidden-layer impl seems over simplified. (I'm not an expert on this.)
Questions for authors
  • It mentions that TTT-MLP's fused kernel supports dimension of 64 only. How would this gap contribute to the 4.9x speedup?
  • Should clarify batch size, memory usage, checkpointing strategy and all the other configs that can affect training speed.
1 comment

We thank Reviewer Nept for the thoughtful questions and positive review of our work! We have strengthened the evidence for the nonlinear memory and expanded the timing and memory details. Please also see our general response for a summary of the revisions.

W: why use only one hidden layer? One hidden layer already gives the nonlinear memory used by TTT-MLP and LaCT. This lets us focus on parallelizing established nonlinear TTT memories and measuring what they add over linear memory. The state-tracking results show that one hidden layer already makes a substantial difference: after 4K training steps, N=2N=2 ParaTTT and Gated DeltaNet achieve the following accuracies (Appendix F.1, Table 3):

Task (length)Gated DeltaNet (%)ParaTTT (%)
Parity (256)63.397.6
Permutation (64)38.162.7

The algorithm and fixed-point result also support deeper memories, with an additional forward scan and shifted transposed scan per hidden layer and iteration. Evaluating the benefit of greater depth remains future work.

Q1: does TTT-MLP's head-size restriction affect the training speedup? The 4.9$\times$ full-model training speedup uses TTT-MLP's supported fused kernel, with its published head dimension of 64 and block size of 16. The sequence-length comparison uses this configuration too, so neither result relies on a slower fallback.

The separate head-dimension sweep uses a compiled scan at larger head sizes; Figure 4 marks those points with hollow markers (Appendix G).

Q2: timing settings, memory and checkpointing. Appendix G now gives the timing and memory setup, with the training hyperparameters in Appendix D. All full-model timings use the same recipe, data loader and hardware: 4K-token sequences, eight sequences per GPU, two accumulation steps and eight MI325X GPUs, in bf16. Throughput is averaged over 50 steps after warm-up. At 340M, we obtain the following throughput and peak memory (Table 11):

Modelk tokens/s (node)GiB/GPU
Gated DeltaNet112.158
TTT-MLP (fused)7.886
ParaTTT, N=2N=237.9145
ParaTTT, N=5N=521.8228

ParaTTT trades more activation memory for faster training. It retains activations across refinement iterations and recomputes chunk-boundary states during the backward pass, without checkpointing across iterations. TTT-MLP checkpoints segments of its sequential scan. Appendix G now states these choices explicitly.

Reviewer_nhZuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ParaTTT (PTTT) proposes an algorithm to parallelize token-by-token online gradient descent in MLP-based Test-Time Training (TTT) layers without resorting to the block-wise stale-gradient approximation of TTT-MLP or LaCT. Instead of iterating on the O(d2)O(d^2) fast-weight matrices directly (as in Newton/DEER methods), ParaTTT iterates on a sequence of guessed per-token hidden-layer pre-activation gradients.

Critical strengths
  • Recognizing that conditioning on the O(d)O(d)-dimensional hidden pre-activation gradients δzt(l)\delta z_t^{(l)} turns the O(d2)O(d^2) fast-weight trajectory into standard chunkwise linear scans, and that the readout layer's exact gradient w.r.t. ϕt\phi_t can be computed by simply transposing the final query/value operands of the existing FLA Gated DeltaNet WY kernel, is a mathematically clean and clever insight.
  • The fused Triton epilogue kernels, careful differentiation through NN Picard iterations, and the revealing ablation in Appendix F.2 (Table 5), showing that removing LaCT's 2048-token sliding-window attention causes its WikiText perplexity to explode from 25.51 to 463.2, provides valuable empirical clarity to the community.
Critical weaknesses
  • As acknowledged in Section 3.3, Theorem 1 only analyzes a simplified toy model (soft clipping instead of hard unit-ball projection, uncentered readout, and a linear-attention hidden update without the hidden delta rule). Yet Table 1 shows that this analyzed "linear-attention body" model collapses on recall and NIAH (scoring 7.2% on 4K NIAH vs. 25.9% for vanilla Gated DeltaNet and 32.6% for PTTT with the delta-rule body!). Thus, the algorithm analyzed in Theorem 1 actually performs much worse than a simple 1-layer linear Gated DeltaNet.
  • Empirical evidence in Table 3 (Appendix F.1) proves that the 1.3B model does NOT converge to a better solution as NN increases at test time: when the 1.3B model (trained at N=2N=2) is evaluated with more refinement iterations (N=2→3→5→7→10N = 2 \to 3 \to 5 \to 7 \to 10), its SWDE recall accuracy steadily degrades from 62.8% down to 44.4%, and its LAMBADA perplexity worsens from 9.00 to 10.38! This directly contradicts the claim that N=2N=2 is merely approximating the ideal sequential fixed point; rather, the network overfits to the specific 2-step unrolled computational graph during prefill, creating a prefill–decode mismatch when switching to exact N=∞N=\infty token-by-token updates during decoding.
Questions for authors

See weaknesses.

1 comment

We thank Reviewer nhZu for the detailed review and positive outlook on our work! Your concerns motivated both the extended analysis and the new training experiments. Please also see our general response for a summary of the revisions we made.

W1: scope of the convergence theorem. The ablation confirms that replacing the hidden-layer delta rule with linear attention substantially worsens recall and NIAH (Table 1). We have therefore extended the convergence analysis to include the hidden-layer delta rule and centered readout (Theorem 2 and Corollary 1, Appendix B).

The extended analysis retains soft clipping, keys with norm at most one, and sufficient contraction conditions. The trained models use a hard projection and RMS-normalized keys, and their learned rates and gates fall outside these conditions (Section 3.3; Appendix F.5).

Although the theorem's guarantee does not apply to these trained models, its conditions are sufficient but not necessary for convergence. Empirically, we find that increasing the iteration count improves their agreement with the exact recurrence, both on short sequences (Figure 5(b,c)) and over whole documents from 4K to 32K (Appendix F.4, Table 9).

W2: degradation with more iterations at 1.3B. Decoding uses exact updates, but starts from the memory state produced by approximate prefill. Increasing NN changes the prompt computation and that starting state, while leaving the decoding update unchanged. Since the model's weights were optimized through the N=2N=2 computation, making prefill more accurate can still hurt its predictions.

To address this mismatch, we trained one variant with a final phase at N=5N=5 and another that randomly samples NN per layer throughout training. Both substantially reduce the degradation when evaluation uses more iterations. The changes from N=2N=2 to N=10N=10 at 1.3B are below (Section 4.1, Table 4):

TrainingLAMBADA ppl.SWDE (%)
Original (N=2N=2)9.00→10.389.00 \to 10.3862.8→44.462.8 \to 44.4
Final phase at N=5N=58.77→9.568.77 \to 9.5667.8→62.667.8 \to 62.6
Sampled NN9.39→9.479.39 \to 9.4763.6→61.963.6 \to 61.9

We also tested exact processing of whole 4K documents and a 16K teacher-forced recurrent continuation after 16K prefill. Both training approaches reduce the perplexity gap with exact recurrent evaluation, as the relative increases over N=2N=2 prefill show below at 1.3B (Appendix F.4, Table 8):

TrainingExact 4K document (%)16K + 16K (%)
Original (N=2N=2)5.110.4
Final phase at N=5N=51.93.8
Sampled NN1.11.7

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all the reviewers for their thoughtful feedback! Your comments prompted us to investigate the effect of finite-iteration training, strengthen the baseline comparisons, and clarify the scope of our claims. The main changes are highlighted in blue in the revised paper. We summarize the main additions below and address each reviewer's questions in the individual replies.

1. Training with different iteration counts. Reviewers nhZu, HVP5, b3EF and AJoJ raised concerns about the mismatch between finite-iteration training and exact recurrent evaluation. In response, we tested two approaches: increasing the iteration count near the end of training, and randomly sampling it for each layer throughout training. Both reduce sensitivity to the evaluation-time iteration count and the perplexity penalty from exact recurrent decoding. Training with randomly sampled iteration counts also improves long-context retrieval. Section 4.1 and Appendix F.4 report these results, including the remaining mismatch on longer documents.

2. Stronger baseline and state-tracking comparisons. Following b3EF and AJoJ's suggestions, we compared LaCT, Gated DeltaNet and ParaTTT with the same sliding-window attention branch. ParaTTT's gains on perplexity and most recall tasks persist in this comparison (Appendix F.2). We also added experiments on parity and permutation tasks to demonstrate the expressivity advantage of nonlinear TTT layers over linear recurrent layers in state-tracking. ParaTTT's gains over Gated DeltaNet persist across random seeds and even when the baseline is trained for more steps (Appendix F.1).

3. Clearer theory and approximation claims. We revised the abstract, introduction and discussion to distinguish finite-iteration training and prefill from exact recurrent decoding. We also extended the convergence analysis to the hidden-layer delta rule and centering, addressing nhZu and AJoJ's concerns about the original simplified model. Section 3.3 now states which assumptions differ from the trained models, with their measured gates and learning rates in Appendix F.5.

4. More complete efficiency measurements. To answer Nept and HVP5's questions, we added decode latency, state size, training memory and checkpointing details, and a speed–error comparison at TTT-MLP's default block size (Section 4.2; Appendices G and I). We also incorporated the figure-label and annotation fixes noted by AJoJ.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.