acceptodds
Under review as a conference paper at ICLR 2027

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

Abstract

Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network's previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, we find that this expected advantage does not consistently materialize in nonlinear memories: a fixed-base parallel TTT baseline outperforms its serial counterpart. Our exploratory experiments point to a key underlying difficulty: nonlinear memories can be harder to optimize than linear ones within a single pass over the sequence. To alleviate this optimization difficulty, we introduce DeltaTTT, which replaces joint inner-loop optimization of a two-layer memory network with layerwise learning. Each layer is assigned a local prediction target and updated through a state-dependent delta rule. This formulation retains a nonlinear readout while enabling chunkwise parallel computation. Experiments on DeltaNet and LaCT backbones show improvements in language modeling and retrieval over their recurrent baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Training

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_wUaMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the inner-loop optimization of nonlinear fast-weight memories in test-time training (TTT). The authors observe that, in a LaCT-based model, fixed-base parallel TTT outperforms sequential TTT on single-needle retrieval tasks. Using a fixed-K/V reconstruction experiment, they argue that nonlinear memories are harder to optimize than linear ones within a single pass. To address this, they propose DeltaTTT, which replaces joint optimization of a two-layer MLP memory with layer-wise learning: each layer regresses the value with its own delta-rule update. This lets both layers use DeltaNet's chunkwise parallel algorithm while keeping exact token-wise updates and a nonlinear readout. On 0.5B models trained on 92.5B tokens with DeltaNet and LaCT backbones, DeltaTTT achieves lower training loss and better single-needle retrieval scores than the recurrent baselines, with higher training throughput than both LaCT variants.

Critical strengths

A simple and efficient method with consistent gains on both backbones. DeltaTTT turns the two-layer memory into two stacked delta-rule regressions, so it can directly reuse DeltaNet's chunkwise parallel algorithm while keeping exact, state-dependent per-token updates and a nonlinear readout. Its training throughput is 1.37× that of full attention, close to DeltaNet's 1.42×. On both backbones, it achieves lower training loss and better single-needle retrieval than the corresponding recurrent baselines; on the LaCT backbone, against a DeltaNet that also uses a learnable initial state, its single-needle score is 50.47 vs. 40.52. Nonlinear TTT has long faced a trade-off between speed and update granularity, and this method is simple to implement and easy to adopt. Even though the layerwise-optimization explanation has not been verified yet, the method's practical value supports acceptance.

Critical weaknesses
  1. The motivating finding is not robust, and the core hypothesis is not directly verified. The paper starts from Table 1, where parallel TTT beats serial TTT, but this comparison uses the average over S1–S3. Broken down by task (the LaCT rows of Table 3), the advantage comes almost entirely from S1, where parallel scores 90.85 and serial 44.85. On S2 and S3, serial is actually better (28.10 vs. 15.80 and 11.80 vs. 6.95), and the two have almost the same training loss. S1 also uses repeated noise as the haystack, which makes it the easiest of the three tasks. The paper attributes this result to nonlinear memories being hard to optimize in a single pass, relying mainly on Figure 2, but the learning rates in that experiment were tuned on the 16-epoch results and then fixed (Appendix B). More importantly, DeltaTTT itself is not part of this controlled experiment, so the claim that layerwise learning is easier to optimize in a single pass is never tested directly. Since the chain from finding to hypothesis to method is the main thread of the paper, this gap weakens its central claim about optimization difficulty.

  2. The sources of the gains are not controlled, which makes it hard to attribute them to layerwise optimization. DeltaTTT differs from the baselines in several ways at once. Each head has two 64×64 states, A and B, which is twice the state of DeltaNet, yet the paper has no state-matched baseline. Compared with LaCT, the memory also changes from SwiGLU to a two-layer SiLU MLP, and the update granularity changes from once every 512 tokens to every token. Any of these factors could explain a substantial part of the improvement, so the current experiments do not support the conclusion that the gains come from layerwise optimization.

Questions for authors
  1. On the DeltaNet backbone, the DeltaNet baseline does not use a learnable initial state, yet on the LaCT backbone this alone raises DeltaNet's single-needle score from 25.00 to 40.52. Could you report DeltaNet and GDN results on the DeltaNet backbone with the same learnable initialization as DeltaTTT?
  2. In Table 5(a), the "selected" Pre variant reports 1.962162 / 54.32, which matches neither DeltaTTT result in Table 3 (1.950079 / 48.78 on the DeltaNet backbone and 1.952344 / 50.47 on the LaCT backbone). Table 8 also reports 1.95 / 48.78 for the same initialization, and Appendix C states that the main-table models use Pre. Which run and configuration does Table 5(a) correspond to? Which backbone is used in Table 5(b)?
  3. Why does the same DeltaNet memory drop from 44.38 to 25.00 in single-needle score when placed in the LaCT backbone? Were the baselines tuned separately for each backbone? Delta MLP reaches only 4.88, and GDN has a higher training loss than DeltaNet. Were these baselines sufficiently tuned?
  4. Apart from both layers sharing the same target v, what is the substantive difference between DeltaTTT and the Delta MLP of Irie et al. (2021)?
  5. What were the measurement conditions for Table 4, including hardware, kernel implementation, batch size, and which backbone the DeltaTTT model uses? LaCT uses 512-token chunks here. What is its throughput at its default chunk size of 2048?
Reviewer_UuGzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DeltaTTT, a method for optimizing nonlinear recurrent memories through layerwise delta-rule updates rather than joint gradient descent. The approach decomposes a two-layer memory into separate linear regression objectives while retaining a nonlinear readout and efficient chunkwise parallel computation. The authors motivate this design by observing that nonlinear TTT memories can be difficult to optimize within a single pass over the sequence. Experiments on DeltaNet and LaCT backbones demonstrate improvements in language modeling and associative retrieval.

Critical strengths
  1. Efficient optimization: The decomposition into layerwise delta-rule updates induces an efficient chunkwise parallel form that invokes existing Tritok kernels in flash-linear-attention. This provides a practical alternative to sequential nonlinear TTT.

  2. Empirical improvements: The authors evaluate the approach on two language model backbones trained on 92.5B tokens and demonstrate improvements in language modeling and retrieval over several recurrent baselines. The scale of the experiments is sufficiently convincing.

Critical weaknesses
  1. Insufficient characterization of nonlinearity: This is the main weakness in my opinion. The paper does not isolate the contribution of nonlinear readout or recurrent dynamics. In particular, there is no ablation that removes the activation function while retaining both adaptive memory matrices, nor a comparison against linear alternatives with matched recurrent state capacity. The existing ablations remove the second memory and nonlinearity together.

  2. Limited evidence for the proposed optimization mechanism and practical advantage: The motivating reconstruction experiment compares linear memories with jointly optimized MLPs but does not include DeltaTTT itself. Furthermore, the larger improvements over LaCT are confounded by differences in update granularity and readout timing, while gains over Gated DeltaNet are relatively modest. Overall, the experiments do not demonstrate that layerwise optimization of nonlinear memories is responsible for the reported improvements.

See the questions below for more specific details

Questions for authors
  1. Role of activation nonlinearity. How does DeltaTTT perform when the activation is replaced by the identity, i.e., ϕ(x)=x\phi(x)=x, giving ht=At−1kth_t=A_{t-1}k_t and ot=BtAtqto_t=B_tA_tq_t, while retaining both adaptive matrices and their original delta-rule updates? I believe this would isolate the contribution of the nonlinearity

  2. Role of recurrent state capacity and coupling. How does DeltaTTT compare against two independent DeltaNet memories with matched total recurrent state capacity? For example, using independently learned query/key/value projections for i∈1,2i\in{1,2}, each memory could follow Mt(i)=Mt−1(i)+ηt(i)(vt(i)−Mt−1(i)kt(i))(kt(i))⊤M_t^{(i)}=M_{t-1}^{(i)}+\eta_t^{(i)}(v_t^{(i)}-M_{t-1}^{(i)}k_t^{(i)})(k_t^{(i)})^\top, with readout ot=∑i=12Mt(i)qt(i)o_t=\sum_{i=1}^2 M_t^{(i)}q_t^{(i)}.

  3. Single-pass optimization advantage. Can the authors directly compare DeltaTTT with jointly optimized nonlinear TTT, using ℓt(A,B)=12∣Bϕ(Akt)−vt∣22\ell_t(A,B)=\frac12|B\phi(Ak_t)-v_t|_2^2, on identical fixed K/V sequences? This would explicitly test whether the proposed layerwise objectives address the optimization difficulty which essentially motivates the paper in the first place

I'm happy to reconsider my score if some/all of the above concerns are sufficiently addressed.

Reviewer_vVadOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates the reason that state-dependent sequential updates don't necessarily outperform fixed-base parallel updates in non-linear TTT memories. The work proposes DeltaTTT that replaced joint optimization of a two-layer memory with two local regression objectives, one matrix that predicts values from keys and the other one reconstructs values from the first matrix's predictions. With delta-rule update applied to both matrices, tokenwise memory evolution can be evaluated with two successive chunkwise scans.

The effectiveness of the proposed method is evaluated with models with DeltaNet and LaCT backbones, using methods like single-needle retrieval and position-dependent held-out language-modeling loss. DeltaTTT is shown to improve the reported retrieval results over recurrent baselines while being substantially faster than serial LaCT implementation.

Critical strengths
  • Simple design for nonlinear recurrent-memory. It constructs the second layer's write address from the first layer's prediction, and this makes sure the memory stages are coupled and we have affine recurrence stage so we can do tokenwise updates through existing delta-rule scans.
  • Empirical results support the proposed method. For single needle retrieval, LaCT parallel/DeltaNet (learned init) improves from 37.87/40.52 to 50.47. Disabling second-layer online updates reduces accuracy to 36.02, supporting the usefulness of that adaptive component.
Critical weaknesses

Evidence for advantages over a strong recurrent alternatives is still a bit weak. In a DeltaNet backbone, DeltaTTT has 48.78 single needle retrieval over 47.60. Without reported uncertainty or repeated runs, the gain seems a bit unclear.

Questions for authors

I only have one critical question: Are uncertainty estimates available for DeltaTTT vs Gated DeltaNet on the DeltaNet backbone? If not, are there any explanations or justifications on the minor gains on DeltaNet?

Reviewer_K6SaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

DeltaTTT investigates why serial nonlinear Test-Time Training (TTT) with a two-layer MLP memory can underperform a "fixed-base parallel" TTT baseline (which evaluates all chunk gradients at the initial state W0W_0) when restricted to a single pass over the sequence. Based on a synthetic key-value reconstruction experiment showing that a linear memory achieves lower MSE after 1 epoch than a jointly optimized 2-layer MLP, the authors propose replacing joint backpropagation through the 2-layer MLP memory, with two decoupled local linear regression objectives.

Critical strengths
  • Identifying that fixed-base parallel chunk updates outperform serial chunk updates in LaCT (37.87% vs. 28.25% single-needle retrieval in Table 1) and linking this to single-epoch underfitting of joint 2-layer MLPs (Figure 2) is an interesting and well-motivated empirical observation.
  • Visualizing the learned initial matrices A0A_0 and B0B_0 (Figure 4) and measuring within-sequence MSE reduction (Table 6) nicely illustrates how BB acts as a near-diagonal residual refinement stage on top of AA's initial value prediction.
  • The two step training process is parallelizable and can reuse the existing kernels.
Critical weaknesses
  • The proposed method shares significant similarity with "Trellis: Learning to Compress Key-Value Memory in Attention Models" (Karami et al., COLM 2025), which also uses two step mapping and also reuse the kernels. The paper has missed these relevant citations and also there is a lack of discuss/comparison with this method. To my understanding the technical approach is too similar to Trellis and the novelty is unclear. It would be beneficial if the authors clarify the differences and provide a comparison. Have been the authors aware of this study?
  • Per head, DeltaNet maintains one dv×dkd_v \times d_k matrix (64×6464 \times 64), whereas DeltaTTT maintains two 64×6464 \times 64 matrices (AtA_t and BtB_t), doubling the recurrent state memory and doubling the scan FLOPs per layer. On the DeltaNet backbone (Table 3), DeltaTTT improves average single-needle retrieval over Gated DeltaNet by only +1.18 points (48.78% vs. 47.60%), and on S1 (99.95% vs. 99.70%) and S3 (7.35% vs. 7.10%) the gains are negligible. The authors never compare DeltaTTT against a state-matched DeltaNet or Gated DeltaNet (e.g., doubling dkd_k or dvd_v, or using 2-step DeltaProduct by Siems et al., 2025, which isn't cited again).
  • Every experiment in the paper includes a 512-token Sliding Window Attention branch. DeltaTTT is never ablated without SWA to show whether its layerwise memory can function as a standalone sequence mixer.
  • Evaluation is restricted entirely to training loss, held-out perplexity on 43 documents, and 3 single-needle RULER tasks (S1/S2/S3). There are no zero-shot commonsense reasoning benchmarks (PIQA, ARC, HellaSwag, WinoGrande), no multi-needle/aggregation RULER tasks (MK, MQ, MV, VT), and no LongBench or real-world in-context retrieval benchmarks (SWDE, FDA, SQuAD).
  • What was the reason/criteria for the baseline selection for Table 2? There are so many missing citations and wrong assignment of contributions. Token-wise TTT in the context of what the authors are discussing in the paper was first proposed as Delta MLP by (Irie et al., 2021). Similarly, the authors are using a wrong citation for DeltaNet, as this architecture first presented by Schlag et al. (2021). Again similarly, the authors are referring to E2-TTT, while it is not clear how it is different from Titans (Behrouz et al., 2025). In Table 2, why Trellis (Karami et al., 2025), Titans (Behrouz et al., 2025), DeltaProduct (Siems et al., 2025), Delta MLP (Irie et al., 2021), ... are missing?
Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.