acceptodds
Under review as a conference paper at ICLR 2027

Modular TTT: Rethinking Test-Time Training as Composable Modules

Abstract

Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule. Despite the growing number of TTT variants, existing approaches typically hard-code each variant separately, which makes it difficult to design new TTT methods and to isolate the role of each component. To address this, we propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph and exposes the fast-weight network, loss function, learning rate, weight decay, and normalization as explicit design dimensions. Once the forward pass of each fast-weight primitive is specified, Modular TTT automatically composes the corresponding backward and query-view computations over the graph. Using Modular TTT, we systematically ablate the components of TTT and find that the learning rate, weight decay, a single-layer nonlinearity, and a dot-product loss improve performance. By contrast, deeper fast-weight networks and normalization tend to hurt performance because they induce excessively large activations, while residual connections, gating, and MSE loss provide little measurable benefit. Guided by these findings, we train the best resulting variant as 410M- and 1.45B-parameter models on 100B tokens, and observe training loss and benchmark performance comparable to Gated DeltaNet.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Test Time Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vK2xOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Modular TTT, a framework for expressing and systematically studying test-time training architectures through composable modules. The framework represents the inner learner as a directed acyclic graph and assembles primitive-level train-view forward, train-view backward, and causal query-view rules into the complete computation, including fast-weight state updates. This formulation enables controlled comparisons of learner architectures, inner objectives, learning-rate initialization, decay mechanisms, and normalization. Through language-modeling ablations at 160M and 410M parameters, the authors find that small initial inner learning rates, weight decay, and simple pointwise nonlinearities improve performance under the evaluated settings, while MSE and inner-product objectives perform similarly. Deeper learners, normalization, and residual or gated extensions do not consistently improve upon the shallow configurations. The paper also analyzes update stability and factor-dependent dynamics to help explain these observations. The implementation achieves approximately 2.2–3.3× higher training throughput than the official TTT implementation in two evaluated configurations. Finally, selected variants trained on 100B tokens at 410M and 1.45B parameters achieve competitive language-modeling and multiple-choice benchmark results relative to Gated DeltaNet, while precise long-context retrieval remains a limitation.

Critical strengths
  1. The paper provides substantial ablation coverage, including losses, learning-rate initialization, decay, nonlinearities, normalization, and deeper learners. The additional stabilization and initialization sweeps strengthen the empirical observation that the tested deeper configurations underperform the shallow alternatives. This provides actionable design guidance for TTT.(But, to mention, similar systematic studies of design choices have already been conducted in vision (see ViT³)).
  2. The scale-up experiments demonstrate that the selected configurations achieve competitive results on several language-modeling and downstream metrics. The paper also acknowledges limitations in precise retrieval, which I consider a strength.
Critical weaknesses
  1. How does Modular TTT differ from prior TTT methods? Do prior methods not already involve primitive-level forward, backward, and causal query computations? What does modularization actually add? For an entirely new inner network, would the update rules not still need to be implemented manually? Is deriving and implementing these rules after modifying the inner network particularly difficult? It remains unclear what practical benefits modularization brings to TTT implementation, especially compared with existing approaches (such as Learning to (Learn at Test Time): RNNs with Expressive Hidden States).
  2. Limitations of the empirical and theoretical evidence. First, Modular TTT shows no clear overall empirical advantage and underperforms GDN in some settings, including certain needle-in-a-haystack retrieval evaluations. Second, the factor-scaling analysis in Section 4.1 establishes that equivalent factorizations can induce different updates depending on the scaling coefficient cc. However, sensitivity to parameterization does not by itself establish greater optimization difficulty; a more direct connection to the observed failures of deeper learners is needed. Finally, the large speed gap between LaCT and Modular TTT warrants further explanation. More details on the benchmark configurations and implementations are needed to assess the fairness of this comparison.
Questions for authors
  1. What implementation effort does Modular TTT eliminate over prior TTT, and what still requires manual implementation?
  2. In which settings does Modular TTT offer a clear empirical benefit over GDN? How do the authors explain its weaker performance in some needle-in-a-haystack retrieval settings?
  3. How does factor-scaling sensitivity imply greater optimization difficulty? Could the authors provide direct experimental evidence linking this mechanism to the observed failures of deeper TTT memories?
  4. What explains the large speed gap with LaCT, and how is comparison fairness ensured?
3 comments

Thank you for your detailed questions about our framework and our comparison with LaCT.

1. Modular TTT is a unified framework rather than a new TTT method.

Modular TTT is not a new TTT method, but a unified execution framework for constructing and comparing TTT variants by automatically composing primitive-level rules ϕtrain\phi^{\text{train}}, ϕtrain-bwd\phi^{\text{train-bwd}}, and ϕquery\phi^{\text{query}} over any valid learner DAG.

  • Prior implementations such as Official TTT hardcode each topology separately, whereas Modular TTT separates graph topology from primitive rules. In Official TTT, TTT-Linear and TTT-MLP maintain separate hand-derived dual-form updates, normalization placements, and state layouts, so modifying the inner network requires rewriting global update and cache code. Modular TTT instead defines local rules once for Linear, Act, Norm, Add, Mul, and Gate in Table 1; recombining them into any new learner topology requires only changing the declarative forward graph, while manual implementation is needed only when adding a brand-new primitive type:
{
  "graph_nodes": [
    {"name": "linear_0", "type": "linear", "inputs": ["input"], "d_out": "output"},
    {"name": "act_0", "type": "act", "inputs": ["linear_0"], "activation": "silu"}
  ],
  "graph_output": "act_0"
}
  • PyTorch automatic differentiation cannot replace causal query composition or torch.compile-compatible execution. Autodiff can compute train-view gradients dV^d\hat{V}, but it generates neither chunkwise causal query readout over s≤ts \le t without materializing per-token weight copies nor the fast-weight state transition:

O=QW−Tril⁡(QK^⊤⊙M)dV^,W←W−dWO = Q W - \operatorname{Tril}(Q \hat{K}^\top \odot M) d\hat{V}, \qquad W \leftarrow W - dW

TTT-E2E uses the same fast weights throughout each mini-batch and updates them at the batch boundary. In Modular TTT, each query incorporates the causal prefix of fast-weight updates within the current chunk. Its analytic backward rules avoid inner-loop autograd.grad so the graph compiles under torch.compile, running 2.00×2.00\times and 3.14×3.14\times faster than Official TTT in pure PyTorch and 2.2×–3.3×2.2\times\text{--}3.3\times faster with Triton in Table 32.

2. Empirical benefit over GDN and weaker retrieval results.

Our goal is not a SOTA model that beats specialized GDN everywhere, as our main contribution is enabling reproducible single-factor TTT comparisons; still, Modular TTT outperforms GDN across several concrete settings while sharing the exact-retrieval limits of fixed-state models.

  • Modular TTT outperforms GDN in inference memory, 160M validation loss, multiple-choice accuracy, and several NIAH settings. Beyond supporting general learner DAGs rather than one hardcoded recurrence, Modular TTT uses lower peak allocated inference memory than GDN across every variant at both evaluated scales in Table 27. On quality and retrieval, Linear-SiLU achieves lower 160M validation loss than GDN across all five seeds in Table 15, while in 100B-token evaluations Modular TTT reaches lower 410M WikiText perplexity and slightly higher 410M and 1.45B multiple-choice accuracy in Table 7, beats GDN on 1.45B niah_single across all four variants at 4K in Table 29, and leads 410M niah_multiquery and niah_multivalue across 1K to 8K in Table 30.
  • Retrieval gaps reflect model capacity, 4K-only pretraining without long-context continual training, and fixed-state compression rather than a uniform deficit against GDN. Relative to GDN, NIAH ordering varies by scale and task: Modular TTT trails on 410M niah_single under general language-modeling selection without retrieval-specific tuning, yet scaling model capacity to 1.45B reverses this at 2K and 4K in Table 29. Meanwhile, at 8K and on harder multi-key tasks in Tables 29 and 30, all models were trained only at 4K context without long-context continual pretraining. Like GDN and LaCT, Modular TTT compresses history into a fixed-size fast state without a per-token KV cache, leaving exact long-context recall an open limitation as noted in Appendix E.

3. How factor-scaling sensitivity creates optimization difficulty and links to deep-memory failures.

In a factorized fast learner, the outer optimizer cannot merely learn an effective product W=W(1)W(2)W = W^{(1)}W^{(2)}; it must simultaneously keep factor scales balanced across chunks so that coupled one-step updates neither vanish nor overshoot.

  • Parameterization sensitivity turns a simple linear write into a coupled conditioning problem. A single fast matrix takes an uncoupled step ΔW=K⊤D\Delta W = K^\top D. Rescaling the factors to cW(1)cW^{(1)} and c−1W(2)c^{-1}W^{(2)} preserves the forward output, but their first-order contributions to the update of the product matrix scale by c−2c^{-2} and c2c^2, respectively (Equation 13). Poorly balanced factor scales therefore distort the effective update even when the forward function is identical.
  • Targeted interventions link factor coupling and normalization conditioning directly to observed deep failures. First, Proposition 5 predicts that all-zero factors trap coupled gradients at zero while leaving a single matrix unaffected; Table 25 confirms that zero initialization works for shallow learners but fails on every multi-factor learner until Gaussian initialization is used. Second, even with Gaussian initialization, two-layer learners fail or degrade at the shallow baseline's η0=10−3\eta_0 = 10^{-3} and require smaller η0\eta_0, while Norm-containing graphs suffer additional 1/σ(z)1/\sigma(z) gradient amplification in Equation 9 unless ε\varepsilon or mean scaling is raised. Below, Failed denotes divergence to NaN or stalled high loss with gradient norms above 101010^{10}. Even after stabilizing and tuning each family, every deep learner still trails shallow Linear-SiLU, and we will present factor coupling alongside normalization conditioning rather than as the sole cause.
Topology at 160MFailure Mechanism & Targeted FixBefore →\to After Loss ↓\downarrow
Linear-SiLU, shallow baselineZero init →\to Gaussian init at η0=10−3\eta_0=10^{-3}3.0247→3.02053.0247 \to 3.0205
Linear-LinearCoupled zero-gradient trap: Zero init →\to Gaussian initFailed→3.1265\text{Failed} \to 3.1265
Linear-SiLU-Linear-SiLUCoupled step scale: Zero or η0=10−3→\eta_0=10^{-3} \to Gaussian, η0=5×10−4\eta_0=5\times 10^{-4}Failed→3.1156\text{Failed} \to 3.1156
Linear-SiLU-Linear-NormNorm 1/σ(z)1/\sigma(z) amplification: ε=10−5→ε=0.5\varepsilon=10^{-5} \to \varepsilon=0.5 at η0=10−4\eta_0=10^{-4}7.7933→3.11447.7933 \to 3.1144
Linear-Norm-LinearCoupled step + residual scale: η0=10−3→η0=10−4\eta_0=10^{-3} \to \eta_0=10^{-4} with inner lossFailed→3.0542\text{Failed} \to 3.0542
Residual SiLUBranch scale amplification: no mean scaling →\to mean scaling at η0=10−4\eta_0=10^{-4}Failed→3.2367\text{Failed} \to 3.2367

4. The speed gap with LaCT and comparison fairness.

Table 27 reports end-to-end inference throughput under an identical workload: during decoding, Modular TTT reuses its fast-weight state and processes one new token per step, whereas the evaluated LaCT implementation recomputes the full prefix.

  • We matched the prompt length, decode length, and batch size across models at each scale. We used a single A100 with BF16 and per-layer torch.compile, a 2,048-token prompt, and 128 decode steps. The batch size was 32 at 160M and 16 at 410M. For LaCT, we used our default training configuration.
  • At 410M, M-SiLU, MSE and LaCT reach the same final loss of 2.5582, while Modular TTT trains nearly 3× faster with 35% lower peak memory. Both were trained for 100,000 steps at sequence length 4,096 under the same effective batch of 16 per device across micro-batch settings, as shown below.
Model at 410M, 100B TokensFinal Loss ↓\downarrowTraining Throughput tok/s ↑\uparrowPeak Memory GiB ↓\downarrow
LaCT2.55827,447.262.95
M-SiLU, MSE2.558221,438.740.94
M-SiLU, Inner2.556921,569.740.97
M, MSE2.564824,101.036.28
M, Inner2.562124,310.834.76
  • In two controlled training comparisons, enabling Muon alone reduces LaCT's throughput by 31% and 40%. We varied only Muon within each pair. Enabling it reduced throughput from 108,020 to 74,235 tokens/s in the smaller model and from 17,912 to 10,751 tokens/s in the larger model.

Dear Reviewer vK2x,

Thank you very much for your insightful questions regarding DAG modularization, GDN retrieval comparisons, deep-memory failure mechanisms, and LaCT benchmark fairness, which have allowed us to provide much stronger support for Modular TTT. We have addressed all your points with targeted comparative experiments and detailed clarifications. We would truly welcome your feedback on our responses and additional results before the discussion window ends. Thank you very much again for your time and thoughtful review.

Reviewer_oiNxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Modular TTT, a framework that represents the test-time-training inner learner as a DAG of primitives. Each primitive defines train-view forward, train-view backward, and causal query-view rules, which compose automatically into the full chunkwise TTT computation. Experiments find that small learning-rate initialization, weight decay, and shallow nonlinear learners improve language modeling, while deeper or gated learners generally underperform. Selected variants scale to 1.45B-parameter and achieve benchmark results comparable to Gated DeltaNet. The implementation accelerates training relative to official TTT implementations, although inference remains slower than Gated DeltaNet and precise long-context retrieval remains limited.

Critical strengths
  1. The Modular TTT framework provides an empirical research instrument to systematically explore the design space of TTT. The three-pass DAG formulation separates train-view computation, backward signals, and causal query updates, enabling changes to learner topology without manually deriving the entire TTT computation for every variant. This is important for TTT practitioners
  2. Ablations provide useful insights and also justify the framework's value. The experiments isolate learning-rate initialization, decay, loss, and learner structure under shared training conditions. Scaling selected variants to 1.45B parameters and approximately 100B tokens establishes that the resulting recipe remains competitive with GDN on multiple-choice benchmarks.
Critical weaknesses
  1. The selected models do not establish a compelling quality–efficiency advantage over the strongest baseline. At 1.45B, all Modular TTT variants have higher language-modeling loss and lower inference speedup than GDN. This limits the demonstrated payoff of TTT architecture search: comparable benchmark accuracy alone does not show a better practical trade-off.
  2. There is no equivalence check against the official TTT code. Official TTT-Linear reaches 3.395 versus about 3.038 for Modular TTT at 160M (Table 26). The gap is attributed to “recipe mismatch,” but the fidelity of the framework can be better established with more detailed analysis.
Questions for authors
  1. When Modular TTT is configured to match official TTT-Linear and TTT-MLP exactly, does it reproduce the official forward outputs and validation loss to numerical tolerance? Is the 2.2–3.3× speedup measured under that exactly matched configuration?
  2. I am curious, did the fast-weight factors share the shallow model’s learning rate and initialization in the deep-memory experiments? If so, the deep models may simply have written much smaller updates. Did you match effective update sizes across depths (e.g., per-factor learning rates or larger init), or log factor and update norms during training? The factor-coupling argument (Equations 10–13) shows the effect is possible, not that it occurred. This would materially affect how strongly I interpret the negative depth results; it is not a request for an unrestricted future architecture search.
4 comments

Thank you for reading our paper carefully and for your questions about our baselines and inner-loop updates.

1. Goal of Modular TTT and comparison with GDN.

Our goal is not to build a SOTA TTT architecture, but to provide a unified DAG framework and controlled experiments testing which TTT designs actually work. Methodological contributions are not limited to new models: lowering the barrier to reproducible comparison helps prevent unverified claims about complex fast learners from spreading. Adding SiLU to our linear learner lowered validation loss for every paired seed at both model sizes (Table 15). At 1.45B, our linear learner with the inner-product loss also achieved slightly higher average multiple-choice accuracy than GDN (Table 7). In our inference benchmarks, GDN was faster, while every Modular TTT variant used less peak memory at both tested sizes (Table 27).

2. Equivalence check, the 3.395 versus 3.038 loss gap, and the 2.2–3.3×2.2\text{--}3.3\times speedup.

We verified numerical equivalence with Official TTT when matching its operator and update configuration; the 3.3951 versus 3.0380 gap in Table 26 arises because unmodified Official TTT bundles several weaker recipe choices that our single-factor ablations quantify directly.

  • Tables 13 and 18 isolate the main factors behind Official TTT-Linear's 3.3951 loss. Table 26 runs the unmodified public Official TTT module, which combines no weight decay, a head-scaled learning-rate gate, fast-weight LayerNorm, and a V−KV - K residual target. In Modular TTT at 160M, removing scalar decay alone raises MSE loss from 3.0380 to 3.2005 in Table 13, using standard near-unit learning-rate initialization raises loss to 3.3035 with decay and 3.6012 without decay, and updating normalization scale as a fast weight degrades loss to 3.1831–3.2719 in Table 18. Together with the V−KV - K target, these factors explain the 3.3951 result without any fidelity discrepancy.
  • Table 32 measures the 2.2×–3.3×2.2\times\text{--}3.3\times speedup on topology-matched 160M norm learners rather than Official TTT's V−KV - K recipe, while toggling TTT_KERNEL_BACKEND inside Modular TTT isolates the exact kernel gain. Analytic primitive backward rules run 1.65×1.65\times and 2.62×2.62\times faster than autograd.grad in Table 31 and enable torch.compile, giving pure PyTorch Modular TTT a 2.00×–3.14×2.00\times\text{--}3.14\times gain over Official TTT, while Triton adds a strictly matched 1.06×–1.08×1.06\times\text{--}1.08\times speedup and 16%–17%16\%\text{--}17\% lower peak memory.
Topology at 160MOfficial tok/sModular PyTorch tok/s / GiBModular Triton tok/s / GiBPyTorch / OfficialTriton / PyTorch
Linear + Norm43,397.586,716.1 / 38.9493,366.9 / 32.152.00×2.00\times1.08×1.08\times
Linear + GELU + Linear + Norm21,336.966,961.6 / 55.5571,101.1 / 46.563.14×3.14\times1.06×1.06\times

3. Effective update sizes and the interpretation of factor coupling.

We raised the initial inner learning rate in the deep models to match the rate we used in the shallow model. Some deep models then had higher validation loss, and others failed to train.

  • Fast-weight nodes share initial η0\eta_0 and Gaussian std=0.02 initialization while predicting node-specific tokenwise ηt\eta_t; we swept η0\eta_0, hidden width, ε\varepsilon, mean scaling, and inner loss rather than setting separate per-factor η0\eta_0 or logging update norms. For Linear-SiLU-Linear-SiLU, validation loss fell as we increased the learning rate up to half the shallow model's rate. At the shallow rate, training failed. For Linear-SiLU-Linear-Norm, using the shallow rate increased validation loss even after we stabilized training by adjusting the normalization parameter. In the table, Failed means that training produced NaN or that loss stayed abnormally high. The shallow Linear-SiLU reached a validation loss of 3.0205. Neither a wider hidden layer nor Residual Norm closed that gap.
Two-Layer Topology at 160MMSE, η0=10−4\eta_0=10^{-4}MSE, η0=10−3\eta_0=10^{-3}Inner, η0=10−4\eta_0=10^{-4}
Linear-SiLU-Linear-SiLU3.1221Failed3.1297
Linear-Linear-SiLU3.12233.13043.1247
Linear-SiLU-Linear3.12403.13723.1213
Linear-Norm-Linear3.1594Failed3.0542
  • We agree that Equations 10–13 describe a structural optimization mechanism rather than proving small updates occurred under Gaussian initialization. With all fast weights initialized to zero, the two-factor learners failed to train, while the shallow Linear-SiLU trained successfully (Table 25). With Gaussian initialization, Linear-SiLU-Linear-SiLU trained at a lower learning rate but failed when we raised it to the rate used by the shallow baseline. Across our sweeps of learning rate, normalization, and inner loss, even the best deep learner had higher validation loss than the shallow baseline.

Thank you for the detailed response. The reported numerical equivalence with Official TTT helps address my concern about implementation fidelity. The additional learning-rate sweeps provide more empirical evidence that deeper learners underperform under the tested settings. Although a unified framework has value for TTT practitioners, I still view the contribution as primarily an assessment of existing designs, without opening up promising new directions for TTT architectures. More fundamentally, it remains unclear whether the TTT formalization itself exposes an effective design space. Therefore, I will keep my score.

Thank you very much for confirming that our numerical equivalence check addresses implementation fidelity, that our learning-rate sweeps strengthen the evidence on deeper learners, and that our unified framework is valuable for TTT practitioners. We offer a brief perspective on your concerns regarding TTT's design space and future directions, and respectfully ask if you might reconsider our contribution in this light.

1. Rigorously determining where the TTT formalization exposes an effective design space is itself a central scientific contribution. Recent works (TTT-MLP [1], Titans [2], ATLAS [3], Nested Learning [4], and LaCT [5]) widely assume that deeper, more expressive inner learners unlock a superior sequence-modeling space, yet bundle multiple changes or omit reproducible inner-learner code. Testing this assumption under controlled single-factor conditions is essential: we show that TTT does expose effective design axes, specifically initial update-spectrum control (Proposition 2), scalar and vector forget-decay (Table 4), and bounded coordinatewise write-gating (Table 5), whereas naive product-form depth, fast-weight normalization, residuals, and gating fail under one-step chunkwise learning. Clarifying which axes genuinely drive gains and which fail under controlled scrutiny prevents costly community-wide detours; revealing a sharper, accurate effective frontier is a core empirical contribution rather than a limitation of the study itself.

2. Beyond assessing prior designs, modularly decomposing TTT fills a gap with no prior counterpart in language-modeling TTT, uncovering stronger recipes and concrete new directions. Prior TTT studies hard-coded bundled variants individually without a unified framework for component-level attribution or mechanistic diagnosis.

  • Stronger, simpler TTT recipes: Isolating Official TTT's [1] bundled choices yielded Linear-SiLU (scalar decay, small-LR init, and bounded post-linear activation without fast-weight LayerNorm or V−KV - K targets), lowering 160M loss from Official TTT-Linear's 3.3951 (Table 26) to 3.0205 (Table 5), winning 10/10 paired seeds over Linear at 160M and 410M (Table 15), beating GDN [6] on all five 160M seeds, and matching LaCT's [5] 410M 100B-token loss (2.5582) at 2.88×2.88\times throughput and 35% lower memory.
  • Concrete new directions from mechanistic diagnosis (Section 4.1):
    1. Scale-invariant or preconditioned multi-factor inner updates: Equation 13 and Proposition 4 show that one-step SGD on W(1)W(2)W^{(1)}W^{(2)} suffers c2c^2 and c−2c^{-2} update distortion under forward-equivalent rescalings (cW(1))(c−1W(2))(cW^{(1)})(c^{-1}W^{(2)}), pointing toward factor-balanced, orthogonalized, or preconditioned inner updates rather than naive multi-layer SGD.
    2. Richer write- and forget-modulation on uncoupled states: Equation 8 (ΔW=K⊤(Do⊙σ′(U))\Delta W = K^\top (D _ o \odot \sigma'(U))) and Table 4 (where vector decay further lowers loss to 3.0038 at 160M and 2.7821 at 410M) show that TTT's productive frontier lies in expressive coordinatewise write-gating and selective decay on well-conditioned uncoupled states rather than stacking forward depth.
    3. Composable prototyping for multi-step and hybrid memories: Because Modular TTT composes ϕtrain\phi^{\text{train}}, ϕtrain-bwd\phi^{\text{train-bwd}}, and causal ϕquery\phi^{\text{query}} rules with torch.compile (2.2×2.2\times to 3.3×3.3\times faster than Official TTT [1]), practitioners can prototype new update rules, multi-step schedules, and hybrid exact-recall plus fast-weight blocks without re-deriving chunkwise dual kernels by hand.

Since our experiments and analyses have addressed your technical questions on implementation fidelity and deep-learner fairness, we sincerely hope you might weigh the value of providing an open, high-throughput composable framework together with a rigorous mechanistic map of where TTT works, why naive depth fails, and where future TTT designs should focus. We would be truly grateful if you could kindly reconsider your assessment before the discussion window closes. Thank you so much again for your time and thoughtful engagement.

[1] Yu Sun et al. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. ICML, 2024.

[2] Ali Behrouz et al. Titans: Learning to Memorize at Test Time. arXiv, 2025.

[3] Ali Behrouz et al. ATLAS: Learning to Optimally Memorize the Context at Test Time. arXiv, 2025.

[4] Ali Behrouz et al. Nested Learning: The Illusion of Deep Learning Architectures. arXiv, 2025.

[5] Tianyuan Zhang et al. Test-Time Training Done Right. arXiv, 2025.

[6] Songlin Yang et al. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR, 2025.

Reviewer_CwahOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

the paper proposes an analysis suit for TTT ingredients. Their modular API allows to easily modify depth, replace nonlinearities, normalization, loss functions, etc to more easily measure their effect. The analyses show some design principles including MSE/inner-product losses being preferable to L1 or RMSE; low LR init. Scaling experiment show competitive performance to a strong linear recurrent method GDN .

Critical strengths

(+) the key benefit of this work is a convenient sandbox to test TTT modules. I can see the codebase being useful to the community, especially given it is twice as fast as the TTT implementation. (+) I appreciate the analyses runs that change one factor at a time which helps identify key ingredients importance

Critical weaknesses

(-) there's no real methodological contribution here. While indeed this is an analysis tool that can be useful for discovering key architectural ingredients, most of the results reported are negative -- what doiesn't help. some results confirm what was already found in current TTT papers like low learning rate. Eventually, performance do not win over existing solutions. In particle, could the GDN be seen as a chunk-of-size-1 TTT variant? If so would the sweep have found it due to increaed performance? (-) increased inner loop depth is found to harm performance. but i'm not convinced due to thee setup. TTT is meant to help with long sequences. In the paper, the expriment runs with a context length is 2K. Given the large chunk size there are only a handful of updates per sequence. I would really like to see this expriment re-run with much longer sequences.

Questions for authors

(-) Is W0 learned in the outer loop?

4 comments

Thank you for your thoughtful evaluation and constructive questions.

1. Methodological contribution, negative results, and GDN at chunk size 1.

We believe that methodological contributions are not limited to proposing new methods: providing a reproducible framework, conducting controlled single-factor ablations, and showing which complex designs fail to help are equally important research contributions.

  • Controlled negative results directly test complexity claims that lacked isolated verification. Titans, ATLAS, and Nested Learning claimed deeper or more complex fast learners beat DeltaNet-style models without releasing reproducible inner-learner code, while LaCT jointly altered the learner, attention branch, update rule, and kernels, preventing component-level attribution. Once an incorrect scientific narrative becomes widespread, correcting it becomes increasingly costly; one purpose of Modular TTT is precisely to lower the barrier to community reproduction and controlled comparison by automatically composing ϕtrain\phi^{\text{train}}, ϕtrain-bwd\phi^{\text{train-bwd}}, and ϕquery\phi^{\text{query}} into causal DAG execution with open-source code.
  • Prior use of individual components is distinct from a matched ablation under a shared backbone, data, budget, codebase, and protocol. Beyond small-learning-rate initialization, Tables 13–16 show that weight decay substantially improves training, MSE and inner-product losses outperform L1 and RMSE, and Linear-SiLU beats Linear across all paired seeds at 160M and 410M, with smaller, higher-variance gains at 410M. Equation 8, ΔW=K⊤(DO⊙σ′(U))\Delta W = K^\top (D_O \odot \sigma'(U)), attributes this gain to bounded elementwise write-gating σ′(U)\sigma'(U), whereas depth, normalization, residuals, and gating do not surpass this shallow frontier.
  • Our objective is neither a global optimum nor a SOTA architecture; "best" means best among the evaluated configurations, and being "close to GDN" is not itself our contribution. At c=1c=1, a scalar-decay Linear learner recovers GDN's inner gated delta-rule update, whereas full GDN also adds outer modules such as short convolutions and key-query/output normalizations outside the inner DAG. Because our chunk sweep in Table 23 uses the same outer backbone for chunk sizes of 128, 256, and 512 tokens, an inner-DAG sweep alone would not introduce those outer modules; Table 7 therefore compares against full GDN directly as an external baseline at 410M and 1.45B, where our shallow configuration remains competitive.

2. Do the negative depth results extend to much longer training sequences?

In our one-step, 2K setting, shallow Linear-SiLU reaches 3.0205 and outperforms every deeper learner we tested, both when increasing updates per sequence and after stabilizing and tuning deep variants, where the best loss is 3.0542.

  • Reducing the chunk size from c=512c=512 to c=128c=128 increases cross-chunk updates from 44 to 1616 per 2K sequence, yet the deep learner slightly degrades rather than closing the gap. In addition, queries within each chunk already incorporate token-level prefix updates through Tril⁡(⋅)\operatorname{Tril}(\cdot) in Table 23.
Learner160M c=512c=512c=256c=256c=128c=128410M c=512c=512c=256c=256c=128c=128
Linear-GELU3.02473.02323.02242.78382.78442.7834
Linear-SiLU3.02173.02053.02102.78452.78492.7833
Linear-Norm3.02853.03003.03752.80662.81922.7961
Linear-SiLU-Linear-Norm3.11323.11443.13732.91282.92582.9295
  • Fixing initialization and normalization instabilities makes failed deep learners trainable, but every tuned deep variant still trails Linear-SiLU. Here Failed denotes runs that diverge to NaN or stall with abnormally high loss and gradient norms exceeding 101010^{10}; following Equations 10–13, using Gaussian initialization and tuning Norm ε\varepsilon, mean scaling, η0\eta_0, and the inner loss at 160M restores stable training, and we will clarify this 2K scope in the paper.
Topology (160M, scalar decay)Stabilization & Tuning AppliedDefault →\to Best Loss ↓\downarrow
Linear-SiLU (shallow baseline)Zero init →\to Gaussian init (η0=10−3\eta_0 = 10^{-3}, MSE)3.0247→3.02053.0247 \to \mathbf{3.0205}
Linear-Norm-LinearGaussian init (η0=10−4\eta_0 = 10^{-4}), MSE →\to Inner product3.1594→3.05423.1594 \to \mathbf{3.0542}
Linear-SiLU-Linear-NormGaussian init, ε:10−5→0.5\varepsilon: 10^{-5} \to 0.5, η0:10−3→10−4\eta_0: 10^{-3} \to 10^{-4}Failed→3.1144\text{Failed} \to 3.1144
Linear-SiLU-Linear-SiLUZero init →\to Gaussian init, η0∈10−5,10−4,5×10−4\eta_0 \in 10^{-5}, 10^{-4}, 5\times 10^{-4}Failed→3.1156\text{Failed} \to 3.1156
Linear-LinearZero init →\to Gaussian init (η0=10−3\eta_0 = 10^{-3}, MSE)Failed→3.1265\text{Failed} \to 3.1265
SwiGLU-NormZero / no-mean →\to Gaussian init + mean scalingFailed→3.2041\text{Failed} \to 3.2041
Residual SiLUGaussian init (η0=10−4\eta_0 = 10^{-4}), no-mean →\to mean scalingFailed→3.2367\text{Failed} \to 3.2367

3. Is W0W_0 learned in the outer loop?

Yes, W0W_0 is a trainable outer-loop parameter optimized end-to-end across training sequences. Outer-loop gradients backpropagate directly into W0W_0 through the inner-loop updates. Each sequence initializes its fast weights from W0W_0 without detaching the computation graph, so outer language-modeling gradients update W0W_0 alongside all other backbone parameters.

i thank the authors for their detailed response, upon reading the discussions i maintain my original score. The main remaining weaknesses presist: the paper offers a tool than confirms known results and allow to search new methods but did not lead to new finding or even the re-discovery of GDN. second, the token sequences are short for the memory size tested. experiments with longer seuquences are cruical.

Thank you very much for reading the discussions and clarifying your concerns. Below we provide direct evidence on both points and respectfully ask if you might reconsider our work.

1. Re-discovering the c→1c \to 1 DeltaNet/GDN limit and new findings beyond known results.

  • Empirical c→1c \to 1 convergence and why search selects Linear-SiLU (c=256c=256): For our Linear MSE learner (no decay, mean scaling, 2K, 10B), shrinking cc monotonically approaches c=1c=1 DeltaNet [4] (and GDN's [5] inner rule under scalar decay) at both scales, while generic-DAG speed (tgs, tok/s) drops 6.3×6.3\times to 7.0×7.0\times without a hardcoded scan kernel:
Metric (2K, 10B)c=256c=256c=128c=128c=64c=64c=32c=32c=16c=16DeltaNet [4] (c=1c=1)
160M Loss ↓\downarrow3.19643.19113.17293.15323.13493.0833
160M tgs ↑\uparrow108,91182,57953,46831,74317,324174,340
410M Loss ↓\downarrow2.92192.91072.89742.88242.86602.8396
410M tgs ↑\uparrow37,16826,72317,2159,8435,30161,740

With scalar decay, small-LR init, and our post-linear SiLU write-gate at c=256c=256, Linear-SiLU reaches 3.0205 at 160M (beating c=1c=1 GDN's [5] 3.0301 across 5/5 seeds, Table 15) and 2.5569 at 410M/4K/100B (vs. GDN's 2.5559, Table 7, with better WikiText PPL 20.98 vs. 21.00 and accuracy 50.59% vs. 50.45%), matching or beating c=1c=1 GDN without the 6×6\times to 7×7\times speed loss.

  • New findings (no prior modular decomposition existed in language-modeling TTT): Prior TTT works bundled changes without component attribution; beyond small-LR init (Prop. 2), we show: (i) Why deep memory fails: contrary to TTT-MLP [1], Titans [2], and LaCT [3], Eq. 13 and Prop. 4 prove multi-factor SGD suffers c2c^2 and c−2c^{-2} distortion under (cW(1))(c−1W(2))(cW^{(1)})(c^{-1}W^{(2)}), explaining why stabilized deep learners trail shallow ones; (ii) Bounded write-gating: isolating Linear-SiLU (ΔW=K⊤(Do⊙σ′(U))\Delta W = K^\top (D _ o \odot \sigma'(U)), Eq. 8) without LayerNorm amplification (Eq. 9) or V−KV - K targets cuts 160M loss from Official TTT's [1] 3.3951 (Table 26) to 3.0205 (Table 5), winning 10/10 paired seeds over Linear (Table 15); and (iii) Selective decay: vector decay lowers loss to 3.0038 (160M) and 2.7821 (410M, Table 4).

2. State-rank saturation at 2K and longer-sequence (4K, 100B-token; 8K) results.

  • 2,048 tokens exceed state rank by 16×16\times: Per head, W∈R128×128W \in \mathbb{R}^{128 \times 128} (Table 9) has max rank 128. By O=QWstart−Tril⁡(QK^⊤)dV^O = QW _ {\text{start}} - \operatorname{Tril}(Q\hat{K}^\top) d\hat{V} (Eqs. A.12 to A.14), every token tt writes k^t⊤dv^t\hat{k} _ t^\top d\hat{v} _ t (not one write per chunk), so 2,048 tokens execute 2,048 writes (16×16\times rank capacity; >20×>20\times the ∼100\sim 100-token horizon under γ≈0.99\gamma \approx 0.99), and 4,096 tokens execute 32×32\times rank capacity.
  • Longer-sequence (4,096 tokens, 100B tokens) comparison at 410M:
Method (410M, 4K, 100B)TopologyLoss ↓\downarrowtgs ↑\uparrowMem (GB)
Official TTT-MLP [1]2-matrix MLP + Norm3.27257,28636.90
Official TTT-Linear [1]1-matrix Linear + Norm2.905112,35737.21
LaCT [3]3-matrix SwiGLU + Muon2.55827,44762.95
Modular M (Inner)1-matrix Linear2.562124,31134.76
Modular M-SiLU (MSE)1-matrix Linear + SiLU2.558221,43940.94
Modular M-SiLU (Inner)1-matrix Linear + SiLU2.556921,57040.97

Doubling context to 4,096 (32×32\times rank) and tokens 10×10\times to 100B widens deep deficits: TTT-MLP's gap behind TTT-Linear grows >30×>30\times (+0.0110 at 2K/10B, Table 26, to +0.3674 at 4K/100B), and 3-matrix LaCT [3] fails to beat M-SiLU at 2.9×2.9\times lower speed and +22 GB memory; Figure 4 confirms stable loss to 8,192 tokens (64×64\times rank).

As these c→1c \to 1, new-finding, state-rank, and 4K/100B results address both concerns, we would be truly grateful if you could kindly reconsider your score before the window closes. Thank you very much again for your time and feedback.

[1] Yu Sun et al. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. ICML, 2024.

[2] Ali Behrouz et al. Titans: Learning to Memorize at Test Time. arXiv, 2025.

[3] Tianyuan Zhang et al. Test-Time Training Done Right. arXiv, 2025.

[4] Songlin Yang et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. NeurIPS, 2024.

[5] Songlin Yang et al. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Additional Results and Clarifications

Dear Area Chair and Reviewers,

We sincerely thank the reviewers and Area Chair for their time and constructive feedback. For ease of reference, we summarize the main conclusions from the additional experiments and clarifications provided across our rebuttal and follow-up responses:

  • Framework value, numerical equivalence, and speedup [Reviewers oiNx, vK2x]. Modular TTT separates the learner graph from primitive-level forward, analytic backward, and causal query-view rules. This automatically composes chunkwise causal readouts and fast-weight state transitions that PyTorch autodiff cannot generate, while enabling torch.compile. Under matched settings, Modular TTT matches Official TTT numerically, isolates why unmodified Official TTT has higher loss, and decomposes our 2.2×2.2\times to 3.3×3.3\times training speedup into pure PyTorch compilation gains and matched Triton kernel gains.
  • Design-space findings and c→1c \to 1 convergence to DeltaNet/GDN [Reviewers Cwah, oiNx, vK2x]. As the first modular decomposition for controlled component attribution in language-modeling TTT, our study shows that small learning-rate initialization, weight decay, and a single-layer SiLU nonlinearity consistently improve performance, while pointing to future directions such as scale-balanced multi-layer updates and selective decay. Shrinking chunk size from c=256c=256 to c=16c=16 monotonically approaches the c=1c=1 DeltaNet limit at both 160M and 410M, but slows general-DAG training by 6×6\times to 7×7\times. By contrast, chunkwise Linear-SiLU at c=256c=256 beats matched GDN across all 160M seeds and matches full GDN at 410M and 100B tokens without this speed penalty.
  • Deep-learner stabilization and longer-sequence evidence [Reviewers Cwah, oiNx, vK2x]. Sweeping inner learning rates, initialization, normalization, and mean scaling across deeper, residual, and gated learners rules out under-sized updates. Deeper learners exhibit an interior learning-rate optimum, directly link zero-gradient trapping and normalization amplification to observed failures, and still trail shallow Linear-SiLU after stabilization. Moreover, every token applies a rank-1 update to the 128×128128 \times 128 per-head fast weight, so 2K sequences already exceed state rank by 16×16\times and 4K sequences by 32×32\times. In matched 410M, 4K-context, 100B-token runs, Official TTT-MLP falls even further behind Official TTT-Linear, 3-matrix LaCT still does not beat shallow Linear-SiLU, and per-token loss stays stable out to 8K.
  • Matched comparisons with LaCT and GDN [Reviewers vK2x, oiNx]. Under identical A100 training and inference setups, shallow Linear-SiLU matches or slightly improves upon LaCT's 410M 100B-token loss while training nearly 3×3\times faster with 35% lower peak memory. Controlled ablations further show that enabling Muon alone slows LaCT training by over 30%. We also clarify where Modular TTT improves over GDN, including inference memory, 160M validation loss, 410M WikiText perplexity, multiple-choice accuracy, and several NIAH tasks, while clearly scoping the exact-retrieval limits shared by fixed-state models.

Detailed tables and derivations are provided in our reviewer-specific responses and follow-ups. We remain available to clarify any remaining questions. Thank you very much again for your time and thoughtful engagement.

Best regards,

The Authors of Submission 39209

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.