- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents Modular TTT, a framework for expressing and systematically studying test-time training architectures through composable modules. The framework represents the inner learner as a directed acyclic graph and assembles primitive-level train-view forward, train-view backward, and causal query-view rules into the complete computation, including fast-weight state updates. This formulation enables controlled comparisons of learner architectures, inner objectives, learning-rate initialization, decay mechanisms, and normalization. Through language-modeling ablations at 160M and 410M parameters, the authors find that small initial inner learning rates, weight decay, and simple pointwise nonlinearities improve performance under the evaluated settings, while MSE and inner-product objectives perform similarly. Deeper learners, normalization, and residual or gated extensions do not consistently improve upon the shallow configurations. The paper also analyzes update stability and factor-dependent dynamics to help explain these observations. The implementation achieves approximately 2.2–3.3× higher training throughput than the official TTT implementation in two evaluated configurations. Finally, selected variants trained on 100B tokens at 410M and 1.45B parameters achieve competitive language-modeling and multiple-choice benchmark results relative to Gated DeltaNet, while precise long-context retrieval remains a limitation.
Critical strengths
- The paper provides substantial ablation coverage, including losses, learning-rate initialization, decay, nonlinearities, normalization, and deeper learners. The additional stabilization and initialization sweeps strengthen the empirical observation that the tested deeper configurations underperform the shallow alternatives. This provides actionable design guidance for TTT.(But, to mention, similar systematic studies of design choices have already been conducted in vision (see ViT³)).
- The scale-up experiments demonstrate that the selected configurations achieve competitive results on several language-modeling and downstream metrics. The paper also acknowledges limitations in precise retrieval, which I consider a strength.
Critical weaknesses
- How does Modular TTT differ from prior TTT methods? Do prior methods not already involve primitive-level forward, backward, and causal query computations? What does modularization actually add? For an entirely new inner network, would the update rules not still need to be implemented manually? Is deriving and implementing these rules after modifying the inner network particularly difficult? It remains unclear what practical benefits modularization brings to TTT implementation, especially compared with existing approaches (such as Learning to (Learn at Test Time): RNNs with Expressive Hidden States).
- Limitations of the empirical and theoretical evidence. First, Modular TTT shows no clear overall empirical advantage and underperforms GDN in some settings, including certain needle-in-a-haystack retrieval evaluations. Second, the factor-scaling analysis in Section 4.1 establishes that equivalent factorizations can induce different updates depending on the scaling coefficient . However, sensitivity to parameterization does not by itself establish greater optimization difficulty; a more direct connection to the observed failures of deeper learners is needed. Finally, the large speed gap between LaCT and Modular TTT warrants further explanation. More details on the benchmark configurations and implementations are needed to assess the fairness of this comparison.
Questions for authors
- What implementation effort does Modular TTT eliminate over prior TTT, and what still requires manual implementation?
- In which settings does Modular TTT offer a clear empirical benefit over GDN? How do the authors explain its weaker performance in some needle-in-a-haystack retrieval settings?
- How does factor-scaling sensitivity imply greater optimization difficulty? Could the authors provide direct experimental evidence linking this mechanism to the observed failures of deeper TTT memories?
- What explains the large speed gap with LaCT, and how is comparison fairness ensured?
3 comments
3. How factor-scaling sensitivity creates optimization difficulty and links to deep-memory failures.
In a factorized fast learner, the outer optimizer cannot merely learn an effective product ; it must simultaneously keep factor scales balanced across chunks so that coupled one-step updates neither vanish nor overshoot.
- Parameterization sensitivity turns a simple linear write into a coupled conditioning problem. A single fast matrix takes an uncoupled step . Rescaling the factors to and preserves the forward output, but their first-order contributions to the update of the product matrix scale by and , respectively (Equation 13). Poorly balanced factor scales therefore distort the effective update even when the forward function is identical.
- Targeted interventions link factor coupling and normalization conditioning directly to observed deep failures. First, Proposition 5 predicts that all-zero factors trap coupled gradients at zero while leaving a single matrix unaffected; Table 25 confirms that zero initialization works for shallow learners but fails on every multi-factor learner until Gaussian initialization is used. Second, even with Gaussian initialization, two-layer learners fail or degrade at the shallow baseline's and require smaller , while
Norm-containing graphs suffer additional gradient amplification in Equation 9 unless or mean scaling is raised. Below,Faileddenotes divergence toNaNor stalled high loss with gradient norms above . Even after stabilizing and tuning each family, every deep learner still trails shallowLinear-SiLU, and we will present factor coupling alongside normalization conditioning rather than as the sole cause.
| Topology at 160M | Failure Mechanism & Targeted Fix | Before After Loss |
|---|---|---|
| Linear-SiLU, shallow baseline | Zero init Gaussian init at | |
| Linear-Linear | Coupled zero-gradient trap: Zero init Gaussian init | |
| Linear-SiLU-Linear-SiLU | Coupled step scale: Zero or Gaussian, | |
| Linear-SiLU-Linear-Norm | Norm amplification: at | |
| Linear-Norm-Linear | Coupled step + residual scale: with inner loss | |
| Residual SiLU | Branch scale amplification: no mean scaling mean scaling at |
4. The speed gap with LaCT and comparison fairness.
Table 27 reports end-to-end inference throughput under an identical workload: during decoding, Modular TTT reuses its fast-weight state and processes one new token per step, whereas the evaluated LaCT implementation recomputes the full prefix.
- We matched the prompt length, decode length, and batch size across models at each scale. We used a single A100 with BF16 and per-layer
torch.compile, a 2,048-token prompt, and 128 decode steps. The batch size was 32 at 160M and 16 at 410M. For LaCT, we used our default training configuration. - At
410M,M-SiLU, MSEand LaCT reach the same final loss of2.5582, while Modular TTT trains nearly3×faster with35%lower peak memory. Both were trained for100,000steps at sequence length4,096under the same effective batch of16per device across micro-batch settings, as shown below.
| Model at 410M, 100B Tokens | Final Loss | Training Throughput tok/s | Peak Memory GiB |
|---|---|---|---|
| LaCT | 2.5582 | 7,447.2 | 62.95 |
| M-SiLU, MSE | 2.5582 | 21,438.7 | 40.94 |
| M-SiLU, Inner | 2.5569 | 21,569.7 | 40.97 |
| M, MSE | 2.5648 | 24,101.0 | 36.28 |
| M, Inner | 2.5621 | 24,310.8 | 34.76 |
- In two controlled training comparisons, enabling Muon alone reduces LaCT's throughput by
31%and40%. We varied only Muon within each pair. Enabling it reduced throughput from 108,020 to 74,235 tokens/s in the smaller model and from 17,912 to 10,751 tokens/s in the larger model.
Dear Reviewer vK2x,
Thank you very much for your insightful questions regarding DAG modularization, GDN retrieval comparisons, deep-memory failure mechanisms, and LaCT benchmark fairness, which have allowed us to provide much stronger support for Modular TTT. We have addressed all your points with targeted comparative experiments and detailed clarifications. We would truly welcome your feedback on our responses and additional results before the discussion window ends. Thank you very much again for your time and thoughtful review.
Thank you for your detailed questions about our framework and our comparison with LaCT.
1. Modular TTT is a unified framework rather than a new TTT method.
Modular TTT is not a new TTT method, but a unified execution framework for constructing and comparing TTT variants by automatically composing primitive-level rules ϕtrain, ϕtrain-bwd, and ϕquery over any valid learner DAG.
TTT-LinearandTTT-MLPmaintain separate hand-derived dual-form updates, normalization placements, and state layouts, so modifying the inner network requires rewriting global update and cache code. Modular TTT instead defines local rules once forLinear,Act,Norm,Add,Mul, andGatein Table 1; recombining them into any new learner topology requires only changing the declarative forward graph, while manual implementation is needed only when adding a brand-new primitive type:torch.compile-compatible execution. Autodiff can compute train-view gradients dV^, but it generates neither chunkwise causal query readout over s≤t without materializing per-token weight copies nor the fast-weight state transition:O=QW−Tril(QK^⊤⊙M)dV^,W←W−dW
TTT-E2E uses the same fast weights throughout each mini-batch and updates them at the batch boundary. In Modular TTT, each query incorporates the causal prefix of fast-weight updates within the current chunk. Its analytic backward rules avoid inner-loop
autograd.gradso the graph compiles undertorch.compile, running 2.00× and 3.14× faster than Official TTT in pure PyTorch and 2.2×–3.3× faster with Triton in Table 32.2. Empirical benefit over GDN and weaker retrieval results.
Our goal is not a SOTA model that beats specialized GDN everywhere, as our main contribution is enabling reproducible single-factor TTT comparisons; still, Modular TTT outperforms GDN across several concrete settings while sharing the exact-retrieval limits of fixed-state models.
160Mvalidation loss, multiple-choice accuracy, and several NIAH settings. Beyond supporting general learner DAGs rather than one hardcoded recurrence, Modular TTT uses lower peak allocated inference memory than GDN across every variant at both evaluated scales in Table 27. On quality and retrieval,Linear-SiLUachieves lower160Mvalidation loss than GDN across all five seeds in Table 15, while in100B-token evaluations Modular TTT reaches lower410MWikiText perplexity and slightly higher410Mand1.45Bmultiple-choice accuracy in Table 7, beats GDN on1.45Bniah_singleacross all four variants at4Kin Table 29, and leads410Mniah_multiqueryandniah_multivalueacross1Kto8Kin Table 30.4K-only pretraining without long-context continual training, and fixed-state compression rather than a uniform deficit against GDN. Relative to GDN, NIAH ordering varies by scale and task: Modular TTT trails on410Mniah_singleunder general language-modeling selection without retrieval-specific tuning, yet scaling model capacity to1.45Breverses this at2Kand4Kin Table 29. Meanwhile, at8Kand on harder multi-key tasks in Tables 29 and 30, all models were trained only at4Kcontext without long-context continual pretraining. Like GDN and LaCT, Modular TTT compresses history into a fixed-size fast state without a per-token KV cache, leaving exact long-context recall an open limitation as noted in Appendix E.