acceptodds
Under review as a conference paper at ICLR 2027

Universal Test-Time Training

Abstract

Recent Test-Time Training (TTT) architectures compress context into fast weights that are updated online and queried as memory. Existing TTT designs keep this memory private to each layer: it recurs only over time, and depth merely indexes separate memories. We argue that memory ownership need not be tied to depth, and introduce Universal Test-Time Training (uTTT), in which all layers read and write one shared memory while retaining layer-specific backbone parameters. The shared memory thus recurs over two dimensions, time and depth, with chunks and layers as their units: a write by a deep layer in one chunk can be read by a shallow layer in the next. We instantiate this idea as uTTT-MoE and uTTT-Dense. uTTT-MoE routes each token head to a few experts in a pool shared by all layers; uTTT-Dense applies the whole shared memory at every layer without routing. In language modeling, uTTT-MoE reaches 15.5 and 27.9 RULER accuracy at 124M and 760M, 2.6 and 2.1 points above its layer-private counterpart at equal state and active compute, the highest among tested bounded-state models, with per-token loss matching or beating full attention. In novel view synthesis, sharing at fixed per-layer compute gains 0.92 dB in view-23 object PSNR in routed models and 0.76 dB in dense models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Training, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6rWwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Universal Test-Time Training (uTTT), which extends test-time training (TTT) and large-chunk TTT (LaCT) by replacing per-layer private fast-weight memories with a single memory shared across layers (and across time). All layers write key-value pairs to, and read queries from, this shared memory, so that information written by one layer can be read by other layers in later chunks, and the training loss can backpropagate through cross-layer writes. The paper instantiates this idea as uTTT-MoE, which routes reads and writes to a shared expert pool, and as uTTT-Dense. Experiments on language modeling (124M and 760M parameters, up to 32K context) and novel view synthesis show that the shared design outperforms a layer-private counterpart at equal total state and active compute, with the strongest claim being higher RULER accuracy than all other bounded-state baselines.

Critical strengths
  1. The paper pursues a worthwhile direction, and its reporting is commendable. It targets long-context models whose memory does not grow with the input length, unlike the KV cache of full attention, and I consider this a direction that deserves continued study. The authors also transparently disclose results that are unfavorable to their own method. Specifically, the fixed-memory models still trail full attention in RULER accuracy at both scales (Table 1, p. 6), and retrieval beyond 32K falls to the level of the sliding-window baseline (Sec. 4.2, p. 7), which suggests a memory-capacity limit of current approaches at long contexts. Finally, the authors report all the configurations they tried (Tables E1 and E2, p. 24), which will help follow-up work.

  2. Shared fast weights improve training efficiency. At 760M, uTTT-MoE reaches the final training loss of TTT-Dense with 15% fewer GPU-hours than TTT-Dense and 27% fewer than TTT-MoE (Sec. 4.3, p. 7). It also attains the highest training throughput in the setting of Figure 5a (p. 8), while per-token forward FLOPs differ by at most 0.6% across the models (Appendix G, p. 27). This matters because training cost is a central practical constraint when scaling LLMs, so an architecture that is cheaper to train is a welcome benefit.

Critical weaknesses
  1. The central claim that sharing fast weights across layers is beneficial is not convincingly supported, especially on language modeling. The paper offers no principled argument for the benefit beyond showing that a cross-layer gradient path exists, and the authors themselves note that this does not establish its magnitude (App. I.1). The empirical evidence is also limited in scope: long-context results are reported mainly on synthetic benchmarks such as RULER (NIAH-style retrieval), and it remains unclear whether the claimed benefit carries over to practical LLM tasks, such as long-document QA, summarization, or code. In addition, "layer-private" is a misleading term for the baselines, since each layer in those models also takes the previous layer's output as input. The only difference is that the fast weights are not shared, so "unshared" would be more accurate.

  2. The paper does not compare inference efficiency, which is a primary motivation for fixed-memory models such as linear attention in the first place. Efficiency is evaluated only for training, and only against TTT variants with unshared fast weights. No decoding throughput, latency, or memory results are reported against the DeltaNet-style, Mamba-2, or other baselines.

Questions for authors
  1. Can the authors explain why a layer-shared memory should help? The gradient-reach result only shows that a cross-layer path exists, not that it is useful, and the RULER gain at 760M comes almost entirely from the 4K setting. Is there evidence that the gain comes from reading what other layers wrote rather than from more flexible expert allocation?
  2. Does the design offer any advantage at inference time? Efficiency is reported only for training, and decoding is slower than TTT-Dense (App. G). Could the authors report decoding throughput, latency, and memory against DeltaNet, Gated DeltaNet, Mamba-2, and others?
  3. Does the method work on practical LLM tasks without degrading generation quality? The evaluation consists of per-token loss and RULER at 124M and 760M. Could the authors report downstream and open-ended generation results compared with full attention and the baselines?
  4. Do the gains at long contexts persist as the model is scaled up? In Table 1, the RULER gain of uTTT-MoE over TTT-MoE is spread across 4K–32K at 124M (+1.15, +4.06, +3.63, +1.70), whereas at 760M it is concentrated at 4K (+8.88) and disappears at 8K–32K. Could the authors explain this trend and show how the gains and the training cost, including the GPU peak memory (App. G), change with scale, for example with an intermediate scale or a longer token budget?
7 comments
AuthorsOpenReview ↗

Why sharing helps (W1, Q1)

We thank the reviewer for the detailed review, for finding bounded-memory long-context models a direction worth pursuing, and for noting our transparent reporting, including the unfavorable results and the full configuration tables (Tables E1 and E2). We answer in three parts: why sharing helps (W1, Q1); evaluation beyond PPL and RULER, and behaviour with scale (W1, Q3, Q4); and inference efficiency (W2, Q2). The new results are route knock-outs at 760M (this reply), six evaluations at 760M (common response, Tables G1–G6), and a decoding benchmark against DeltaNet, Gated DeltaNet and Mamba-2 (Table R3).

Weakness 1 (why sharing helps; "layer-private") and Q1.
We agree that Proposition 1 only shows that a cross-layer route exists. In short: sharing lets a shallow layer read what a deep layer wrote, which an unshared model cannot do, and removing cross-layer reads makes the trained models worse, in view synthesis and in language modeling at both scales.
1. What sharing adds.
In an unshared model, what a layer wrote for earlier context is read later only by the same layer, so information from beyond the attention window passes through at most L layers on its way to the output (proof of Proposition 1(ii)). With a shared memory, a shallow layer can read what a deep layer wrote and process it again, so the depth applied to earlier context is no longer bounded by L, as Feedback Transformers (Fan et al., 2020) argue for standard Transformers.
2. The trained models use this route.
The test is simple: take a trained model, block some of its reads at inference, keep everything else fixed (all parameters, including the router), and measure how much the loss rises. If a model does not use a route, blocking it costs nothing.
In language models, following Appendix I.2, we block all cross-layer reads: each layer reads only its own writes, which is all an unshared model can do. This measures exactly the route that sharing adds. The table gives the loss increase in mnat (0.001 nat), averaged over chunks 2–8 of 1,024 Books3 documents. The 124M rows are from Figure 4b; the 760M row is new and uses the checkpoint of Tables G1–G6.

ModelCross-layer reads blocked (each layer reads only its own writes)
124M uTTT-MoE0.57
124M Cut (cross-layer gradients stopped in training)0.10
760M uTTT-MoE1.81
  • The models use cross-layer reads. The cost is positive for uTTT-MoE at both scales. At 760M, 93% of the documents get worse, and the cost grows with the amount of earlier context (0.12 mnat in chunk 2, 4.11 in chunk 8). The same holds in view synthesis: blocking layer 3 from reading layer 8's writes costs 0.69 dB (common response, point 2).
  • Cross-layer reads account for a substantial part of the advantage over unshared models. At 760M, blocking them shrinks uTTT-MoE's advantage over TTT-MoE at positions 30K–32K by 36% on the same documents.
  • The models use the direction predicted in section 1: top-down. At 760M, blocking only the reads of deeper layers' writes costs 0.87 mnat, and blocking only the reads of shallower layers' writes costs 0.30. In view synthesis, too, the most important route is layer 3 reading layer 8.
  • The gradient path of Proposition 1 teaches the layers to use this route. Cut has the same architecture and forward computation as the 124M uTTT-MoE; only the cross-layer gradient of Proposition 1 is stopped in training, so a layer's read sends no gradient to other layers' writes. Cut can still read other layers' writes, yet barely does: blocking them costs it about one sixth as much as the 124M uTTT-MoE (Cut row).

3. Reading other layers' writes and expert allocation are not rival explanations.
In uTTT-MoE, routing decides whose writes a token can read: a layer reads another layer's write only if both pick the same expert. Expert allocation is therefore how cross-layer reading is carried out, and the knock-outs above measure what this reading is worth. The common response ("Why uTTT outperforms TTT", point 2) makes the same argument in view synthesis and shows that the router does send different layers to the same experts.
Naming. We agree and will call the baselines "unshared" throughout the paper: they differ from uTTT only in that their fast weights are not shared.

AuthorsOpenReview ↗

Practical tasks, generation quality and scale (W1, Q3, Q4)

Weakness 1 (practical tasks), Q3 and Q4: evaluation beyond PPL and RULER, and behaviour with scale.
The common response adds six evaluations of eight 760M models (Tables G1–G6). They include the tasks the reviewer names, long-document QA, summarization and code (LongBench, Table G1), and open-ended generation compared with full attention (Table G5).
Long context: uTTT-MoE leads, including at 8K–32K .

  • Beyond single-needle retrieval, BABILong (Table G2) tests reasoning over facts scattered through real book text; most of its tasks require combining several facts. uTTT-MoE is the most accurate of the eight models at every length from 4K to 32K, including against TTT-MoE, the unshared baseline of Table 1; at 16K and 32K, full attention falls far behind.
  • At positions 30K–32K of the PG-19 and GovReport test sets (Table G3), uTTT-MoE has the lowest PG-19 perplexity of all eight models and the lowest GovReport perplexity among the bounded-state models.
  • On LongBench (Table G1), uTTT-MoE has the highest multi-document QA and code scores of the eight models.

General ability and generation quality: no degradation .

  • The same evaluations cover recall-intensive extraction and QA (Table G4), zero-shot commonsense reasoning (Table G6) and open-ended generation (Table G5). uTTT-MoE has the highest recall mean, commonsense average, FDA and HellaSwag of the eight models; in open-ended generation, evaluated with the standard SimCTG and Contrastive Decoding protocol, it has the highest MAUVE and higher coherence than full attention and TTT-MoE.
AuthorsOpenReview ↗

Inference efficiency (W2, Q2)

Table R3. Decoding efficiency (760M, one H200, BF16, 32K-token prompt, greedy decoding).

ModelLatency, batch 1 (ms/token): released → optimisedThroughput b8 (tok/s)b16b32b64GPU memory per sequence (GiB)
Transformer-SWA17.5 → 1.902,7032,3443,0603,2140.60
DeltaNet-SWA28.7 → 3.091,9471,9762,8073,0220.61
Gated DeltaNet-SWA29.8 → 3.581,7361,8922,7192,9740.61
Mamba-2-SWA30.3 → 3.131,6431,7062,2742,4780.76
TTT-Dense37.1 → 1.932,5212,2622,8743,0000.89
TTT-MoE52.7 → 1.962,5382,2592,8542,9880.89
uTTT-MoE54.9 → 2.002,4972,2022,8402,9820.89

W2 / Q2: Inference efficiency. We thank the reviewer and add decoding latency, throughput and memory against DeltaNet, Gated DeltaNet, Mamba-2 and Transformer-SWA (Table R3). We will add the table to App. G and release the code.

In short: uTTT-MoE's gains on multiple downstream tasks come at no extra inference cost over unshared TTT; despite training at test time, it decodes within 5% of DeltaNet/Gated DeltaNet at batch 32–64 and about 20% faster than Mamba-2 at batch 64, at ~0.28 GiB more state per sequence.

How Table R3 is measured.

- Latency (batch 1): released → optimised. Released is each model's released code in eager PyTorch, as in App. G; at this scale it mostly measures CPU kernel-launch overhead (DeltaNet-SWA: GPU busy 2.9 of 28.7 ms). Optimised captures every model's decoding step in a CUDA graph; the TTT models also use a fused decoding step, with outputs within run-to-run variation. Because batch-1 latency still depends on kernel fusion, throughput at batch ≥ 32 is the fairest comparison.
- Throughput. Same CUDA-graph setting. TTT numbers include the full cost of test-time training (gradient writes every 256 tokens, updates every 4,096 tokens).
- Memory. GPU memory per sequence; ~0.60 GiB is the 4K-window KV of every SWA model. The FP32 state used only at chunk-boundary updates (TTT-Dense, uTTT-MoE) is kept in host memory (bit-identical outputs; ≤ 4% throughput cost, included).

Does the design offer an advantage at inference time?

- Test-time training is cheap. Reading the fast weights costs ~85 MB per sequence per step (1/7 of the window KV), and gradient writes and updates take 2–4% of decoding time at batch 32–64.
- State is TTT's main cost. Beyond the window KV, TTT holds 0.29 GiB per sequence (fast weights 0.08, test-time-training state 0.16, write buffers 0.05), against ~0.01 GiB for DeltaNet/GDN and 0.16 GiB for Mamba-2. This limits the largest batch, not the speed at a given batch.
- Sharing adds no cost. uTTT-MoE matches TTT-Dense/MoE in latency and memory and is within 5% in throughput. The App. G slowdown came from the released decoding code, which concatenated the whole pool at every token.

Reviewer_3SV2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes universal test-time training (uTTT), which allows different layers to jointly read from and write to a shared fast-weight memory updated through online gradient steps, while retaining layer-specific backbone parameters. Within each chunk, the model reads the previous memory state and aggregates writes from all layers at the chunk boundary, allowing deeper-layer information to be accessed by shallower layers in subsequent chunks and overcoming the unidirectional constraint on information flow across layers. The authors present Dense and MoE variants that improve language modeling and novel view synthesis, and investigate the role of cross-layer memory exchange through gradient probes and path ablation experiments.

Critical strengths
  1. The paper presents a clear motivation: sharing a memory module across layers while using MoE to control the amount of active memory. The experimental results demonstrate the method’s effectiveness, particularly its superior retrieval recall compared with similar methods in long-context settings.

  2. The analytical experiments support the motivation by showing that shared memory indeed enables cross-layer information flow. These experiments are carefully designed and convincing.

Critical weaknesses
  1. The average RULER improvement of the 760M model mainly comes from the 4K and 8K settings, while its performance at 16K and 32K is slightly below that of TTT-MoE with layer-private memory. In Appendix F, the 124M model’s accuracy drops to 3.7% at 64K, close to the sliding-window baseline. It therefore remains unclear whether this architecture offers a significant advantage in retrieval recall for longer contexts, particularly sequence lengths beyond the training length.

  2. The language modeling experiments focus on PPL and RULER, with limited evaluation beyond these metrics, making it difficult to comprehensively assess the strengths and weaknesses of sharing fast-weight memory across layers.

Questions for authors

Were the models in Table 1 trained with multiple random seeds? Reporting the mean and variance of their performance would make the results more convincing.

3 comments

We thank the reviewer for the positive assessment of our motivation and our analyses.
W1 and W2: long-context ability, and evaluation beyond PPL and RULER.
The common response adds six evaluations of eight 760M models (Tables G1–G6). They cover both the abilities the reviewer asks about: long-context use, and general language ability on tasks other than perplexity and retrieval.

  • Long context: uTTT-MoE leads.
    • Beyond single-needle retrieval, BABILong (Table G2) tests reasoning over facts scattered through real book text; most of its tasks require combining several facts. uTTT-MoE is the most accurate of the eight models at every length from 4K to 32K, including against TTT-MoE, the unshared baseline of Table 1; at 16K and 32K, full attention falls far behind.
    • At positions 30K–32K of the PG-19 and GovReport test sets (Table G3), uTTT-MoE has the lowest PG-19 perplexity of all eight models and the lowest GovReport perplexity among the bounded-state models.
    • On LongBench (Table G1), uTTT-MoE has the highest multi-document QA and code scores of the eight models.
  • General ability: no degradation.
    The same evaluations cover recall-intensive extraction and QA (Table G4), zero-shot commonsense reasoning (Table G6) and open-ended generation (Table G5). uTTT-MoE has the highest recall mean, commonsense average, FDA and HellaSwag of the eight models; in open-ended generation, evaluated with the standard SimCTG and Contrastive Decoding protocol, it has the highest MAUVE and higher coherence than full attention and TTT-MoE.
  • Beyond the training length.
    We agree that retrieval does not extrapolate, and we will say so in Section 4.2. At 64K every 32K-trained model falls toward the level of its attention window: full attention scores 1.4 and uTTT-MoE 3.7 (paper, Table F2). In view synthesis, where attention never spans views, the gain from sharing still holds at twice the number of views seen in training (Section 5.2; common response).

For our model at 124M, Appendix E.1 (Figure E2a) reports six training seeds of the 124M uTTT-MoE configuration of Table 1 (seeds 7, 42, 43, 101, 713 and 2027, all at step 10,240): per-token loss 3.0863 ± 0.0064 and RULER 15.60 ± 0.75 (mean ± standard deviation over seeds; ranges 3.0753–3.0939 and 14.47–16.60). The run in Table 1 is seed 42 (3.0857 and 15.46). It ranks third of the six on RULER, slightly below the mean, and second on loss: a typical seed, not the best one.

Thanks for your reply. My fundamental assessment of this paper remains unchanged, and I will maintain my score.

Thank you for your positive assessment and helpful comments! We greatly appreciate your time and insightful comments throughout the review process.

Reviewer_XFtEOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

In test-time training, a function is trained online to learn key–value associations, with its weights serving as a compressed memory of the input context. This paper proposes sharing this memory across depth instead of maintaining a separate memory for each layer. THis allows layers to retrieve information written by other layers in previous chunks. The authors introduce dense and MoE variants of the shared memory and evaluate them on language modeling using 124M and 760M parameters neural networks. They show higher training throughput compare to TTT-MoE.

Critical strengths
  1. Simple architectural approach to sharing adaptive memory across layers.
  2. Impressive throughput gains (tho I have a question regarding this)
  3. Matched baselines and targeted ablations help isolate the benefits of sharing.
Critical weaknesses
  1. The benefit of memory sharing for long-context retrieval appears limited at the larger evaluated scale (Section 4)
  2. Sharing memory across depth complicates pipeline parallelism because the fast-weight state can no longer be partitioned naturally alongside the layers. Keeping memory accesses local would require replicating the shared pool across pipeline stages. This is concerning for both training and inference.
  3. Limited model size, 760M is a small size model to proof that method is scalable.
Questions for authors
  1. Throughput comparison.

Figure 5 reports up to approximately 39% higher training throughput for uTTT-MoE than for TTT-MoE. The paper attributes this advantage to aggregating updates into fewer, larger kernel launches. However, in TTT-MoE, the inner gradient computation and memory update for layer ll could potentially overlap with the forward pass of layer l+1l+1. Alternatively, independent updates to private memories could be batched across layers. Could the authors clarify whether the baseline uses either optimization and provide an ablation or more detailed profiling to distinguish the benefit of memory sharing from differences in execution schedules?

  1. Peak memory.

Appendix G reports that uTTT-MoE uses 29–37% more peak training memory than TTT-Dense in the tested settings. Could the authors provide a comparison with TTT-MoE under matched configurations and explain which components account for the difference?

3 comments
AuthorsOpenReview ↗

Weaknesses W1–W3

We thank the reviewer for recognizing the simplicity of the approach, the throughput gains, and the matched baselines and targeted ablations. Below we answer the three weaknesses and then the two questions; the common response adds six new evaluations at 760M (Tables G1–G6).

W1: We add long-context tasks at 760M, and they show the advantage of uTTT-MoE.

The common response evaluates all eight 760M models on them (Tables G1–G3):

  • BABILong (Table G2) requires combining several facts scattered through book text. uTTT-MoE is the most accurate of the eight models at every length from 4K to 32K (26.0, 15.2, 9.6, 7.0), against 22.2, 11.4, 5.2 and 3.2 for TTT-MoE.
  • On long-document perplexity at positions 30K–32K (Table G3), it is the lowest of all eight models on PG-19 (14.79 vs. 15.43 for full attention) and the lowest of the bounded-state models on GovReport (6.56 vs. 6.67).
  • On LongBench (Table G1), it has the highest multi-document QA score , and its code score is above full attention and the other fast-weight models.

W2: Pipeline-parallelism trade-offs.

We agree that the shared pool complicates pipeline parallelism. Because the pool is fixed within each chunk (Sec. 3.1), a possible design passes each chunk through the pipeline stages with a pool snapshot and accumulated write gradients, applying the update at the final stage. Independent sequences can be interleaved, but successive chunks of the same sequence must wait for the preceding update.

Our experiments use FSDP; this pipeline design remains unimplemented and unbenchmarked. We will discuss its communication overhead, state-management costs, and scheduling constraints in the limitations.

W3: Our model scale is within the range used in this line of work, and we train at a much longer context (32K).
We agree that evaluation at larger scales would strengthen the evidence for scalability. Our 760M setting follows LaCT [1] and gives a controlled comparison of eleven models, all trained at 32K context. This context is 8–16 times longer than the one used to train several of our baselines in their original papers: 2K for MoM [2] and Mamba-2 [3], and 4K for Gated DeltaNet [4] and Kimi Linear [5]. Compute constraints limited the model sizes in this study; we plan to extend the comparison to 1.3B models in future work to assess how the benefits scale with model size.

[1] Zhang et al. Test-Time Training Done Right. arXiv:2505.23884, 2025.
[2] Du et al. MoM: Linear Sequence Modeling with Mixture-of-Memories. arXiv:2502.13685, 2025.
[3] Dao, Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML 2024.
[4] Yang, Kautz, Hatamizadeh. Gated Delta Networks: Improving Mamba2 with Delta Rule. ICLR 2025.
[5] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692, 2025.

AuthorsOpenReview ↗

Throughput and peak memory (Q1, Q2)

Q1: Throughput comparison.

We thank the reviewer and agree that execution scheduling must be controlled. The two models have the same fast-weight state and nearly identical forward FLOPs (within 0.6%; App. G); sharing changes how updates are grouped rather than reducing their computational workload.

The original TTT-MoE uses neither overlap nor cross-layer batching. Table R1 adds cross-layer batching while keeping its memories private, improving throughput by 20–23% and reducing the uTTT-MoE advantage from 25%/41% to 5%/14% at 124M/760M. After further optimizations to both models and matched recomputation, uTTT-MoE remains 18%/15% faster. Thus, scheduling explains a substantial part of the original gap; the remaining difference should not be attributed solely to memory sharing.
We will add Table R1 and the batched baseline, and clarify that the original reported speedup compares the original implementations.

Table R1. Cumulative training-throughput ablation (8×H200, 32K tokens, batch 1 per GPU as in Fig. 5a). Cells give tokens/s/GPU. Ratio is uTTT-MoE ÷ TTT-MoE in the same row.

Cumulative stepTTT-MoE 124MuTTT-MoE 124MRatio 124MTTT-MoE 760MuTTT-MoE 760MRatio 760M
Original15,82619,8341.257,739108871.41
+ Cross-layer batched update (schedule)18,94919,8341.059542108871.14
+ Merged raw-gradient call across layers¹21,68424,8211.141089612,2901.13
+ Kernel-level optimisations²29,33334,5201.181462716,8341.15

Original: the unmodified implementations behind all results in the paper (Fig. 5, App. G).

¹ For uTTT-MoE this step also removes an extra checkpoint around its update, so both models recompute the same amount.
² Additional fused routing/FFN kernels, launch caching, FlashAttention/KV reuse in recomputation, and compiled read/write pre- and post-processing.

Q2: Peak training memory.

We thank the reviewer. We have added a matched comparison with TTT-MoE, including a variant with cross-layer batched updates to align the update schedule with uTTT-MoE.

Table R2 reports peak training memory at 760M parameters, batch size 1, and sequence length 32K on one H200, using BF16 parameters as in the 8-GPU runs. uTTT-MoE uses 35.49 GiB, compared with 27.49 GiB for the original TTT-MoE and 29.50 GiB for batched TTT-MoE—29.1% and 20.3% higher, respectively.

The model-state contribution is identical across all four configurations at 11.35 GiB. The additional memory comes from update-related tensors (raw gradients, momentum, Newton–Schulz intermediates, and update application) and activations retained for backward. Relative to the original TTT-MoE, these categories increase by 3.90 GiB and 4.10 GiB, respectively. Relative to batched TTT-MoE, the corresponding increases are 1.56 GiB and 4.42 GiB. Table R2 therefore makes explicit the memory cost accompanying the throughput and downstream-task results.

We will add this matched comparison and memory breakdown to Appendix G and discuss the throughput–memory tradeoff in Sec. 4.3. We will also distinguish this single-GPU diagnostic, which shows a 26.7% increase over TTT-Dense, from the 29–37% reported under the original Appendix G settings.

Table R2 note. Each entry measures tensors live at that implementation’s overall memory peak, attributed to the code that allocated them; entries are not independently measured component maxima.

Table R2. Where the training peak goes (GiB; 760M, batch 1 × 32K, one H200, BF16 parameters as in the 8-GPU runs). Every tensor live at the peak is attributed to the code that allocated it.

ComponentTTT-DenseTTT-MoETTT-MoE + batched updateuTTT-MoE
Model state (parameters, gradients, AdamW state, KV storage, LM head/loss)11.3511.3511.3511.35
Fast-weight update (raw gradients, momentum, Newton–Schulz, apply)2.081.674.015.57
Activations kept for the backward pass (attention, residual/LayerNorm, MLP, routing, update inputs)14.5814.4714.1518.57
Peak28.0127.4929.5035.49

Dear Reviewer,

Thank you again for your valuable comments.

As the discussion period is less than 10h, we would like to kindly follow up on our previous responses. We hope our detailed explanations have adequately addressed your concerns.

Please let us know if you need any further clarification.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Six new evaluations at 760M (Tables G1–G6)

Table G1. LongBench v1 (by category) and LongBench v2

ModelSingle-doc QA ↑Multi-doc QA ↑Summarization ↑Few-shot ↑Synthetic ↑Code ↑LongBench v2 ↑
Transformer12.618.311.323.02.734.327.0
Transformer-SWA16.119.37.920.20.731.328.0
GDN-SWA13.413.99.422.70.933.822.7
DeltaNet-SWA15.819.38.129.40.234.728.0
Mamba-2-SWA15.016.110.528.41.835.320.5
TTT-Dense13.316.28.831.32.936.028.0
TTT-MoE16.020.29.332.20.835.623.5
uTTT-MoE16.120.313.532.21.737.130.6

Table G2. BABILong

Model4K ↑8K ↑16K ↑32K ↑Mean ↑
Transformer17.68.83.62.28.1
Transformer-SWA7.67.44.41.85.3
GDN-SWA10.25.04.42.85.6
DeltaNet-SWA22.811.07.64.811.6
Mamba-2-SWA10.68.46.83.27.3
TTT-Dense17.410.86.86.810.5
TTT-MoE22.211.45.23.210.5
uTTT-MoE26.015.29.67.014.5

Table G3. Short- vs. long-context perplexity

ModelWikiText PPL ↓LAMBADA PPL ↓PG-19 PPL 30K+ ↓GovReport PPL 30K+ ↓
Transformer23.5022.2315.436.48
Transformer-SWA22.8318.9215.717.10
GDN-SWA22.5020.2015.337.02
DeltaNet-SWA22.5218.1315.326.97
Mamba-2-SWA21.7617.1215.156.84
TTT-Dense21.6216.3614.876.67
TTT-MoE21.8617.8114.976.67
uTTT-MoE21.5815.2814.796.56

Table G4. Recall-intensive tasks

ModelSWDEFDASQuAD-completionDROPTriviaQANQMean ↑
Transformer59.677.219.324.947.820.841.6
Transformer-SWA62.078.634.921.450.120.844.6
GDN-SWA63.573.441.122.747.920.744.9
DeltaNet-SWA62.276.736.324.950.720.945.3
Mamba-2-SWA65.980.630.322.749.921.245.1
TTT-Dense61.280.337.222.250.421.445.5
TTT-MoE63.278.430.023.951.121.544.7
uTTT-MoE65.584.140.223.952.821.848.1

Table G5. Open-ended generation

ModelDiversity ↑MAUVE ↑Coherence ↑
Transformer93.690.40.460
Transformer-SWA93.791.90.494
GDN-SWA90.390.10.508
DeltaNet-SWA92.391.50.495
Mamba-2-SWA92.991.50.508
TTT-Dense92.190.90.508
TTT-MoE93.391.10.490
uTTT-MoE93.792.50.504

Table G6. Zero-shot commonsense reasoning

ModelLAMBADAHellaSwagPIQAARC-eARC-cWinoGrandeOpenBookQAAverage ↑
Transformer37.038.464.945.725.052.229.041.7
Transformer-SWA39.239.364.947.126.251.828.042.4
GDN-SWA38.638.965.848.927.750.730.843.1
DeltaNet-SWA40.139.764.947.625.950.528.442.4
Mamba-2-SWA41.841.467.249.427.650.631.244.2
TTT-Dense43.341.666.245.127.452.629.043.6
TTT-MoE40.341.465.648.425.551.830.043.3
uTTT-MoE42.241.766.448.527.554.330.244.4
AuthorsOpenReview ↗

Evaluation protocol and references for Tables G1–G6

All results above are for our paper's 760M models, with the configurations of Table 1. We evaluate the final checkpoint of each base model zero-shot (no instruction tuning), with identical prompts, decoding and truncation for every model. The protocol for each table is given below.

  • G1. LongBench [1] under the official protocol, with inputs truncated to 32K tokens; each category is the mean over its datasets. LongBench v2 [2]: the 99 questions whose full prompt fits in 32K tokens.

  • G2. BABILong [3], tasks qa1–qa5, 100 samples per task and length, with the official prompts and scoring; the metric is accuracy.

  • G3. Short context: word-level perplexity on WikiText-2 and perplexity on LAMBADA. Long context: the PG-19 and GovReport test sets; 30K+ is the perplexity at positions 30,000–32,767.

  • G4. The recall-intensive suite of Based [4] and JRT [5], zero-shot; a prediction is correct if it contains the gold answer.

  • G5. Following SimCTG [6] and Contrastive Decoding [7], prefixes from WikiText-103, WikiNews and BookCorpus are continued for 256 tokens with nucleus sampling (p = 0.95). Diversity, MAUVE and Coherence (the SimCSE similarity between prefix and continuation) are computed as in these two papers.

  • G6. Zero-shot accuracy on LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, WinoGrande and OpenBookQA.HellaSwag, ARC-Challenge and OpenBookQA use length-normalized accuracy, following Mamba [8].Average is the mean over the seven tasks.

References
[1] Bai et al. LongBench: A bilingual, multitask benchmark for long context understanding. ACL 2024.
[2] Bai et al. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. ACL 2025.
[3] Kuratov et al. BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack. NeurIPS 2024 (Datasets and Benchmarks).
[4] Arora et al. Simple linear attention language models balance the recall-throughput tradeoff. ICML 2024.
[5] Arora et al. Just read twice: Closing the recall gap for recurrent language models. arXiv:2407.05483, 2024.
[6] Su et al. A contrastive framework for neural text generation. NeurIPS 2022.
[7] Li et al. Contrastive decoding: Open-ended text generation as optimization. ACL 2023.
[8] Gu & Dao. Mamba: Linear-time sequence modeling with selective state spaces. COLM 2024.

AuthorsOpenReview ↗

Why uTTT outperforms TTT

Why uTTT outperforms TTT: the gain comes from sharing
Reviewer 6rWw asks why a layer-shared memory should help: does the gain come from reading what other layers wrote, or from more flexible expert allocation? Our view-synthesis experiments (Section 5; Appendices H and I) answer this under tight controls.
Setup. The task maps onto long-context language modeling. The model reads photos of an object (views), one view per chunk, and renders the object from a new camera. Attention stays within each view, so information about earlier views can only pass through the fast-weight memory. Models train on 11 views and are tested on up to 23, about twice the training length; since attention never spans more than one view, this is a clean length-extrapolation test of the memory itself. Quality is PSNR (dB, higher is better); for scale, full attention leads Gated DeltaNet by 1.0 dB averaged over 23 views.
The experiments show three things:

  1. The gain grows as more layers share, at fixed compute, parameters and state.
  2. It comes from cross-layer reads and writes, which expert routing carries out.
  3. It does not come from more candidate experts or more capacity per layer.

1. The more layers share a pool, the larger the gain (Section 5.2; Figure 6, row A; Table H1).
We split the eight layers into p groups, each sharing one pool, and divide the 64 experts evenly among the pools. p = 8 is TTT-MoE (one pool per layer) and p = 1 is uTTT-MoE (one pool for all layers). Compute, parameters and total state stay fixed; only the degree of sharing changes.

PSNR (dB) ↑p = 8 (TTT-MoE)p = 4p = 2p = 1 (uTTT-MoE)
View 2324.5624.8924.9825.48
Mean over views 1–2323.9024.1124.2024.68

On Objaverse/GSO (1,019 test objects), PSNR rises each time the pools are halved (8 → 4 → 2 → 1), at view 23 and averaged over all views.
2. Cross-layer reading is carried out by expert allocation.
In uTTT-MoE, routing decides whose writes a token can read: a layer reads another layer's write only if both pick the same expert. So reading other layers' writes and expert allocation are not rival explanations. The real question is whether the router sends different layers to the same experts, since it could also keep them apart. It does:

  • Repeated content is routed together, and more so as more layers share. Two patches from different views that show the same surface point are like a fact mentioned twice in a document. Written at two different layers, they go to the same expert 1.40× as often as a nearby patch showing a different point, on every object; when the later patch reads, it goes to the earlier patch's experts 1.34× as often. The ratio rises with sharing, from 1.15× at p = 4 to 1.39× at p = 2 and 1.40× at p = 1, while at p = 8 no two layers can share an expert (Appendix I.3, Figure I1).
  • The model relies most on a route TTT cannot have: a shallow layer reading a deep layer's writes. Blocking layer 3 from reading layer 8's writes costs 0.69 dB (Section 5.2, Figure 4c; Appendix I.2).
AuthorsOpenReview ↗

General Response to Reviewers and Area Chair

We thank the reviewers for their constructive reviews. They recognized the simple design with matched baselines and targeted ablations (XFtE), the clear motivation and convincing analyses of cross-layer information flow (3SV2), and the worthwhile direction with transparent reporting (6rWw).

The reviewers raised three main concerns: (1) why sharing helps, beyond showing that a cross-layer gradient path exists (6rWw); (2) whether the gains carry over to practical tasks beyond perplexity and RULER, especially at 760M (6rWw, 3SV2, XFtE); and (3) efficiency: whether the training speedup comes from sharing or from the update schedule and what accounts for the peak memory (XFtE), and the inference cost (6rWw). We summarize how the discussion addressed each. Following Reviewer 6rWw, we call the baselines with per-layer fast weights “unshared”.

1. Why sharing helps. Sharing unifies time and depth in one memory recurrence (Section 3.4). In an unshared model, a layer can use only representations computed by itself and the layers below it (Proposition 1(ii)). With sharing, every layer can read what any layer has written, and each memory update runs through the whole network, which turns a stacked recurrence into a deep-transition one. The trained models use this route: blocking cross-layer reads makes them worse at both scales; at 760M, 93% of documents get worse, the cost grows with the amount of earlier context, and blocking top-down reads, which an unshared model cannot have, costs more than blocking bottom-up ones (0.87 vs 0.30 mnat). The gradient path is what teaches this use: a model trained with the cross-layer gradient stopped can still read other layers' writes, yet barely does. View synthesis, where attention stays within each view, gives the same picture under tight controls: at fixed compute, parameters and total state, PSNR rises each time more layers share a pool (24.56 → 25.48 dB at view 23, from one pool per layer to one pool for all eight layers), and the model relies most on a shallow layer reading a deep layer's writes.

2. Practical tasks at 760M. Six evaluations of eight of the 760M models in Table 1 show that the gains carry over (Tables G1–G6). On long-context tasks, uTTT-MoE is the most accurate of the eight on BABILong at every length from 4K to 32K (mean 14.5, against 10.5 for TTT-MoE and 8.1 for full attention), has the lowest PG-19 perplexity at positions 30K–32K, below full attention (14.79 vs 15.43), and the lowest GovReport perplexity among bounded-state models, and has the highest LongBench multi-document QA and code scores. On recall-intensive tasks, zero-shot commonsense reasoning and open-ended generation it shows no degradation, with the highest recall mean (48.1, against at most 45.5 for the other models), commonsense average and MAUVE. At equal state, uTTT-MoE also reaches a lower long-context loss than TTT-MoE in essentially every run we trained, at both scales (Tables E1–E2).

3. Efficiency. For training, uTTT-MoE and TTT-MoE have the same fast-weight state and nearly identical forward FLOPs. Following Reviewer XFtE, we added cross-layer batched updates to TTT-MoE: the update schedule explains a substantial part of the original speedup, and uTTT-MoE remains faster once the schedules are matched (Table R1). Table R2 attributes the difference in peak training memory to update-related tensors and activations kept for the backward pass; the model state is identical. At inference, uTTT-MoE has comparable cost to unshared TTT-MoE in decoding latency, throughput and memory; despite training at test time, it decodes within 5% of DeltaNet and Gated DeltaNet at batch 32–64 and about 20% faster than Mamba-2 at batch 64 (Table R3).

On scale (XFtE), our 760M setting follows LaCT and trains all eleven models at 32K context, 8–16 times longer than several baselines were trained on in their original papers. Retrieval beyond the 32K training length is not achieved by any 32K-trained model in our comparison (3SV2). We thank the reviewers again for their engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.