- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Universal Test-Time Training (uTTT), which extends test-time training (TTT) and large-chunk TTT (LaCT) by replacing per-layer private fast-weight memories with a single memory shared across layers (and across time). All layers write key-value pairs to, and read queries from, this shared memory, so that information written by one layer can be read by other layers in later chunks, and the training loss can backpropagate through cross-layer writes. The paper instantiates this idea as uTTT-MoE, which routes reads and writes to a shared expert pool, and as uTTT-Dense. Experiments on language modeling (124M and 760M parameters, up to 32K context) and novel view synthesis show that the shared design outperforms a layer-private counterpart at equal total state and active compute, with the strongest claim being higher RULER accuracy than all other bounded-state baselines.
Critical strengths
-
The paper pursues a worthwhile direction, and its reporting is commendable. It targets long-context models whose memory does not grow with the input length, unlike the KV cache of full attention, and I consider this a direction that deserves continued study. The authors also transparently disclose results that are unfavorable to their own method. Specifically, the fixed-memory models still trail full attention in RULER accuracy at both scales (Table 1, p. 6), and retrieval beyond 32K falls to the level of the sliding-window baseline (Sec. 4.2, p. 7), which suggests a memory-capacity limit of current approaches at long contexts. Finally, the authors report all the configurations they tried (Tables E1 and E2, p. 24), which will help follow-up work.
-
Shared fast weights improve training efficiency. At 760M, uTTT-MoE reaches the final training loss of TTT-Dense with 15% fewer GPU-hours than TTT-Dense and 27% fewer than TTT-MoE (Sec. 4.3, p. 7). It also attains the highest training throughput in the setting of Figure 5a (p. 8), while per-token forward FLOPs differ by at most 0.6% across the models (Appendix G, p. 27). This matters because training cost is a central practical constraint when scaling LLMs, so an architecture that is cheaper to train is a welcome benefit.
Critical weaknesses
-
The central claim that sharing fast weights across layers is beneficial is not convincingly supported, especially on language modeling. The paper offers no principled argument for the benefit beyond showing that a cross-layer gradient path exists, and the authors themselves note that this does not establish its magnitude (App. I.1). The empirical evidence is also limited in scope: long-context results are reported mainly on synthetic benchmarks such as RULER (NIAH-style retrieval), and it remains unclear whether the claimed benefit carries over to practical LLM tasks, such as long-document QA, summarization, or code. In addition, "layer-private" is a misleading term for the baselines, since each layer in those models also takes the previous layer's output as input. The only difference is that the fast weights are not shared, so "unshared" would be more accurate.
-
The paper does not compare inference efficiency, which is a primary motivation for fixed-memory models such as linear attention in the first place. Efficiency is evaluated only for training, and only against TTT variants with unshared fast weights. No decoding throughput, latency, or memory results are reported against the DeltaNet-style, Mamba-2, or other baselines.
Questions for authors
- Can the authors explain why a layer-shared memory should help? The gradient-reach result only shows that a cross-layer path exists, not that it is useful, and the RULER gain at 760M comes almost entirely from the 4K setting. Is there evidence that the gain comes from reading what other layers wrote rather than from more flexible expert allocation?
- Does the design offer any advantage at inference time? Efficiency is reported only for training, and decoding is slower than TTT-Dense (App. G). Could the authors report decoding throughput, latency, and memory against DeltaNet, Gated DeltaNet, Mamba-2, and others?
- Does the method work on practical LLM tasks without degrading generation quality? The evaluation consists of per-token loss and RULER at 124M and 760M. Could the authors report downstream and open-ended generation results compared with full attention and the baselines?
- Do the gains at long contexts persist as the model is scaled up? In Table 1, the RULER gain of uTTT-MoE over TTT-MoE is spread across 4K–32K at 124M (+1.15, +4.06, +3.63, +1.70), whereas at 760M it is concentrated at 4K (+8.88) and disappears at 8K–32K. Could the authors explain this trend and show how the gains and the training cost, including the GPU peak memory (App. G), change with scale, for example with an intermediate scale or a longer token budget?
7 comments
Practical tasks, generation quality and scale (W1, Q3, Q4)
Weakness 1 (practical tasks), Q3 and Q4: evaluation beyond PPL and RULER, and behaviour with scale.
The common response adds six evaluations of eight 760M models (Tables G1–G6). They include the tasks the reviewer names, long-document QA, summarization and code (LongBench, Table G1), and open-ended generation compared with full attention (Table G5).
Long context: uTTT-MoE leads, including at 8K–32K .
- Beyond single-needle retrieval, BABILong (Table G2) tests reasoning over facts scattered through real book text; most of its tasks require combining several facts. uTTT-MoE is the most accurate of the eight models at every length from 4K to 32K, including against TTT-MoE, the unshared baseline of Table 1; at 16K and 32K, full attention falls far behind.
- At positions 30K–32K of the PG-19 and GovReport test sets (Table G3), uTTT-MoE has the lowest PG-19 perplexity of all eight models and the lowest GovReport perplexity among the bounded-state models.
- On LongBench (Table G1), uTTT-MoE has the highest multi-document QA and code scores of the eight models.
General ability and generation quality: no degradation .
- The same evaluations cover recall-intensive extraction and QA (Table G4), zero-shot commonsense reasoning (Table G6) and open-ended generation (Table G5). uTTT-MoE has the highest recall mean, commonsense average, FDA and HellaSwag of the eight models; in open-ended generation, evaluated with the standard SimCTG and Contrastive Decoding protocol, it has the highest MAUVE and higher coherence than full attention and TTT-MoE.
Inference efficiency (W2, Q2)
Table R3. Decoding efficiency (760M, one H200, BF16, 32K-token prompt, greedy decoding).
| Model | Latency, batch 1 (ms/token): released → optimised | Throughput b8 (tok/s) | b16 | b32 | b64 | GPU memory per sequence (GiB) |
|---|---|---|---|---|---|---|
| Transformer-SWA | 17.5 → 1.90 | 2,703 | 2,344 | 3,060 | 3,214 | 0.60 |
| DeltaNet-SWA | 28.7 → 3.09 | 1,947 | 1,976 | 2,807 | 3,022 | 0.61 |
| Gated DeltaNet-SWA | 29.8 → 3.58 | 1,736 | 1,892 | 2,719 | 2,974 | 0.61 |
| Mamba-2-SWA | 30.3 → 3.13 | 1,643 | 1,706 | 2,274 | 2,478 | 0.76 |
| TTT-Dense | 37.1 → 1.93 | 2,521 | 2,262 | 2,874 | 3,000 | 0.89 |
| TTT-MoE | 52.7 → 1.96 | 2,538 | 2,259 | 2,854 | 2,988 | 0.89 |
| uTTT-MoE | 54.9 → 2.00 | 2,497 | 2,202 | 2,840 | 2,982 | 0.89 |
W2 / Q2: Inference efficiency. We thank the reviewer and add decoding latency, throughput and memory against DeltaNet, Gated DeltaNet, Mamba-2 and Transformer-SWA (Table R3). We will add the table to App. G and release the code.
In short: uTTT-MoE's gains on multiple downstream tasks come at no extra inference cost over unshared TTT; despite training at test time, it decodes within 5% of DeltaNet/Gated DeltaNet at batch 32–64 and about 20% faster than Mamba-2 at batch 64, at ~0.28 GiB more state per sequence.
How Table R3 is measured.
- Latency (batch 1): released → optimised. Released is each model's released code in eager PyTorch, as in App. G; at this scale it mostly measures CPU kernel-launch overhead (DeltaNet-SWA: GPU busy 2.9 of 28.7 ms). Optimised captures every model's decoding step in a CUDA graph; the TTT models also use a fused decoding step, with outputs within run-to-run variation. Because batch-1 latency still depends on kernel fusion, throughput at batch ≥ 32 is the fairest comparison.
- Throughput. Same CUDA-graph setting. TTT numbers include the full cost of test-time training (gradient writes every 256 tokens, updates every 4,096 tokens).
- Memory. GPU memory per sequence; ~0.60 GiB is the 4K-window KV of every SWA model. The FP32 state used only at chunk-boundary updates (TTT-Dense, uTTT-MoE) is kept in host memory (bit-identical outputs; ≤ 4% throughput cost, included).
Does the design offer an advantage at inference time?
- Test-time training is cheap. Reading the fast weights costs ~85 MB per sequence per step (1/7 of the window KV), and gradient writes and updates take 2–4% of decoding time at batch 32–64.
- State is TTT's main cost. Beyond the window KV, TTT holds 0.29 GiB per sequence (fast weights 0.08, test-time-training state 0.16, write buffers 0.05), against ~0.01 GiB for DeltaNet/GDN and 0.16 GiB for Mamba-2. This limits the largest batch, not the speed at a given batch.
- Sharing adds no cost. uTTT-MoE matches TTT-Dense/MoE in latency and memory and is within 5% in throughput. The App. G slowdown came from the released decoding code, which concatenated the whole pool at every token.
Why sharing helps (W1, Q1)
We thank the reviewer for the detailed review, for finding bounded-memory long-context models a direction worth pursuing, and for noting our transparent reporting, including the unfavorable results and the full configuration tables (Tables E1 and E2). We answer in three parts: why sharing helps (W1, Q1); evaluation beyond PPL and RULER, and behaviour with scale (W1, Q3, Q4); and inference efficiency (W2, Q2). The new results are route knock-outs at 760M (this reply), six evaluations at 760M (common response, Tables G1–G6), and a decoding benchmark against DeltaNet, Gated DeltaNet and Mamba-2 (Table R3).
Weakness 1 (why sharing helps; "layer-private") and Q1.
We agree that Proposition 1 only shows that a cross-layer route exists. In short: sharing lets a shallow layer read what a deep layer wrote, which an unshared model cannot do, and removing cross-layer reads makes the trained models worse, in view synthesis and in language modeling at both scales.
1. What sharing adds.
In an unshared model, what a layer wrote for earlier context is read later only by the same layer, so information from beyond the attention window passes through at most L layers on its way to the output (proof of Proposition 1(ii)). With a shared memory, a shallow layer can read what a deep layer wrote and process it again, so the depth applied to earlier context is no longer bounded by L, as Feedback Transformers (Fan et al., 2020) argue for standard Transformers.
2. The trained models use this route.
The test is simple: take a trained model, block some of its reads at inference, keep everything else fixed (all parameters, including the router), and measure how much the loss rises. If a model does not use a route, blocking it costs nothing.
In language models, following Appendix I.2, we block all cross-layer reads: each layer reads only its own writes, which is all an unshared model can do. This measures exactly the route that sharing adds. The table gives the loss increase in mnat (0.001 nat), averaged over chunks 2–8 of 1,024 Books3 documents. The 124M rows are from Figure 4b; the 760M row is new and uses the checkpoint of Tables G1–G6.
3. Reading other layers' writes and expert allocation are not rival explanations.
In uTTT-MoE, routing decides whose writes a token can read: a layer reads another layer's write only if both pick the same expert. Expert allocation is therefore how cross-layer reading is carried out, and the knock-outs above measure what this reading is worth. The common response ("Why uTTT outperforms TTT", point 2) makes the same argument in view synthesis and shows that the router does send different layers to the same experts.
Naming. We agree and will call the baselines "unshared" throughout the paper: they differ from uTTT only in that their fast weights are not shared.