- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies In-Place Test-Time Training (TTT), where pretrained MLP parameters are updated online as fast weights. It compares Frobenius, Delta, gated, and decay-based update rules on Qwen3-4B and LLaMA-3.1-8B across RULER, HELMET, and pretrained-capability benchmarks. The main finding is that TTT-aware training often improves the underlying model even when inference-time fast-weight updates are disabled; the direct TTT effect is usually small and sometimes negative. The authors also show that training suppresses update magnitude, likely because larger updates interfere with pretrained capabilities, while query-aware prompting makes writing more selective.
Critical strengths
- The on/off disentanglement is a simple, valuable protocol, and it exposes a real gap in how TTT-for-LLM results are reported. Comparing the same trained model with fast-weight updates on and off separates the effect of inference-time adaptation from everything else that TTT-aware training changes. Prior In-Place TTT work does not report this comparison. Across six learning rules, two backbones, two training stages and two benchmark families, TTT-Effect stays within about ±1 point almost everywhere. That is notable for models trained explicitly with active updates. Because it is a paired comparison on the same model and the same examples, it is the most robust result in the paper. It should change how future work in this area evaluates and attributes improvements, and this is the main reason I see value in publishing.
- Thia paper provides a systematic, mechanism-level look at how training controls update magnitude across learning rules. The paper goes beyond benchmark tables. It tracks the relative output change, ‖V‖, the β gates and β selectivity over training.
Critical weaknesses
- The headline claim, that TTT-Benefit comes largely from the outer loop, rests on differences that are probably within run-to-run noise. Tables 2–5 report many differences of less than one percentage point, but the paper does not provide results across multiple independent training or evaluation repetitions. This makes it difficult to assess whether the small TTT-Benefit and TTT-Effect values are reliable.
- The proposed explanation for suppression (preserving pretrained capabilities) is not tested against an obvious alternative: full attention already makes the fast weights redundant. In-Place TTT keeps full causal attention over the whole context, and every evaluation fits inside the attention window. Attention already has lossless access to every needle, so outer-loop training has little reason to rely on a lossy fast-weight memory, whatever the cost to pretrained computation.
- Narrow experimental scope for general claims about pretrained parameters as dynamic memory. All conclusions come from one framework with one fixed set of hyperparameters: MLP down-projections in 6 of 36 layers, 4K chunks, η=3, β and α initializations taken from or near the original. There is no sweep over the number or placement of TTT layers, chunk size, inner-loop learning rate, or which parameters act as fast weights. Small TTT-Effect may well be a property of this configuration rather than of fast weights in pretrained LLMs in general. The LLaMA replication is partial: Stage 1 only, three rules, up to 64K. The capability-preservation test (Figure 2b) covers only GDN and GDN-Residual. The paper also does not reconcile its results with the original In-Place TTT paper. As I understand it, that paper reports clear long-context gains over its baseline under a nominally matched setup. Here, the per-sample fast-weight reset with padding to chunk boundaries is at least one visible deviation. Without that reconciliation, readers cannot tell whether this is a negative finding about the method or a reproduction gap. This limits how far the conclusions in the title and abstract can be generalized.
Questions for authors
- How much do scores vary across seeds for standard-CPT and for at least one TTT variant? Can you report bootstrap confidence intervals for TTT-Benefit, TTT-Effect and the outer-loop effect in Tables 2, 3 and 5? If the outer-loop effect does not exceed CPT seed variance, the central claim should be reframed.
- Did the authors reproduce the In-Place TTT paper’s reported RULER numbers with the Frobenius variant?
- For the delta variants, what are TTT-Effect and the relative output change at step 0, is the decline in Figure 1a mostly undoing the large initial erasure caused by V≈0, E≈−O and η=3?
4 comments
2. Are fast weights unused only because full attention already sees everything? (W2)
We agree that full attention may reduce reliance on fast-weight updates by keeping relevant information directly accessible. To test this, we constructed three hybrid models by replacing 25%, 50%, and 75% of the full-attention layers in Qwen3-4B-Base with sliding-window attention (SWA, 4K window), and trained both a standard-CPT baseline and a Frobenius TTT variant for each, following the Stage 1 recipe (32K context, 23B tokens).
As shown in the table below, restricting attention substantially limits the backbone's long-context performance: at 32K, standard-CPT drops from 81.5 with full attention to 56.9 with 75% SWA. In this setting, the fast weights are far less redundant with attention, and TTT-Benefit increases to +2.1/+2.4 at 16K/32K. However, TTT-Effect remains small (+0.7/−0.1), so the improvement still arises primarily from changes learned by the underlying model rather than from inference-time fast-weight updates. Thus, redundancy with full attention alone does not explain the limited fast-weight utilization.
| SWA Ratio | (16K) Standard-CPT | (16K) Frobenius | (16K) TTT-Benefit | (16K) TTT-Effect | (32K) Standard-CPT | (32K) Frobenius | (32K) TTT-Benefit | (32K) TTT-Effect |
|---|---|---|---|---|---|---|---|---|
| Qwen3-4B-Base | 88.5 | — | — | — | 82.2 | — | — | — |
| 0% | 90.0 | 89.8 | -0.2 | -0.1 | 81.5 | 82.0 | +0.5 | 0.0 |
| 25% | 88.0 | 88.2 | +0.3 | +0.3 | 73.5 | 71.5 | -2.0 | -1.1 |
| 50% | 86.4 | 85.7 | -0.7 | -0.1 | 73.8 | 74.2 | +0.4 | +0.6 |
| 75% | 81.2 | 83.2 | +2.1 | +0.7 | 56.9 | 59.3 | +2.4 | -0.1 |
Furthermore, our update-amplification experiments (Section 4.3, Figure 2) show that stronger fast-weight updates can degrade pretrained capabilities even when long-context retrieval remains intact. Together, these results suggest that capability preservation may also constrain fast-weight utilization, beyond the availability of full attention. We have added the SWA results and this discussion to Appendix C.6 (Table 9).
3. Experimental scope and generalizability. (W3, Q2)
We address these concerns through additional experiments and clarifications.
(1) Training configurations. To examine sensitivity to the configuration, we conducted additional ablations on inner-loop learning rate, number of TTT layers, and chunk size under stage-1 training (32K context, 23B tokens).
Reducing η from 3 to 0.3 yields TTT-Effects of 0.0, −0.1, and −0.5 points on RULER at 16K, 32K, and 64K, comparable to the original configuration (−0.1, 0.0, −0.3). Reducing the chunk size from 4,096 to 1,024 yields TTT-Effects of 0.0, +0.2, and −1.0, despite a +1.0-point TTT-Benefit at 32K.
Doubling the number of TTT layers from 6 to 12 (every third layer instead of every sixth) yields TTT-Effects of +0.1, −0.4, and −0.9 points, compared with −0.1, 0.0, and −0.3 for the original 6-layer configuration. Adding more writable layers thus makes the effect of inference-time updates slightly more negative rather than more useful, consistent with interference with pretrained computation.
| Variant | (16K) Perf. | (16K) TTT-Benefit | (16K) TTT-Effect | (32K) Perf. | (32K) TTT-Benefit | (32K) TTT-Effect | (64K) Perf. | (64K) TTT-Benefit | (64K) TTT-Effect |
|---|---|---|---|---|---|---|---|---|---|
| standard-CPT | 90.0 | — | — | 81.5 | — | — | 68.1 | — | — |
| Frobenius (η=3, 6 layers, chunk-size=4,096) | 89.8 | -0.2 | -0.1 | 82.0 | +0.5 | 0.0 | 67.4 | -0.7 | -0.3 |
| Frobenius (η=0.3) | 90.0 | 0.0 | 0.0 | 81.5 | 0.0 | -0.1 | 67.2 | -0.9 | -0.5 |
| Frobenius (chunk-size=1,024) | 90.1 | +0.1 | 0.0 | 82.8 | +1.0 | 0.2 | 68.3 | +0.1 | -1.0 |
| Frobenius (12 layers) | 89.6 | -0.4 | +0.1 | 80.9 | -0.6 | -0.4 | 68.0 | -0.1 | -0.9 |
Together with the six learning rules spanning update objectives, write gating, and decay, these ablations show limited or negative TTT-Effects across configurations, suggesting that the finding is not tied to a single setting.
(2) Extended LLaMA replication. We trained LLaMA-3.1-8B through Stage 2 (128K context, 17B tokens) and evaluated on three data seeds. The results remain consistent with our central observation: TTT-Effect is generally small (positive effects ≤+0.5), while reaching −1.8 ± 1.0 on RULER 128K. This further supports that limited fast-weight utilization beyond Qwen3-4B.
| Benchmark | Length | CPT Perf. | TTT Perf. | TTT-Benefit | TTT-Effect |
|---|---|---|---|---|---|
| RULER | 16K | 93.3 ± 0.8 | 93.1 ± 0.2 | −0.2 ± 0.6 | +0.1 ± 0.3 |
| 32K | 90.2 ± 0.8 | 89.8 ± 0.7 | −0.4 ± 0.1 | 0.0 ± 0.2 | |
| 64K | 82.3 ± 0.5 | 81.9 ± 0.5 | −0.4 ± 0.1 | −0.7 ± 0.5 | |
| 128K | 65.2 ± 1.5 | 65.4 ± 1.3 | +0.2 ± 0.8 | −1.8 ± 1.0 | |
| HELMET | 16K | 50.8 ± 0.9 | 50.0 ± 1.2 | −0.9 ± 0.3 | 0.0 ± 0.2 |
| 32K | 51.9 ± 1.0 | 50.9 ± 1.4 | −1.0 ± 0.4 | +0.2 ± 0.2 | |
| 64K | 54.6 ± 1.0 | 54.5 ± 0.6 | −0.1 ± 0.8 | +0.5 ± 0.3 | |
| 128K | 52.0 ± 0.9 | 51.6 ± 0.2 | −0.3 ± 0.7 | 0.0 ± 0.0 |
(3) Capability-preservation analysis. Figure 2b originally focused on GDN and GDN-Residual because these variants maintain retrieval performance under substantial update amplification, allowing us to examine capability interference without concurrent retrieval degradation. Following the reviewer's suggestion, we have updated Figure 2b to include all six learning rules and the standard-CPT baseline in the revised manuscript. The extended results show that Frobenius degrades substantially more under amplification than GDN and GDN-Residual, further illustrating how learning rules differ in their ability to preserve pretrained capabilities.
(4) Reproduction gap (Q2). Our Frobenius variant implements the original In-Place TTT learning rule using the publicly released codebase and follows its training configuration as closely as possible, but is trained on our own long-context corpus, as the original training data is not publicly available. Our corpus consists of 25B unique tokens curated from publicly available instruction-following, math, code, and heavily filtered web documents, with approximately 75% long documents. Standard continued pretraining on this corpus substantially improves long-context performance of Qwen3-4B-Base (RULER 65.8→78.6 at 64K and 50.5→69.5 at 128K), suggesting the data is suitable for continued pretraining.
Regarding per-sample fast-weight resets and chunk-boundary padding, these choices were made to prevent updates from carrying over between independent samples. Since the original data preprocessing details are unavailable, we cannot confirm whether these choices differ from the original setup. Thus, while our implementation closely follows the original method, differences in data and preprocessing may account for some of the discrepancy.
Importantly, our central finding does not depend on matching the original gains. The original work reports gains over its baseline, which corresponds to TTT-Benefit, whereas TTT-Effect is a paired comparison within the same trained model. A larger TTT-Benefit therefore does not by itself imply greater reliance on inference-time updates, as our SWA experiments illustrate (response to W2).
(5) Scope of conclusions. With these additions, the small TTT-Effect holds across six learning rules varying the update objective, write gating, and decay; two backbones (Llama and Qwen), both trained up to 128K context; different inner-loop learning rates, numbers of TTT layers, and chunk size; hybrid models with sliding-window attention (response to W2); standard and query-aware prompting; and both training- and evaluation-seed variation (response to W1). Thus, within the setting where pretrained parameters are repurposed as fast weights, limited fast-weight utilization is consistent across all configurations we examined.
4. Is the decline in update magnitude just recovery from a disruptive initialization? (Q3)
We agree that the initial updates in prediction-error rules can be disruptive, since V ≈ 0 at initialization leads to a large residual E ≈ −O.
The table below extends the relative MLP output changes shown in Figure 1a to include initialization (step 0), with exact values reported at each training step.
| Step | Delta | Delta + β | GDN |
|---|---|---|---|
| 0 | 0.348 | 0.098 | 0.097 |
| 1500 | 0.292 | 0.069 | 0.068 |
| 3000 | 0.284 | 0.058 | 0.058 |
| 4500 | 0.279 | 0.054 | 0.054 |
| 6000 | 0.277 | 0.053 | 0.051 |
| 7500 | 0.275 | 0.052 | 0.050 |
| 9000 | 0.275 | 0.052 | 0.050 |
| 11077 | 0.275 | 0.052 | 0.050 |
At step 0, the relative output change is 0.348 for Delta, 0.098 for Delta+β, and 0.097 for GDN, decreasing to 0.292 / 0.069 / 0.068 by step 1.5K; the corresponding step-0 TTT-Effects on RULER 16K/32K are −0.6/−7.4 points for Delta, +0.2/+1.0 for Delta+β, and +0.1/+1.2 for GDN. For these rules, a substantial part of the decline indeed occurs early in training, consistent with the reviewer's hypothesis, particularly for ungated Delta, whose initial updates substantially hurt retrieval.
However, recovery from initialization does not fully explain the attenuation in the gated variants. For Delta+β and GDN, the fast-weight updates at initialization (step 0) actually improve retrieval (TTT-Effect on RULER of +1.0 and +1.2 at 32K), yet training reduces this effect to +0.2 and +0.5, respectively, and their write gates continue to decrease throughout training (Figure 1c). If outer-loop training optimized only for retrieval, there would be little reason to suppress updates that already help it. This suggests that the attenuation is driven by another pressure, consistent with our update-amplification results (Figure 2b), where stronger updates degrade pretrained capabilities even when retrieval is preserved.
Dear reviewer b3ze, Thank you again for your thoughtful and constructive feedback. As we near the end of the discussion period, we hope our responses and additional experiments have addressed your concerns. We are happy to discuss any aspects of the paper that may require further clarification.
Dear Reviewer b3ze,
We sincerely thank the reviewer for the thoughtful and constructive assessment of our work. We especially appreciate the reviewer's recognition of the value and robustness of our TTT-ON/OFF comparison protocol, its potential impact on future TTT research, and our systematic, mechanism-level analyses of fast-weight updates.
1. Are the small TTT-Benefit and TTT-Effect values reliable given training and evaluation variance? (W1, Q1)
We agree that small performance differences relative to standard-CPT should be interpreted cautiously, and we conducted two additional analyses to quantify variability.
Training-seed variability. We trained additional independent runs of standard-CPT (2 seeds) and Frobenius (3 seeds), and evaluated all checkpoints on the same fixed evaluation set. The table below reports mean ± std across training seeds.
Across independent training runs, TTT-Benefit is task- and context-dependent: it can yield clear improvements over standard-CPT (e.g., +1.8 ± 0.9 on HELMET at 64K) while remaining negligible or negative elsewhere. In contrast, TTT-Effect is consistently small: its positive values are at most +0.5 ± 0.3, even where TTT-Benefit is largest. The only notable TTT-Effect is negative, on RULER at 128K (−1.6 ± 0.4), where it matches TTT-Benefit (−1.5 ± 1.4), so disabling the updates recovers standard-CPT performance. Thus, in no setting does TTT-aware training induce substantial positive reliance on inference-time fast-weight updates.
Evaluation-seed variability. We additionally re-evaluated all trained checkpoints on three independently sampled evaluation sets (seeds 42, 43, 44). Unlike bootstrapping, which can only resample examples from a single fixed set, this draws new evaluation instances and therefore also captures variability that a single draw cannot reveal. The table below reports results averaged across the six TTT variants (mean ± std across evaluation seeds), for the standard (CQ) and query-aware (QCQ) RULER formats and HELMET.
The same pattern holds: TTT-Effect is within ±0.2 points in every setting, including under query-aware prompting (QCQ), except RULER (CQ) at 128K (−1.1 ± 0.2), while TTT-Benefit varies across tasks and lengths. We have updated Tables 2 and 3 to report mean performance across evaluation seeds, with the corresponding standard deviations in Appendix C.2 (Table 5) and C.5 (Table 8).
Given that TTT-Benefit varies across tasks and is not consistently positive, we agree with the reviewer that the central claim should be reframed, and we apologize for the lack of clarity in the original manuscript: our wording around "gains" from TTT-aware training overstated what the results support. Our central claim is that TTT-aware training can change model performance without inducing corresponding reliance on inference-time fast-weight updates, rather than that outer-loop training yields large or consistent gains. This claim is further supported beyond benchmark differences: the training dynamics in Section 4.2 directly show that outer-loop training progressively attenuates the functional effect of fast-weight updates. We have revised the Abstract, Contributions, and Conclusion accordingly.