- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AdaWrite, a method for adaptive computation in test-time training architectures. For each input chunk, a learned router determines the execution depth, skipping subsequent Transformer layers and their fast-weight updates while preserving the corresponding memory states. The method uses intermediate readouts, a full-depth teacher for routing supervision, and a compute regularizer. Experiments on LaCT and TTT-E2E demonstrate reduced analytical prefill FLOPs, with particularly strong improvements on long-context needle retrieval.
Critical strengths
-
AdaWrite jointly controls Transformer execution and memory updates at the chunk level. The method is parameter-efficient and can be applied to existing TTT backbones without modifying their native update rules.
-
On LaCT, AdaWrite substantially reduces analytical prefill FLOPs while improving S-NIAH-1 accuracy beyond the training context length. The observation that the router allocates deeper computation to needle-containing chunks without explicit needle supervision is particularly compelling. The ablations support the importance of the proposed training components.
Critical weaknesses
-
The strongest gains are concentrated on S-NIAH-1, where the context contains highly repetitive filler. On natural text and other retrieval tasks, the router generally executes nearly all layers, resulting in limited computational savings. PG-19 perplexity deteriorates slightly, while the TTT-E2E results show smaller benefits. Moreover, the reported efficiency improvements are based on analytical FLOPs rather than measured inference latency.
-
AdaWrite simultaneously skips Transformer computation, suppresses fast-weight updates, and substitutes attention K/V representations in skipped layers. Consequently, the experiments do not isolate whether the retrieval improvements arise from selective memory writing, changes in attention representations, or reduced execution depth. The random-update controls are informative but do not resolve this attribution.
-
The router pools representations over the complete chunk, so routing decisions for earlier tokens can depend on later tokens within that chunk. This complicates the causal interpretation of the next-token training objective. Furthermore, the GTCF comparator uses a full-depth teacher with a different memory-update history from the routed student, making its gradient a heuristic rather than an exact same-state counterfactual. For TTT-E2E, the teacher-free inference comparator also changes the gradients used for fast-weight updates relative to training.
Questions for authors
-
Could the authors report measured prefill latency and throughput, particularly on natural-text benchmarks where the analytical FLOP savings are relatively small?
-
Could the authors evaluate learned write-only routing, where all Transformer layers execute normally and maintain their original attention K/V representations, but selected fast-weight updates are suppressed? This would help isolate the contribution of adaptive memory writing.
-
How does full-chunk pooling in the router affect the causal interpretation of the language-modeling objective? Have the authors considered routing based only on previously observed chunks?
-
Can the authors further justify the GTCF gradient when the teacher and student have different memory histories? For TTT-E2E, how important is the difference between training-time and inference-time surrogate gradients for the resulting fast-weight updates?
-
The no-update and random-update controls outperform AdaWrite on LongBench-v2. Could the authors investigate why reducing memory updates improves performance in this setting and whether a simpler adaptive write policy could achieve comparable results without skipping Transformer layers?
-
Could the authors clarify how AdaWrite relates to GradMem and segment-level recurrent memory architectures? These methods use different mechanisms to compress context into persistent memory, including gradient-optimized memory embeddings and explicit associative updates. It would be useful to understand whether the proposed adaptive writing principle generalizes to these architectures or depends on the layerwise fast-weight updates of LaCT and TTT-E2E.
4 comments
3. Full-chunk pooling and causality. No reported number involves look-ahead. In training, the routed loss is not strictly autoregressive within a chunk, but later tokens reach earlier predictions only through one six-valued route per chunk.
Inference. Only complete chunks are routed, and a route affects only the predictions of tokens after its chunk. Every scored or generated token is therefore predicted from routes that depend only on tokens preceding it, and held-out targets never enter router pooling (Sec. 4.1). On PG-19 the 256 scored tokens follow the routed context.
Training. You are right that a routed chunk's own next-token loss is read out at a depth chosen from the whole chunk, so the routed loss is not a strictly autoregressive likelihood, and we will say so in the paper. The dependence is narrow. The route is the only path from later tokens to earlier predictions, and it takes one of six values (exit at 4, 8, 12, 16 or 20, or full depth). With the routes held fixed, every prediction depends only on earlier tokens: attention is causal, fast weights are read before the chunk's own write, and substituted K/V are position-aligned. Later tokens can therefore reach earlier predictions through at most log₂ 6 ≈ 2.6 bits per 2,048-token chunk.
This gives one precise statement about the value of the loss. Take the chunks in order, each given the earlier chunks and their routes. Let be the causal model obtained by forcing the current chunk's route to , and let be the uniform mixture of the six. Then is strictly autoregressive and , so the routed loss can fall below the loss of this causal model by at most ln 6 nats per chunk, about 0.0009 nats per token. The bound concerns the loss value only. The router itself is meant to use the whole chunk: at prefill a chunk is fully observed before it is routed, and its own predictions are not used. We will describe the routed loss accordingly, as a training surrogate for chunk-level routing: a causal likelihood conditioned on one six-valued route per chunk.
Routing from previous chunks only. We have not trained this variant; we chose the current chunk deliberately. The decision is how deeply to process and store this chunk, and LaCT's native write already aggregates all of the chunk's tokens [1], so we route at the same granularity with the same information. A previous-chunk router cannot see what it gates: in S-NIAH-1 the chunks before a needle are indistinguishable filler, yet 254 of 268 full-depth routes fall on needle-bearing chunks (App. E), and giving the same depths to the wrong chunks drops accuracy from 81.0% to 23.6% (depth-shuffle control, reply to X1wD). Previous-chunk routing is the natural variant for token-streaming generation, where a chunk must be processed before it is complete, and we will discuss it.
PG-19. The scoring convention routes the context chunk immediately before the continuation, and the first continuation token is predicted from that chunk's last position. The PG-19 difference arises here:
| PG-19 books | LaCT PPL | AdaWrite PPL | Write-only PPL |
|---|---|---|---|
| 37 books: last context chunk at full depth | 16.80 | 16.78 | 16.76 |
| 2 books: last context chunk exits at depth 4 | 17.13 | 24.60 | 17.24 |
| All 39 | 16.82 | 17.11 | 16.79 |
The split is by route, not by outcome: these are the only two books in which the last context chunk exits, a decision fixed before any scored token. There the continuation's local attention reads K/V substituted from the depth-4 state, and with native K/V (write-only, item 2) both books return to the full-depth level. On the other 37 books AdaWrite matches full depth while exiting about 15% of chunks. This is a real but local cost of exiting next to the read position, not look-ahead and not a loss of long-range memory. Its natural remedy is to keep the chunk adjacent to the prediction span at full depth, as the terminal-chunk rule already does for the final chunk. We will state the scoring convention and this boundary effect precisely in the revision.
[1] Test-Time Training Done Right.
5. Higher LongBench v2 scores without updates. The controls score higher because their predictions concentrate on one answer label. A simpler write-only policy matches AdaWrite's quality but gives up the compute saving.
In the predictions underlying the paper's results, no-update chooses C on 320 of 358 examples and gets 110 correct; random-50% chooses C on 272 and gets 113 correct; always choosing C gets 112 correct.
| Setting | Task-macro accuracy (%) |
|---|---|
| LaCT | 25.95 |
| AdaWrite | 27.19 |
| No updates | 30.84 |
| Random 50% updates | 30.46 |
| Always C, diagnostic | 29.85 |
The no-update control is within two questions of the constant answer, so its score is consistent with the answer-label prior and is not evidence of better use of the document. (Task-macro accuracy weights tasks equally, so its ordering can differ from total correct counts.) On tasks that depend on the context, removing updates is clearly harmful: S-NIAH-1 falls from 0.73 to 0.08 and PG-19 perplexity rises from 16.82 to 20.64 (Table 4).
On your second question: yes on quality, no on compute. The write-only controls of item 2 are that simpler adaptive write policy. Without skipping any layer they match AdaWrite on S-NIAH-1 (80.67% vs 81.00%) and are at parity with LaCT on PG-19 (16.79 vs 16.82). Their analytical prefill cost, however, is 0.90–0.91 of full depth on S-NIAH-1, against 0.45–0.51 for AdaWrite. Coupling the write decision to the exit is what converts selective writing into a 2× saving.
6. Other memory architectures. The idea is general, but each architecture needs its own state-preserving skip. Gated Delta Networks combine learned retention and targeted delta-rule updates [1]. GradMem optimizes prefix memory embeddings through context reconstruction while keeping model weights fixed at test time [2]. RMT carries memory tokens between segments [3], and Infini-attention combines local attention with compressive associative memory [4]. They share the problem of deciding what information should enter persistent memory, but use different states and update rules.
AdaWrite couples selective writing with execution depth in the layerwise fast weights of LaCT and TTT-E2E. The underlying idea, deciding per segment whether a write is worth its cost and skipping the associated computation when it is not, carries over in principle to any architecture that writes a persistent memory segment by segment. A transfer requires defining a state-preserving skip, including retention and normalization state, and identifying which computation it can avoid; these transfers remain future work.
[1] Gated Delta Networks: Improving Mamba2 with Delta Rule.
[2] GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent.
[3] Recurrent Memory Transformer.
[4] Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention.
4. GTCF with different memory histories. GTCF changes only the exit gate's training derivative, and the reference it replaces is in the paper and is clearly worse.
GTCF alters only the straight-through derivative of the selected exit gate. The current chunk's logits and loss value are unchanged, and no teacher is used at inference. The question is which reference the gate is compared against when it learns whether stopping was acceptable.
The in-pass reference is our "GTCF off" ablation. Without GTCF the gate is compared against the continuation of the exited chunk computed within the student's own routed pass (Fig. 7b; Table 4), which, as Sec. 3.4 notes, already reflects the student's earlier routing decisions. With it, S-NIAH-1 falls from 81.0% to 56.2% at similar depth (9.7 vs 10.4 layers), from 79% to 19% at 96K and from 40% to 7% at 128K.
Why a teacher with a different history is a sound reference. (i) It does not depend on the policy, so the target does not move as the router changes. (ii) It errs on the safe side: a wrongly skipped write is lost to every later chunk, whereas a wrongly continued chunk costs only compute. (iii) The mismatch is limited: teacher and student share all slow weights and are identical through the unconditional four-layer stem; deeper states diverge only as a consequence of the student's own exits. (iv) Write and for the exit and teacher readouts, for the hard gate, for the exit logit with , and for the cross-entropy. For a hard exit with , convexity of gives
=p(1-p)\langle\nabla F(R),R-T\rangle \geq p(1-p)\bigl(F(R)-F(T)\bigr).$$ The direct term thus pushes against EXIT at least in proportion to the teacher's advantage, and the efficiency loss pushes toward it. A degraded reference shrinks or reverses that advantage, so the gate receives the wrong signal about which chunks may stop: with correct-class margins 0 / +2 / −2 for the exit, the teacher and a degraded continuation, the term is +0.19 with the teacher and −0.19 with the degraded reference at $p=0.75$. Neither reference measures the full effect of a decision, because the value of a write appears in later chunks; measuring it would need a separate rollout per chunk and exit site. We will describe GTCF as a teacher-referenced surrogate, not an exact counterfactual. *TTT-E2E.* LaCT's update is a local key–value objective that never sees the comparator, so this part concerns only the TTT-E2E study, whose savings are smaller by construction because only the last six of its 24 layers carry fast weights and are routed. There the comparator enters the inner update on write-enabled EXIT chunks through the gate derivative. Substituting the training-time teacher at inference on a fixed panel of eight PG-19 books (same executable for both arms): | Inference comparator | PG-19 PPL | Mean depth | | --- | ---: | ---: | | Teacher-free (the paper's inference rule) | 13.34 | 22.87 | | Training-time teacher | 12.95 | 23.81 | The difference matters, and it acts through routing. The comparator changes the fast-weight update on EXIT chunks and, through it, later routes: with the training-time teacher the same routers continue on almost every chunk and perplexity returns to the full-depth level (12.94 on the same books), whereas with the teacher-free rule they exit more and perplexity is 3% higher on this panel. In six local probes, a single changed update moved the next chunk's mean NLL by at most 0.005 nats, in either direction. Using the teacher at inference needs a full-depth pass, which removes the saving, so every TTT-E2E number in the paper uses the teacher-free rule and already includes this effect. We will state it as a limitation of the TTT-E2E study.
Thank you for your time and detailed questions. Each item starts with the short answer. In quality tables, LaCT and AdaWrite keep the paper's values; the other rows are new controls on the same frozen checkpoints.
1. Prefill latency and throughput. Measured S-NIAH-1 speedups match or exceed the analytical ratios (1.99×/2.28×/2.37× against 1.98×/2.15×/2.21× at 32K/64K/128K); on natural text the measured gain is 1.02–1.06× (analytical 1.04–1.06×). We used BF16, batch size one and RTX 6000 Ada GPUs. Each input has six paired trials across two GPU assignments, balanced model order, two warmups per model per round, fresh memory/KV state and synchronized timing. The table reports cached prefill; tokenization, transfer, decoding and task scoring are outside the timed region.
Latency averages per-input six-trial medians; throughput divides total input tokens by their summed latency. Intervals use a paired input bootstrap. PG-19 uses the 65,536-token context alone.
2. Write-only routing. The retrieval gain survives with every layer executed, so it comes from selective writing; early exit turns the same write schedule into a 2× saving. Both controls use the existing checkpoint without retraining. They execute all 24 layers with native attention K/V and fast-weight reads, and suppress only the fast-weight updates (weights and momentum) that the route would skip. Replay fixes the write schedule recorded from AdaWrite. Online re-evaluates the learned routers along the full-computation trajectory.
F/full is the analytical prefill cost relative to full depth. Per length (8K to 128K), write-only accuracy is 92/91/93/89/78/41, against 92/91/94/90/79/40 for AdaWrite and 100/100/100/89/50/0 for LaCT. The write schedule alone therefore reproduces AdaWrite's whole accuracy profile, including the gains at 96K and 128K: the improvement is not produced by substituted K/V or by reduced execution depth. On PG-19, write-only is at 16.79 against 16.82 for LaCT, so the learned write selection carries no perplexity cost there. The LongBench v2 column lies inside the answer-label-bias band discussed in item 5, and we draw no conclusion from it. Skipping the layers is what delivers the compute: F/full falls from about 0.90 to 0.45–0.51 on S-NIAH-1 at unchanged accuracy.
This also explains the binary design. In an early prototype with three actions (UPDATE, READ without writing, EXIT), we qualitatively observed that the router favoured READ over EXIT, with little apparent quality improvement over the binary design; READ keeps the forward cost. Coupling the write decision to the exit makes not writing also save computation, and the controls above quantify what that coupling buys.