acceptodds
Under review as a conference paper at ICLR 2027

AdaWrite: Adaptive Write Depth Routing for Test-Time Training

Abstract

Test-time training (TTT) models retain information from long contexts in fast weights that are updated during inference. Common chunkwise implementations process each chunk at full depth, even though the benefit of deeper processing can vary. We introduce AdaWrite, a parameter-efficient method that reduces inference computation during context processing, or prefill. AdaWrite learns where each updating chunk stops in a frozen TTT backbone. An early exit skips deeper computation and memory updates while leaving skipped layers' fast states unchanged. We first train intermediate readouts, then learn the routing policy with a detached full-depth teacher as a reference and a compute penalty. Our main experiments on LaCT cover retrieval, document understanding, and language modeling. On S-NIAH-1, AdaWrite substantially reduces analytical prefill FLOPs and improves retrieval beyond the training context length. The router concentrates deeper updates near the needle without needle supervision. On other retrieval tasks and natural text, the same policy uses more layers and achieves prediction quality close to the full-depth baseline. An additional study on TTT-E2E shows modest computational savings with a different memory update rule.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-time training, Early exit, Routing, Fast-weight memory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XCSROpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaWrite, a method for adaptive computation in test-time training architectures. For each input chunk, a learned router determines the execution depth, skipping subsequent Transformer layers and their fast-weight updates while preserving the corresponding memory states. The method uses intermediate readouts, a full-depth teacher for routing supervision, and a compute regularizer. Experiments on LaCT and TTT-E2E demonstrate reduced analytical prefill FLOPs, with particularly strong improvements on long-context needle retrieval.

Critical strengths
  1. AdaWrite jointly controls Transformer execution and memory updates at the chunk level. The method is parameter-efficient and can be applied to existing TTT backbones without modifying their native update rules.

  2. On LaCT, AdaWrite substantially reduces analytical prefill FLOPs while improving S-NIAH-1 accuracy beyond the training context length. The observation that the router allocates deeper computation to needle-containing chunks without explicit needle supervision is particularly compelling. The ablations support the importance of the proposed training components.

Critical weaknesses
  1. The strongest gains are concentrated on S-NIAH-1, where the context contains highly repetitive filler. On natural text and other retrieval tasks, the router generally executes nearly all layers, resulting in limited computational savings. PG-19 perplexity deteriorates slightly, while the TTT-E2E results show smaller benefits. Moreover, the reported efficiency improvements are based on analytical FLOPs rather than measured inference latency.

  2. AdaWrite simultaneously skips Transformer computation, suppresses fast-weight updates, and substitutes attention K/V representations in skipped layers. Consequently, the experiments do not isolate whether the retrieval improvements arise from selective memory writing, changes in attention representations, or reduced execution depth. The random-update controls are informative but do not resolve this attribution.

  3. The router pools representations over the complete chunk, so routing decisions for earlier tokens can depend on later tokens within that chunk. This complicates the causal interpretation of the next-token training objective. Furthermore, the GTCF comparator uses a full-depth teacher with a different memory-update history from the routed student, making its gradient a heuristic rather than an exact same-state counterfactual. For TTT-E2E, the teacher-free inference comparator also changes the gradients used for fast-weight updates relative to training.

Questions for authors
  1. Could the authors report measured prefill latency and throughput, particularly on natural-text benchmarks where the analytical FLOP savings are relatively small?

  2. Could the authors evaluate learned write-only routing, where all Transformer layers execute normally and maintain their original attention K/V representations, but selected fast-weight updates are suppressed? This would help isolate the contribution of adaptive memory writing.

  3. How does full-chunk pooling in the router affect the causal interpretation of the language-modeling objective? Have the authors considered routing based only on previously observed chunks?

  4. Can the authors further justify the GTCF gradient when the teacher and student have different memory histories? For TTT-E2E, how important is the difference between training-time and inference-time surrogate gradients for the resulting fast-weight updates?

  5. The no-update and random-update controls outperform AdaWrite on LongBench-v2. Could the authors investigate why reducing memory updates improves performance in this setting and whether a simpler adaptive write policy could achieve comparable results without skipping Transformer layers?

  6. Could the authors clarify how AdaWrite relates to GradMem and segment-level recurrent memory architectures? These methods use different mechanisms to compress context into persistent memory, including gradient-optimized memory embeddings and explicit associative updates. It would be useful to understand whether the proposed adaptive writing principle generalizes to these architectures or depends on the layerwise fast-weight updates of LaCT and TTT-E2E.

4 comments

Thank you for your time and detailed questions. Each item starts with the short answer. In quality tables, LaCT and AdaWrite keep the paper's values; the other rows are new controls on the same frozen checkpoints.

1. Prefill latency and throughput. Measured S-NIAH-1 speedups match or exceed the analytical ratios (1.99×/2.28×/2.37× against 1.98×/2.15×/2.21× at 32K/64K/128K); on natural text the measured gain is 1.02–1.06× (analytical 1.04–1.06×). We used BF16, batch size one and RTX 6000 Ada GPUs. Each input has six paired trials across two GPU assignments, balanced model order, two warmups per model per round, fresh memory/KV state and synchronized timing. The table reports cached prefill; tokenization, transfer, decoding and task scoring are outside the timed region.

WorkloadLaCT msAdaWrite msSpeedup [95% CI]Tokens/s LaCT → AdaWrite
S-NIAH-1 8K229.74176.691.300 [1.273, 1.329]34289 → 44582
S-NIAH-1 16K511.27291.861.752 [1.732, 1.773]30664 → 53716
S-NIAH-1 32K1107.57556.351.991 [1.981, 2.001]29323 → 58377
S-NIAH-1 64K2288.731002.032.284 [2.277, 2.292]28346 → 64746
S-NIAH-1 96K3498.551505.192.324 [2.320, 2.329]27977 → 65027
S-NIAH-1 128K4812.512030.682.370 [2.366, 2.374]27195 → 64450
S-NIAH-1 overall, 600 inputs2074.73927.132.238 [2.235, 2.241]28089 → 62858
LongBench v1, 550 inputs489.12460.481.062 [1.053, 1.072]29858 → 31716
LongBench v2, 358 inputs2775.402728.551.017 [1.011, 1.024]27413 → 27883
PG-19, 39 contexts2509.392444.791.026 [1.013, 1.041]26116 → 26806

Latency averages per-input six-trial medians; throughput divides total input tokens by their summed latency. Intervals use a paired input bootstrap. PG-19 uses the 65,536-token context alone.

2. Write-only routing. The retrieval gain survives with every layer executed, so it comes from selective writing; early exit turns the same write schedule into a 2× saving. Both controls use the existing checkpoint without retraining. They execute all 24 layers with native attention K/V and fast-weight reads, and suppress only the fast-weight updates (weights and momentum) that the route would skip. Replay fixes the write schedule recorded from AdaWrite. Online re-evaluates the learned routers along the full-computation trajectory.

SettingS-NIAH-1 acc. (%)LongBench v2 acc. (%)PG-19 PPLS-NIAH-1 F/full at 32K / 64K / 128KPG-19 F/full
LaCT73.1725.9516.821.000 / 1.000 / 1.0001.000
AdaWrite81.0027.1917.110.506 / 0.466 / 0.4520.940
Write-only online80.6729.1916.790.911 / 0.898 / 0.8960.988
Write-only replay80.6729.4516.790.911 / 0.898 / 0.8960.988

F/full is the analytical prefill cost relative to full depth. Per length (8K to 128K), write-only accuracy is 92/91/93/89/78/41, against 92/91/94/90/79/40 for AdaWrite and 100/100/100/89/50/0 for LaCT. The write schedule alone therefore reproduces AdaWrite's whole accuracy profile, including the gains at 96K and 128K: the improvement is not produced by substituted K/V or by reduced execution depth. On PG-19, write-only is at 16.79 against 16.82 for LaCT, so the learned write selection carries no perplexity cost there. The LongBench v2 column lies inside the answer-label-bias band discussed in item 5, and we draw no conclusion from it. Skipping the layers is what delivers the compute: F/full falls from about 0.90 to 0.45–0.51 on S-NIAH-1 at unchanged accuracy.

This also explains the binary design. In an early prototype with three actions (UPDATE, READ without writing, EXIT), we qualitatively observed that the router favoured READ over EXIT, with little apparent quality improvement over the binary design; READ keeps the forward cost. Coupling the write decision to the exit makes not writing also save computation, and the controls above quantify what that coupling buys.

3. Full-chunk pooling and causality. No reported number involves look-ahead. In training, the routed loss is not strictly autoregressive within a chunk, but later tokens reach earlier predictions only through one six-valued route per chunk.

Inference. Only complete chunks are routed, and a route affects only the predictions of tokens after its chunk. Every scored or generated token is therefore predicted from routes that depend only on tokens preceding it, and held-out targets never enter router pooling (Sec. 4.1). On PG-19 the 256 scored tokens follow the routed context.

Training. You are right that a routed chunk's own next-token loss is read out at a depth chosen from the whole chunk, so the routed loss is not a strictly autoregressive likelihood, and we will say so in the paper. The dependence is narrow. The route is the only path from later tokens to earlier predictions, and it takes one of six values (exit at 4, 8, 12, 16 or 20, or full depth). With the routes held fixed, every prediction depends only on earlier tokens: attention is causal, fast weights are read before the chunk's own write, and substituted K/V are position-aligned. Later tokens can therefore reach earlier predictions through at most log₂ 6 ≈ 2.6 bits per 2,048-token chunk.

This gives one precise statement about the value of the loss. Take the chunks in order, each given the earlier chunks and their routes. Let q(r)q^{(r)} be the causal model obtained by forcing the current chunk's route to rr, and let mm be the uniform mixture of the six. Then mm is strictly autoregressive and m(x)≥q(r(x))(x)/6m(x)\geq q^{(r(x))}(x)/6, so the routed loss can fall below the loss of this causal model by at most ln 6 nats per chunk, about 0.0009 nats per token. The bound concerns the loss value only. The router itself is meant to use the whole chunk: at prefill a chunk is fully observed before it is routed, and its own predictions are not used. We will describe the routed loss accordingly, as a training surrogate for chunk-level routing: a causal likelihood conditioned on one six-valued route per chunk.

Routing from previous chunks only. We have not trained this variant; we chose the current chunk deliberately. The decision is how deeply to process and store this chunk, and LaCT's native write already aggregates all of the chunk's tokens [1], so we route at the same granularity with the same information. A previous-chunk router cannot see what it gates: in S-NIAH-1 the chunks before a needle are indistinguishable filler, yet 254 of 268 full-depth routes fall on needle-bearing chunks (App. E), and giving the same depths to the wrong chunks drops accuracy from 81.0% to 23.6% (depth-shuffle control, reply to X1wD). Previous-chunk routing is the natural variant for token-streaming generation, where a chunk must be processed before it is complete, and we will discuss it.

PG-19. The scoring convention routes the context chunk immediately before the continuation, and the first continuation token is predicted from that chunk's last position. The PG-19 difference arises here:

PG-19 booksLaCT PPLAdaWrite PPLWrite-only PPL
37 books: last context chunk at full depth16.8016.7816.76
2 books: last context chunk exits at depth 417.1324.6017.24
All 3916.8217.1116.79

The split is by route, not by outcome: these are the only two books in which the last context chunk exits, a decision fixed before any scored token. There the continuation's local attention reads K/V substituted from the depth-4 state, and with native K/V (write-only, item 2) both books return to the full-depth level. On the other 37 books AdaWrite matches full depth while exiting about 15% of chunks. This is a real but local cost of exiting next to the read position, not look-ahead and not a loss of long-range memory. Its natural remedy is to keep the chunk adjacent to the prediction span at full depth, as the terminal-chunk rule already does for the final chunk. We will state the scoring convention and this boundary effect precisely in the revision.

[1] Test-Time Training Done Right.

5. Higher LongBench v2 scores without updates. The controls score higher because their predictions concentrate on one answer label. A simpler write-only policy matches AdaWrite's quality but gives up the compute saving.

In the predictions underlying the paper's results, no-update chooses C on 320 of 358 examples and gets 110 correct; random-50% chooses C on 272 and gets 113 correct; always choosing C gets 112 correct.

SettingTask-macro accuracy (%)
LaCT25.95
AdaWrite27.19
No updates30.84
Random 50% updates30.46
Always C, diagnostic29.85

The no-update control is within two questions of the constant answer, so its score is consistent with the answer-label prior and is not evidence of better use of the document. (Task-macro accuracy weights tasks equally, so its ordering can differ from total correct counts.) On tasks that depend on the context, removing updates is clearly harmful: S-NIAH-1 falls from 0.73 to 0.08 and PG-19 perplexity rises from 16.82 to 20.64 (Table 4).

On your second question: yes on quality, no on compute. The write-only controls of item 2 are that simpler adaptive write policy. Without skipping any layer they match AdaWrite on S-NIAH-1 (80.67% vs 81.00%) and are at parity with LaCT on PG-19 (16.79 vs 16.82). Their analytical prefill cost, however, is 0.90–0.91 of full depth on S-NIAH-1, against 0.45–0.51 for AdaWrite. Coupling the write decision to the exit is what converts selective writing into a 2× saving.

6. Other memory architectures. The idea is general, but each architecture needs its own state-preserving skip. Gated Delta Networks combine learned retention and targeted delta-rule updates [1]. GradMem optimizes prefix memory embeddings through context reconstruction while keeping model weights fixed at test time [2]. RMT carries memory tokens between segments [3], and Infini-attention combines local attention with compressive associative memory [4]. They share the problem of deciding what information should enter persistent memory, but use different states and update rules.

AdaWrite couples selective writing with execution depth in the layerwise fast weights of LaCT and TTT-E2E. The underlying idea, deciding per segment whether a write is worth its cost and skipping the associated computation when it is not, carries over in principle to any architecture that writes a persistent memory segment by segment. A transfer requires defining a state-preserving skip, including retention and normalization state, and identifying which computation it can avoid; these transfers remain future work.

[1] Gated Delta Networks: Improving Mamba2 with Delta Rule.

[2] GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent.

[3] Recurrent Memory Transformer.

[4] Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention.

4. GTCF with different memory histories. GTCF changes only the exit gate's training derivative, and the reference it replaces is in the paper and is clearly worse.

GTCF alters only the straight-through derivative of the selected exit gate. The current chunk's logits and loss value are unchanged, and no teacher is used at inference. The question is which reference the gate is compared against when it learns whether stopping was acceptable.

The in-pass reference is our "GTCF off" ablation. Without GTCF the gate is compared against the continuation of the exited chunk computed within the student's own routed pass (Fig. 7b; Table 4), which, as Sec. 3.4 notes, already reflects the student's earlier routing decisions. With it, S-NIAH-1 falls from 81.0% to 56.2% at similar depth (9.7 vs 10.4 layers), from 79% to 19% at 96K and from 40% to 7% at 128K.

Why a teacher with a different history is a sound reference. (i) It does not depend on the policy, so the target does not move as the router changes. (ii) It errs on the safe side: a wrongly skipped write is lost to every later chunk, whereas a wrongly continued chunk costs only compute. (iii) The mismatch is limited: teacher and student share all slow weights and are identical through the unconditional four-layer stem; deeper states diverge only as a consequence of the student's own exits. (iv) Write RR and TT for the exit and teacher readouts, gg for the hard gate, uu for the exit logit with p=σ(u)p=\sigma(u), and FF for the cross-entropy. For a hard exit with H=gR+(1−g)sg⁡(T)H=gR+(1-g)\operatorname{sg}(T), convexity of FF gives

=p(1-p)\langle\nabla F(R),R-T\rangle \geq p(1-p)\bigl(F(R)-F(T)\bigr).$$ The direct term thus pushes against EXIT at least in proportion to the teacher's advantage, and the efficiency loss pushes toward it. A degraded reference shrinks or reverses that advantage, so the gate receives the wrong signal about which chunks may stop: with correct-class margins 0 / +2 / −2 for the exit, the teacher and a degraded continuation, the term is +0.19 with the teacher and −0.19 with the degraded reference at $p=0.75$. Neither reference measures the full effect of a decision, because the value of a write appears in later chunks; measuring it would need a separate rollout per chunk and exit site. We will describe GTCF as a teacher-referenced surrogate, not an exact counterfactual. *TTT-E2E.* LaCT's update is a local key–value objective that never sees the comparator, so this part concerns only the TTT-E2E study, whose savings are smaller by construction because only the last six of its 24 layers carry fast weights and are routed. There the comparator enters the inner update on write-enabled EXIT chunks through the gate derivative. Substituting the training-time teacher at inference on a fixed panel of eight PG-19 books (same executable for both arms): | Inference comparator | PG-19 PPL | Mean depth | | --- | ---: | ---: | | Teacher-free (the paper's inference rule) | 13.34 | 22.87 | | Training-time teacher | 12.95 | 23.81 | The difference matters, and it acts through routing. The comparator changes the fast-weight update on EXIT chunks and, through it, later routes: with the training-time teacher the same routers continue on almost every chunk and perplexity returns to the full-depth level (12.94 on the same books), whereas with the teacher-free rule they exit more and perplexity is 3% higher on this panel. In six local probes, a single changed update moved the next chunk's mean NLL by at most 0.005 nats, in either direction. Using the teacher at inference needs a full-depth pass, which removes the saving, so every TTT-E2E number in the paper uses the teacher-free rule and already includes this effect. We will state it as a limitation of the TTT-E2E study.
Reviewer_X1wDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaWrite, a way to cut prefill computation in frozen TTT models. The starting point is that models like LaCT run every chunk through all layers, even though not every chunk needs that much processing. AdaWrite adds routers at a few layer boundaries, and at each one a chunk either continues or exits. Once a chunk exits, the remaining layers are skipped and their fast weights are not updated, so they keep their previous states. Training happens in two steps. The intermediate readouts are trained first, and then the routers. For the routers, the authors use a teacher that always runs at full depth as the reference for the exit decision (they call this GTCF), plus a penalty on compute. The method adds less than 0.2% parameters and leaves the backbone frozen. On a 1B LaCT model, S-NIAH-1 uses only 9.7 of 24 layers on average. Accuracy at 96K and 128K is clearly higher than the baseline, but it is lower up to 32K. The router gets no needle supervision, yet most deep updates land on chunks that contain the needle. On the other tasks the model uses almost all layers and performs about the same as the baseline.

Critical strengths

1.The idea is new, and the design is clean and lightweight. When early exit is applied to TTT, an exit does more than save computation. It also decides which layers write to their fast weights. The authors let the skipped layers keep their states unchanged, so one decision controls both computation and memory, and forcing CONTINUE everywhere recovers the full-depth path exactly. The method trains only about 0.2% of the parameters and keeps the backbone frozen. It works on both LaCT and TTT-E2E, which use different update rules, so it is easy to add to existing TTT models.

2.The routing behavior is also interpretable, and the ablations are fairly complete. The router gets no needle supervision, yet on S-NIAH-1, 254 of the 268 full-depth chunks contain the needle, and the other 14 all come right after a needle-bearing chunk. This shows that the router does learn to allocate depth based on content. The paper ablates GTCF, the two-stage training, the z-loss, fixed depths, no updates and random 50% updates, and reports results for each context length. The authors are also open about the limitations: the FLOPs are analytical estimates rather than measured latency, and the random controls cannot isolate the effect of content-based placement.

Critical weaknesses

1.Almost all of the benefit shows up on a single synthetic task, S-NIAH-1, and it is not clear where it comes from. The filler text in S-NIAH-1 is a repeated sentence, and the router basically learns to run needle chunks through all 24 layers and to stop more than 99% of the other chunks at layer 8. On natural text the model uses almost every layer and saves only about 5% of the FLOPs, and PG-19 perplexity goes up from 16.82 to 17.11. So the main goal, cutting prefill compute, does not carry over to real inputs. Even on S-NIAH-1, the gain in average accuracy comes entirely from 96K and 128K, which are beyond the training length, while accuracy up to 32K drops from 1.00 to 0.91 to 0.94. Randomly dropping half of the updates, with no training at all, already reaches 0.38 at 128K, close to AdaWrite’s 0.40. Much of the long-context improvement may therefore come from writing less and overwriting less, not from the learned routing, so I cannot tell how much of the gain is due to content-based routing.

2.The other evaluations tell us very little, they do not support the claim that quality stays close to the baseline. LongBench v2 is four-way multiple choice, so random guessing gives 0.25. The baseline gets 0.260 and AdaWrite gets 0.272. Even the controls that never update the fast weights (0.308) or keep a random half of the updates (0.305) score higher. On S-NIAH-2/3, the baseline is at 0.01 to 0.07 from 32K on, which leaves almost no room for comparison. Matching the baseline on these tasks says little about whether routing hurts the model on real long text.

Questions for authors
  1. The gain on S-NIAH-1 could come from two things: writing less overall, so older memory gets overwritten less, or sending the deep updates to the chunks that contain the needle. The random 50% update control differs from AdaWrite both in how much is written and in where, so it can't tell these apart. Could you add a control that randomly shuffles AdaWrite's per-chunk depths within each sample? That keeps the amount of writing and the depth distribution the same, and only randomizes which chunks go deep.

  2. On natural text, AdaWrite saves only about 5% of compute, and with the z-loss turned off, the average depth on PG-19 drops to 18.8 layers while perplexity rises from 17.11 to 18.28. Does a larger compute penalty λ_eff buy meaningful savings at an acceptable quality cost? A quality–FLOPs curve on PG-19, and on a long-context task where the full-depth model is clearly above chance (e.g., single-document QA in LongBench v1), would answer this.

7 comments

Thank you for your careful review and concrete suggestions. We added the depth-shuffle control you proposed, single-document QA, and an inference-threshold study, and we examined the LongBench v2 controls. In the S-NIAH-1 table, LaCT, AdaWrite and Random 50% keep the paper's values; the other columns are new controls on the same checkpoints.

1. Does depth placement matter? Yes. On all 600 S-NIAH-1 examples we recorded AdaWrite's depths and shuffled them among the updating chunks of each example (mean over five permutations per example). This preserves the depth distribution and the per-layer write counts and randomizes only which chunks go deep. We also ran a write-only control that executes all 24 layers with native attention K/V and applies only AdaWrite's write schedule (reply to XCSR, item 2).

Accuracy (%):

LengthLaCTAdaWriteRandom 50%Depth shuffleWrite-only
8K100925676.492
16K100915039.091
32K100946414.093
64K8990658.089
96K5079562.678
128K040381.841
Mean73.1781.0054.8323.6380.67

These controls separate the two explanations you raise. Writing less helps extrapolation by itself: random thinning lifts 128K from 0 to 38. It is not what makes AdaWrite work, however: with exactly the same amount of writing placed on the wrong chunks, accuracy collapses to 23.6%, far below both LaCT and random thinning. Content-based placement is what holds accuracy at 90–94% through 64K, where random thinning reaches only 50–65%, and at 79% at 96K. The write-only column shows that the effect is carried by the memory writes: with every layer executed, the write schedule alone reproduces AdaWrite within one point at every length.

Accuracy up to 32K. The write-only control shows the same loss with all computation restored, so the loss comes from which chunks are written deeply, not from the computation skipped on filler. It follows the needles the router misses. In the per-example records of the control runs at 8K, 16K and 32K, every example whose needle chunk reaches full depth is answered correctly, and every error falls on an example whose needle chunk exits early. The paper's counts agree at 64K: 85 of the 95 routed needle chunks reach full depth and 5 needles lie in the full-depth terminal chunk (App. E), and 85 + 5 = 90 matches the 90% accuracy. The short-context loss is therefore a matter of router recall, not an inherent cost of early exit.

2. Single-document QA. We evaluated all 550 English single-document QA examples in LongBench v1 (200 NarrativeQA, 200 Qasper, 150 MultiFieldQA-en) with the official prompts, answer-token F1, and greedy generation limited to 128/128/64 new tokens. No input required truncation.

SettingNarrativeQA F1 (%)Qasper F1 (%)MultiFieldQA-en F1 (%)Mean F1 (%)Mean depthF/full
LaCT1.9635.90912.1676.68024.0001.0000
AdaWrite1.9295.93612.2296.69822.6680.9592

F/full is the analytical prefill cost relative to full depth. AdaWrite matches LaCT on all three tasks at 4.08% lower cost. The absolute level is set by the frozen backbone, not by routing: full-depth LaCT with no router scores 6.68. This is consistent with the original LaCT work, which reports no QA and omits the RULER tasks beyond needle retrieval because even its full-attention baseline "achieved trivial results beyond a 16K sequence length" [1]. Our claims are relative to the same frozen backbone at full depth, and at this level QA is less sensitive to routing than PG-19, which we treat as the primary natural-text evidence (item 3). On S-NIAH-2/3, at the lengths where the backbone is above floor, AdaWrite is level with LaCT on average: at 8K/16K it scores 0.62/0.24 against 0.62/0.19 on S-NIAH-2 and 0.53/0.26 against 0.59/0.19 on S-NIAH-3 (Tables 6 and 7).

[1] Test-Time Training Done Right.

3. Quality–compute tradeoff. We varied the exit rule P(EXIT)≥τP(\mathrm{EXIT})\geq\tau on the same frozen checkpoint, with the trained policy and efficiency penalty fixed. This moves the operating point at inference without retraining; it changes the decision threshold, not the softmax temperature.

SettingPG-19 PPLDepthF/fullQA F1 (%)DepthF/full
LaCT16.8224.01.00006.68024.0001.0000
AdaWrite, τ = 0.7516.8224.0001.00006.68024.0001.0000
AdaWrite, τ = 0.50 (default)17.1122.10.94006.69822.6680.9592
AdaWrite, τ = 0.2533.534.0000.37165.9694.3640.4076

These rows are the default and the two limits of the dial: at τ = 0.75 no chunk exits and the checkpoint reproduces full-depth quality at full-depth cost, and at τ = 0.25 every PG-19 chunk stops at layer 4. They establish two things.

(i) The threshold is a working inference-time control. The same checkpoint can be moved along the tradeoff, and returned to full-depth quality, without retraining.

(ii) At the default, the cost measured on natural text is a boundary cost. It is carried by the two books whose last context chunk exits; on the other 37, where about 15% of chunks exit, AdaWrite is at 16.78 against 16.80 for LaCT (reply to XCSR, item 3). The all-exit limit is a different regime: only the four-layer stem writes to memory and the chunk next to the continuation always exits, so its 33.53 does not measure what moderate, well-placed exits cost.

We regard these operating points, together with the per-book analysis, as sufficient to characterize the tradeoff for this checkpoint: removing more computation costs comparatively little on QA (F1 5.97 at 41% of the cost), while on PG-19 the cost at the default is confined to the boundary case above.

4. Higher LongBench v2 scores without updates. You are right that the no-update and random-update controls score higher than AdaWrite on this benchmark. The predictions underlying the paper's results show why: these controls concentrate on one answer label. No-update chooses C on 320 of 358 examples and gets 110 correct; random-50% chooses C on 272 and gets 113 correct; always choosing C gets 112 correct.

SettingTask-macro accuracy (%)
LaCT25.95
AdaWrite27.19
No updates30.84
Random 50% updates30.46
Always C, diagnostic29.85

The no-update control is within two questions of the constant answer, so its score is consistent with the answer-label prior and is not evidence of better use of the document. (Task-macro accuracy weights tasks equally, so its ordering can differ from total correct counts.) On tasks that depend on the context, removing updates is clearly harmful: S-NIAH-1 falls from 0.73 to 0.08 and PG-19 perplexity rises from 16.82 to 20.64 (Table 4). We therefore do not read LongBench v2 at this accuracy level as evidence for or against an update policy, and we evaluate selective writing with the controls of item 1 and with PG-19.

Thank you for the detailed response and additional experiments.

I understand that AdaWrite's computational savings depend on input redundancy. However, I remain concerned about its practical applicability. While AdaWrite achieves substantial speedups on the synthetic S-NIAH-1 benchmark, the gains on natural-text datasets are only 1.02–1.06×.

The authors explain that this is due to the conservative routing strategy, but I believe this remains an important limitation of the current method.

I have two follow-up questions:

  1. Could the authors demonstrate meaningful speedups on realistic workloads, such as logs or templated documents, where input redundancy is common?

  2. Could a larger and more capable TTT backbone enable more aggressive early exiting on natural text, potentially overcoming the current limitation?

Reviewer_q9MyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces early exit to TTT models such as LaCT and TTT-E2E, achieving efficient computation while preserving performance. The proposed method, AdaWrite, places routers at several layer boundaries of a frozen TTT backbone. For each context chunk, a router decides whether to continue or exit. An exit skips both the deeper computation and the fast-weight updates, while the skipped layers keep their incoming states. The routers are trained in two stages: first the exit readouts, then the routers with a gate-teacher counterfactual (GTCF) and a compute penalty. On LaCT S-NIAH-1, AdaWrite improves accuracy from 0.73 to 0.81 at an average depth of 9.7 of 24 layers, with up to 2.21× lower analytical prefill FLOPs. On the other benchmarks, the same policy keeps most layers and stays close to the full-depth baseline.

Critical strengths
  1. The paper introduces early-exit techniques to TTT models and reduces computational cost while preserving the performance of the full-depth model.
  2. On an easy retrieval benchmark such as S-NIAH-1, the reduction ratio is substantial (up to 2.21× fewer analytical FLOPs). This shows that early exit can be successfully adapted to TTT models.
Critical weaknesses
  1. On LongBench v2 and PG-19, the FLOPs reduction is much smaller than on S-NIAH-1 (about 1.05–1.06× vs. 1.98–2.21× for LaCT). The paper does not discuss why. Since reducing computation is the main contribution, the method looks less promising if the savings appear only on specific datasets. A discussion of the cause is needed in depth.
Questions for authors

In Figure 1(b), why does AdaWrite not execute all 24 layers at the last chunk C₈? C₈ appears to exit at depth 4. In my understanding, the terminal chunk should run at full depth without a fast-weight update.

1 comment

Thank you for your time, careful reading, and suggestions. We agree that the figure needs correcting and that the difference in savings across tasks deserves a clearer explanation.

Terminal chunk. Your understanding is correct. Both LaCT and AdaWrite execute the terminal chunk at all 24 layers without updating the fast weights. This is already how our evaluations run. Figure 1(b) illustrates the routing idea but depicts this chunk incorrectly; we will correct the figure and caption.

Why the savings differ. The savings follow the redundancy of the input, by design. At an exit, the gate is trained by comparing the shallow readout with the full-depth teacher on the chunk's own tokens (Sec. 3.4), so it learns to stop a chunk only when a shallow readout predicts it about as well as the full model. The router was trained only on ProLong-64K natural text, with no retrieval data and no needle labels. On S-NIAH-1 it stops repeated-filler chunks at layer 8 and keeps needle-bearing chunks deep: needle chunks are 1.6–6.6% of routed chunks but receive 92–98% of the full-depth routes (Sec. 4.4). This produces the 1.98–2.21× analytical saving, and it holds in wall-clock time: 2.24× measured over all 600 inputs, rising from 1.30× at 8K to 2.37× at 128K.

On books and long-context QA the router keeps 22.1–22.7 of 24 layers, and the measured speedup is 1.02–1.06×. Two points put this in context.

(i) Away from the read position, exits on natural text do not cost quality. On 37 of the 39 PG-19 books AdaWrite exits about 15% of chunks and is at 16.78 against 16.80 for LaCT. The reported increase to 17.11 is carried by the two books in which the chunk adjacent to the scored continuation exits at depth 4 (reply to XCSR, item 3).

(ii) The criterion is deliberately safe, and stricter than prefill needs. Prefill uses a chunk's memory writes and K/V, not its own next-token predictions, yet the gate protects those predictions. The small natural-text saving therefore reflects, at least in part, a conservative training signal rather than a limit of the mechanism. A prefill-aware objective, finer routing granularity (AdaWrite currently makes one decision per 2,048-token chunk) and a full-depth guard on the chunk next to the read position are direct routes to larger natural-text savings.

Redundant context is the regime the paper targets (Sec. 1) and is common in practice: repeated or templated text, boilerplate and logs. S-NIAH-1 is its controlled extreme, and the router finds that redundancy without having been trained on it.

Backbone capability. Absolute task levels are set by the frozen backbone before routing. Our backbone has the depth and width of LaCT's 760M configuration (24 layers, dimension 1,536) with the larger Llama-3 vocabulary, so the published LaCT results are the relevant reference. That work evaluates Book-3 next-token prediction and six RULER retrieval tasks, and omits the remaining RULER tasks because even its full-attention baseline "achieved trivial results beyond a 16K sequence length" [1]; its 760M Muon model scores 7.8%/5.2% on S-NIAH-2/3 at 32K. AdaWrite also keeps most layers on S-NIAH-2/3 (mean depth 22.5/22.8), whose haystacks are natural-text essays. LongBench v2 is demanding at this scale: its published Llama-3.1-8B-Instruct result is 30.0% without chain-of-thought [2].

We will make this task dependence explicit in the revision.

[1] Test-Time Training Done Right.

[2] LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks.

Comments on the paper

2 comments

Thank you to the reviewers and the Area Chair for the careful feedback. The shared questions are whether analytical savings become real speedups, what produces the retrieval gain, and why savings differ across tasks. In short: the savings are input-adaptive, 2.2× measured where the context is redundant and 2–6% on natural text, where the router keeps most layers; and the retrieval gain comes from selective memory writing. Details are in the individual replies. On the paper's benchmarks, LaCT and AdaWrite keep the paper's reported values; all additional measurements use the same frozen checkpoints.

1. The savings are real in wall-clock time. Cached prefill, BF16, batch size one, RTX 6000 Ada, six paired trials per input:

WorkloadLaCT msAdaWrite msSpeedup [95% CI]Prefill tokens/s LaCT → AdaWrite
S-NIAH-1, 600 inputs2074.73927.132.238 [2.235, 2.241]28089 → 62858
LongBench v1, 550 inputs489.12460.481.062 [1.053, 1.072]29858 → 31716
LongBench v2, 358 inputs2775.402728.551.017 [1.011, 1.024]27413 → 27883
PG-19, 39 contexts2509.392444.791.026 [1.013, 1.041]26116 → 26806

On S-NIAH-1 the speedup grows with context length, from 1.30× at 8K to 2.37× at 128K, and matches or exceeds the analytical ratio at each of the paper's three lengths.

2. The retrieval gain comes from where the router writes. Two controls on all 600 S-NIAH-1 examples isolate the mechanism. Shuffling AdaWrite's own depths among the chunks of each example keeps the depth distribution and per-layer write counts, yet accuracy falls from 81.0% to 23.6%. Executing all 24 layers with native attention K/V and applying only AdaWrite's write schedule gives 80.7%, against 73.2% for LaCT, and stays within one point of AdaWrite at every length. Selective writing therefore accounts for the gain, and early exit converts the same schedule into a 2× saving. The 6–9 point loss up to 32K has the same origin: it follows the needle chunks that the router does not send deep (reply to X1wD).

3. On natural text, quality matches full depth except at one boundary. The PG-19 increase from 16.82 to 17.11 is carried by two of the 39 books, in which the context chunk adjacent to the scored continuation exits at depth 4 and the continuation attends to substituted K/V. On the other 37 books AdaWrite is at 16.78 against 16.80 for LaCT while exiting about 15% of chunks, and with native K/V the two books return to the LaCT level (reply to XCSR, item 3). This is a local-attention cost of exiting next to the read position, not a loss of long-range memory, and keeping that one chunk at full depth is its natural remedy. On LongBench v1 single-document QA, mean F1 is 6.698 for AdaWrite and 6.680 for LaCT at 4.08% lower analytical prefill cost.

4. Why savings differ across tasks. At an exit, the gate is trained by comparing the shallow readout with the full-depth teacher on the chunk's own tokens, so it learns to stop a chunk only when a shallow readout predicts it about as well as the full model. The router was trained only on ProLong-64K natural text, with no needle supervision. On S-NIAH-1 it stops repeated-filler chunks at layer 8 and keeps needle chunks deep (mean depth 9.7 of 24); on books and long-context QA it keeps 22.1–22.7 layers. The savings therefore follow the redundancy of the input, by design. The criterion is also stricter than prefill needs, because prefill uses a chunk's memory and K/V rather than its own predictions, which leaves room for larger natural-text savings with the same mechanism.

5. Backbone and absolute scores. The backbone (about 1.09B parameters, pretrained from scratch on ProLong-64K) is frozen, so absolute task levels are set before routing. The original LaCT work evaluates Book-3 next-token prediction and six RULER retrieval tasks; it omits the remaining RULER tasks because even its full-attention baseline "achieved trivial results beyond a 16K sequence length" [1], and its 760M Muon model scores 7.8%/5.2% on S-NIAH-2/3 at 32K. AdaWrite's claims are relative to the same frozen backbone at full depth.

6. LongBench v2 controls. The no-update and random-50% controls (30.84%, 30.46%) choose answer C on 320 and 272 of the 358 examples, and always answering C scores 29.85%. Their margin over AdaWrite (27.19%) and LaCT (25.95%) is consistent with answer-label bias and is not evidence of stronger long-context reasoning.

The revision will include these results, the corrected Figure 1(b), and precise statements of the training objective and the PG-19 scoring convention.

[1] Test-Time Training Done Right.

We thank the reviewers again. One result obtained since our first replies concerns a point raised in the reviews: the small saving on natural text (1.02–1.06×) and the PG-19 perplexity increase from 16.82 to 17.11. Our replies traced that increase to the context chunk next to the read position and named a full-depth guard on that chunk as its remedy. We have now run the guard. The frozen checkpoint is unchanged and nothing is retrained; τ is the inference exit threshold (default 0.50).

SettingPG-19 PPLPG-19 speedup [95% CI]CostLongBench v1 mean F1 (%)LongBench v1 speedup [95% CI]Cost
LaCT16.821.0001.0006.6801.0001.000
AdaWrite, as reported17.111.026 [1.013, 1.041]0.9406.6981.062 [1.053, 1.072]0.959
AdaWrite + guard16.801.024 [1.011, 1.039]0.9436.7501.054 [1.046, 1.063]0.963
+ guard, τ = 0.4816.831.139 [1.107, 1.171]0.8606.7871.142 [1.124, 1.163]0.891
+ guard, τ = 0.4716.861.249 [1.201, 1.301]0.7966.7751.191 [1.170, 1.214]0.857
+ guard, τ = 0.4616.901.458 [1.385, 1.539]0.7046.6391.288 [1.258, 1.320]0.800
+ guard, τ = 0.4517.031.727 [1.631, 1.833]0.6166.7581.463 [1.424, 1.504]0.722

Speedup is the measured cached-prefill time of LaCT divided by that of AdaWrite, with the protocol of our latency table, on the 65,536-token PG-19 contexts and on the full LongBench v1 prompts. Cost is the paper's analytical prefill cost relative to full depth. PG-19 uses all 39 books and LongBench v1 all 550 single-document QA inputs.

  1. Natural-text quality is preserved. With the guard, AdaWrite is at 16.80 on PG-19 against 16.82 for LaCT, which confirms that the 17.11 we reported was a boundary cost.
  2. Natural-text savings are no longer small. With the guard, lowering the threshold gives measured prefill speedups of 1.14× to 1.46× on PG-19 with perplexity within 0.5% of LaCT, and 1.73× for +1.3%. On LongBench v1 the same settings make measured prefill 1.08× to 1.22× faster than AdaWrite as reported, and 1.38× faster at τ = 0.45, with F1 at the full-depth level (6.64–6.79 against 6.68).
  3. The guard is the enabling step. Without it, τ = 0.47 gives 18.55 on PG-19, because the chunk next to the continuation then exits in 11 of the 39 books. The threshold study in our reply to X1wD was run without the guard, which is why its low-threshold limit looked costly (33.53 at τ = 0.25).

These are inference-time settings for natural-text prefill on the same checkpoint. All results in the paper use the default threshold and are unchanged. Details are in our follow-up to Reviewer q9My, and the revision will include the guard and this table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.