- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes GAP-TTT, a test-time training framework that stores predictive residuals instead of complete future representations. A lightweight semantic predictor and a temporal-consistency formulation jointly determine memory targets and transition-aware write features. Fast memory is constructed through causal ridge regression, avoiding iterative test-time optimization. Experiments on RULER and LongBench-v2 show improvements over several TTT baselines.
Critical strengths
-
The idea of allocating fast-memory capacity to information not explained by the pretrained model is compelling. The joint derivation of predictive residual targets and transition-aware write features is particularly interesting, and the ablations support its effectiveness.
-
Closed-form causal ridge regression eliminates iterative test-time optimization and enables parallel prefix computation. GAP-TTT improves long-context performance with substantially fewer trainable adapter parameters than the evaluated TTT baselines.
Critical weaknesses
-
The main RULER results use joint backbone and adapter training, while the parameter-efficiency experiments use a frozen backbone. It is unclear which setting is used for other benchmarks. Moreover, there is no parameter-matched standard LoRA baseline trained on the same ProLong data, making it difficult to isolate the benefits of dynamic memory from additional trainable capacity.
-
The long-context evaluations also appear to keep relevant information within the backbone's attention window. Therefore, they do not establish whether GAP-TTT can retain and retrieve information after it becomes inaccessible through attention, which is an important motivation for persistent memory architectures.
-
Equation (3) is algebraically correct, but minimizing the sampled temporal-difference residual does not necessarily recover optimal predictive corrections. Correlations involving the next-position key can introduce bias, and the derivation assumes a shared candidate memory although online inference uses evolving memory states.
-
The broader motivation of prioritizing unpredictable information has precedents in Titans and GradMem, while TTCD explicitly uses representation discrepancies to guide memory updates. The distinctive contribution appears to be the particular Bellman-consistent formulation and its closed-form optimization. The advantages of this formulation over existing surprise- and residual-based memory mechanisms could be established more clearly.
Questions for authors
-
Which training paradigm was used for LongBench-v2 and the general capability evaluations? Could the authors report adapter-only RULER results at 64K and clarify whether continued-pretraining budgets were matched across baselines?
-
Could the authors compare GAP-TTT against standard LoRA continued pretraining on ProLong-64K, matching trainable parameter counts and training budgets? This would help isolate the contribution of dynamic test-time memory.
-
Can the authors evaluate a streaming setting where earlier tokens are evicted from the KV cache? Such an experiment would establish whether the proposed memory can preserve information beyond the accessible attention window.
-
Under what assumptions does minimizing Eq. (4) recover useful predictive corrections? In particular, how does the estimator behave when the slow predictor is already optimal, and how does evolving memory affect Bellman consistency?
-
Could the authors clarify the differences from Titans, GradMem, and particularly TTCD, which also uses prediction-related discrepancies to guide memory updates? A controlled comparison would help establish the contribution of the proposed residual objective.
5 comments
Rebuttal for Reviewer 4fk1 (2/3)
W3 & Q4. Assumptions behind Eq. (4)
Eq. (3) rewrites temporal consistency into a linear relation between the write feature d_t and the gap target g_t. Eq. (4) uses this relation to construct a prefix-specific predictive correction through causal ridge regression.
When is the correction useful? Suppose the gap follows a stable linear relation, g_t = M* d_t + ε_t, with the residual uncorrelated with the complete write feature d_t. With finite second moments, sufficient feature coverage, and transitions whose empirical moments converge, ridge estimates the linearly predictable component of the gap. For the summed loss in Eq. (4), this estimate converges to the best linear approximation as the sample size grows and λ/n → 0. Positive regularization stabilizes finite-sample fitting. When the gap is nonlinear, the memory captures its predictable linear component. The orthogonality assumption applies to the full transition-aware feature, including the next-position key. This directly specifies the condition under which next-key correlations do not bias recovery. Each fit uses only completed transitions, while causal masking preserves the information boundary and stop-gradient stabilizes the training target.
What happens when the slow predictor is already optimal? When its remaining gap is noise uncorrelated with the write feature, there is no systematic correction to learn. Ridge shrinks the finite-sample correction, and the estimate approaches zero as observations accumulate under the conditions above. Table 6 is consistent with this behavior: correction magnitudes are smaller on known contexts (0.11) than unknown contexts (0.31). Table 8 further supports the effectiveness of transition-aware writing, improving the score from 86.85 with k_t to 87.63.
How does evolving memory affect Bellman consistency? The shared candidate M is used to fit all completed transitions within a given prefix. As new observations arrive, the next prefix obtains an updated estimate M_q from the expanded transition set. The same temporal-consistency relation defines every prefix’s objective, while the observations and fitted memory evolve. Thus, sharing M within a fit is fully compatible with updating memory across prefixes. Positive regularization ensures a unique solution at each prefix, and recursive updates compute the same ridge solution as the corresponding prefix-wise fit. We will clarify this interpretation in §3.1.
Rebuttal for Reviewer 4fk1 (3/3)
W4 & Q5. Relation to Titans, GradMem, and TTCD
These methods share the motivation of retaining information that existing predictions miss. GAP-TTT contributes a formulation that jointly derives the predictive residual target and the transition-aware write feature, then constructs memory through causal ridge regression. This focuses limited fast-memory capacity on context-specific corrections that complement the slow predictor. The main differences are summarized below.
| Method | Signal guiding memory | Memory update |
|---|---|---|
| Titans | Surprise (gradient of prediction loss) | Gradient updates with momentum and forgetting |
| GradMem | Model-level context reconstruction loss | Gradient descent on prefix memory tokens |
| TTCD | Representation discrepancy between long-window and short-window views | Context-distillation updates to fast weights |
| GAP-TTT | Temporal semantic residual relative to the slow predictor | Closed-form causal ridge regression |
The closest connection is TTCD, whose discrepancy captures the predictive contribution of additional context at the same position. GAP-TTT instead uses consistency across successive positions to derive both the gap target and the write feature . This connects multi-horizon semantic prediction to memory content and addressing within one regression formulation. This formulation also gives GAP-TTT practical advantages: memory construction requires no test-time backpropagation or heuristic auxiliary inner-loop objective, and causal updates operate without a fixed chunk size controlling update frequency. Parallel prefix scans during prefill and recursive updates during decoding support efficient execution, with per-token memory-update cost independent of context length.
To assess the contribution of the target–feature design to accuracy, we compare alternative memory targets and simplified variants on RULER:
| Variant | RULER Avg. | Difference from GAP-TTT |
|---|---|---|
| Hidden-State | 85.36 | −2.27 |
| KV-Binding | 85.01 | −2.62 |
| Direct semantic residual, | 86.44 | −1.19 |
| GAP target with write feature | 86.85 | −0.78 |
| GAP-TTT | 87.63 | — |
GAP-TTT outperforms the alternative memory targets and both simplified variants. The Hidden-State and KV-Binding comparisons support the benefit of its memory target and design; the direct residual supports the multi-step temporal formulation; and the k_t variant isolates the benefit of transition-aware writing. Together, these results support jointly deriving memory content and addressing from temporal consistency.
We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.
Rebuttal for Reviewer 4fk1 (1/3)
We sincerely thank the reviewer for their time and expertise in providing constructive feedback. Thank you for recognizing the novelty and parameter efficiency of our work. We conducted additional experiments and analyses to address your concerns.
W1 & Q1 & Q2. Training paradigms, 64K adapter-only evaluation, and parameter-matched LoRA
Parameter-matched LoRA-CPT. We place LoRA on the FFN down-projection every four layers, matching GAP-TTT's insertion points; rank r=171 gives 25.21M trainable parameters against 25.24M for GAP-TTT, and both are trained on the same data with the same setting. The results are shown below:
At the same capacity, GAP-TTT is superior at every length, and its advantage grows with context. The results demonstrate that the dynamic memory advantage of GAP-TTT becomes prominent in longer contexts.
Training paradigms. Two settings are used: joint backbone-and-adapter training for the main evaluations, and frozen-backbone, adapter-only training for the parameter-efficiency experiments and ablations. We summarize the experimental setups for different tables, and the results are as follows:
LongBench-v2 and general capability assessment use the full CPT model, while Table 4 and ablation use a frozen skeleton, with only the adapter trained; we will explain this in the table headings.
Adapter-only evaluation at 64K. We extended the adapter-only evaluation for each method to 64K, with the following results:
The results show that GAP-TTT is the only method that outperforms the baseline model when training only the adapter. Both Inplace-TTT and TTT-NTP rely on full-parameter CPT for performance gains, which is expensive in most cases. GAP-TTT provides a more parameter-efficient approach to this.
W2 & Q3. Streaming with KV-cache eviction
To test whether the memory still helps once the context is no longer accessible through attention, we evict it from the KV cache before decoding. We added KV cache eviction tests on RULER with lengths of 32K and 64K, using 50 samples per task at each length, and employing the same hints and answer parsing as the main evaluation. We report the mean and include results for the full KV for comparison. We compared the Base model with the GAP-TTT model using only the adapter on LLaMA3.1-8B. Both models were pre-populated with the full context and then decoded using a fixed 4K or 8K KV window. Tags outside the window were removed before answer decoding, and the memory built during pre-population in GAP-TTT was kept separate from the KV cache. The results are as follows.
Eviction lowers both models’ scores, but GAP-TTT loses less performance and maintains a larger advantage at both context lengths. At 32K, this advantage increases as the KV window shrinks. Thus, GAP-TTT’s benefit persists when earlier tokens are removed from the accessible KV cache, directly testing its performance beyond the full-cache evaluations. We will add the full protocol and per-task results to the appendix.