acceptodds
Under review as a conference paper at ICLR 2027

Gap-Aware Predictive Memory for Test Time Training

Abstract

Test-Time Training (TTT) expands recurrent sequence modeling by treating hidden states or KV-Binding as input-adaptive fast weights that continuously learn during inference. Existing TTT frameworks typically optimize these weights using self-supervised next-token prediction or full-state reconstruction. We argue that fitting fast memory directly to raw future representations is fundamentally redundant: a substantial portion of the target state is already explained by the frozen model’s pretrained predictive prior. Under bounded memory budgets, this redundancy forces fast weights to expend limited capacity on static, task-invariant dynamics rather than novel, context-specific interactions. To address this limitation, we introduce **GAP-TTT** (**G**ap-**A**ware **P**redictive Memory), a TTT framework that explicitly decomposes sequence prediction into a pretrained slow prior and an adaptive fast correction. Instead of modeling full target representations, **GAP-TTT** trains fast memory exclusively on the predictive gap, the semantic residual unaccounted for by the frozen backbone. Specifically, a lightweight semantic predictor exposes the slow model’s future latent expectations, and temporal consistency maps the resulting prediction error into both the memory write target and transition-aware key features. This unifies what to memorize and how to index it without requiring auxiliary inner-loop objectives. GAP-TTT updates these weights via online causal ridge regression, enabling hardware-efficient parallel prefix computation during inference, eliminating test-time backpropagation and iterative gradient descent. In the RULER and LongBench-v2 tests, GAP-TTT consistently outperforms other TTT baseline models while requiring only 25% of their training parameters, validating our argument that fast memory should focus on local contextual residuals.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM, Test Time Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4fk1OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes GAP-TTT, a test-time training framework that stores predictive residuals instead of complete future representations. A lightweight semantic predictor and a temporal-consistency formulation jointly determine memory targets and transition-aware write features. Fast memory is constructed through causal ridge regression, avoiding iterative test-time optimization. Experiments on RULER and LongBench-v2 show improvements over several TTT baselines.

Critical strengths
  1. The idea of allocating fast-memory capacity to information not explained by the pretrained model is compelling. The joint derivation of predictive residual targets and transition-aware write features is particularly interesting, and the ablations support its effectiveness.

  2. Closed-form causal ridge regression eliminates iterative test-time optimization and enables parallel prefix computation. GAP-TTT improves long-context performance with substantially fewer trainable adapter parameters than the evaluated TTT baselines.

Critical weaknesses
  1. The main RULER results use joint backbone and adapter training, while the parameter-efficiency experiments use a frozen backbone. It is unclear which setting is used for other benchmarks. Moreover, there is no parameter-matched standard LoRA baseline trained on the same ProLong data, making it difficult to isolate the benefits of dynamic memory from additional trainable capacity.

  2. The long-context evaluations also appear to keep relevant information within the backbone's attention window. Therefore, they do not establish whether GAP-TTT can retain and retrieve information after it becomes inaccessible through attention, which is an important motivation for persistent memory architectures.

  3. Equation (3) is algebraically correct, but minimizing the sampled temporal-difference residual does not necessarily recover optimal predictive corrections. Correlations involving the next-position key can introduce bias, and the derivation assumes a shared candidate memory although online inference uses evolving memory states.

  4. The broader motivation of prioritizing unpredictable information has precedents in Titans and GradMem, while TTCD explicitly uses representation discrepancies to guide memory updates. The distinctive contribution appears to be the particular Bellman-consistent formulation and its closed-form optimization. The advantages of this formulation over existing surprise- and residual-based memory mechanisms could be established more clearly.

Questions for authors
  1. Which training paradigm was used for LongBench-v2 and the general capability evaluations? Could the authors report adapter-only RULER results at 64K and clarify whether continued-pretraining budgets were matched across baselines?

  2. Could the authors compare GAP-TTT against standard LoRA continued pretraining on ProLong-64K, matching trainable parameter counts and training budgets? This would help isolate the contribution of dynamic test-time memory.

  3. Can the authors evaluate a streaming setting where earlier tokens are evicted from the KV cache? Such an experiment would establish whether the proposed memory can preserve information beyond the accessible attention window.

  4. Under what assumptions does minimizing Eq. (4) recover useful predictive corrections? In particular, how does the estimator behave when the slow predictor is already optimal, and how does evolving memory affect Bellman consistency?

  5. Could the authors clarify the differences from Titans, GradMem, and particularly TTCD, which also uses prediction-related discrepancies to guide memory updates? A controlled comparison would help establish the contribution of the proposed residual objective.

5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 4fk1 (1/3)

We sincerely thank the reviewer for their time and expertise in providing constructive feedback. Thank you for recognizing the novelty and parameter efficiency of our work. We conducted additional experiments and analyses to address your concerns.

W1 & Q1 & Q2. Training paradigms, 64K adapter-only evaluation, and parameter-matched LoRA

Parameter-matched LoRA-CPT. We place LoRA on the FFN down-projection every four layers, matching GAP-TTT's insertion points; rank r=171r{=}171 gives 25.21M trainable parameters against 25.24M for GAP-TTT, and both are trained on the same data with the same setting. The results are shown below:

MethodParameters4K8K16K32K64KAverage
Base090.8488.5988.1780.1878.8985.33
LoRA-CPT25.21M89.6888.5287.4480.5579.1285.06
GAP-TTT25.24M89.8889.4788.5482.6181.2086.34

At the same capacity, GAP-TTT is superior at every length, and its advantage grows with context. The results demonstrate that the dynamic memory advantage of GAP-TTT becomes prominent in longer contexts.

Training paradigms. Two settings are used: joint backbone-and-adapter training for the main evaluations, and frozen-backbone, adapter-only training for the parameter-efficiency experiments and ablations. We summarize the experimental setups for different tables, and the results are as follows:

ExperimentBackboneTrainable parametersCPT budget
RULER, Table 1UnfrozenFull model + adapter25B
RULER, Table 4FrozenAdapter only25B
LongBench-v2, Table 2UnfrozenFull model + adapter25B
General capability, Table 3UnfrozenFull model + adapter25B

LongBench-v2 and general capability assessment use the full CPT model, while Table 4 and ablation use a frozen skeleton, with only the adapter trained; we will explain this in the table headings.

Adapter-only evaluation at 64K. We extended the adapter-only evaluation for each method to 64K, with the following results:

Method4K8K16K32K64KAverage
Base90.8488.5988.1780.1878.8985.33
In-Place TTT88.9287.3385.6777.6075.8683.08
TTT-NTP88.7585.9686.4380.2978.1283.91
GAP-TTT89.8889.4788.5482.6181.2086.34

The results show that GAP-TTT is the only method that outperforms the baseline model when training only the adapter. Both Inplace-TTT and TTT-NTP rely on full-parameter CPT for performance gains, which is expensive in most cases. GAP-TTT provides a more parameter-efficient approach to this.

W2 & Q3. Streaming with KV-cache eviction

To test whether the memory still helps once the context is no longer accessible through attention, we evict it from the KV cache before decoding. We added KV cache eviction tests on RULER with lengths of 32K and 64K, using 50 samples per task at each length, and employing the same hints and answer parsing as the main evaluation. We report the mean and include results for the full KV for comparison. We compared the Base model with the GAP-TTT model using only the adapter on LLaMA3.1-8B. Both models were pre-populated with the full context and then decoded using a fixed 4K or 8K KV window. Tags outside the window were removed before answer decoding, and the memory built during pre-population in GAP-TTT was kept separate from the KV cache. The results are as follows.

ContextKV windowBaseGAP-TTTΔ
32KFull80.1882.61+2.43
32K8K64.969.6+4.7
32K4K57.663.6+6.0
64KFull78.8981.20+2.31
64K4K51.257.8+6.6

Eviction lowers both models’ scores, but GAP-TTT loses less performance and maintains a larger advantage at both context lengths. At 32K, this advantage increases as the KV window shrinks. Thus, GAP-TTT’s benefit persists when earlier tokens are removed from the accessible KV cache, directly testing its performance beyond the full-cache evaluations. We will add the full protocol and per-task results to the appendix.

AuthorsOpenReview ↗

Rebuttal for Reviewer 4fk1 (2/3)

W3 & Q4. Assumptions behind Eq. (4)

Eq. (3) rewrites temporal consistency into a linear relation between the write feature d_t and the gap target g_t. Eq. (4) uses this relation to construct a prefix-specific predictive correction through causal ridge regression.

When is the correction useful? Suppose the gap follows a stable linear relation, g_t = M* d_t + ε_t, with the residual uncorrelated with the complete write feature d_t. With finite second moments, sufficient feature coverage, and transitions whose empirical moments converge, ridge estimates the linearly predictable component of the gap. For the summed loss in Eq. (4), this estimate converges to the best linear approximation as the sample size grows and λ/n → 0. Positive regularization stabilizes finite-sample fitting. When the gap is nonlinear, the memory captures its predictable linear component. The orthogonality assumption applies to the full transition-aware feature, including the next-position key. This directly specifies the condition under which next-key correlations do not bias recovery. Each fit uses only completed transitions, while causal masking preserves the information boundary and stop-gradient stabilizes the training target.

What happens when the slow predictor is already optimal? When its remaining gap is noise uncorrelated with the write feature, there is no systematic correction to learn. Ridge shrinks the finite-sample correction, and the estimate approaches zero as observations accumulate under the conditions above. Table 6 is consistent with this behavior: correction magnitudes are smaller on known contexts (0.11) than unknown contexts (0.31). Table 8 further supports the effectiveness of transition-aware writing, improving the score from 86.85 with k_t to 87.63.

How does evolving memory affect Bellman consistency? The shared candidate M is used to fit all completed transitions within a given prefix. As new observations arrive, the next prefix obtains an updated estimate M_q from the expanded transition set. The same temporal-consistency relation defines every prefix’s objective, while the observations and fitted memory evolve. Thus, sharing M within a fit is fully compatible with updating memory across prefixes. Positive regularization ensures a unique solution at each prefix, and recursive updates compute the same ridge solution as the corresponding prefix-wise fit. We will clarify this interpretation in §3.1.

AuthorsOpenReview ↗

Rebuttal for Reviewer 4fk1 (3/3)

W4 & Q5. Relation to Titans, GradMem, and TTCD

These methods share the motivation of retaining information that existing predictions miss. GAP-TTT contributes a formulation that jointly derives the predictive residual target and the transition-aware write feature, then constructs memory through causal ridge regression. This focuses limited fast-memory capacity on context-specific corrections that complement the slow predictor. The main differences are summarized below.

MethodSignal guiding memoryMemory update
TitansSurprise (gradient of prediction loss)Gradient updates with momentum and forgetting
GradMemModel-level context reconstruction lossGradient descent on prefix memory tokens
TTCDRepresentation discrepancy between long-window and short-window viewsContext-distillation updates to fast weights
GAP-TTTTemporal semantic residual relative to the slow predictorClosed-form causal ridge regression

The closest connection is TTCD, whose discrepancy captures the predictive contribution of additional context at the same position. GAP-TTT instead uses consistency across successive positions to derive both the gap target gtg_t and the write feature dt=kt−γkt+1d_t = k_t - \gamma k_{t+1}. This connects multi-horizon semantic prediction to memory content and addressing within one regression formulation. This formulation also gives GAP-TTT practical advantages: memory construction requires no test-time backpropagation or heuristic auxiliary inner-loop objective, and causal updates operate without a fixed chunk size controlling update frequency. Parallel prefix scans during prefill and recursive updates during decoding support efficient execution, with per-token memory-update cost independent of context length.

To assess the contribution of the target–feature design to accuracy, we compare alternative memory targets and simplified variants on RULER:

VariantRULER Avg.Difference from GAP-TTT
Hidden-State85.36−2.27
KV-Binding85.01−2.62
Direct semantic residual, γ=0\gamma = 086.44−1.19
GAP target with ktk_t write feature86.85−0.78
GAP-TTT87.63—

GAP-TTT outperforms the alternative memory targets and both simplified variants. The Hidden-State and KV-Binding comparisons support the benefit of its memory target and design; the direct residual supports the multi-step temporal formulation; and the k_t variant isolates the benefit of transition-aware writing. Together, these results support jointly deriving memory content and addressing from temporal consistency.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.

Reviewer_QF9xOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GAP-TTT adds a temporary semantic correction memory to a pretrained model. A temporal-consistency equation jointly defines the predictive residual and transition-aware write feature, and causal ridge regression fits the memory at each prefix. The paper evaluates full-model and adapter-only training on long-context tasks and reports accuracy, trainable-parameter, and inference-cost tradeoffs.

Critical strengths
  1. The joint temporal construction and causal implementation provide a coherent methodological contribution beyond simply storing prediction residuals.

  2. RULER gains across two backbones, positive component ablations, and measured efficiency support practical value. The adapter-only model improves the frozen backbone using roughly one quarter of the adapter baselines' trainable parameters.

Critical weaknesses
  1. The target's benefit is not isolated from architecture and capacity. Table 5's Hidden-State row matches Table 4's TTT-NTP scores at all four lengths, although Table 4 lists 100.66M parameters for TTT-NTP versus 25.24M for GAP-TTT. This needs reconciliation with the shared-configuration claim before attributing the difference to the target. The adapter-only gain is modest (+0.69 points on average, with a drop at 4K), making a matched comparison consequential.

  2. The LongBench-v2 comparison lacks a clearly common protocol. Its six base scores match TTT-NTP Table 2, which uses a 32k-token budget and head-and-tail truncation; the manuscript describes 32K–128K tokens and its displayed averages match unweighted domain means rather than the cited paper's example-weighted accuracy. The cited paper's own TTT-NTP row exceeds GAP-TTT's reported scores in five of six domains. These cross-paper differences do not establish a ranking, but leave the reported advantage difficult to assess.

Questions for authors
  1. [Clarification] How do the Table 5 variants differ in trainable capacity and training configuration, and were the matching Hidden-State scores obtained independently?
  2. [Clarification] Which LongBench-v2 rows were rerun, and what input budget, truncation, and aggregation were applied consistently across methods?
4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer QF9x

We sincerely thank the reviewer for their time and expertise in providing constructive feedback. Thank you for recognizing the novelty and parameter efficiency of our work. We conducted additional experiments and analyses to address your concerns.

W1 & Q1. Memory targets and capacity-matched comparison

The original Table 5 contains three different configurations. Its Hidden-State row reuses the 100.66M-parameter TTT-NTP run from Table 4, rather than an independently trained 25.24M variant, because TTT-NTP uses the NTP hidden-state target. The KV-Binding row also uses 100.66M trainable parameters and follows In-Place TTT’s KV-binding objective, but omits its Conv1D component, explaining the slightly different scores despite the same parameter count (85.01 vs. 84.88). GAP-TTT uses 25.24M trainable parameters, consistent with Table 4. We apologize for the unclear description of these configurations.

To provide a capacity-controlled comparison, we independently retrained Hidden-State and KV-Binding with 25.24M trainable parameters, using the same ProLong-64K data, optimizer, and adapter insertion positions as GAP-TTT. The results on LLaMA3.1-8B over RULER 4K–32K are as follows:

VariantParams4K8K16K32KAvg
TTT-NTP100.66M88.7585.9686.4380.2985.36
Hidden-State25.24M87.9485.5585.5879.8984.74
KV-Binding25.24M87.4086.1084.2079.3084.25
GAP-TTT25.24M89.8889.4788.5482.6187.63

At matched capacity, GAP-TTT outperforms Hidden-State and KV-Binding by 2.89 and 3.38 points, respectively. It also exceeds the original, larger-capacity variants. These results support the effectiveness of GAP-TTT’s gap target and memory design beyond differences in trainable capacity.

Slight advantage. GAP-TTT is the only method that outperforms Base in Table 4, and the matched comparison further supports its architectural advantages. It can operate as an external adapter for TTT, whereas the published gains of TTT-NTP and In-Place TTT use full-parameter CPT. This makes GAP-TTT a lightweight and efficient option when full-parameter training is expensive.

W2 & Q2. LongBench-v2 protocol and aggregation

“32K–128K” describes document lengths in the medium-length split, rather than the model input budget. Every method is evaluated with the same 32K-token input budget, head–tail truncation, prompts, answer parsing, and official A/B/C/D exact-match scoring. The Base row is the published reference for the released checkpoint from TTT-NTP Table 2; CPT, In-Place TTT, TTT-NTP, and GAP-TTT come from our own training runs. We corrected a mismatch in the question lists used for the trained rows and re-scored them on the same 215-question medium-length split. The paper’s Avg is the unweighted mean of the six domain accuracies; example-weighted overall accuracy is the total number of correct answers divided by 215. We report both below.

Table B. LongBench-v2 medium-length split, common 32K-token protocol.

MethodSingle-DocMulti-DocICLDialogueCodeStructMacroWeighted
Baseline26.025.027.915.833.326.125.725.6
CPT29.927.325.621.133.326.127.227.4
In-place TTT29.927.327.921.133.326.127.627.9
TTT-NTP31.227.327.915.833.330.427.728.4
GAP-TTT31.229.530.221.144.430.431.130.2

GAP-TTT achieved an overall accuracy of 30.2% and a macro accuracy of 31.1%, while our rerun of TTT-NTP achieved overall accuracies of 28.4% and 27.7%, respectively. These results were obtained using the publicly available TTT-NTP code trained on the same 25B ProLong-64K dataset as GAP-TTT. Under this general protocol, GAP-TTT achieved the highest scores in both aggregation methods.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Reviewer_QF9xOpenReview ↗

Updated assessment of the RULER and LongBench-v2 results

Thank you for the clarification regarding the Hidden-State row in Table 5, which originally used the TTT-NTP run also shown in Table 4. Moreover, I appreciate you having re-run the two controls, each with 25.24M trainable parameters, and I am very relieved that GAP-TTT outperforms both controls at all four context lengths. In order to make the relevant data point in Table 5 more understandable, it would be great if in the revised version of Table 5 you could also specify which of the write features were kept fixed in the various variants, and which of the update rules were kept fixed.

The corrected LongBench-v2 results resolve my question about the evaluation protocol. After re-scoring the four trained models on the same 215 questions with a 32K-token input limit, GAP-TTT answered 65 questions correctly, compared with 61 for your TTT-NTP run. I view this as modest supporting evidence. The capacity-matched RULER results are the main reason I now favor weak acceptance.

Reviewer_opN2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GAP-TTT adds a trainable, Linear-Attention-like memory module to selected MLP blocks of a pretrained model. Its most interesting idea is deriving both the residual target and the transition-aware write key from a Bellman formulation, so that the module learns only what the model failed to predict. This requires an auxiliary loss on top of the next-token prediction loss.

The paper claims better RULER scores than other test-time training methods on Qwen3-4B and LLaMA3.1-8B-Base models, and better LongBench-v2 scores on LLaMA3.1-8B-Base model. For LLaMA model it only uses a quarter of their trainable parameters. Its ablations indicate that GAP-TTT outperforms a method that directly predicts future representations, makes smaller corrections on text the model already predicts well, and outperforms other gap formulations. It also reports comparable memory and latency overhead than the other test-time training methods.

Critical strengths
  1. The motivation is reasonable and the proposed method, particularly the Bellman formulation, is interesting. The derivation is mathematically correct.
  2. GAP-TTT shows general improvement on two long-context benchmarks with two models.
  3. The method adds modest memory and latency overhead over the base model, and less overhead than the other TTT methods.
  4. The paper provides several ablations of its design.
Critical weaknesses
  1. Several category scores do not appear to match accuracy on the 215 questions in the medium length LongBench-v2 subset. For example, the code section has only 9 questions in this subset, so its scores should be 33.3 or 44.4, yet the table reports values such as 34.4 and 35.3. The paper does not state its scoring rule or number of runs, so these may reflect averaging or a different metric. Can the author explain how were these scores computed?
  2. On LLaMA3.1-8B, In-Place TTT and TTT-NTP score 1.1 to 1.5 points below CPT in Table 1, and on average both fall below the base model in Table 4 when the backbone is frozen. Both methods appear not to be effective in the paper's setup, whereas their original papers report gains (for example, TTT-NTP reports +3.9 over the base model on RULER). Moreover, the base model scores 86.94 here over 4K–32K, against 55.80 in the TTT-NTP paper. Since the paper states that it follows the baselines' original evaluation settings, could the author explain why there're big differences on the base model's score and the baseline methods' effectiveness?
  3. The method uses a ridge-regression linear memory, similar to MesaNet, rather than a simpler update such as DeltaNet. There's no ablation on the solver. It would be informative to see how much of the reported advantage depends on this choice.
Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer opN2

We sincerely thank the reviewers for their meticulous and constructive review. We are pleased that the reviewers recognized the reasonable motivation, novel methodology, and more efficient implementation of GAP-TTT. We have added additional experiments and explanations to answer your questions.

W1. LongBench-v2 scores

Thank you for pointing that out. We sincerely apologize for mixing question lists from different LongBench-v2 length splits. We have corrected this data-processing error and re-scored the results on the same 215-question medium-length split using the official A/B/C/D exact-match criteria.

MethodSingle Document (n=77)Multiple Document (n=44)ICL (n=43)Dialogue (n=19)Code (n=9)Structure (n=23)Macro Avg.Weighted Avg. (n=215)
LLaMA3.1-8B26.025.027.915.833.326.125.725.6
CPT29.927.325.621.133.326.127.227.4
In-Place TTT29.927.327.921.133.326.127.627.9
TTT-NTP31.227.327.915.833.330.427.728.4
GAP-TTT31.229.530.221.144.430.431.130.2

We report both the macro-average across domains and the example-weighted overall accuracy. GAP-TTT achieves the highest score under both metrics. The corrected results continue to support GAP-TTT’s effectiveness on LongBench-v2.

W2. Base-model score and baseline effectiveness

The published gains of both baselines come from their original recipes, which differ from ours in both corpus and budget. Table 1 retrains both on ProLong-64K for 25B tokens so that they are compared with GAP-TTT under the same data and budget; the table below records what their own papers used, which is why the reproduced and published numbers differ in strength.

MethodData in original paperTokens in original paperRetrained for Table 1
In-Place TTTAuthors' own corpus~20BProLong-64K, 25B, joint
TTT-NTPLong-Data-Collections0.4BProLong-64K, 25B, joint
GAP-TTT (Table 1)ProLong-64K25BProLong-64K, 25B, joint
GAP-TTT (Table 4)ProLong-64K25BProLong-64K, 25B, frozen

Both Inplace-TTT and TTT-NTP rely on full-parameter CPT, and their papers do not report the performance of training only the TTT adapter. Table 4 evaluates their adapter-only performance with the backbone frozen. In this setting, GAP-TTT improves Base from 86.94 to 87.63 using only 25.24M trainable parameters, while both baselines remain below Base despite using 100.66M parameters. This demonstrates GAP-TTT’s parameter efficiency.

Base Scores and Evaluation Protocol. The two numbers come from different evaluators, and we do not know TTT-NTP's evaluation settings, so they are not comparable across papers. We asked the TTT-NTP authors for their settings — in particular whether they used OpenCompass, and why LLaMA3.1-8B scored so poorly under their evaluation — but did not receive this information. We therefore evaluated the released LLaMA3.1-8B-Base checkpoint with OpenCompass ourselves, using the same hints and answer parsing for every method.

SourceEvaluated checkpointRULER evaluation4K–32K Avg.
TTT-NTPLLaMA3.1-8B-BaseTheir evaluation setup55.80
GAP-TTTLLaMA3.1-8B-BaseOpenCompass86.94

Since the number of their evaluation rulers is unknown, we tested on our own RULER to ensure a fair comparison. In summary, based on our real-world testing, the actual performance of LLaMA3.1-8B-Base should be significantly higher than the 55.80 score reported in TTT-NTP. We will replace the statement "strictly adhere to the original evaluation settings" with a clear description of our shared training scheme and OpenCompass evaluation protocol.

W3. Solver ablation

Ridge regression is an established solver, also used in MesaNet. Our contribution is the predictive-gap target and its causal construction. To measure the solver’s contribution, we replaced the ridge solve with a DeltaNet-style update on LLaMA3.1-8B (adapter-only training, RULER 4K–32K), keeping the gap target, write feature, adapter dimensions, and parameter count fixed.

Memory updateRULER Avg. (4K–32K)
DeltaNet-style delta rule87.20
Ridge solve (GAP-TTT)87.63

DeltaNet reaches 87.20, retaining a 0.26-point improvement over Base’s 86.94. Ridge adds another 0.43 points, bringing the total gain to 0.69. The solver supplies about two-thirds of the gain.GAP-TTT therefore remains effective with the alternative update, while the ridge solver contributes to its final performance.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity, completeness, and expressiveness of our paper. We hope our response has addressed your concerns. If you have any further questions, please feel free to let us know; we are happy to assist you.

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair,

We thank you for handling our submission, and Reviewers opN2, QF9x, and 4fk1 for detailed, technically engaged reviews. We submitted point-by-point responses, with new experiments or analyses for every major concern. However, it was frustrating that no reviewers participated in the discussion a week after our rebuttal was submitted.. We ask you to weigh the absence of discussion when deciding, and to assess our responses directly where they differ from the reviews. Below is a summary of this rebuttal.

Strengths

Novelty. Fast memory should hold only what the model fails to predict; the residual target and transition-aware write key are derived jointly from a temporal-consistency relation. opN2 calls the derivation mathematically correct.

Evidence. Consistent RULER gains on two backbones and LongBench-v2 gains on LLaMA3.1-8B, with component ablations.

Efficiency. Closed-form causal ridge removes test-time optimization and allows parallel prefix computation; adapter-only training with 25.24M parameters beats a frozen base, versus 100.66M for the baselines.

Common concerns and how we addressed them

Capacity and setup confounds (capacity-matched baselines, joint vs. frozen training, LoRA, 64K adapter-only). Added a parameter-matched LoRA-CPT baseline (25.21M vs. 25.24M; GAP-TTT +1.28 on average), retrained Hidden-State and KV-Binding at 25.24M (−2.89 / −3.38), and added a setup table plus 64K results, where GAP-TTT is the only method that beats the frozen base (86.34 vs. 85.33). This directly solves the following problem: reviewer 4fk1 W1 & Q1–Q2; reviewer QF9x W1 & Q1; reviewer opN2 W2.

LongBench-v2 protocol. Corrected a question-list mismatch and re-scored every trained row on the same 215-question split (32K budget, head–tail truncation, official exact match). GAP-TTT is highest under both aggregations (31.1 macro / 30.2 weighted vs. 27.7 / 28.4 for TTT-NTP). This directly solves the following problem: reviewer opN2 W1; reviewer QF9x W2 & Q2.

Base score and baseline reproduction. Differences come from corpus, budget, and evaluation stack; we retrained both baselines on ProLong-64K (25B tokens) and evaluated LLaMA3.1-8B-Base ourselves with OpenCompass. We will replace the "original settings" claim with an explicit protocol. This directly solves the following problem: reviewer opN2 W2.

Beyond the attention window. RULER at 32K/64K with 4K and 8K KV windows shows the gap widening as the window shrinks: +4.7, +6.0, +6.6 versus +2.3 with the full cache. This directly solves the following problem: reviewer 4fk1 W2 & Q3.

Theory and novelty (Eq. 4, Bellman consistency, Titans/GradMem/TTCD, solver). We state when ridge recovers the predictable gap and why next-key correlations do not bias it, add a method-by-method comparison, and show a DeltaNet-style update still reaches 87.20 vs. 87.63. This directly solves the following problem: reviewer 4fk1 W3–W5; reviewer opN2 W3.

We will revise the manuscript accordingly and are glad to provide any further material.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.