acceptodds
Under review as a conference paper at ICLR 2027

All Memory Is Local: Measuring the Horizon of Persistent Test-Time Training

Abstract

Test-time training (TTT) lets a language model update a small set of fast weights as it reads. The drop-in methods considered here reset those weights at document boundaries, so nothing carries across documents. When the reset is removed, the model recalls symbols from earlier documents well after they have left the context, which looks like persistent memory. We ask how much of this signal is memory. Delayed recall has at least three other sources: information the model already had (pre-training contamination, in-context copying, synthetic vocabularies shared across sessions), properties of the evaluation (scoring order, length-normalized accuracy), and adaptation to the stream as a whole. We build a protocol, Streaming-TTT, that separates them: per-session obfuscated vocabularies, first-mention targets, fixed-lag re-scoring, and a control of sessions that were never streamed. On 180 code repositories and 200 prose sessions with Qwen3 models from 0.6B to 4B, persistence raises first-mention recall to 7.6 times the frozen model's immediately after a session, but the estimated session-specific component falls by about four fifths within five sessions; by lag 15 what remains cannot be distinguished from the gain on never-streamed sessions. A linear-interference model predicts that this decay depends on subsequent write mass rather than on the learning rate, and the measured curves collapse accordingly. Resetting at session boundaries keeps 60% of the immediate benefit and avoids the general-capability cost that unrestricted persistence incurs. In this setting, delayed recall alone did not establish persistent memory; the relevant quantity is what remains after adaptation is subtracted.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time training, fast weights, continual learning, catastrophic forgetting, evaluation methodology, language models, memory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BHHDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the temporal impact of test-time training (TTT) in data streaming scenarios (at inference time). They study a series of TTT policies that vary how frequently fast weights are updated and incorporated at inference time: persist, session-reset, consistency, random (Table 1). They find that the benefits of test time training decay pretty rapidly after the initial update to fast weights, and after several additional streamed documents, it is hard to disambiguate the performance gain from TTT from simple contextual knowledge from never-streamed sessions. They recommend using a “session-reset” TTT policy.

Critical strengths
  1. I liked how this paper attempts to disambiguate different factors that lead to performance gains in the streaming protocol during TTT. They account for base model knowledge by renaming cross-file identifiers for code and session-defined symbols for prose. They account for the test-time knowledge by scoring a “chunk” before it affects fast weights. They study the impact of fast-weights on sessions that haven't been streamed.
Critical weaknesses
  1. The main takeaway from the conclusion (lines 403-405) don’t seem to be supported by the “Main results” Table 1. For example, the authors write "Resetting at session boundaries captures most of the benefit without the benchmark cost or privacy exposure”; but in table 1 for “PPL at lag” session-reset was the second worst method. If anything, across both the coding and prose tasks it seems the “random” baseline beats session-reset and many of the other policies. I am assuming higher recall is better and lower PPL is better.

  2. I found this paper really challenging to understand. I had several instances while reading where I wasn’t sure what a term meant and when I re-read prior parts of the paper I wasn’t able to figure out the definition (I have enumerated some of those questions below). For example in the interpretation of Figure 4.a (lines 301-302) it is written: “the adaptation reshapes per-token likelihood in a way the normalized rule rewards without changing which answers are correct. We report both rules.” What are the “rules”?

Questions for authors
  1. In table 1 is the “frozen” policy just the base model w/ no TTT? If so, how do you control for the computational cost of the non-frozen TTT policies vs the frozen case which incurs no TTT overhead?

  2. How do you define a “write budget”? Is this the number of tokens that are streamed? Does it account for the excess compute that goes into training the fast weights?

  3. In figure 1, were the “never streamed repositories” evaluated once? I am trying to figure out why the blue line doesn’t change across sessions. If so, were they evaluated after lag 40? Is the never streamed line (blue) reporting the “stream adaption” score (lines 192 - 196).

  4. Clarification request: What is the “First mention recall” metric for code? I can see on line 131 that prose is evaluated with log likelihood which is a metric I am familiar with. On line 178-179 could you give an example of a prompt and the evaluation criteria to evaluate what it means for a model to recall cross-file identifiers?

  5. What is the main takeaway from the raw vs. length normalized accuracies in Figure 4.a section 5.6? There seems to be no trend in the figure and therefore I am not sure what the actionable takeaways are. The written text description just summarizes what is in the figure and I did not glean additional interpretation or insight from the main text discussion of the figure.

6 ) I find the citation format a bit strange: I notice the publication venues are abbreviated (e.g., ‘ICLR’ instead of ‘International Conference on Learning Representations’). I have typically seen the fully publication venue spelled out in most reference sections, but perhaps I am unfamiliar with this format.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer BHHD

"Session-reset is second worst in PPL at lags 5–40, so the conclusion is unsupported." At lags ≥ 5, session-reset equals the frozen model by construction: its fast weights were zeroed at the session boundary, so there is nothing to probe (Remark 1). Those cells measure cross-session carry-over, of which the session-reset policy has none; they are not a ranking of quality. The conclusion "captures most of the benefit" refers to lag 0, where session reaches 4.6% recall and 1229 PPL against persist's 7.1% and 398 (60% of the recall excess over frozen). Random is not a competitor to session-reset: it is a control for the consistency gate (same gate values, permuted), and it carries the same capability cost as persist. The revised Table 1 caption states both points. You are right that higher recall and lower PPL are better.

Undefined terms. The revision adds a worked example in §4: a held-out file contains from utils import qzrvmt; the model is scored at the position predicting the first sub-token of qzrvmt, a name defined in an earlier file of the same repository and never seen in this file. "First-mention recall" is top-1 accuracy at such positions. "Scoring rules" are raw accuracy and length-normalized accuracy as implemented in lm-eval-harness (acc vs. acc_norm); the point of the raw-vs-normalized panel (Fig. 4a in the submitted version, Fig. 5a in the revision) is that a policy can raise acc_norm on ARC-Easy by 3.7 pt while raw accuracy moves 0.1 pt, so reporting only acc_norm would misstate the capability effect—the actionable takeaway is to report both, and the revised §5.6 says so explicitly. "Write budget" meant the expected gate value, not tokens or compute; the revision uses "mean gate" to avoid the ambiguity.

Frozen vs. TTT compute. Frozen is the base model with no TTT and incurs no TTT cost; TTT policies run at ≈0.4× its throughput (§5.7). I do not equalize compute, because the question is what persistence buys at its cost, and the cost is reported (§5.7 now says this explicitly).

Never-streamed line in Fig. 1. Evaluated once, with the fast-weight state at the end of the stream (after all 178 sessions), on 65 repositories that were never streamed; it is the stream-adaptation estimate of §3.2 and is constant by construction. The caption now says so.

Citation format. I use the ICLR style file; venue names are now spelled out.

Reviewer_sKGYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study whether test-time training retains information from earlier sessions. They use renamed symbols and delayed recall probes, and conclude that most session-specific memory disappears within a few sessions. They also propose a gate that is intended to improve retention by changing where updates are applied while keeping their size fixed.

Critical strengths

The attempt to separate recall of a particular session from general adaptation is useful. Independent renaming and first-mention probes address real evaluation problems, and the reported costs of persistent adaptation are worth examining.

Critical weaknesses
  1. The main conclusion about memory decay does not follow from the control. In Section 3.2, recall at earlier checkpoints is compared with performance on fresh sessions measured only at the final checkpoint. General adaptation may have changed between those checkpoints. The sessions included also change with the delay. The reported decline can therefore reflect either of these changes rather than forgetting. I do not think this supports the claim that roughly 80% of session-specific memory has disappeared.

  2. The gate does not preserve update size as claimed. A gate with mean=1 does not preserve the norm of the update it multiplies. For example, an update (m,m/100)(m,m/100) multiplied by gates (0,2)(0,2) and (2,0)(2,0) gives norms m/50m/50 and 2m2m. Both gates have the same mean and satisfy the stated cap. Consequently, the experiment cannot attribute better recall to where an equal amount was written. This is a core contribution which simply does not hold.

  3. Supplementary material referenced but not attached. The authors state: "Experimental code, stream construction scripts, obfuscation pipelines, and unit tests are provided as supplementary material.", but no supplementary material exists.

Questions for authors
  1. How do the retention curves change when new controls are evaluated at the same checkpoints and the same sessions are followed across delays? Does the implementation additionally match the norms after gating?

  2. The authors refer to supplementary material containing the code and logs. It does not exist in the reviewer submission. Section 5.1 also refers to a per-document-reset result in Table 1, but that row is absent.

  3. The related work needs to distinguish the new measurement from existing adaptation and forgetting experiments. Rannen-Triki et al. already compare continuous adaptation across books with resets at book boundaries. Please discuss the following papers. The proposed lagged-symbol test could add something, but its control must first establish what it measures.

  4. The references also contain hallucinations. All seven first names in In-Place Test-Time Training are incorrect; for example, “Xiao Feng” should be Guhao Feng (record).

1 comment
AuthorsOpenReview ↗

Reply to Reviewer sKGY

(a) "The decline can reflect composition or adaptation changes rather than forgetting." Both readings were tested on the submitted 1.7B runs (3 seeds).

Composition. Restricting to the 288 held-out documents that reach all four lags (a balanced panel, identical sessions at every lag) gives a persist−frozen excess of 6.12 → 1.93 → 1.12 → 0.67 pt (percentage points of first-mention recall) at lags 0/5/15/40, versus 6.19 → 1.57 → 0.92 → 0.67 for the full set. The shape is unchanged.

Adaptation changing over the stream. The average stream position at which a lag-L measurement is taken is 91, 93, 98 and 110 (of 178) for L = 0, 5, 15, 40: nearly matched, so the lag axis is not a proxy for stream position. If adaptation grows along the stream, it biases later measurements upward; since lag-40 measurements are taken slightly later on average than lag-0 measurements, this bias would reduce the measured decay, not create it. Splitting each lag by measurement position shows lag-0 excess rising from 4.96 (early third) to 6.73 (late third), while lag-40 excess is flat (0.54 / 0.87 / 0.57); the opposite alternative, a fast-weight state that degrades along the stream and so depresses late measurements, is contradicted by the rising lag-0 excess. The late-position lag-40 excess (0.57, measured near the end of the stream) is the value most directly comparable to the end-of-stream never-streamed control (0.35), and the two are within each other's intervals.

One position effect does exist, and the revision reports it: lag-5 excess falls along the stream (2.35 [1.48, 3.29], 1.74 [0.80, 2.77], 0.67 [0.05, 1.33] in the three terciles), i.e. later sessions lose their lag-0 advantage faster. This does not rescue the long-lag residual (it makes the decay steeper for late sessions, not shallower), but it is a time-varying effect; the interference model of §5.5 (Prop. 1) with uniform write masses does not predict it, and the revision says so (§5.3, Fig. 2a).

I agree that a cleaner design evaluates the never-streamed control at several checkpoints; I will add that run. I also agree the original phrasing "falls by about four fifths" should be qualified: the balanced-panel estimate of P(5)/P(0) (§3.2) is 0.27 rather than 0.21; the revision reports both and now says "three quarters to four fifths" throughout.

(b) "Mean-one gates do not preserve the update norm." Correct. The normalization fixes the mean of the gate α to one; it does not make the Frobenius norm of the gated update α ⊙ G equal to that of the ungated update G. If α correlates with |G|, the consistency gate writes more than the random gate despite identical gate histograms. The submitted logs do not record write norms, so I cannot establish that the norms were equal. I have (i) renamed the comparison "equal mean gate" throughout, (ii) removed the claim that differences reflect placement alone, (iii) restated H3's result as "consistency > random at equal mean gate; attribution to placement vs. magnitude is open", and (iv) committed to a norm-matched rerun (rescale the gated update to the Frobenius norm of the ungated update at every boundary) for the revision. The comparison that does not depend on this, random ≈ persist at every lag, is unaffected.

(c) Supplementary material and Table 1. The supplementary zip was not attached at submission; it is attached to the revision. The per-document-reset row was never run at 1.7B; the sentence in §5.1 now cites the 0.6B result (reset at η=1e-2 and 3e-2 on the earlier 50-repository stream with a shared synthetic vocabulary: cross-file first-mention recall 1.0% / 0.8% vs 0.9% frozen, App. F) and unit test (3) of App. D, which asserts that reset equals processing each document in isolation.

(d) Related work. Rannen-Triki et al. (2024) compare continuous online adaptation with resets at book boundaries and report that a continuously adapted model often underperforms shortly after boundaries and that resetting at boundaries gave the lowest total loss; this is the perplexity-level precursor of the persist-vs-session comparison here, with the same practical conclusion, and it is now cited as such in §2. What this protocol adds is the separation of session-specific retention from stream adaptation via the never-streamed control and the fixed-lag recall probe; dynamic evaluation measures the aggregate. Krause et al. (2018), Tirumala et al. (2022), AND-mask and SAND-mask are also added.

(e) References. The In-Place TTT author list was wrong (correct: Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Di He, Wenhao Huang, Tianle Cai).

(f) Ethics note. The AI-use statement discloses the AI assistant's role in full. Every number in the paper is regenerated from the raw logs by the scripts now in the supplement; the reference error was a failure of my verification process, which I own and have fixed.

Reviewer_GQ2tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether persistent test-time training (TTT) can provide memory across sessions. The authors introduce Streaming-TTT, an evaluation protocol that uses session-specific symbol obfuscation, first-mention evaluation, fixed-lag probing, and never-streamed controls to separate session-specific retention from other sources of improvement. Experiments show that persistent updates improve short-term recall, but the estimated memory benefit decays rapidly after accounting for stream adaptation. The paper also finds that consistency-based gating mainly improves short-term retention, while unrestricted persistence can hurt general capabilities.

Critical strengths
  1. An important problem with a clearer evaluation framework. The paper challenges the assumption that delayed recall necessarily indicates persistent memory. By separating prior knowledge, evaluation artifacts, and stream adaptation from session-specific retention, it provides a useful framework for studying memory in TTT.

  2. Careful experimental controls. The protocol combines session-specific obfuscation, never-streamed controls, randomized gate placement, unit tests for evaluation ordering, and session-level bootstrap confidence intervals. These practices help reduce common confounding factors and could benefit other research on continual adaptation and model memory.

  3. Useful analysis of practical trade-offs. The paper compares persistence, session resets, and selective writing across model sizes and data types. By examining both retention and general capability degradation, it offers useful insights into the costs of persistent adaptation.

Critical weaknesses
  1. Limited scope and insufficient discussion of related methods. The experiments use clipped NTP gradients on pretrained models, which differ from the local update objectives in In-Place TTT and TTT-NTP. The findings may not extend to other TTT mechanisms. The paper also overlooks Dynamic Evaluation [1] and the closely related gradient sign agreement methods AND-mask [2] and SAND-mask [3]. These connections deserve clearer discussion.

  2. The memory estimator may not fully isolate session-specific retention. The never-streamed control is evaluated using the final model state, while streamed sessions at different lags are evaluated at different points in the stream. Changes in adaptation over time could therefore bias (P(L))(P(L)). Also, first-mention recall only measures the first sub-token, leaving it unclear whether complete identifiers are remembered.

  3. The equal write budget comparison is not fully justified. Normalizing gate values to a mean of one does not guarantee equal Frobenius norms of the actual updates. Consistency and random gates may therefore differ in effective write magnitude despite having identical gate distributions. This makes it difficult to attribute the observed improvement solely to better write placement.

  4. The evidence for write-load-driven forgetting remains indirect. The learning rate sweep shows similar normalized recall curves, but does not isolate write load from session lag. It is also unclear whether the clipping threshold changes with the learning rate. Moreover, the theoretical signal-to-noise ratio is not quantitatively linked to the observed recall decay.

[1] Krause, Ben, et al. "Dynamic evaluation of neural sequence models." International Conference on Machine Learning. PMLR, 2018.

[2] Parascandolo, Giambattista, et al. "Learning explanations that are hard to vary." ICLR 2021

[3] Shahtalebi, Soroosh, et al. "Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization." arXiv preprint arXiv:2106.02266 (2021).

Questions for authors
  1. Unexplained benefit of persistence: At lag 0, persistence outperforms session reset by 2.5 points, while stream adaptation accounts for only 0.35 points. Does this gap increase for sessions appearing later in the stream, as the proposed warm-start explanation might suggest?

  2. Robustness across lags: How many sessions and first-mention targets contribute to each lag? Does the forgetting curve remain similar when the analysis is restricted to sessions that reach all four evaluation lags?

  3. Cost of session resets: Session reset restores the base model after each session, but does general capability degrade during a session before the reset? Could the authors report this using a small fixed evaluation subset?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer GQ2t

Scope and related methods. Agreed on both. The update is a clipped NTP gradient on a pretrained model; it borrows In-Place TTT's fast-weight location and chunk schedule but not its learned local objective, and the revision says so in §3.1. Dynamic evaluation, AND-mask and SAND-mask are now discussed in §2 (sign-agreement masking is the closest ancestor of the consistency gate, applied there across environments rather than across documents).

Estimator bias from evaluating the control at the final state. See the position analysis in my reply to sKGY (a): measurement positions are matched across lags (91/93/98/110 of 178), the balanced panel reproduces the decay, and any growth of adaptation along the stream biases against the decay rather than toward it. The one time-varying effect I do find is that lag-5 excess is smaller for sessions late in the stream (2.35 → 1.74 → 0.67 pt across terciles); it steepens the decay for late sessions rather than explaining the long-lag residual, and the revision reports it (§5.3, Fig. 2a). I agree a multi-checkpoint control is the right fix and will add it.

First sub-token only. Correct; I score the position predicting the first sub-token because it is the only position at which the identifier is not already visible. I will report full-identifier exact match in the next version; I expect it to be strictly lower and have not measured it. The revised limitations paragraph states this.

Equal write budget. Agreed; see reply to sKGY (b). The claim is now "equal mean gate", and the attribution to placement is withdrawn pending a norm-matched rerun.

Write-load evidence is indirect. Agreed on two of three points. The clipping threshold does scale with the learning rate (m = η throughout, App. C), so the sweep does not separate "larger steps" from "larger clip"; the revision states this (§5.5). The SNR of Prop. 1 is linked to recall only through monotonicity (App. A), and I make no quantitative claim. The sweep does not separate write load from lag, but the balanced panel and matched measurement positions above remove the alternative that lag is acting as a proxy for position.

Q1 (does the persist−session gap grow along the stream?). Yes. Splitting the paired lag-0 difference by stream position (3 seeds pooled, 124 held-outs per tercile): +1.62 [0.45, 2.84] in the first third, +2.18 [1.11, 3.34] in the middle, +3.76 [2.08, 5.82] in the last third. This is the pattern a warm-start explanation predicts; it is added to §5.7 and Fig. 2c with the caveat that it is consistent with, not a test of, that explanation.

Q2 (coverage per lag). Seed 0: 124 / 120 / 113 / 96 held-out documents and 2045 / 1957 / 1877 / 1553 first-mention targets at lags 0 / 5 / 15 / 40. Restricted to the 96 (288 over three seeds) that reach all four lags: 6.12 → 1.93 → 1.12 → 0.67 pt. Added to §4 and Fig. 2b.

Q3 (capability during a session, before reset). I did not measure it and could not within the discussion window. It can be bounded only loosely: within a session the write sequence equals persist's, so the exposure is at most what persist accumulates over one session rather than over all 178; I will report a fixed-subset evaluation at session midpoints in the revision. The revised limitations paragraph states that this was not measured.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

I thank the reviewers for their careful reading. Three criticisms are correct and I have fixed them in the revised PDF: (i) the "equal write budget" claim was stronger than the normalization supports; I now state it as an equal-mean-gate comparison and downgrade the placement claim accordingly (§3.3, §5.8, §6); (ii) §5.1 referred to a per-document-reset row that is not in Table 1—that experiment was run at 0.6B only, and the text now says so; (iii) the author list of In-Place TTT (Feng et al., 2026) was wrong and has been corrected, and every reference has been re-checked against arXiv/DBLP. I have also added the dynamic-evaluation line of work (Krause et al., 2018; Rannen-Triki et al., 2024; Tirumala et al., 2022) and the gradient-sign-agreement methods (AND-mask, SAND-mask) to §2, and uploaded the supplementary material (code, stream construction, unit tests, result logs), which was missing from the original submission through my error.

Two criticisms I believe are not supported by the data, and I add offline analyses of the submitted runs to show why: the decay of session-specific retention is not an artifact of changing session composition across lags or of adaptation changing over the stream (reply to sKGY, GQ2t). I also answer GQ2t's questions 1–2 with the submitted data; question 3 and a norm-matched rerun of the gates require new runs, which I commit to for the revised version but could not complete in the discussion window.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.