- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the temporal impact of test-time training (TTT) in data streaming scenarios (at inference time). They study a series of TTT policies that vary how frequently fast weights are updated and incorporated at inference time: persist, session-reset, consistency, random (Table 1). They find that the benefits of test time training decay pretty rapidly after the initial update to fast weights, and after several additional streamed documents, it is hard to disambiguate the performance gain from TTT from simple contextual knowledge from never-streamed sessions. They recommend using a “session-reset” TTT policy.
Critical strengths
- I liked how this paper attempts to disambiguate different factors that lead to performance gains in the streaming protocol during TTT. They account for base model knowledge by renaming cross-file identifiers for code and session-defined symbols for prose. They account for the test-time knowledge by scoring a “chunk” before it affects fast weights. They study the impact of fast-weights on sessions that haven't been streamed.
Critical weaknesses
-
The main takeaway from the conclusion (lines 403-405) don’t seem to be supported by the “Main results” Table 1. For example, the authors write "Resetting at session boundaries captures most of the benefit without the benchmark cost or privacy exposure”; but in table 1 for “PPL at lag” session-reset was the second worst method. If anything, across both the coding and prose tasks it seems the “random” baseline beats session-reset and many of the other policies. I am assuming higher recall is better and lower PPL is better.
-
I found this paper really challenging to understand. I had several instances while reading where I wasn’t sure what a term meant and when I re-read prior parts of the paper I wasn’t able to figure out the definition (I have enumerated some of those questions below). For example in the interpretation of Figure 4.a (lines 301-302) it is written: “the adaptation reshapes per-token likelihood in a way the normalized rule rewards without changing which answers are correct. We report both rules.” What are the “rules”?
Questions for authors
-
In table 1 is the “frozen” policy just the base model w/ no TTT? If so, how do you control for the computational cost of the non-frozen TTT policies vs the frozen case which incurs no TTT overhead?
-
How do you define a “write budget”? Is this the number of tokens that are streamed? Does it account for the excess compute that goes into training the fast weights?
-
In figure 1, were the “never streamed repositories” evaluated once? I am trying to figure out why the blue line doesn’t change across sessions. If so, were they evaluated after lag 40? Is the never streamed line (blue) reporting the “stream adaption” score (lines 192 - 196).
-
Clarification request: What is the “First mention recall” metric for code? I can see on line 131 that prose is evaluated with log likelihood which is a metric I am familiar with. On line 178-179 could you give an example of a prompt and the evaluation criteria to evaluate what it means for a model to recall cross-file identifiers?
-
What is the main takeaway from the raw vs. length normalized accuracies in Figure 4.a section 5.6? There seems to be no trend in the figure and therefore I am not sure what the actionable takeaways are. The written text description just summarizes what is in the figure and I did not glean additional interpretation or insight from the main text discussion of the figure.
6 ) I find the citation format a bit strange: I notice the publication venues are abbreviated (e.g., ‘ICLR’ instead of ‘International Conference on Learning Representations’). I have typically seen the fully publication venue spelled out in most reference sections, but perhaps I am unfamiliar with this format.
Reply to Reviewer BHHD
"Session-reset is second worst in PPL at lags 5–40, so the conclusion is unsupported." At lags ≥ 5, session-reset equals the frozen model by construction: its fast weights were zeroed at the session boundary, so there is nothing to probe (Remark 1). Those cells measure cross-session carry-over, of which the session-reset policy has none; they are not a ranking of quality. The conclusion "captures most of the benefit" refers to lag 0, where session reaches 4.6% recall and 1229 PPL against persist's 7.1% and 398 (60% of the recall excess over frozen). Random is not a competitor to session-reset: it is a control for the consistency gate (same gate values, permuted), and it carries the same capability cost as persist. The revised Table 1 caption states both points. You are right that higher recall and lower PPL are better.
Undefined terms. The revision adds a worked example in §4: a held-out file contains
from utils import qzrvmt; the model is scored at the position predicting the first sub-token ofqzrvmt, a name defined in an earlier file of the same repository and never seen in this file. "First-mention recall" is top-1 accuracy at such positions. "Scoring rules" are raw accuracy and length-normalized accuracy as implemented in lm-eval-harness (accvs.acc_norm); the point of the raw-vs-normalized panel (Fig. 4a in the submitted version, Fig. 5a in the revision) is that a policy can raiseacc_normon ARC-Easy by 3.7 pt while raw accuracy moves 0.1 pt, so reporting onlyacc_normwould misstate the capability effect—the actionable takeaway is to report both, and the revised §5.6 says so explicitly. "Write budget" meant the expected gate value, not tokens or compute; the revision uses "mean gate" to avoid the ambiguity.Frozen vs. TTT compute. Frozen is the base model with no TTT and incurs no TTT cost; TTT policies run at ≈0.4× its throughput (§5.7). I do not equalize compute, because the question is what persistence buys at its cost, and the cost is reported (§5.7 now says this explicitly).
Never-streamed line in Fig. 1. Evaluated once, with the fast-weight state at the end of the stream (after all 178 sessions), on 65 repositories that were never streamed; it is the stream-adaptation estimate of §3.2 and is constant by construction. The caption now says so.
Citation format. I use the ICLR style file; venue names are now spelled out.