- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that counting updates is not enough to compare models that learn during use. It proposes Control Map to record what was updated, where, and in what order before each answer. Experiments show that these choices can change performance even when update counts are equal, and that learning new information immediately does not mean remembering it later or leaving unrelated knowledge unaffected.
Critical strengths
- The paper examines whether update counts are enough for a fair comparison. With the same number of updates, changing their placement produces accuracies from 19.7% to 34.2% in sec 4. This provides clear evidence that evaluations need to consider how updates are arranged.
The paper shows that learning new information immediately does not guarantee remembering it later or preserving other knowledge in sec 5. Evaluating these abilities separately gives a more complete picture of whether an update method works.
Critical weaknesses
It is unclear what Control Map adds beyond recording updates carefully and using existing evaluation metrics. The paper gives useful advice, but does not clearly show why a new framework is needed.
Too many key results are left in the appendix . The main text mostly describes retention, locality, and comparisons across settings in words, making it difficult to judge how strong the evidence is. A summary table with the main results and uncertainty estimates would help.
Several headline findings are limited to narrow settings. The 19.7%–34.2% accuracy range comes from one model checkpoint and one retrieval task in sec 4, while the LoRA findings use six selected settings in sec 5. These results are useful, but their broader relevance remains uncertain.
The theoretical guarantee assumes that the recorded history captures everything relevant to the score in sec 2, but the control map does not establish whether this assumption holds for a given system.
Questions for authors
Too many key results are left in the appendix. Figure 2 mainly shows the effect of update placement, while the main text describes retention, locality, and cross-setting results mostly in words. Bringing the key results into a main-text table would make the paper easier to follow and assess.
4 comments
Response to Reviewer cSL3 (2/2)
(Continued) In response to:
Narrow settings.
- Placement: there are now six complete equal-count surfaces on four tasks; a prediction frozen from the step-17,827 surface and checked against the surfaces at steps 21,000 and 23,000; and a reversal between one fixed pair of schedules, preregistered and confirmed in 3/3 additional independently trained models and at step 50,000 (see the general response). We do not claim that the full 24-schedule ranking carries over between tasks. Its rank correlation with Single-NIAH is −0.35 on BoolQ and +0.47 on LAMBADA, and this lack of transfer is exactly our point.
- LoRA: the six settings cover five models from four families (Pythia-410M at two checkpoints, GPT-2 Large, GPT-2XL, Qwen2.5-0.5B, Llama-3.2-1B), each with 32 CounterFact families and four LoRA initializations, under protocols fixed before we saw any outcomes. The failures are not borderline. In the four settings where retention is measured after later writes, it is far below zero (e.g., −10.95 nats [−13.04, −8.85]); in the other two, too few edits install in the first place. Locality changes by −31 to −53 pp, against a −5 pp threshold.
The guarantee assumes the recorded history captures everything relevant. Every evaluation assumes something like this implicitly: that the logged treatment is the one that actually ran. The Control Map states the assumption and tests it, much like an A/A test in online experiments:
- Conformance. Recorded executions are checked against the declared trace. In a live test on a neural model, all 16 injected trace corruptions are caught and the restored trace passes.
- History-identical pairs. If the recorded history missed something that affects the score, runs with identical recorded histories could differ. In 1,600 of 1,600 cross-dose pairs, the final parameters and locality are identical, and the six SocialIQA swaps give identical logits on all 1,953 examples where their histories match.
- Sensitivity. If something is missed and its reach can be bounded, Proposition 1 bounds the error by the share of examples whose histories differ plus the share affected by the missing path. A hidden-cache example shows this is tight: leaving out a cache read on k of N examples makes the exact bound fail by exactly k/N.
Like an A/A test, these checks catch violations. For the internals of third-party editors, the guarantee is stated as conditional on the recorded updates.
Key results in the appendix (your question). Retention, locality, and cross-setting results are now in the two main-text tables above. The placement breadth results are in a new main-text paragraph (Sec. 4.1), with the full table in Appendix B.7 (Table 12).
Thank you again. We hope the revision makes the evidence easier to judge, and we would appreciate it if you would reconsider your score.
I think this is very hard to follow the author's rebutall. e.g., "A/A test" "−2.69 pp" and this table "Main-text summary table with uncertainty" hard to follow.
Response to Reviewer cSL3 (we make it easier to follow)
Thanks for following up, and sorry, we packed too much into our first reply. Let us try again more simply.
-
What the Control Map adds beyond logging. A log tells you what happened. The Control Map also tells you, before you run anything, whether a comparison can show a difference at all. For example, in our MAC model more than half of the PIQA examples get no update before the answer, so lots of schedule comparisons on PIQA can't show a meaningful difference no matter what. Plain logging wouldn't warn you about that.
-
Key results in the main text. We added two tables on page 10. Table 2 lists the main results with their uncertainty ranges. Table 3 takes seven of our studies and shows, side by side, what the usual evaluation would conclude vs. what the Control Map concludes on the same data. The conclusion changes in every row.
-
Narrow settings. The placement result now covers four tasks and four independently trained models (Appendix B.7). The main takeaway is that the best schedule depends on the task: the schedule that's best on BoolQ is the worst of all 24 on LAMBADA, where a different schedule wins. That reversal between the two held on all three extra models we trained.
-
Does the recorded history capture everything? We check it directly. We compare runs that record exactly the same updates; if something unrecorded mattered, those runs would come out different. Across all 1,600 such pairs, the final model parameters and locality scores were identical. That can't prove nothing is ever missing in every system, but it's a direct check that would catch this kind of problem.
On the two things you pointed out: - "A/A test" just means the check in point 4: run the same treatment twice and confirm nothing hidden changes the result. - "−2.69 pp": the schedule that's best on the retrieval task (Single-NIAH) is 2.69 percentage points worse on BoolQ than a simple reference schedule that updates the earliest layers most often. The 95% range, [−3.95, −1.34], doesn't include zero, so the drop is unlikely to be noise. In short, a winner on one task can lose on another. Hope this is clearer. Happy to explain anything else.
Thank you again. We hope these makes the evidence easier to judge, and we would appreciate it if you would reconsider your score.
Response to Reviewer cSL3 (1/2)
Thank you for the careful review, and for recognizing the placement and feasibility results.
What the Control Map adds beyond careful logging and existing metrics. Logging records what happened. The Control Map adds four steps that logging alone does not, and each one changed a conclusion in our studies:
This matters for test-time training. In our MAC model, the update often never happens before the answer on short-prompt benchmarks (57.4% of PIQA examples), so those scores cannot be credited to TTT. The best schedule depends on the task, so a schedule tuned on one benchmark should not simply be reused on another. And a persistent update can learn right away and still fail retention or damage unrelated behavior (LoRA learns in 6/6 settings and retains in 0/6). Longer term, we hope placement, timing, and strength become reported, tunable settings of TTT, as temperature is for decoding. The revision also includes the side-by-side table from our reply to 881L.
Main-text summary table with uncertainty. Added as Table 2. shortened here for easy to read (95% intervals):