acceptodds
Under review as a conference paper at ICLR 2027

BEYOND UPDATE COUNTS: A CONTROL MAP FOR EVALUATING TEST-TIME MODEL UPDATES

Abstract

Test-time updating lets a language model change persistent state after training—such as model weights, adapter parameters, or a memory module—before or while answering queries. Existing evaluations often compare systems using update count or frequency and performance measured immediately afterward. These summaries omit what is updated, where, when, and in what order. The same total can therefore hide an important difference: one system may answer after one update while another answers after two. We introduce the Control Map, an evaluation framework that records the committed updates preceding each scored answer and separately evaluates behavioral changes, immediate learning (acquisition), later recall (retention), preservation of unrelated behavior (locality), and performance in new settings. We prove when counts alone determine final state and show, under stated assumptions, how differences between pre-answer histories limit differences in average scores. Our experiments establish three lessons. First, choices hidden by update count matter. At one 760M-parameter checkpoint, 24 equal-count schedules span 19.7%–34.2% accuracy; updating the early and late layer groups more often than the middle groups performs best in this experiment. Second, apparent count effects can disappear after proper matching: in 1,600 pairs of runs assigned different nominal dose labels but executing the same updates in the same order, both runs end with the same model state and locality. Third, immediate learning does not guarantee a lasting or well-contained change. Across six selected CounterFact LoRA settings, new information is acquired in all six; in a separate ROME study, edits are retained at all 13 tested operating points but fail locality at all 13. The relative ordering of editors also changes across the tested combinations of models, benchmarks, and tests. Therefore, evaluations should record which updates precede each answer and assess these questions separately.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test Time Training, Control, Evaluations, Memory, Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cSL3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that counting updates is not enough to compare models that learn during use. It proposes Control Map to record what was updated, where, and in what order before each answer. Experiments show that these choices can change performance even when update counts are equal, and that learning new information immediately does not mean remembering it later or leaving unrelated knowledge unaffected.

Critical strengths
  1. The paper examines whether update counts are enough for a fair comparison. With the same number of updates, changing their placement produces accuracies from 19.7% to 34.2% in sec 4. This provides clear evidence that evaluations need to consider how updates are arranged.

The paper shows that learning new information immediately does not guarantee remembering it later or preserving other knowledge in sec 5. Evaluating these abilities separately gives a more complete picture of whether an update method works.

Critical weaknesses

It is unclear what Control Map adds beyond recording updates carefully and using existing evaluation metrics. The paper gives useful advice, but does not clearly show why a new framework is needed.

Too many key results are left in the appendix . The main text mostly describes retention, locality, and comparisons across settings in words, making it difficult to judge how strong the evidence is. A summary table with the main results and uncertainty estimates would help.

Several headline findings are limited to narrow settings. The 19.7%–34.2% accuracy range comes from one model checkpoint and one retrieval task in sec 4, while the LoRA findings use six selected settings in sec 5. These results are useful, but their broader relevance remains uncertain.

The theoretical guarantee assumes that the recorded history captures everything relevant to the score in sec 2, but the control map does not establish whether this assumption holds for a given system.

Questions for authors

Too many key results are left in the appendix. Figure 2 mainly shows the effect of update placement, while the main text describes retention, locality, and cross-setting results mostly in words. Bringing the key results into a main-text table would make the paper easier to follow and assess.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cSL3 (1/2)

Thank you for the careful review, and for recognizing the placement and feasibility results.

What the Control Map adds beyond careful logging and existing metrics. Logging records what happened. The Control Map adds four steps that logging alone does not, and each one changed a conclusion in our studies:

  1. Checking, before any outcome is seen, whether each comparison can show an effect and which effects the design can estimate. (on PIQA and WinoGrande, 12 equal-count schedule pairs per benchmark make identical updates, and only 6 of the 9 directions in the layer-group × frequency model can be estimated)
  2. Matching runs on their complete update histories instead of that on labels. (removed an apparent dose curve in final locality)
  3. Separate pass/fail/inconclusive checks for acquisition, retention, and locality, with thresholds set in advance, so a strong acquisition result cannot make up for weak retention or locality. (LoRA passes acquisition in 6/6 settings and retention in 0/6)
  4. Evaluating each new setting directly instead of carrying a ranking over. (the GPT-2XL locality ranking flips on 2 of 3 endpoints, and the best schedule on Single-NIAH is significantly worse than both controls on BoolQ (−2.69 pp [−3.95, −1.34] and −2.42 [−3.55, −1.25]).)

This matters for test-time training. In our MAC model, the update often never happens before the answer on short-prompt benchmarks (57.4% of PIQA examples), so those scores cannot be credited to TTT. The best schedule depends on the task, so a schedule tuned on one benchmark should not simply be reused on another. And a persistent update can learn right away and still fail retention or damage unrelated behavior (LoRA learns in 6/6 settings and retains in 0/6). Longer term, we hope placement, timing, and strength become reported, tunable settings of TTT, as temperature is for decoding. The revision also includes the side-by-side table from our reply to 881L.

Main-text summary table with uncertainty. Added as Table 2. shortened here for easy to read (95% intervals):

ResultEstimate
Placement, Single-NIAH (paired rows)best − worst 14.50 pp [13.60, 15.40]
Frozen step-17,827 prediction vs. surfaces at steps 21K / 23Kρ = 0.81 / 0.84; source winner beats both controls, all Bonferroni intervals > 0
BoolQ–LAMBADA reversal of a fixed schedule pair (3 more independently trained models)+7.80 pp [6.21, 9.40]
Timing, GPT-2 S/M/L/XL (4 LoRA inits)+0.12 [0.10, 0.14], +0.06 [0.05, 0.07], +0.43 [0.29, 0.57], +0.17 [0.14, 0.20] nats
Timing, Pythia-410M+0.16 [−0.45, +0.77]
LoRA, 6 settings / 5 models (CounterFact families)acquisition 6/6; retention 0/6 (e.g., −10.95 nats [−13.04, −8.85]); locality −31 to −53 pp (threshold −5)
MEMIT-high vs. ROME-low (160 families)damage −2.56 pp [−4.27, −1.06]; MEMIT passes, ROME inconclusive
GPT-2XL locality contrast, 3 endpoints+2.56 [1.06, 4.27], −7.50 [−11.02, −4.22], −15.92 [−21.00, −10.94]
Llama-3.2-3B, same 3 endpoints+21.22 [17.84, 24.61], +4.45 [1.17, 7.81], 0.00 [−3.81, 3.91]
WikiRecent (64 families, prospective)gain +0.21 [0.11, 0.34] nats, but locality −43.4 [−52.7, −34.0] → fail
AuthorsOpenReview ↗

Response to Reviewer cSL3 (2/2)

(Continued) In response to:

Narrow settings.

  • Placement: there are now six complete equal-count surfaces on four tasks; a prediction frozen from the step-17,827 surface and checked against the surfaces at steps 21,000 and 23,000; and a reversal between one fixed pair of schedules, preregistered and confirmed in 3/3 additional independently trained models and at step 50,000 (see the general response). We do not claim that the full 24-schedule ranking carries over between tasks. Its rank correlation with Single-NIAH is −0.35 on BoolQ and +0.47 on LAMBADA, and this lack of transfer is exactly our point.
  • LoRA: the six settings cover five models from four families (Pythia-410M at two checkpoints, GPT-2 Large, GPT-2XL, Qwen2.5-0.5B, Llama-3.2-1B), each with 32 CounterFact families and four LoRA initializations, under protocols fixed before we saw any outcomes. The failures are not borderline. In the four settings where retention is measured after later writes, it is far below zero (e.g., −10.95 nats [−13.04, −8.85]); in the other two, too few edits install in the first place. Locality changes by −31 to −53 pp, against a −5 pp threshold.

The guarantee assumes the recorded history captures everything relevant. Every evaluation assumes something like this implicitly: that the logged treatment is the one that actually ran. The Control Map states the assumption and tests it, much like an A/A test in online experiments:

  1. Conformance. Recorded executions are checked against the declared trace. In a live test on a neural model, all 16 injected trace corruptions are caught and the restored trace passes.
  2. History-identical pairs. If the recorded history missed something that affects the score, runs with identical recorded histories could differ. In 1,600 of 1,600 cross-dose pairs, the final parameters and locality are identical, and the six SocialIQA swaps give identical logits on all 1,953 examples where their histories match.
  3. Sensitivity. If something is missed and its reach can be bounded, Proposition 1 bounds the error by the share of examples whose histories differ plus the share affected by the missing path. A hidden-cache example shows this is tight: leaving out a cache read on k of N examples makes the exact bound fail by exactly k/N.

Like an A/A test, these checks catch violations. For the internals of third-party editors, the guarantee is stated as conditional on the recorded updates.

Key results in the appendix (your question). Retention, locality, and cross-setting results are now in the two main-text tables above. The placement breadth results are in a new main-text paragraph (Sec. 4.1), with the full table in Appendix B.7 (Table 12).

Thank you again. We hope the revision makes the evidence easier to judge, and we would appreciate it if you would reconsider your score.

I think this is very hard to follow the author's rebutall. e.g., "A/A test" "−2.69 pp" and this table "Main-text summary table with uncertainty" hard to follow.

AuthorsOpenReview ↗

Response to Reviewer cSL3 (we make it easier to follow)

Thanks for following up, and sorry, we packed too much into our first reply. Let us try again more simply.

  1. What the Control Map adds beyond logging. A log tells you what happened. The Control Map also tells you, before you run anything, whether a comparison can show a difference at all. For example, in our MAC model more than half of the PIQA examples get no update before the answer, so lots of schedule comparisons on PIQA can't show a meaningful difference no matter what. Plain logging wouldn't warn you about that.

  2. Key results in the main text. We added two tables on page 10. Table 2 lists the main results with their uncertainty ranges. Table 3 takes seven of our studies and shows, side by side, what the usual evaluation would conclude vs. what the Control Map concludes on the same data. The conclusion changes in every row.

  3. Narrow settings. The placement result now covers four tasks and four independently trained models (Appendix B.7). The main takeaway is that the best schedule depends on the task: the schedule that's best on BoolQ is the worst of all 24 on LAMBADA, where a different schedule wins. That reversal between the two held on all three extra models we trained.

  4. Does the recorded history capture everything? We check it directly. We compare runs that record exactly the same updates; if something unrecorded mattered, those runs would come out different. Across all 1,600 such pairs, the final model parameters and locality scores were identical. That can't prove nothing is ever missing in every system, but it's a direct check that would catch this kind of problem.

On the two things you pointed out: - "A/A test" just means the check in point 4: run the same treatment twice and confirm nothing hidden changes the result. - "−2.69 pp": the schedule that's best on the retrieval task (Single-NIAH) is 2.69 percentage points worse on BoolQ than a simple reference schedule that updates the earliest layers most often. The 95% range, [−3.95, −1.34], doesn't include zero, so the drop is unlikely to be noise. In short, a winner on one task can lose on another. Hope this is clearer. Happy to explain anything else.

Thank you again. We hope these makes the evidence easier to judge, and we would appreciate it if you would reconsider your score.

Reviewer_881LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an evaluation framework called the "Control Map." This framework records the update operations performed prior to each scored response and evaluates the following aspects: behavioral changes, immediate learning (acquisition), later recall (retention), preservation of unrelated behavior (locality), and performance in new settings. The author also prove the conditions under which the final state can be determined solely by counting, and points out that, under certain assumptions, differences in the reaction process constrain the gap between average scores.

Critical strengths
  1. The research question addressed in this paper has a certain degree of practical significance.
  2. The paper proposes a relatively systematic evaluation framework.
Critical weaknesses
  1. This paper appears to merely consolidate various evaluation concepts, lacking a direct comparison of evaluation processes. Furthermore, the experimental section in the main text comes across as quite rudimentary. The presentation could be improved through some revisions. Directly demonstrating the advantages of the new framework is essential.
  2. The core experiments in the paper are overly simplistic, featuring only a very limited range of models and task types.
  3. The experimental results in the main text are presented only as aggregated figures scattered throughout various paragraphs, lacking detailed and intuitive comparisons.
  4. The key theories in the paper lack sufficient information, and the upper limit of many theories is the potential maximum gap in scoring. Further analysis is needed to determine the necessity and importance of these theories.
Questions for authors

Please see the critical weeknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 881L (1/2)

Thank you for the review, and for noting the practical significance of the question and the systematic framework.

W1/W3 (no direct comparison; results scattered in the text). We added a main-text table (Table 3) that evaluates the same data both ways, with the usual summary next to the Control Map:

StudyUsual summary → conclusionControl Map → conclusion
Placement, MAC-760MEqual update counts → budget-matched; report the best schedule14.5 pp spread [13.6, 15.4]; the best schedule depends on the task (BoolQ's best is LAMBADA's worst; its reversal against LAMBADA's best replicates in 3 independently trained models, +7.80 pp [6.21, 9.40])
Schedule sweep on PIQA / WinoGrandeEqual counts → valid comparisons; a null suggests placement doesn't matter12 pairs per benchmark make identical updates; Theorem 1 caps 48 more at 0.33 / 0.87 pp; only 6 of the 9 directions in the layer-group × frequency model can be estimated
Dose–locality, editor gridsLocality changes with dose → dose effectMatched on full update order, 1,600/1,600 pairs have identical final parameters and locality → the dose curve in final locality was an ordering artifact
LoRA, 6 settings (5 models)Acquisition passes 6/6 → the edit worksRetention 0/6, locality 0/6 → not usable
ROME, 13 schedule cellsAcquisition and retention pass 13/13 → reliableLocality passes 0/13 (9 not certified, 4 inconclusive) → not shown usable
GPT-2XL, MEMIT vs. ROMEMEMIT has better locality on CounterFact (+2.56 pp [1.06, 4.27])Evaluated directly, the ranking flips: −7.50 [−11.02, −4.22] (zsRE), −15.92 [−21.00, −10.94] (WikiData-CF)
WikiRecent (prospective)Mean gain MEMIT > ROME (+0.21 nats [0.11, 0.34]) → prefer MEMITBoth fail locality (−43.4 / −41.4 pp) → neither usable

A second new main-text table (Table 2) lists every headline estimate with its 95% interval, inference unit, and scope.

W2 (few models and task types). The study is broader than the main text made clear. It covers nine model configurations from five families (MAC-760M with four independently trained models; GPT-2 S/M/L/XL; Pythia-410M at two checkpoints; Qwen2.5-0.5B; Llama-3.2-1B and 3B), six update methods (MAC memory writes, LoRA/SGD, ROME, MEMIT, AlphaEdit, FT-L), and 14 evaluation sets (Single-NIAH, 8 LM/commonsense benchmarks, CounterFact, zsRE, WikiData-CounterFact, WikiRecent, and a constructed association set). The revision collects these in one table (Table 9). Placement is studied in MAC-760M, where per-layer write schedules can be set exactly. There, the evidence covers four tasks and four independently trained models: six full 24-schedule surfaces, frozen predictions checked at two later checkpoints, and the BoolQ–LAMBADA reversal retested in three more models and at a later checkpoint.

AuthorsOpenReview ↗

Response to Reviewer 881L (2/2)

W4 (the theory needs more information; the bound is often the maximum gap). The bound is a check of whether a comparison can show an effect at all (not an estimate of the effect). It reaches the maximum gap only when the two plans' pre-answer updates differ on every example; in that case it simply places no restriction. When histories partly overlap, the bound is tighter, and that happens often. Over all 2,208 equal-count comparisons in our atlas:

Share of examples whose updates differComparisonsLargest accuracy difference allowed by Theorem 1
024 (1.1%)none (identical treatments)
< 0.01132 (6.0%)< 1 pp
< 0.1156 (7.1%)< 10 pp
< 0.5528 (23.9%)< 50 pp

The reason is simple: on short prompts many examples get few or no updates before the answer (on PIQA, 57.4% get none).

This matters in practice in three ways:

  1. Before a sweep, Corollary 1 gives the smallest share of differing examples needed to claim an effect of a given size, so comparisons that cannot show it can be dropped or moved to longer prompts.
  2. After a sweep, a null result may simply mean the two treatments were nearly identical when few examples differ; when nearly all differ, that explanation is ruled out, and the usual uncertainty analysis decides the rest.
  3. For the design, the estimability result shows which effects repeating the same design can never separate: on PIQA and WinoGrande, only 6 of the 9 directions in the layer-group × frequency model can be estimated.

The observed results agree with the bound. On SocialIQA, the six evaluated swaps have identical accuracy, and their logits match on every example where the histories match.

In the revision, each formal result is first stated in one plain sentence, and Table 11 (Appendix B.1) reports this share for every benchmark and pair type.

We hope the side-by-side table and the new results make the value of the framework clearer. We would appreciate it if you would reconsider your score, and we are happy to answer follow-up questions.

Thank you for the author's response. I carefully reviewed the discussion of other reviewers and the author's reply. The author has addressed some of my concerns. The organization of the paper still seems to need further improvement, and the authors should present their results more clearly. I have decided to maintain my score.

AuthorsOpenReview ↗

Follow-up: Ask of the remaining concerns

Thank you for reading the discussion and our revision, and we are glad most of your concerns has been addressed. Could you tell us which concern you feel is still not addressed, and what in the organization gets in the way most? For example: the order of the sections, which results should be in the main text rather than the appendix, or the any notations? Even a short pointer would help us a lot. Thank you again for your time.

Reviewer_ztJZOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary
  • Test-time model updates are often compared by update count.
  • The paper proposes the Control Map, an evaluation framework.
    • Record which updates precede each answer.
    • Evaluate acquisition, retention, locality separately.
  • Theory: same histories give same scores.
  • Experiments: equal-count schedules span 19.7--34.2% on one checkpoint. LoRA and ROME fail retention or locality. Editor rankings change across settings.
Critical strengths
  • An potentially important point is made explicit: what happened before each answer, not the total count, determines what is compared. The checklist could be useful across TTT, memory and editing work.
Critical weaknesses
  • No evidence that the problem is real.
    • Important but trivial point.
    • No existing work shown that drew a wrong conclusion from update counts.
    • Separate acquisition/retention/locality is already standard in model editing.
    • The 1,600 pairs are a tautology. Same updates in same order give same state.
    • Feels like forcing a problem on what is already done well.
  • Trivial results under heavy formalism.
    • Theorem 1: same histories give same scores. Authors call it "classical" (L237).
    • Filtration, action kernel, contrast laws are never used in experiments.
    • Many uncommon terms without precise definition. Sec. 2 is hard to follow.
Questions for authors
  • Can you point to published work that reached a wrong conclusion by comparing update counts only?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ztJZ (1/2)

Thank you for the review, and for noting the strength: "what happened before each answer, not the total count, determines what is compared." We respectfully disagree that the point is trivial or already handled, and explain why below.

1. The problem is real and not yet handled in practice

Evidence from prior work. Layer location, update timing, strength, and frequency are all being studied as design choices in test-time training (Lee et al., 2023; Tandon et al., 2025; Wakayama, 2026), and so is the number of edits applied together (Meng et al., 2023; Yoon et al., 2024). Each of these studies looks at one or two factors and reports them as a count or a frequency. We are not saying these studies are wrong. Our point is that a summary along one axis can miss differences that matter, and the literature already has examples:

  • Gupta et al. (2024b) followed ROME and MEMIT over thousands of sequential edits and concluded that ROME's "disabling edits" are "a fundamental limitation of ROME." The same authors later found (2024a) that these edits are "an artifact of irregularities in the implementation." This only came out when they looked at the update that was actually executed; neither the method name nor the edit count could show it.
  • Yoon et al. (2024) report that larger edit batches hurt Llama-3 more than smaller sequential batches "for equal number of edits," so "N edits" does not pin down the treatment.
  • Test-time memory models are usually compared with one accuracy per benchmark on short-prompt suites (e.g., Titans on PIQA, HellaSwag, WinoGrande, ARC, SIQA, and BoolQ). In our MAC reconstruction, which writes memory in 16-token chunks, 57.4% of PIQA examples and 28.1% of WinoGrande examples get no complete write before the answer.

How large the effects are. With exactly the same number of updates:

  • schedules differ by 14.5 pp [13.6, 15.4] on Single-NIAH;
  • the best of 24 schedules on BoolQ is the worst on LAMBADA, a reversal that holds in a preregistered test on three independently trained models (+7.80 pp [6.21, 9.40]) and in a test fixed in advance at a later checkpoint (+9.35 pp [7.88, 10.82]);
  • of 2,208 equal-count comparisons on standard benchmarks, 24 make identical updates on every example and 132 cannot differ by more than 1 pp.

Acquisition, retention, and locality. The metric names do come from model editing; what differs is when they are measured. Editing benchmarks score edits at one fixed point, without linking each score to the updates before it, and test-time memory papers usually report one score per model. Linking each metric to its pre-answer history changes the verdict. ROME passes acquisition and retention at all 13 operating points but locality at none. LoRA passes acquisition in all six settings (five models, four families) and retention in none. A check of immediate success alone would call both methods effective. As for the 1600 pairs, these would be a tautology only if we already knew the recorded history was complete. In practice, the same list of updates does not always give the same state: optimizer state, random-number streams, caching, and nondeterministic kernels can all leak in.

What the paper adds. To our knowledge, this is the first formal framework for test-time updates that ties every score to the updates that preceded it, checks before any outcome whether a comparison can show an effect, and tests acquisition, retention, locality, and transfer separately. Longer term, we want the choices that counts hide (where, when, how strongly, and in what order a model updates) to become reported, tunable settings of test-time training, the way temperature is for decoding. That needs a standard record of updates, which the Control Map provides.

AuthorsOpenReview ↗

Response to Reviewer ztJZ (2/2)

2. On "trivial results under heavy formalism"

Why the formalism is there. Part 1 shows that update counts hide differences that matter. Replacing counts with a standard record means answering questions that are easy to leave vague in prose: what belongs to an answer's history (optimizer state, caches, random streams?), when two evaluations are actually comparable, and what an update policy may depend on. The definitions in Section 2 pin these down so they can be checked.

Theorem 1. Equal histories are only the special case where the histories differ on no example, so the bound is zero. In general, the theorem bounds the difference in average scores by the score range times the fraction of examples whose pre-answer histories differ. As we say at L237 of the submitted version (L240 in the revision), the inequality itself is classical. What is new is what it is applied to, and that it can be computed from the inputs alone, before running any model. We use it this way in the experiments:

  • on SocialIQA it caps six schedule swaps at 0.0512 pp, and the observed difference is exactly zero;
  • across 2,208 equal-count comparisons on eight benchmarks, it finds 24 identical treatments and 132 comparisons that cannot differ by more than 1 pp;
  • its assumption is tested directly: 1,600 of 1,600 history-matched pairs are identical, and a live check catches 16 of 16 injected trace errors.

A second result gives the exact condition under which counts are enough: the updates must commute. Our editor grids show that real updates need not, since mixing update orders alone produced an apparent dose curve.

Filtration, action kernel, and contrast laws. All three are used.

  • The filtration says what an update policy may depend on: only information available before the update. Every schedule we test satisfies it, and the trace atlas is built without reading any model output (184 checks).
  • The action kernel is the update decision itself: what to write, which layers, which method, how strongly, how often, and when relative to the query. The experiments vary these parts: layers and frequency (the 24 placement schedules), timing, method (LoRA, ROME, MEMIT, AlphaEdit), strength, and dose.
  • Contrast laws define comparisons (Sec. 2.2, Definition 2). A contrast weights plans so the weights sum to zero (A vs. B is +1/−1), and its two laws are the update-history distributions on each side; Theorem 1 bounds the score gap by how far apart they are. From the design alone, this tells us that on PIQA and WinoGrande no example gets four or more updates before the answer, so 3 of the 9 layer-group × frequency effects (every 4 vs. every 8 steps) cannot be estimated with any amount of data.

The general, stochastic form also covers adaptive and randomized policies, such as learned update controllers; our experiments use the deterministic case. Appendix A adds a table showing where each formal object is used.

Terms and readability. Each definition in Section 2 now comes with a plain-language sentence, and Appendix A adds a glossary. If any term is still unclear, please tell us which one and we will define it more precisely.

We hope this, together with Part 1, addresses your concerns, and we would appreciate it if you would reconsider your score.

Thank you for the response, which clarifies Theorem 1 and the 1,600 matched pairs, and for the new tables. However, no published conclusion is shown to be wrong because update counts were compared: the Gupta et al. study had already ruled out edit accumulation and its error was a code bug, and the "usual summary" conclusions in Table 3 are not attributed to published work. The revision itself concludes that placement effects are "real but task-specific", which is what I expect of any hyperparameter.

AuthorsOpenReview ↗

Follow-up response to Reviewer ztJZ

Thank you for following up. We agree with you. In the final version we will say both plainly and present the contribution as a check.

Where we'd ask you to reconsider is the hyperparameter point. With an ordinary hyperparameter, two settings are two different treatments, and the question is how much the result depends on the setting. What we show is different. On short-prompt benchmarks often used to compare test-time memory models, many schedules with different labels and the same update count are not different treatments at all. On PIQA and WinoGrande, 24 schedule pairs make identical updates on every example, and across eight benchmarks 132 of 2,208 comparisons are capped below 1 pp before any model is run. A sweep reported by update count would treat all of these as valid comparisons, and its null results would look like evidence. That isn't task-specific tuning; it's whether the comparison exists at all, and the Control Map checks it before anything is run. We think this result is new, and we hope it changes your view of what the paper adds.

Thanks again for the careful discussion. We would appreciate it if you would reconsider your score. Thanks.

Comments on the paper

1 comment

We thank the reviewers for their time and comments. In brief, the Control Map is, to our knowledge, the first evaluation framework for test-time updates that ties each score to the updates that came before it, checks in advance whether a comparison can show an effect, and tests acquisition, retention, locality, and transfer as separate requirements. The reviews mainly asked for four things: evidence that the problem is real, placement results beyond one checkpoint and one task, key results in the main text with uncertainty, and an easier Section 2. We summarize the changes here; new and changed text in the revised PDF is in blue.

Placement beyond one checkpoint and one task. We had run full 24-schedule surfaces on more tasks and checkpoints, with the same frozen MAC-760M harness as Section 4 and identical per-example update counts. They were cut when we reframed the paper around the evaluation framework, and only Single-NIAH stayed in. The new Appendix B.7 reports all of them. Equal-count schedules differ on all four tasks (Single-NIAH, BoolQ, LAMBADA, HellaSwag), and a surface fixed in advance predicts the surfaces at two later checkpoints (Spearman ρ = 0.81 and 0.84). Rankings, however, do not carry over between tasks. The best of the 24 schedules on BoolQ is the worst on LAMBADA. This reversal holds in a preregistered test on three independently trained models (+7.80 pp [6.21, 9.40]) and in a test fixed in advance at a later checkpoint (+9.35 pp [7.88, 10.82]).

Comparisons that cannot show an effect. Using only prompt lengths and the schedules, we checked all 2,208 equal-count comparisons in our atlas (8 benchmarks × 276 schedule pairs). In 24 of them the two schedules make exactly the same updates on every example, and in 132 the accuracy difference cannot exceed 1 pp. On PIQA, 57.4% of examples get no complete update before the answer. Update counts show none of this.

Side-by-side comparison. A new main-text table (Table 3) puts the usual summary (update count, immediate score, source ranking, or mean gain) next to the Control Map on the same data. see our reply to 881L.

Presentation and theory. Each definition in Section 2 now comes with a one-sentence plain explanation, and Appendix A adds a glossary and a table showing where each formal object is used. As the paper says, the inequality behind Theorem 1 is classical; what we add is a way to compute it before any outcome is seen. Table 2 (main text) lists every headline estimate with its interval and inference unit. Table 9 lists the nine model configurations (five families), six update methods, and 14 evaluation sets.

Replies to each review follow, and we are happy to answer follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.