acceptodds
Under review as a conference paper at ICLR 2027

Correct Numbers, Wrong Constructs: Acquisition-Aware Evaluation Across Model Scales

Abstract

We built a program to learn, at a small scale, which adaptations make a model forget, and to test whether that knowledge survives an order of magnitude of scale. It returned a striking headline: roughly 98% of the relationships it selected failed to transfer. This paper is the audit that stopped us reporting it, and what the audit found was not one mistake but five instances of one. Each has the same shape: a quantity was computed correctly, and a different quantity was inferred from it. A count of records read as a count of independent observations, where 186 "pairs" were 62 directed pairs recorded three times, byte-identical across all 558 matched groups. A failure rate read as evidence that selection failed, at 98.7% against 98.8% for choosing at random on the same population. Deterioration read as forgetting, after an update that trained repeatedly on one 32-sequence batch and left held-out loss worse in 60 checks across three scales. A shared recipe's collapse read as a property of scale, when a per-scale recipe acquires in every confirmation cell and a width-scaled learning rate alone moves 1.4B from +0.043 to +0.239. And a preregistered median read as agreement between task families, where with three families the median is the middle family's value, certifying a match that held at a 0.0019 nat spread in one family while another disagreed by 0.5079. None required an unusual mistake, and preregistration caught none of them because one of them is the preregistration. A sixth arrived while we built the corrected experiment, this one caught prospectively: the per-scale recipe our own median rule certified for 160M acquires in just 32 of 62 pairs at that scale. We give an acquisition-aware protocol and a checklist with one item per substitution. Across 18 published papers a screen finds an explicit held-out new-task metric in 11 and does not resolve the other 7, so the defect is not universal and the screen does not establish how far it reaches. Asked at last under a preregistration written before any fit existed, the original question does not resolve: our candidate reaches a held-out R^2 of -0.004 and fails its margin against the best baseline, which is the one comparator with a positive point estimate, +0.023. On 15 held-out pairs we cannot separate absence of signal from absence of power, and we report both.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, catastrophic forgetting, evaluation methodology, acquisition-aware evaluation, scaling, negative results, reproducibility, adaptation, muP, research practice

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_fjAHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is meant to be a case study on avoiding common experimental mistakes in research. It is an audit of the authors’ investigations on whether small-scale measurements can predict which model adaptations will cause forgetting at larger scales. The original analysis appeared to show that roughly 98% of selected relationships failed to transfer with scale. They argue that this arose from several “inferential substitutions”: quantities were computed correctly but interpreted as measuring different scientific constructs than they actually did. The identified issues include treating duplicated records as independent, interpreting a high failure rate without comparison to the corresponding base rate, evaluating relative improvements when absolute predictive performance was poor, measuring “forgetting” even when the model had not successfully acquired the new task, transferring an adaptation recipe across model scales where it was no longer appropriate, and using a preregistered median across three heterogeneous task families as evidence of cross-scale matching. The authors then redesign parts of the experiment to explicitly verify held-out acquisition of the new task, select adaptation recipes separately by scale, isolate validation from confirmation data, and evaluate predictors on distinct task/data units. Under this corrected protocol, the original scientific question remains unresolved.

Critical strengths
  1. It's very transparent and concrete. I enjoyed the process of reading this. The step-by-step isolation of failure modes of their original study is interesting to follow

  2. These kinds of errors are not altogether uncommon and they turn the case study into a reusable protocol to avoid these errors. They mention useful steps like comparing against base rates, checking your units, etc.

Critical weaknesses
  1. The contribution is interesting to read but I'm not convinced it's substantive enough to be a main-conference submission. All of the individual lessons/ errors are not novel. In a sense, they're just indicative of poor research hygiene. I understand that the premise is that they're easy mistakes to make, but that doesn't make them a meaningful scientific contribution. The literature screening doesn't really cover this gap. It covers 18 papers and isn't done in a particularly scientific/ data-driven way. This could potentially make for a good position paper with a reframing and a broader literature analysis. There exist papers which point out poor research hygiene in a more substantive way. Consider the paper "Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision," which does a thorough study of how many papers' headline results are likely just consequences of noise. This is to say that there is an avenue and a need for this kind of work, but it needs to be done scientifically and speak to the prevalence of such issues in the broader research community.

  2. The original research question is basically unanswered.

Questions for authors
  1. Do you have quantitative evidence for these kinds of mistakes being a broader pattern in the research community as opposed to failings of your research design?

  2. What is the precise methodological novelty relative to the existing evaluation literature you cite?

Reviewer_j5ibOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

My summary could be wrong, as I did not understand the AI generated paper. It is very specific too early, sentences do not connect well together -- I actually could not tell what exactly their experimental setup was, what does the results mean. That being said, this is what I understood:

The paper studies whether the cheap signals at a small model scale transfer to a larger model. The task is measuring forgetting from finetuning -- how much fine-tuning on a new task B makes the model forget an old task A. They varied aspects of the data in task B from task A, and measure how it changes across model scales. The main claim is the data filtering/association results almost always do not transfer across scale, i.e. how data selection creates forgetting is scale dependent.

Critical strengths

None.

Critical weaknesses

I cannot assess whether the experiments are sound because I can't understand what each section even means. I have worked extensively in this field, and know most cited papers however I could not understand the experimental evidence and how it connects to the main claim. I read thrice but I still did not understand the experimental sections.

  • The original pipeline is never properly defined
  • The abstract is very dense with numbers. These can’t be understood before reading the paper.

The paper centers around the concept of inferential substitutions seems weird to me "a quantity was computed correctly, and a different quantity was inferred from it" -- this looks to me like poor analysis in a paper. I don't think these problems are present in works cited in related work, so I don't understand what problem the paper is fixing. Do I understand the claim correctly that the paper doesn't do a pipeline well and presents this as a negative result?

Questions for authors

No questions, I am unlikely to be convinced to support this paper. Sorry for being overly harsh.

Reviewer_c2bWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a diagnosis procedure of an experimental failure. The authors design an experiment to test whether a list of measured cheaper pre-adaptation signals, including gradient alignment, representation overlap, task novelty, planned update size and their pairwise combinations, can predict the adaptation performance. Then they analyze the reasons why this prediction fails at 98.7% rate.

Critical strengths

(1) The authors target at a critical problem in task acquicision evaluation and provided a set of reasons to diagnose a failure rate reported in their experiment.

Critical weaknesses

(1) The writing is hard to follow with low readability. Especially, the experiment setup, targeted research questions, suite of criterions they apply as assessment are not clearly stated.

(2) Limited novelty and generalizability of the claims: the "improve on training while forgetting on held-out" statement in Table 1 and the "select separately by validation at each scale, then verify acquisition on data the selector never saw. " proposed in Section 7 are common knowledge and experimental practice for experiment design. Also, the authors transparently claim that "We do not claim that published methods share our defects, nor that acquisition matching is new.", which gives a weird posit of the value of this work.

Questions for authors

(1) What is the research question you are targeting to solve? Can you provide a clarification on your research question, experiment setup and clear explanations of each of your presented experiments?

(2) Have you conducted the acquisition audit to any published cross-scale forgetting study?

(3) What is your observation on the scale-dependence of acquisition and forgetting, which is the main goal of section 6? Have you fit any scaling curves given your cross-scale experiments?

(4) Can you restate the novelty and contribution of this work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.