- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper is meant to be a case study on avoiding common experimental mistakes in research. It is an audit of the authors’ investigations on whether small-scale measurements can predict which model adaptations will cause forgetting at larger scales. The original analysis appeared to show that roughly 98% of selected relationships failed to transfer with scale. They argue that this arose from several “inferential substitutions”: quantities were computed correctly but interpreted as measuring different scientific constructs than they actually did. The identified issues include treating duplicated records as independent, interpreting a high failure rate without comparison to the corresponding base rate, evaluating relative improvements when absolute predictive performance was poor, measuring “forgetting” even when the model had not successfully acquired the new task, transferring an adaptation recipe across model scales where it was no longer appropriate, and using a preregistered median across three heterogeneous task families as evidence of cross-scale matching. The authors then redesign parts of the experiment to explicitly verify held-out acquisition of the new task, select adaptation recipes separately by scale, isolate validation from confirmation data, and evaluate predictors on distinct task/data units. Under this corrected protocol, the original scientific question remains unresolved.
Critical strengths
-
It's very transparent and concrete. I enjoyed the process of reading this. The step-by-step isolation of failure modes of their original study is interesting to follow
-
These kinds of errors are not altogether uncommon and they turn the case study into a reusable protocol to avoid these errors. They mention useful steps like comparing against base rates, checking your units, etc.
Critical weaknesses
-
The contribution is interesting to read but I'm not convinced it's substantive enough to be a main-conference submission. All of the individual lessons/ errors are not novel. In a sense, they're just indicative of poor research hygiene. I understand that the premise is that they're easy mistakes to make, but that doesn't make them a meaningful scientific contribution. The literature screening doesn't really cover this gap. It covers 18 papers and isn't done in a particularly scientific/ data-driven way. This could potentially make for a good position paper with a reframing and a broader literature analysis. There exist papers which point out poor research hygiene in a more substantive way. Consider the paper "Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision," which does a thorough study of how many papers' headline results are likely just consequences of noise. This is to say that there is an avenue and a need for this kind of work, but it needs to be done scientifically and speak to the prevalence of such issues in the broader research community.
-
The original research question is basically unanswered.
Questions for authors
-
Do you have quantitative evidence for these kinds of mistakes being a broader pattern in the research community as opposed to failings of your research design?
-
What is the precise methodological novelty relative to the existing evaluation literature you cite?