- It is not known whether the empirical basis is accurate and complete.
Section 8 and Appendix K.3 directly affirm that extraction accuracy has not been estimated independently. This is critical, since the key claim depends on being able to find true author-run tests to associate with the right model-task combination. Primary records that can retain cited rows without explicit evidence of reruns, prose statements that can qualify without an additional outcome check, and format-dependent handling of citation information are all discussed in Appendix K.1. Using the paper’s title or abstract to assign a task is also a possible fallback (Appendix D). Limits in dictionary coverage and text recovery can also exclude eligible tests.
The sensitivity experiments share the same machinery, and so do not test precision and recall on independently-evaluated source data. They also reveal substantial sensitivity: for the 3,512 papers selected by the numerical table filter, the upper quartile of model count drops from six in the raw data to four with the filter applied (Table 27). This does not mean the excluded cases are wrong: it demonstrates the importance of the data filtering decisions, and their accuracy. It is encouraging that the stable family cutoff is not sensitive, but it cannot exclude cases where the same mistake was made for all filters, and the bootstrapped confidence intervals are conditioned on the extracted labels.
In order to evaluate the key contribution, it is essential that the analysis be accompanied by an independently annotated audit of papers included and excluded (including paper-level panel-count errors and their effect on the headline results). There is already a precedent for this sort of human accuracy assessment in closely related work cited by the manuscript: Park et al. (2025, §3.1) compare extracted attributes to human labels. The size of the present dataset is no substitute for this type of evaluation. aclanthology.org
- It has not been demonstrated that the proposed numeric guidelines aid in evaluating claim sufficiency.
The 25th percentile age and 75th percentile breadth cutoffs for models used characterize observed data, but the paper does not test whether meeting them would make a study’s central conclusions more transferable across families, or more stable across additional relevant models. (The explicit acknowledgment that these are planning guidelines rather than strict minima is much appreciated (§8; Appendix L). But even for this restricted use, it is not clear that they are useful: their rationale is a descriptive convention, not any measured link between the evidence requirements for the claim and the data.)
This is important, since the data do not differentiate between experiments that are key to a paper’s claims and others that support them (Appendix F.2), and the most important claim of the paper is not even specified for every study (Appendix L.2). The coverage metrics cannot be used to show that missing tests indicate an evidentiary problem. Table 1 even recommends three families each for systems and math, when the within-task reference count was only two – this is an additional judgment beyond the quantile. Furthermore, these planning guide values are not validated by the inter-paper ranking experiments, and Appendix L.4 explicitly states that stability across additional panels is left for future work. The paper succeeds better at characterizing the observed reference distributions than it does at developing practical guidance, which is the major framing argument.
The contribution to ML evaluation and ICLR
The paper makes an empirical contribution to ML experimental design: it shows how model choice shapes the evidence behind an LLM research claim.
Thank you for recognizing the value of the findings. ICLR’s call welcomes all areas of machine learning [1], and its reviewer guide explicitly recognizes empirical and practitioner contributions [2]. ICLR 2025’s “(Mis)Fitting: A Survey of Scaling Laws” is a clear example of work that studies research choices and their effect on ML conclusions [3].
Our study distinguishes the number of models from family coverage and the evidence for each task. Among 2,531 papers with multiple tasks and at least three families overall, 1,042 (41.2%) have a task covered by one family in the extracted records. This gives a concrete design lesson: show which model comparisons support each focal task. HELM motivates coverage across models and evaluation scenarios [4]. Perlitz et al. show that changing the models in a comparison can change the benchmark leader [5]. Our work measures the panels researchers use and gives task- and contribution-specific references for choosing and explaining those panels.
The method uses contextual rules for author execution, copied results, and supporting roles, with saved source passages. The new 100-paper author audit measures model-inventory agreement at 89.3% precision, 87.2% recall, and 88.3% F1 (Section 4.2; Table 1). We therefore ask that the paper be assessed as empirical ML evaluation research.
References [1] ICLR, “ICLR 2027 call for papers,” 2026. [Online]. Available: https://iclr.cc/Conferences/2027/CallForPapers [2] ICLR, “ICLR 2027 reviewer guide,” 2026. [Online]. Available: https://iclr.cc/Conferences/2027/ReviewerGuidelines [3] M. Li, S. Kudugunta, and L. Zettlemoyer, “(Mis)Fitting: A survey of scaling laws,” in Proc. ICLR, 2025. [4] P. Liang et al., “Holistic evaluation of language models,” Trans. Mach. Learn. Res., 2023. [5] Y. Perlitz et al., “Efficient benchmarking (of language models),” in Proc. NAACL-HLT, 2024, pp. 2519–2536, doi: 10.18653/v1/2024.naacl-long.139.