acceptodds
Under review as a conference paper at ICLR 2027

The Models Behind the Claim: Empirical Standards for LLM Evaluation

Abstract

Selecting an evaluation panel is a consequential but largely unguided part of Large Language Model (LLM) research: we characterize the primary approaches researchers can use to determine how many and which models to include in an LLM evaluation, and we examine the advantages and limitations of each approach, including global standards derived from freshly published, year-long research on arXiv and local standards within the ICLR community. To see how these choices are made in practice, we trace model use across 11,428 arXiv papers that report LLM evaluations and compare this broader record with 486 eligible papers from the ICLR'26. Across this record, a consistent pattern emerges: most evaluations rely on a small set of models drawn from even fewer model families, so evaluation results may only represent a narrow portion of the LLM landscape; the median paper tests 3 models from 2 families on one task family, and 39.1% test a only single family. The ICLR comparison bolsters this point: papers associated with the venue tend to include more model variants, with most of the additional breadth coming from releases within the same families. Collectively, no single minimum model count can determine whether a panel is adequate; adequacy depends on what a paper claims, what it evaluates, how current its evidence is, and which model populations it intends to represent. We therefore translate these patterns into condition-specific standards for panel breadth, family and task coverage, recency, and model access. The resulting framework gives researchers a practical basis for planning LLM evaluation panels and gives the community a clearer basis for judging whether the evidence supports the claims.

Primary area:
datasets and benchmarks
Keywords:
Large language model evaluation, Model selection, Evaluation methodology, Benchmarking, Evaluation panel design, Meta-research, Empirical standards

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_23JYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper examines which models researchers choose for LLM evaluation, using 65,863 paper-model-task relations extracted from 11,428 arXiv papers. It distinguishes model identifiers, operational families, task coverage, access, and release age, then offers descriptive references for planning evaluations around different claims. It also compares these patterns with 486 eligible ICLR 2026 papers.

Critical strengths

The paper provides a massive, rule-based text audit of 11,428 arXiv preprints and 486 ICLR 2026 papers. It concretely quantifies the field’s heavy baseline concentration (e.g., the median paper tests only 3 models from 2 families, heavily skewed toward Qwen, GPT, and Llama). This offers valuable descriptive insight into how researchers currently select evaluation panels.

Critical weaknesses

ICLR focuses on representation learning, deep learning architectures, optimization, and foundational ML methodologies. This submission is essentially a bibliometric audit and meta-science study tracking model keyword co-occurrences and reporting frequencies across arXiv preprints. It introduces no new learning algorithms, representation theories, or actionable evaluation benchmarks. It belongs in a scientometrics or computational social science venue, not ICLR.

Questions for authors

Beyond documenting publication habits across arXiv, what concrete machine learning insights or contributions to representation learning does this paper offer the ICLR community?

1 comment
AuthorsOpenReview ↗

The contribution to ML evaluation and ICLR

The paper makes an empirical contribution to ML experimental design: it shows how model choice shapes the evidence behind an LLM research claim.

Thank you for recognizing the value of the findings. ICLR’s call welcomes all areas of machine learning [1], and its reviewer guide explicitly recognizes empirical and practitioner contributions [2]. ICLR 2025’s “(Mis)Fitting: A Survey of Scaling Laws” is a clear example of work that studies research choices and their effect on ML conclusions [3].

Our study distinguishes the number of models from family coverage and the evidence for each task. Among 2,531 papers with multiple tasks and at least three families overall, 1,042 (41.2%) have a task covered by one family in the extracted records. This gives a concrete design lesson: show which model comparisons support each focal task. HELM motivates coverage across models and evaluation scenarios [4]. Perlitz et al. show that changing the models in a comparison can change the benchmark leader [5]. Our work measures the panels researchers use and gives task- and contribution-specific references for choosing and explaining those panels.

The method uses contextual rules for author execution, copied results, and supporting roles, with saved source passages. The new 100-paper author audit measures model-inventory agreement at 89.3% precision, 87.2% recall, and 88.3% F1 (Section 4.2; Table 1). We therefore ask that the paper be assessed as empirical ML evaluation research.

References [1] ICLR, “ICLR 2027 call for papers,” 2026. [Online]. Available: https://iclr.cc/Conferences/2027/CallForPapers [2] ICLR, “ICLR 2027 reviewer guide,” 2026. [Online]. Available: https://iclr.cc/Conferences/2027/ReviewerGuidelines [3] M. Li, S. Kudugunta, and L. Zettlemoyer, “(Mis)Fitting: A survey of scaling laws,” in Proc. ICLR, 2025. [4] P. Liang et al., “Holistic evaluation of language models,” Trans. Mach. Learn. Res., 2023. [5] Y. Perlitz et al., “Efficient benchmarking (of language models),” in Proc. NAACL-HLT, 2024, pp. 2519–2536, doi: 10.18653/v1/2024.naacl-long.139.

Reviewer_m8fqOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how researchers choose models for LLM evaluations, analyzing 11,428 arXiv papers and 486 ICLR 2026 papers. It lays out seven strategies for choosing evaluation models and observes that most evaluations draw from few model families, with larger model counts typically originating from variants within the same family. The paper proposes descriptive references for panel breadth, task coverage, recency, and access conditions, with recommendations tailored to the claims and tasks being studied. It documents that paper‑level overall diversity can conceal restricted coverage on individual tasks.

Critical strengths

The paper addresses a practical and widely relevant question in LLM research: how should researchers choose models for evaluation? Deciding how many models, families, or tasks are needed to support a paper’s claims remains a common challenge for authors.

Critical weaknesses

The method does not seem sufficiently rigorous and thus cannot support the paper's claim. Specifically,

(1) The entire empirical contribution rests on an unvalidated rule-based extraction pipeline. Despite the detailed inclusion/exclusion rules, the regex-based extractor cannot guarantee coverage: models may be silently missed, and papers that merely mention a model without actually experimenting on it may be miscounted. There is no independent validation of extraction accuracy.

(2) The dataset selection lacks a clear scientific basis. The paper does not explain why ICLR 2026 is the only venue used for comparison, and the arXiv corpus is not limited to papers that have undergone peer review. It is unclear how local statistics from these two sources can reflect the true value of evaluation practice more broadly. In addition, choices such as models, time, budget, and platform differ across papers and may reflect the constraints of each study. The observed configurations may therefore be tradeoffs under different circumstances, and it remains questionable whether they can serve as reference setups for others to follow.

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Reply to W1: Human validation of the extractor

Thank you for raising the need for direct evidence. The revised paper now includes a 100-paper human check, with 89.3% precision, 87.2% recall, and 88.3% micro-F1 for model-inventory agreement. Section 4.2, Table 1, and Appendix I.4 report this.

Explicit rules are an established way to extract scientific information. STEREO learns rules for reported statistics [1], while ChemDataExtractor uses rules suited to prose, captions, and tables alongside entity recognition [2]. Our rules give the same processing steps across papers, and saved source passages make the records easy to inspect.

The human check adds evidence for our own extraction task. We report agreement against an explicit human reference and keep the full comparison, in line with the emphasis on annotation quality in NLP [3] and scientific metadata extraction [4]. The revision therefore supports the method with both prior work and measured agreement in the study corpus.

References [1] S. Epp, M. Hoffmann, N. Lell, M. Mohr, and A. Scherp, “STEREO: A pipeline for extracting experiment statistics, conditions, and topics from scientific papers,” in Proc. iiWAS, 2021, pp. 340–349, doi: 10.1145/3487664.3487712. [2] M. C. Swain and J. M. Cole, “ChemDataExtractor: A toolkit for automated extraction of chemical information from the scientific literature,” J. Chem. Inf. Model., vol. 56, no. 10, pp. 1894–1904, 2016, doi: 10.1021/acs.jcim.6b00207. [3] J.-C. Klie, B. Webber, and I. Gurevych, “Annotation error detection: Analyzing the past and present for a more coherent future,” Comput. Linguist., vol. 49, no. 1, pp. 157–198, 2023, doi: 10.1162/coli_a_00464. [4] M. D. Turner et al., “Large language models can extract metadata for annotation of human neuroimaging publications,” Front. Neuroinform., vol. 19, Art. no. 1609077, 2025, doi: 10.3389/fninf.2025.1609077.

AuthorsOpenReview ↗

Reply to W2: Why this corpus, this venue, and these planning references

The two sources answer two clear questions: arXiv shows evaluation practice across fields, and ICLR provides a focused comparison within a peer-reviewed ML community. We start from the complete arXiv feed across categories, so papers can enter screening even when their titles and abstracts lack LLM keywords. A recent full year covers an annual research cycle while keeping the findings close to the current model ecosystem. The overall family reference stays at three across all four three-month cohorts. LLMEvalDB provides a related example of using arXiv to study reported LLM experiments [1].

ICLR 2026 gives a completed conference cycle within the study period and includes model development, methods, and applications. Its 486 linked papers let us examine the same coverage patterns within one research community. We account for recorded differences in month, subject area, tasks, and contribution using overlap weighting [2], with 2,000 bootstrap repetitions that recalculate the weights (Section 4.4; Appendix H).

Study budgets and access needs are also part of the framework. Section 3 covers cost-constrained design, bounded candidate sets, and controls chosen for the claim. HELM likewise treats coverage and feasibility as central design choices [3]. Our references help authors compare and explain a panel suited to their purpose. Table 2 uses the observed task-specific family counts and asks authors to justify the evaluation period for claims about current capability.

References [1] J. Park, J. Kang, G. Stanovsky, and A. Ritter, “Can LLMs help uncover insights about LLMs? A large-scale, evolving literature analysis of frontier LLMs,” in Proc. ACL, 2025, pp. 20412–20433, doi: 10.18653/v1/2025.acl-long.998. [2] F. Li, K. L. Morgan, and A. M. Zaslavsky, “Balancing covariates via propensity score weighting,” J. Amer. Stat. Assoc., vol. 113, no. 521, pp. 390–400, 2018, doi: 10.1080/01621459.2016.1260466. [3] P. Liang et al., “Holistic evaluation of language models,” Trans. Mach. Learn. Res., 2023.

The added human validation solved my concern about extraction accuracy. However, how the references help authors compare and explain a panel suited to their purpose and why this number is a proper setting remain unclear. Thus, I will raise my score to 2.

Reviewer_9ReXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Composition of LLM evaluation panels. The paper explores the composition of LLM evaluation panels, and presents a framework for model selection relative to claims in a study. It summarizes 7 different approaches for selecting panels. It uses a deterministic extraction pipeline to extract from a 1-year arXiv cohort 65,863 paper model task relationships, across 11,428 papers, with 486 papers connected to ICLR 2026. It separates model identifiers as reported from models' actual family utilization, task scope, access requirements, and model age at release. It finds mostly small, highly-clustered panels; whole paper family coverage can mask limited task family coverage; ICLR-related papers contain more model variants, with smaller spread in family coverage. Recommended planning references: contribution/task-specific 75% breadth, 25% newest model age. Secondary analyses: other inclusion criteria, family clustering, other submission cohorts, bootstrap interval, re-adjusted ICLR comparison. exploratory cross-paper rankings. Independent extraction correctness is not verified. Practical usefulness of proposed thresholds is not tested. (§§3-6, Appendices E-L)

Critical strengths
  1. A useful unit of analysis for a key design of evaluation problem.

Tracking model identifiers, families and tasks the models were evaluated on is more revealing than just the number of models in the paper.

Eg, the result of 1042/2,531 papers with multiple tasks and at least 3 families overall which still have at least one task being evaluated with only a single family shows how aggregate panel size can be misleading about the evidence on the tasks themselves. (§5.1, Appendix F.2).

The wide base arXiv population and deliberate effort to separate out author-performed evaluations from mere mentions or secondary use of models means it has potentially useful scope.

Authors/reviewers can use this evidence map to examine what comparisons actually support a result.

  1. Significant internal robustness testing and thoughtful handling of venue comparison

The alternative evidence standards, grouping families, submission cohorts are explored. The general upper quartile 3 family reference remains so under the more restrictive evidence checks and for all 4 cohorts.

The ICLR analysis controls for timing and research composition measured differences, re-estimates weighting model for its main bootstrap, and studies the specification w/o task indicators as well (Appendix I) in balanced way.

This enhances the descriptive analysis, the authors carefully leave aside interpretation of venue association as having an effect on acceptance (nice of them!).

Critical weaknesses
  1. It is not known whether the empirical basis is accurate and complete.

Section 8 and Appendix K.3 directly affirm that extraction accuracy has not been estimated independently. This is critical, since the key claim depends on being able to find true author-run tests to associate with the right model-task combination. Primary records that can retain cited rows without explicit evidence of reruns, prose statements that can qualify without an additional outcome check, and format-dependent handling of citation information are all discussed in Appendix K.1. Using the paper’s title or abstract to assign a task is also a possible fallback (Appendix D). Limits in dictionary coverage and text recovery can also exclude eligible tests.

The sensitivity experiments share the same machinery, and so do not test precision and recall on independently-evaluated source data. They also reveal substantial sensitivity: for the 3,512 papers selected by the numerical table filter, the upper quartile of model count drops from six in the raw data to four with the filter applied (Table 27). This does not mean the excluded cases are wrong: it demonstrates the importance of the data filtering decisions, and their accuracy. It is encouraging that the stable family cutoff is not sensitive, but it cannot exclude cases where the same mistake was made for all filters, and the bootstrapped confidence intervals are conditioned on the extracted labels.

In order to evaluate the key contribution, it is essential that the analysis be accompanied by an independently annotated audit of papers included and excluded (including paper-level panel-count errors and their effect on the headline results). There is already a precedent for this sort of human accuracy assessment in closely related work cited by the manuscript: Park et al. (2025, §3.1) compare extracted attributes to human labels. The size of the present dataset is no substitute for this type of evaluation. aclanthology.org

  1. It has not been demonstrated that the proposed numeric guidelines aid in evaluating claim sufficiency.

The 25th percentile age and 75th percentile breadth cutoffs for models used characterize observed data, but the paper does not test whether meeting them would make a study’s central conclusions more transferable across families, or more stable across additional relevant models. (The explicit acknowledgment that these are planning guidelines rather than strict minima is much appreciated (§8; Appendix L). But even for this restricted use, it is not clear that they are useful: their rationale is a descriptive convention, not any measured link between the evidence requirements for the claim and the data.)

This is important, since the data do not differentiate between experiments that are key to a paper’s claims and others that support them (Appendix F.2), and the most important claim of the paper is not even specified for every study (Appendix L.2). The coverage metrics cannot be used to show that missing tests indicate an evidentiary problem. Table 1 even recommends three families each for systems and math, when the within-task reference count was only two – this is an additional judgment beyond the quantile. Furthermore, these planning guide values are not validated by the inter-paper ranking experiments, and Appendix L.4 explicitly states that stability across additional panels is left for future work. The paper succeeds better at characterizing the observed reference distributions than it does at developing practical guidance, which is the major framing argument.

Questions for authors
  1. Validation of the current empirical findings: Is there an independent stratified audit of the source papers, those selected and those not selected, with precision and recall for the selection decisions, model classification, author choices and task categorisation? What is the error in the number of model/family/task counts at paper level? To what extent do sensitivity or correction for these errors alter the main focus, R75R_{75} and ICLR comparisons? A simple audit of the papers selected would not address this point about missing papers/models.
  2. Justification of the standards contribution: Is there evidence that the suggested proportions of recent and broad references would lead to increased stability of a paper's main claim in the presence of other models to which the paper could reasonably be compared? How is the set of tasks and potential claims selected with the admitted limitations in Appendices F.2 and L.2? In particular what justification is there for suggesting three families in systems and maths when the evidence shows two? (An option would be to use a panel expansion approach, but that depends on whether there is any evidence that the suggested choices are useful.)
  3. Justification and sensitivity of the time alignment: The evidence is mined from later versions of the papers, but the model dates are based on the initial submission. Appendix G.1 states that 178 records with post-submission model releases are not included in the analysis, but this does not address cases in which a model was released prior to version 1, but was only introduced in later versions. What would be the effect on the use of 170 days, task-specific age thresholds, and venue comparisons, if all the dates were aligned to a single version of the manuscripts?
3 comments
AuthorsOpenReview ↗

Reply to W1 and Q1: Human validation and extraction quality

Thank you for asking for direct validation. We added a human check of model inventories. The extractor reaches 89.3% precision, 87.2% recall, and 88.3% micro-F1. Section 4.2 and Table 1 now report these results. Appendix I.4 gives the procedure.

Rule-based extraction has a strong basis in scientific text analysis. STEREO reports about 95% precision for statistical results outside standard APA style [1]. ChemDataExtractor combines entity recognition with rules for text, captions, and tables, reaching 93.4% F1 for chemical-identifier records [2]. Park et al. also show the value of extracting structured evidence from LLM papers [3]. These studies support the approach, and our human check provides evidence for model recovery in this corpus. The evidence-rule checks give a second view of the findings. Across all four stricter checks, the family reference stays at three and the task reference at two; the model reference is four, compared with five in the main analysis. For the same 3,512 papers, the numerical-table check reports six models under the primary rules and four under the table filter. Comparing the same papers makes the effect of the evidence rule clear (Appendix I.5).

References [1] S. Epp, M. Hoffmann, N. Lell, M. Mohr, and A. Scherp, “STEREO: A pipeline for extracting experiment statistics, conditions, and topics from scientific papers,” in Proc. iiWAS, 2021, pp. 340–349, doi: 10.1145/3487664.3487712. [2] M. C. Swain and J. M. Cole, “ChemDataExtractor: A toolkit for automated extraction of chemical information from the scientific literature,” J. Chem. Inf. Model., vol. 56, no. 10, pp. 1894–1904, 2016, doi: 10.1021/acs.jcim.6b00207. [3] J. Park, J. Kang, G. Stanovsky, and A. Ritter, “Can LLMs help uncover insights about LLMs? A large-scale, evolving literature analysis of frontier LLMs,” in Proc. ACL, 2025, pp. 20412–20433, doi: 10.18653/v1/2025.acl-long.998.

AuthorsOpenReview ↗

Reply to W2 and Q2: How the references help panel design

The claim sets the comparisons a study needs. Our references then help authors explain how their panel compares with studies of similar tasks and contributions.

The upper quartile gives a clear reference for broader panels already used in practice, with little influence from a few very large panels. Authors can use it to explain why they chose a given number of models or families. Table 2 and Section 6.2 now use the observed two-family reference for systems and mathematics. The benefit is clearest at the task level. Among 2,531 papers with multiple tasks and at least three families overall, 1,042 (41.2%) have a task covered by one family in the extracted records. A model-by-task table shows exactly where another family would add evidence for that task. The framework also explains what each addition contributes: more releases test variation within a series, more families widen coverage across series, and more tasks test transfer. This builds on work linking evaluation coverage to the claims a study can support [1], [2]. Sections 6.1–6.3 and Appendix J give authors a concrete process: state the claim, identify the needed comparisons, consult the matching references, and explain each model’s role.

References [1] P. Liang et al., “Holistic evaluation of language models,” Trans. Mach. Learn. Res., 2023. [2] I. D. Raji, E. M. Bender, A. Paullada, E. Denton, and A. Hanna, “AI and the everything in the whole wide world benchmark,” in Proc. NeurIPS Datasets Benchmarks Track, 2021.

AuthorsOpenReview ↗

Reply to Q3: Paper versions and model recency

The revised paper states both timing choices and ties recency advice to the evaluation period of the claim.

First submission defines the cohort and the date used to calculate release age. The latest recovered text supplies the evaluation evidence. Section 4.3 now states this convention directly.

The 170-day overall value and the task-specific values describe newest-model ages under this convention. Table 2 and Section 6.3 ask authors to report when they selected and tested the panel, explain the evaluation period, and choose a current comparator suited to that period. This also gives long-running studies a clear way to state the time frame of their conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.