acceptodds
Under review as a conference paper at ICLR 2027

From Scores to Claims: A Validity Argument Framework for Large Language Model Evaluation

Abstract

Benchmark scores are widely used as evidence of large language model capabilities, yet their meaning depends on the tasks, protocols, and scoring procedures that produce them. We introduce Score-to-Claim Auditing (S2C), a validity-argument framework grounded in language testing and educational measurement. Given an intended interpretation, S2C examines annotated item demands, changes in scores and rankings across protocols, and automated ratings through expert comparisons and controlled tests. Benchmark Validity Cards synthesize the evidence into supported interpretations and conditions of use. We apply S2C to multiple-choice tasks, mathematical word problems with numerical answers, and automated summary evaluation. In multiple-choice tasks, annotations reveal different demand mixtures beneath a shared response format. Small score changes across protocols can alter model rankings. Among 18 equal-accuracy comparisons, 12 contain offsetting item-level gains and losses. Construct-linked analysis also identifies opposing net changes across demand groups. In summarization, positive correlations with expert ratings coexist with numerical discrepancies. On the same summaries, the judges' point-estimate ordering by consistency correlation with experts reverses between summary-level and system-level aggregation. Controlled source reversals produce substantially larger consistency-rating changes than meaning-preserving expression changes or identical-input repeats, supporting a factual-consistency interpretation under the tested rubric and materials. These findings motivate validity profiles alongside benchmark scores, making the evidence and conditions behind capability and quality claims explicit. Analysis code and numerical data are available at https://anonymous.4open.science/r/s2c-review-E603/.

Primary area:
datasets and benchmarks
Keywords:
large language model evaluation, benchmark validity, construct validity, reliability, score interpretation, language testing, educational measurement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gSc2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper brings argument-based validity theory from educational measurement into LLM evaluation and proposes Score-to-Claim Auditing (S2C). S2C produces a Benchmark Validity Card that links three kinds of information: item-demand annotations, variation across evaluation protocols, and evidence for interpreting scores. The method is demonstrated in three settings: six model pipelines on four multiple-choice benchmarks (120 items each) plus a GSM8K extension; two LLM judges compared with expert ratings on SummEval; and a controlled factual-consistency test.

Critical strengths
  1. The problem is important and timely. Over-interpreting benchmark scores is a widespread issue, and the paper systematically brings well-established validity theory (Kane, Messick) into LLM evaluation. The perspective is fresh and has interdisciplinary value.
  2. The framework is broad. It places item demands, protocol stability, and score interpretation in a single argument structure, and it is demonstrated in three quite different settings: multiple-choice, numeric answers, and LLM judges.
  3. The item-level analysis at equal accuracy is insightful. Of the 96 protocol comparisons, 18 leave net accuracy unchanged, and 12 of these hide offsetting gains and losses. This shows clearly how much an aggregate score can conceal.
  4. The controlled factual-consistency test is well designed: changing source facts shifts ratings by about 2 points, while rewording and repeated scoring shift them by only about 0.1. The contrast is clear.
  5. The reporting is transparent. The authors openly disclose limitations such as repair and parsing procedures and missing metadata, and the reported counts are internally consistent.
Critical weaknesses
  1. The paper is hard to read, and the figures need work. The text is very dense and heavily hedged, so it is hard to tell what the authors do and do not claim, and the main text does not show a single prompt example. As for the figures, Figure 1 is a raster image with text overlaid rather than a vector graphic; Figure 2(a) encodes data with circle areas and sectors, which makes the values hard to read; and Figure 3(b) shows only connected point estimates without intervals, although the "reversal" claim depends on it. I suggest tightening the text, stating the core claims and their scope plainly, including one complete worked example in the main text, and redesigning the main figures.
  2. The protocol comparisons lack a noise baseline. Item flips and rank changes across protocols are attributed to the protocol, but the multiple-choice and GSM8K studies have no repeated runs on identical inputs. The paper's own judge changes 4.4–18.8% of its ratings on identical inputs, which is comparable to the 15% turnover in the main example. Running each protocol twice and reporting turnover relative to that baseline would make the conclusions much more convincing.
  3. Some conclusions rest on single examples. The "opposite net change across demand groups" comes from one comparison with ±2 items per group, and the system-level judge "reversal" (0.9425 vs. 0.9602, 16 systems) has no interval. I suggest adding bootstrap intervals and reporting how often such effects occur across all comparisons; if the interval includes 0, the "reversal" should be stated more cautiously.
  4. The framework itself is not yet validated. The Cards currently rest on the authors' own judgment, and there are no explicit criteria for what counts as "supporting" or "qualifying" evidence. Providing such criteria, testing agreement between independent analysts, or showing one case where S2C changes an existing conclusion would greatly strengthen this contribution.
  5. Parts of the setup are hard to trace. Decoding settings and prompt templates are not given, the 3,600 replacement rows (25% of all records) carry a GPT model ID, and the generation of paraphrased items is not described. Please add this information to the appendix.

Minor: "Samuel, 1995" should be Messick (1995); the given and family names are swapped in the reference list. Related work could include multi-prompt evaluation, item flips at equal accuracy, and construct validity in ML evaluation.

Questions for authors
  1. What are the decoding and reasoning settings for each pipeline? Are they the same across protocols?
  2. Do the counts of 18 and 12 still hold when only first-attempt records (without repair or retry) are kept?
  3. Who (or which model) generated the paraphrased items, and how was meaning preservation checked?
  4. Some GPQA baseline scores are low (e.g., Gemini 2.5 Pro at 59.17%). Is this related to whether reasoning mode was enabled, or to the answer format?

Overall, I find the perspective of this work fresh and valuable, but the current presentation and experimental evidence do not yet support several of its core claims. If the authors can adequately address the questions above, I will consider raising my score.

2 comments

Thank you for the detailed review. We have clarified the contribution, revised the presentation and added focused validation.

  1. Clearer presentation and explicit Card criteria

Section 3.4 now defines supporting evidence as evidence that tests and is consistent with a claim's observable implication under stated conditions. Qualifying evidence identifies a counterpattern or uncertainty. A contradicted or untested prerequisite leaves the dependent inference unsupported. Table 2 gives a complete prompt and worked example: 103/120 correct in both presentations supports equal observed accuracy, while nine gains and nine losses contradict unchanged item-level success. The Card records this distinction. Figure 1 has been redrawn as a vector diagram, Figure 2(a) shows readable counts, and Figure 3(b) includes intervals.

  1. Identical-input baseline

We attempted two fresh responses under each of the original and plain-instruction presentations for GPT-5.2 on the 120 preserved HellaSwag items. Routing, reasoning, parsing and output limits were fixed, with provider-default sampling and no retries or replacements. Every contrast uses the same 117 complete items.

Correctness disagreementEstimatePointwise 95% item-block bootstrap interval
Mean within-presentation2.99%[1.28%, 5.13%]
Between-presentation (four cross-pairs averaged)8.12%[3.85%, 12.82%]
Between minus within5.13 pp[1.71, 9.40] pp

The positive paired excess supports presentation sensitivity beyond measured repeat variability in this focal comparison (Section 4.2.4, Appendix H.4). The historical 15% remains a descriptive result from its original pipeline.

  1. Demand-group frequency and aggregation uncertainty

Opposing demand-group net changes occur in 22/96 comparisons with adjudicated labels and 20/96 and 23/96 under the two annotators' labels. Fifteen comparison identities are shared across all three views. The focal GPQA example retains opposing signs: +2/-2 after adjudication and +1/-1 under each annotator. These are descriptive counts of overlapping comparisons (Appendix H.5).

On the matched 75-document, 16-system set, the Sonnet-minus-GPT consistency-correlation difference is -0.1111 at summary level, with a 95% paired document-bootstrap interval of [-0.1491, -0.0707]. At system level it is +0.0178, with an interval of [-0.1312, 0.0842]. These support GPT's higher summary-level correlation and leave the system-level ordering unresolved. The revised text and Figure 3(b) show this distinction (Section 4.4.2, Appendix H.6).

  1. Settings, replacement metadata and first-response sensitivity

Appendices B and H.1 provide prompt templates and distinguish recorded requested settings, preserved code defaults and unrecovered settings. GPT, Gemini, Llama and Mistral record temperature 0 and minimal reasoning, with initial/retry caps of 1024/2048 on the original three benchmarks and 2048/4096 on GPQA, constant across presentations within each model-benchmark cell. For Claude and DeepSeek, temperature 0 is documented in runner code. Initial Claude sources omit the reasoning parameter. Initial direct DeepSeek sources record disabled thinking and 32/128-token caps. Claude's actual caps and some later repair settings remain unverified.

The 3,600 GPT identifiers are inherited metadata in Claude/DeepSeek replacement rows. Returned-model fields and matched source outputs identify the replacement responses. We exclude inherited fields from configuration and retry evidence.

After linking final observations to initial collection sources and excluding recorded retries and repair replacements, the counts remain 18 equal-accuracy cases and 12 with turnover among 92 evaluable comparisons. Membership changes and four comparisons become unavailable. The focal HellaSwag result retains all 120 pairs unchanged. This source-based sensitivity does not reconstruct every discarded first response (Appendix H.3).

  1. Paraphrases and Gemini GPQA

The preserved paraphrase generators default to GPT-5.2, temperature 0.2 and output caps of 512/768. The historical invocation was not recovered, so these are code defaults, not verified run settings. Two specialists in pragmatics and discourse analysis, a professor and an associate professor with a PhD, reviewed the final paraphrases for meaning preservation (Appendix H.2).

Gemini's GPQA run requested minimal reasoning. All 119 completed baseline responses report positive reasoning-token counts, return valid answer letters and end normally. The remaining response is an API failure. Thus, widespread format or truncation failures do not explain the 71/120 result (59.17%). The effect of a different reasoning budget was not tested (Appendix H.1).

We have corrected the Messick citation and clarified the relationship to multi-prompt and validity-centered evaluation in Section 2.

Thank you again for your thoughtful feedback and for helping us strengthen the paper.

Dear Reviewer gSc2,

Thank you again for your thoughtful review. We have posted our detailed response and revised the manuscript, including the identical-input repeat control addressing your question about variability. We would be grateful to hear whether these additions and clarifications resolve your main concerns, and whether there is any remaining point we can address before the discussion closes.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_C9RJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a validity argument framework for LLM evaluation, namely, the score-to-claim (S2C) auditing. Three types of tasks were considered: multiple choice, mathematical word problems, and automated summary. 6 models, 4 benchmarks, and 4 protocols (formal-exam framing, plain instruction, shuffled options, and semantic paraphrase) were considered. The main conclusion is that validity profiles should accompany benchmark scores.

Critical strengths

The paper applies argument-based validity principles from educational measurement to clarify what benchmark scores can support. It claims analyses beyond aggregate accuracy but instead emphasizes explicit interpretations and supporting evidence. The Benchmark validity cards provide a structured way to document supported interpretations, evaluation conditions, and evidence gaps, making the implications and limitations of benchmark results more transparent.

Critical weaknesses

The innovation and generalization are the two major concerns. Although having a coherent argument-based validity framework stems from educational measurement, the recent abundance of LLM-psychometrics literature is already implicitly implementing the validity framework, in that constructs need to be first determined, followed by scoring rules. Hence, the study’s contribution is mainly organizing established ideas in a formatted framework.

Second, due to the time-consuming tasks of human annotation, only 6 models and 4 benchmarks were considered. Hence, the conclusions drawn from the study are mainly tied up to the specific model and benchmarks, which limit its generalizability.

Questions for authors

None.

6 comments

Thank you for recognizing the value of making explicit benchmark interpretations and their supporting evidence. We address the concerns about contribution and generalizability below.

  1. Contribution beyond construct specification and scoring rules

We agree that argument-based validity is established. Section 2 discusses the claim-centered perspective of Measurement to Meaning, benchmark profiling and evaluation documentation. S2C builds on these foundations. Its contribution is a linked empirical audit of what particular score interpretations require, not a claim that validity theory or construct specification is new.

Defining constructs and scoring rules does not by itself establish what an observed score supports. For example, GPT-5.2’s original/plain GPQA comparison preserves 99/119 correct while mathematical-reasoning items gain two correct answers and subject-knowledge items lose two. Meanwhile, GPT-5.2 ranks first across all five recorded protocols (Section 4.2.4, Table 2). These findings support a persistent comparative lead on the audited tasks but do not support interpreting the unchanged total as unchanged performance across demand groups. S2C makes that distinction explicit through linked observations and annotations.

The contribution also includes controlled evidence about score meaning. Section 4.4.3 holds summaries fixed while reversing their source support, with expression changes and identical-input repeats as controls. This tests whether consistency ratings follow the relation specified by the rubric. The Cards document the resulting interpretations and their limits. They are not the sole contribution.

  1. Generalizability and annotation requirements

The six-model/four-benchmark design describes the multiple-choice audit. Sections 4.3-4.4 additionally examine exact numeric-answer scoring and summary-quality ratings. These settings demonstrate the workflow under different score-generating procedures.

We agree that this coverage does not establish universal applicability. The numerical conclusions remain conditional on the sampled tasks, recorded pipelines and scoring choices. Table 2 and Appendix G explicitly distinguish supported task-level interpretations from untested extensions. Applying S2C elsewhere requires corresponding evidence, not assuming that our observed rankings or effect sizes transfer.

Item-demand annotation is also performed on the sampled items and reused across the model panel. As Section 3.4 states, suitable annotations can be reused. Adding models therefore does not inherently require annotating the same items again, although extending task coverage may require additional annotation.

  1. Presentation

We take the readability concern seriously. We will state the empirical contribution and its scope more directly and streamline the explanation around the existing GPQA example. The central distinction is between producing a score and establishing which interpretation that score supports.

Dear Reviewer C9RJ,

Thank you again for reviewing our paper. We have posted our detailed response and revised the manuscript to clarify S2C's contribution, its relationship to existing evaluation approaches, and the evidence supporting it. We would welcome your assessment of these clarifications and would be happy to address any remaining questions before the discussion closes.

Thank you for your time and consideration.

Best regards,

The Authors

Thanks for addressing the comments. I very much appreciate the changes the author made. I still felt that the proposed validity framework lacks novel contribution. For instance, the discussion about score meanings have already been somewhat established in LLM field and very-well established in psychometrics field.

Reviewer_mubxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a validity-argument framework for interpreting large language model benchmark scores, motivated by the gap between reported scores and the capability or quality claims attached to them.

The framework links construct annotations, protocol comparisons, paired item outcomes, rankings, and automated scoring analyses to a specific intended interpretation, and it records the resulting argument in a Benchmark Validity Card.

Critical strengths

The paper addresses an important and underexamined problem: benchmark scores are often treated as direct evidence for broad capability claims, while the validity argument behind those claims is rarely made explicit. A claim-specific framework is useful for evaluation practice.

The combination of construct annotations, protocol variation, paired correctness analysis, ranking analysis, and scoring checks is a sensible evidence profile. The equal-accuracy turnover examples and the construct-linked redistribution examples are especially useful because they show why aggregate scores alone can mislead.

The controlled factual-consistency study is a genuine strength. Holding summaries fixed while reversing source support provides stronger evidence for rating responsiveness than observational correlation alone, and the comparison with expression changes and repeats makes the result more interpretable.

The paper is generally transparent about scope and limitations. It distinguishes judged item demands from internal model processes, acknowledges retrospective protocol comparisons, separates analysis populations, and reports several evidence gaps.

Critical weaknesses

The framework is conceptually valuable but its methodological novelty relative to existing validity theory and evaluation-reporting work is limited. Much of the contribution is a careful synthesis of known validity concerns into a reporting workflow, rather than a new measurement model or a decisive empirical method.

The Benchmark Validity Cards are authored narratives and are not empirically validated as a reporting intervention. There is no study showing that cards improve readers’ interpretations, reduce overclaiming, or change decisions compared with ordinary benchmark reporting. The small classification pilot in the appendix does not test the full cards or their utility.

The framework lacks clear decision rules for adjudicating evidence. It is unclear how much construct, protocol, ranking, and scoring evidence is sufficient for a given claim, how conflicting evidence should be weighed, and when an evidence gap should qualify versus overturn an interpretation. Without such rules, the cards risk becoming structured commentary rather than a validity argument with explicit warrants.

The empirical audits are selective and sometimes underpowered for the breadth of the claims. The multiple-choice audit uses one hundred twenty items per benchmark and a six-model panel; the summary study uses a limited document set and two judges; several analyses are descriptive without uncertainty intervals. The paper is careful about scope, but the evidence base is still illustrative rather than comprehensive.

Construct-linked conclusions depend on item-demand annotations whose reliability is uneven. Agreement for the later expert-science items is only moderate, and assisted labeling performs poorly on one reasoning benchmark. Because construct-linked redistribution is central to the argument, stronger annotation reliability and clearer treatment of label uncertainty would materially strengthen the paper.

The protocol audit is partly confounded by retrospective pipeline variation. The authors acknowledge historical repair, parsing, and metadata inconsistencies, but these limit causal interpretation of protocol effects. The paired comparisons are useful descriptively, yet they cannot fully isolate presentation effects from pipeline history.

The summary-evaluation evidence is narrower than the framing suggests. The prompt variations change only the opening instruction, the expert reference is a fallible mean, and only two judges are studied. The aggregation-dependent ordering reversal is interesting, but the paper does not resolve which evaluation unit should be preferred or how practitioners should choose among units.

The controlled factual-consistency materials are purposive, news-heavy, and sports-heavy, and the expression controls may not fully guarantee meaning preservation. The result is strong for the tested rubric and materials, but generalization to other domains, error types, and rubrics remains untested.

The paper argues that validity profiles should accompany benchmark scores, but it does not evaluate the cost, scalability, or practical burden of producing such profiles. For a framework intended for evaluation practice, evidence about feasibility and usefulness would be important.

The scope is limited to selected multiple-choice tasks, one numeric-answer benchmark, and summarization. Broader claims about benchmark validity would require open-ended generation, agentic evaluation, safety evaluation, and other scoring procedures.

Questions for authors

Can you provide a user study or decision experiment showing that Benchmark Validity Cards improve readers’ interpretations or reduce overclaiming compared with ordinary benchmark reporting?

What are the decision rules for determining when the assembled evidence is sufficient for a claim, when it merely qualifies a claim, and when an evidence gap should block the claim?

How sensitive are the construct-linked conclusions to annotation uncertainty, especially for the expert-science items where agreement is only moderate?

Can you separate protocol effects from historical pipeline variation more cleanly, for example by rerunning a controlled subset under standardized parsing and generation settings?

In the summary study, which evaluation unit do you recommend for practical comparison, and why should practitioners prefer summary-level or system-level aggregation?

How well do the controlled factual-consistency findings generalize beyond the purposive news and sports materials, and across other error types and rubrics?

What is the practical cost of producing a validity card for a new benchmark, and how much of the process can be automated without weakening the authored validity argument?

2 comments

Thank you for the careful review. The revision adds explicit warrant rules and focused validation.

  1. Contribution, decision rules and Card utility

S2C builds on validity theory and evaluation reporting. Its contribution is a claim-specific audit linking construct, protocol and scoring evidence to the interpretations they warrant. For example, a model's persistent lead can support comparative performance while opposing demand-group changes challenge unchanged performance across demands (Table 3).

Section 3.4 now specifies the rules. Supporting evidence tests and is consistent with a claim's observable implication under stated conditions. Counterpatterns or uncertainty qualify the interpretation or block a stronger inference. A contradicted or untested prerequisite leaves the dependent inference unsupported. An untested extension to a new domain does not overturn an observed result within the audited domain. Appendix G.1 explains how these rules guide the authored warrant.

The Cards make that reasoning inspectable. We have not measured their effects on readers' decisions or independent analysts' agreement on complete Cards. The earlier 12-profile pilot tests constrained classification, as clarified in Appendix G.4. Tables 2-3 demonstrate the present contribution through a complete prompt, paired outcomes and worked interpretations.

  1. Annotation uncertainty

We repeated the demand-group analysis under both annotators' original labels. Opposing changes occur in 22/96 comparisons after adjudication and 20/96 and 23/96 under the two annotators, with 15 comparison identities shared across all three views. The focal GPQA mathematics/subject-knowledge changes are +2/-2 after adjudication and +1/-1 under each annotator. The directions persist while magnitudes and group membership vary (Section 4.2.5, Appendix H.5). The assisted-labeling pilot's predictions do not supply the human labels used in the main audit (Appendix A.3).

  1. Standardized protocol control

Following your suggestion, we attempted two fresh responses per original/plain presentation for GPT-5.2 on 120 preserved HellaSwag items. Routing, reasoning, parsing and output limits were fixed, with provider-default sampling and no retries or replacements. All comparisons use the same 117 complete items.

Correctness disagreementEstimatePointwise 95% item-block bootstrap interval
Mean within-presentation2.99%[1.28%, 5.13%]
Between-presentation8.12%[3.85%, 12.82%]
Between minus within5.13 pp[1.71, 9.40] pp

Between-presentation disagreement averages all four cross-pairs, with all four responses retained together in each bootstrap draw. The positive excess supports presentation sensitivity beyond measured repeat variation in this current pipeline (Appendix H.4). A separate source-based sensitivity excludes recorded retries and repair replacements (H.3). These checks do not retrospectively isolate every historical comparison.

  1. Choosing the evaluation unit

We recommend choosing the unit from the intended decision. System means address average quality across a common document set. Individual-output evidence addresses individual-summary judgments, and within-document comparisons address selection among summaries of the same source (Table E5). The unit should be chosen before inspecting the correlations.

On the matched 75-document, 16-system set, the Sonnet-minus-GPT judge-expert correlation difference is -0.1111 [95% interval -0.1491, -0.0707] at summary level and +0.0178 [-0.1312, 0.0842] at system level. These support GPT's higher summary-level correspondence with the expert mean and leave system-level ordering unresolved (Section 4.4.2, Appendix H.6).

  1. Controlled materials and transfer

The controlled study holds summaries unchanged while reversing source support, crossed with two expression forms and identical-input repeats. Its 45 groups cover quantity/time, relation/role and entity/participant changes. Independent material review and revisions preceded scoring (Appendix F.1). This supplies evidence that the ratings track the tested factual relation beyond expression and repeat differences. Transfer to other domains, error types and rubrics remains untested. The three task settings demonstrate the workflow across distinct scoring procedures.

  1. Production effort and automation

For a new benchmark, the work includes specifying the claim, annotating demands, collecting relevant observations, analyzing them and writing the warrant. Suitable annotations can be reused across models. Scripts compile scores, paired outcomes, denominators and uncertainty, while analysts review materials and explain the inference (Section 3.4, Appendix G.1). We have not measured end-to-end human time or total cost per Card.

Thank you again for your thoughtful feedback and for helping us strengthen the paper.

Dear Reviewer mubx,

Thank you again for your careful review. Our response and revised manuscript clarify S2C's contribution and the rules used to support or qualify an interpretation. We have also added sensitivity and uncertainty analyses addressing your questions about the evidence. We would appreciate hearing whether these additions address your main concerns and whether any further clarification would be helpful before the discussion closes.

Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.