- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper brings argument-based validity theory from educational measurement into LLM evaluation and proposes Score-to-Claim Auditing (S2C). S2C produces a Benchmark Validity Card that links three kinds of information: item-demand annotations, variation across evaluation protocols, and evidence for interpreting scores. The method is demonstrated in three settings: six model pipelines on four multiple-choice benchmarks (120 items each) plus a GSM8K extension; two LLM judges compared with expert ratings on SummEval; and a controlled factual-consistency test.
Critical strengths
- The problem is important and timely. Over-interpreting benchmark scores is a widespread issue, and the paper systematically brings well-established validity theory (Kane, Messick) into LLM evaluation. The perspective is fresh and has interdisciplinary value.
- The framework is broad. It places item demands, protocol stability, and score interpretation in a single argument structure, and it is demonstrated in three quite different settings: multiple-choice, numeric answers, and LLM judges.
- The item-level analysis at equal accuracy is insightful. Of the 96 protocol comparisons, 18 leave net accuracy unchanged, and 12 of these hide offsetting gains and losses. This shows clearly how much an aggregate score can conceal.
- The controlled factual-consistency test is well designed: changing source facts shifts ratings by about 2 points, while rewording and repeated scoring shift them by only about 0.1. The contrast is clear.
- The reporting is transparent. The authors openly disclose limitations such as repair and parsing procedures and missing metadata, and the reported counts are internally consistent.
Critical weaknesses
- The paper is hard to read, and the figures need work. The text is very dense and heavily hedged, so it is hard to tell what the authors do and do not claim, and the main text does not show a single prompt example. As for the figures, Figure 1 is a raster image with text overlaid rather than a vector graphic; Figure 2(a) encodes data with circle areas and sectors, which makes the values hard to read; and Figure 3(b) shows only connected point estimates without intervals, although the "reversal" claim depends on it. I suggest tightening the text, stating the core claims and their scope plainly, including one complete worked example in the main text, and redesigning the main figures.
- The protocol comparisons lack a noise baseline. Item flips and rank changes across protocols are attributed to the protocol, but the multiple-choice and GSM8K studies have no repeated runs on identical inputs. The paper's own judge changes 4.4–18.8% of its ratings on identical inputs, which is comparable to the 15% turnover in the main example. Running each protocol twice and reporting turnover relative to that baseline would make the conclusions much more convincing.
- Some conclusions rest on single examples. The "opposite net change across demand groups" comes from one comparison with ±2 items per group, and the system-level judge "reversal" (0.9425 vs. 0.9602, 16 systems) has no interval. I suggest adding bootstrap intervals and reporting how often such effects occur across all comparisons; if the interval includes 0, the "reversal" should be stated more cautiously.
- The framework itself is not yet validated. The Cards currently rest on the authors' own judgment, and there are no explicit criteria for what counts as "supporting" or "qualifying" evidence. Providing such criteria, testing agreement between independent analysts, or showing one case where S2C changes an existing conclusion would greatly strengthen this contribution.
- Parts of the setup are hard to trace. Decoding settings and prompt templates are not given, the 3,600 replacement rows (25% of all records) carry a GPT model ID, and the generation of paraphrased items is not described. Please add this information to the appendix.
Minor: "Samuel, 1995" should be Messick (1995); the given and family names are swapped in the reference list. Related work could include multi-prompt evaluation, item flips at equal accuracy, and construct validity in ML evaluation.
Questions for authors
- What are the decoding and reasoning settings for each pipeline? Are they the same across protocols?
- Do the counts of 18 and 12 still hold when only first-attempt records (without repair or retry) are kept?
- Who (or which model) generated the paraphrased items, and how was meaning preservation checked?
- Some GPQA baseline scores are low (e.g., Gemini 2.5 Pro at 59.17%). Is this related to whether reasoning mode was enabled, or to the answer format?
Overall, I find the perspective of this work fresh and valuable, but the current presentation and experimental evidence do not yet support several of its core claims. If the authors can adequately address the questions above, I will consider raising my score.
2 comments
Dear Reviewer gSc2,
Thank you again for your thoughtful review. We have posted our detailed response and revised the manuscript, including the identical-input repeat control addressing your question about variability. We would be grateful to hear whether these additions and clarifications resolve your main concerns, and whether there is any remaining point we can address before the discussion closes.
Thank you for your time and consideration.
Best regards,
The Authors
Thank you for the detailed review. We have clarified the contribution, revised the presentation and added focused validation.
Section 3.4 now defines supporting evidence as evidence that tests and is consistent with a claim's observable implication under stated conditions. Qualifying evidence identifies a counterpattern or uncertainty. A contradicted or untested prerequisite leaves the dependent inference unsupported. Table 2 gives a complete prompt and worked example: 103/120 correct in both presentations supports equal observed accuracy, while nine gains and nine losses contradict unchanged item-level success. The Card records this distinction. Figure 1 has been redrawn as a vector diagram, Figure 2(a) shows readable counts, and Figure 3(b) includes intervals.
We attempted two fresh responses under each of the original and plain-instruction presentations for GPT-5.2 on the 120 preserved HellaSwag items. Routing, reasoning, parsing and output limits were fixed, with provider-default sampling and no retries or replacements. Every contrast uses the same 117 complete items.
The positive paired excess supports presentation sensitivity beyond measured repeat variability in this focal comparison (Section 4.2.4, Appendix H.4). The historical 15% remains a descriptive result from its original pipeline.
Opposing demand-group net changes occur in 22/96 comparisons with adjudicated labels and 20/96 and 23/96 under the two annotators' labels. Fifteen comparison identities are shared across all three views. The focal GPQA example retains opposing signs: +2/-2 after adjudication and +1/-1 under each annotator. These are descriptive counts of overlapping comparisons (Appendix H.5).
On the matched 75-document, 16-system set, the Sonnet-minus-GPT consistency-correlation difference is -0.1111 at summary level, with a 95% paired document-bootstrap interval of [-0.1491, -0.0707]. At system level it is +0.0178, with an interval of [-0.1312, 0.0842]. These support GPT's higher summary-level correlation and leave the system-level ordering unresolved. The revised text and Figure 3(b) show this distinction (Section 4.4.2, Appendix H.6).
Appendices B and H.1 provide prompt templates and distinguish recorded requested settings, preserved code defaults and unrecovered settings. GPT, Gemini, Llama and Mistral record temperature 0 and minimal reasoning, with initial/retry caps of 1024/2048 on the original three benchmarks and 2048/4096 on GPQA, constant across presentations within each model-benchmark cell. For Claude and DeepSeek, temperature 0 is documented in runner code. Initial Claude sources omit the reasoning parameter. Initial direct DeepSeek sources record disabled thinking and 32/128-token caps. Claude's actual caps and some later repair settings remain unverified.
The 3,600 GPT identifiers are inherited metadata in Claude/DeepSeek replacement rows. Returned-model fields and matched source outputs identify the replacement responses. We exclude inherited fields from configuration and retry evidence.
After linking final observations to initial collection sources and excluding recorded retries and repair replacements, the counts remain 18 equal-accuracy cases and 12 with turnover among 92 evaluable comparisons. Membership changes and four comparisons become unavailable. The focal HellaSwag result retains all 120 pairs unchanged. This source-based sensitivity does not reconstruct every discarded first response (Appendix H.3).
The preserved paraphrase generators default to GPT-5.2, temperature 0.2 and output caps of 512/768. The historical invocation was not recovered, so these are code defaults, not verified run settings. Two specialists in pragmatics and discourse analysis, a professor and an associate professor with a PhD, reviewed the final paraphrases for meaning preservation (Appendix H.2).
Gemini's GPQA run requested minimal reasoning. All 119 completed baseline responses report positive reasoning-token counts, return valid answer letters and end normally. The remaining response is an API failure. Thus, widespread format or truncation failures do not explain the 71/120 result (59.17%). The effect of a different reasoning budget was not tested (Appendix H.1).
We have corrected the Messick citation and clarified the relationship to multi-prompt and validity-centered evaluation in Section 2.
Thank you again for your thoughtful feedback and for helping us strengthen the paper.