acceptodds
Under review as a conference paper at ICLR 2027

Can Saturated Benchmarks Be Recovered by LLM-as-a-Meta-Judge?

Abstract

Widely used language-model benchmarks are increasingly saturated, with frontier systems receiving near-tied scores that standard metrics cannot resolve. Rather than constructing harder alternatives, we ask whether existing benchmarks can regain informativeness through improved evaluation of the same candidate outputs. Therefore, we present Seeded Elimination with Adaptive LLM-as-a-Meta-Judge (SEAL), which combines a seeded tournament with evolving checklist criteria anchored to fixed task-level principles. To validate the recovered rankings, we construct a human-verified dense reference (HVR) by auditing exhaustive pairwise judgment traces against task and response evidence. Experiments span multiple saturated benchmarks such as code generation, mathematical reasoning, knowledge-intensive question answering, and tool calling. SEAL achieves the highest or joint-highest Spearman agreement with HVR across all settings, averaging 0.733 compared with 0.623 for exhaustive pairwise judging. On code generation, it uses 57.5% fewer judge calls and 65.8% fewer tokens than exhaustive comparison. Beyond the tournament, reusing evolved rubrics in pointwise scoring improves HVR agreement, raising mean correlation from 0.565 to 0.647. Together, these results demonstrate its effectiveness in adapting both comparisons and criteria to recover task-grounded ranking signal from saturated benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-Judge, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1x8fOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an approach to improve the informativeness of saturated benchmarks by using checklist-based tournaments grounded on human-verified dense references. This approach achieves high agreement with human references while using fewer judge calls across code generation, math reasoning, knowledge-based QA and tool calling.

Critical strengths
  1. The idea behind repurposing saturated benchmark is interesting but the authors should prove it on more recent benchmarks (that are also saturated) rather than those that are 5 years old and no longer relevant given how easy the prompts are.
Critical weaknesses
  1. The Human validated Reference dataset collection process was described poorly. The paper contains no information on the background on the seven reviewers, how much time they spend on the task, what guidance they were given. In addition, no subset of tasks were labelled by two annotators, which means it’s not possible to figure out how reliable the labels even are.
  2. The benchmarks chosen seem outdated and overly easy. For instance, GSM8k is a benchmark developed in 2021 that tests grade school problems when models in 2026 are already capable of contributing to research-level problem. Therefore, comparing how say PhD-level mathematicans (or LLMs) can do on grade school problem solution process is not that useful, even if some can describe the process of getting to the final answer more comprehensively.
Questions for authors
  1. Why did the authors choose dated evals over more recent evaluations (e.g. GSM8K rather than AIME or IMO)?
5 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for the feedback. However, we respectfully note that the review contains fundamental misunderstandings of our research setting, SEAL's core design, and our primary contributions.

Below, we clarify the misconceptions with corresponding manuscript updates.


1. Clarification on Core Research Setting & Role of HVR

SEAL is completely autonomous and does not use human-verified references (HVR) at any stage of execution or rubric evolution. HVR is strictly an external evaluation ground truth to benchmark SEAL against baseline evaluators.

  • SEAL does not rely on human labels: The review states that SEAL is "grounded on human-verified dense references". This is incorrect. SEAL operates automatically using an LLM-as-a-Meta-Judge. Human-verified labels in HVR are strictly held-out evaluation targets: "Labels are reserved for evaluation rather than checklist evolution or pointwise scoring" (Section 3).
  • Target Problem: This paper addresses a recognized and pressing problem in AI evaluation: re-differentiating saturated benchmarks without changing their tasks or responses. Landmark studies confirm that nearly half of 60 standard LLM benchmarks have hit performance ceilings [Akhtar et al., 2026; Ott et al., 2022]. Rather than abandoning these heavily benchmarked suites, SEAL demonstrates that evolving fine-grained checklist rubrics can extract critical, task-grounded ranking signals that standard binary pass/fail metrics miss.

2. W1 & Q1: HVR Annotation Protocol, Transparency, and Double-Blind Reliability

HVR annotation guidelines were fully established in Section 3 and detailed in Appendix C.3. To eliminate all doubts regarding label reliability, we conducted a full double-blind re-annotation of all 12,236 pairs, achieving 96.96% agreement (κ=0.927\kappa = 0.927).

  • Detailed Protocol & Documentation: Contrary to the statement that the paper "contains no information", Section 3 specified the core audit principles: checking whether cited defects exist, excluding non-applicable criteria, and converting ungrounded LLM preferences into ties. In Appendix C.3, we have further documented:

  • Annotator Background: Professional annotators from a qualified vendor holding STEM/CS degrees (code generation tasks were evaluated by annotators with CS coursework).

  • Training & Qualification: All annotators completed formal qualification guidelines and a pilot training round.

  • Time Metrics: ~310 person-hours spent over two weeks.

  • Full Double-Blind Re-annotation Results: We re-annotate all 12,236 pairs (100% coverage, not a subset) under strict double-blind conditions (blind to model names, original labels, and LLM judge traces):

BenchmarkTotal PairsBoth DefiniteAgreement RateCohen's κ\kappaAgreement
HumanEval4,4104,10295.81%0.87180.25%
GSM8K4,1853,82399.61%0.98597.48%
MMLU3,0562,65994.92%0.91890.38%
BFCL-v258552196.93%0.93991.49%
Overall12,23611,10596.96%0.92788.94%

Out of 11,105 pairs with definite labels from both runs, only 25 pairs (0.22%0.22\%) resulted in direct preference inversions. Detailed in Section 3 and Appendix C.4.

AuthorsOpenReview ↗

Response (2/2)

3. W2 & Q2: Rationale for Benchmark Selection (GSM8K vs. AIME/IMO) & Substance of Evolved Criteria

Competition-level math benchmarks (AIME/IMO) are not saturated and do not exhibit ceiling effects. Standard suites like GSM8K and HumanEval are heavily saturated (models score 96%–100%), yet conceal subtle reasoning bugs and edge-case errors that SEAL successfully uncovers without rewarding superficial verbosity.

  • Why not AIME or IMO? AIME and IMO still provide non-trivial score separation among frontier models under standard metrics. They do not suffer from saturation. By contrast, on GSM8K and HumanEval, frontier candidate models score within narrow ranges of 0.960–0.978 and 0.963–1.000, creating near-tied ranks. Testing SEAL on these benchmarks directly targets the core research question: Can we recover discriminative signal from saturated benchmarks?

  • Saturated Benchmarks Conceal Genuine Errors: Prior work proves that even on "simple" benchmarks, removing evaluation artifacts reveals genuine model failures [Vendrow et al., 2025; Liu et al., 2023]. SEAL uncovers real technical defects:

  • HumanEval/124: Two generated programs both pass standard unit tests. However, one program incorrectly accepts 2-29-2020 despite the prompt strictly mandating mm-dd-yyyy format. SEAL identifies this edge-case violation (Section 4.2).

  • SEAL Does NOT Reward Cosmetic Verbosity: The review suggests that SEAL might merely reward models that describe the process "more comprehensively". This is explicitly prevented by our design (Section 2.2): judges strictly evaluate adherence to task requirements, and HVR converts ungrounded preferences into ties.

  • GSM8K/93: Winning model (50 words) correctly applies speed-time inverse proportionality (40/1.1≈36.36s40 / 1.1 \approx 36.36\text{s}). Losing model (225 words) incorrectly treats a 10% speed increase as a 10% time reduction (36s36\text{s}). The longer response lost due to reasoning invalidity, proving SEAL prioritizes correctness over length.

  • GSM8K/73: Losing response arbitrarily assumes 2% interest is monthly without task support. SEAL flags the hallucinated premise and penalizes the output.


Direct Mapping of Revisions in Manuscript

Point RaisedResponse Summary & Action TakenRevised Manuscript Location
Research Setting & HVR RoleClarified that SEAL is fully automated and does not use HVR labels during evaluation.Section 1, Section 3
HVR Annotation TransparencyAdded full details on annotator STEM qualifications, pilot training, and 310 person-hours budget.Appendix C.3
Label Reliability & Dual ReviewAdded full double-blind re-annotation results (96.96%96.96\% agreement, κ=0.927\kappa=0.927).Section 3, Appendix C.4
Substantive vs. Cosmetic SeparationProvided case studies (GSM8K/93, HumanEval/124) showing SEAL penalizes real logical bugs.Section 4.2, Section 4.5

We hope these clarifications resolve the fundamental misunderstandings regarding our research objectives, evaluation protocol, and experimental findings. Given these clarifications and new empirical evidence, we respectfully hope that you would reconsider the solidness and overall rating of our work. Please let us know if you have any further questions.

References

[Akhtar et al., 2026] Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, et al. When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

[Etzine et al., 2025] Bryan Etzine, Masoud Hashemi, Nishanth Madhusudhan, Sagar Davasam, Roshnee Sharma, Sathwik Tejaswi Madhusudhan, and Vikas Yadav. Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performance. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP), pages 511–523, 2025.

[Kiela et al., 2021] Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, et al. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of NAACL-HLT, pages 4110–4124, 2021.

[Liu et al., 2023] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023.

[Ott et al., 2022] Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner, and Matthias Samwald. Mapping Global Dynamics of Benchmark Creation and Saturation in Artificial Intelligence. Nature Communications, 13:6793, 2022.

[Vendrow et al., 2025] Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do Large Language Model Benchmarks Test Reliability? arXiv preprint arXiv:2502.03461, 2025.

Reviewer_1x8fOpenReview ↗

Reviewer Response

Hi thanks for the response!

I will keep my original ratings for the following reasons:

W1: As a researcher with multiple years of experience doing data collection, the current evidence in the paper and author response makes me doubt the validity of the Human Verification dataset. Some suspicious pointers:

a. The authors claim to curate data on 12,236 datapoints in 310 person-hours with just seven reviewers over two weeks. This works out to be about 1.5min per datapoint. I think it's hard to believe that the data is of high quality when the reviewer only has 1.5 minutes to read/understand the question, think through the correct answer might be and then read through the answers of both models. Bearing in mind that some evals like MMLU tests knowledge across 57 categories (across science, social science and humanities) so it's unlikely people can grade all questions without doing internet search which will then take extra time (since the annotators have STEM/CS degrees).

b. Upon requesting for the agreement rate, the authors managed to arrange for 310 person-hours of annotation with 7 reviewers in around a week (reviews released on 5 Nov and author response by 11 Nov). This is basically 45 hours of work per reviewer done within a week (possible but unlikely to result in high quality data). Even under such urgency, the inter-annotator agreement rate seems too good to be true at ~95-100%. As a note, the error rate within MMLU itself is ~6.5% (see https://aclanthology.org/2025.naacl-long.262.pdf )

c. The authors claim "formal qualification guidelines and a pilot training round" but did not provide evidence of such.

W2: I'm not convinced that say outdated evals such as GSM8K (grade school problems) with recalibrated judges is a useful proxy to a model's math capabilities today.

Reviewer_XjgFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the growing problem of saturated LLM benchmarks by proposing SEAL, an evaluation protocol that keeps tasks and candidate responses fixed while combining seeded single-elimination tournaments with adaptive checklist refinement under fixed evaluation principles. The authors construct a human-verified dense reference (HVR) by auditing exhaustive pairwise judgment traces and use it to validate model rankings. Across four domains—code generation, mathematical reasoning, knowledge-intensive QA, and tool calling—SEAL achieves the highest or joint-highest Spearman correlation with HVR while reducing estimated token consumption relative to exhaustive pairwise judging. The paper also shows that the evolved checklists can improve pointwise evaluation when reused.

Critical strengths
  1. The paper addresses a timely and important problem: widely used benchmarks increasingly struggle to distinguish among strong models. Improving evaluation while preserving existing tasks and responses is a valuable direction.
  2. The experiments cover a broad range of candidate models across four distinct task domains, providing a useful assessment of the approach across different evaluation settings.
  3. SEAL’s ability to outperform FullPair in agreement with HVR while using substantially fewer estimated tokens is impressive. These results support the effectiveness of combining seeded elimination with rubric evolution.
  4. The baselines are thoughtfully selected, covering different evaluation formats and rubric configurations. In particular, FlatBrk and the rubric-reuse experiments help clarify the contribution of checklist refinement.
Critical weaknesses

Major weaknesses

  1. The definition and consistency of confidence scores need clarification. The pairwise judge prompt specifies confidence only as a number between 0 and 1, without explaining what the values represent or how they should be assigned. Since these scores weight principle votes and contribute to ranking candidates eliminated in the same round, inconsistent use of the scale across matches could affect the final rankings. A clearer definition, supported by examples or explicit scoring guidelines, would help standardize confidence reporting. The authors should also assess whether these values are comparable across matches and reliably reflect judgment correctness, and consider calibration if they are intended as probabilities of correctness.
  2. HVR would benefit from additional validation. Given the scale of the annotation effort, assigning one reviewer to each pair is a practical choice. Nevertheless, having multiple reviewers independently assess a subset of the comparisons and reporting inter-annotator agreement would provide valuable evidence about the reliability of the reference.
  3. Robustness to the choice of meta-judge is not established. The paper uses GPT-5.5 for both judging and checklist evolution. The meta-judge’s biases may influence which distinctions are incorporated into the evolved checklists and, consequently, the resulting rankings. Comparing multiple meta-judge models while holding the pairwise judge fixed would help determine how sensitive the results are to this choice.

Minor weaknesses and suggestions

  1. Established benchmarks are appropriate for studying saturation. Nevertheless, including at least one more recent benchmark that also exhibits saturation would strengthen the evidence that the approach applies beyond long-established evaluation suites.
  2. The “Stability Under Task Sampling and Repeated Runs” section provides encouraging evidence of robustness. However, for uncertainty estimates, it would be more informative if confidence intervals is reported among repeated runs.
Questions for authors
  1. Have you assessed whether confidence values are comparable across matches and reliably reflect judgment correctness? If they are intended as probabilities of correctness, have you evaluated whether calibration is needed?
  2. Was any subset of the HVR comparisons independently assessed by multiple reviewers? If so, could you report inter-annotator agreement and explain how disagreements were resolved? If not, could you provide such an assessment for a subset to help establish the reliability of HVR?
  3. How sensitive are the evolved checklists and final rankings to the choice of meta-judge? Have you evaluated an alternative meta-judge while keeping the pairwise judge fixed?
  4. Why were subsets used instead of the full benchmark test sets, and how were these subsets selected? The task-sampling analysis in Section 4.5 draws samples from the already selected subsets, so it demonstrates stability within those subsets rather than robustness to the initial subset selection. Could you clarify the rationale for this design and whether the findings hold for independently selected subsets of the original benchmarks?
4 comments
AuthorsOpenReview ↗

Response (1/3)

We sincerely thank the reviewer for the detailed and constructive comments. We are grateful for the recognization that our problem focus is timely and important. Below, we address each of the specific comments and detailed suggestions with corresponding manuscript updates.


1. W1 / Q1: Definition, Calibration, and Normalization of Pairwise Confidence Scores

Confidence ckc_k is defined as the judge's self-reported certainty regarding its vote on principle kk, serving strictly to weight principles within a match rather than representing absolute probabilities of correctness (thus not requiring probabilistic calibration). Standardizing confidence via explicit guidelines changes mean scores by less than 0.007, and normalizing confidence per match leaves all 3,106 match outcomes 100% unchanged.

  • Formal Definition & Calibration Scope: ckc_k represents the judge's self-reported confidence that the behavior cited in its reasoning supports its vote on principle kk, leveraging LLMs' established ability to verbalize uncertainty [Kadavath et al., 2022; Tian et al., 2023]. Because ckc_k operates strictly as a relative weight among principles within a single match, it does not function as an absolute probability of correctness and does not require probabilistic calibration.
  • Explicit Prompt Guidelines & Worked Examples Experiment: To evaluate confidence score stability under explicit instructions, we augmented the pairwise judge prompt with standard calibration guidelines and 4 worked reference examples (clear preference ≈0.95\approx 0.95, clear tie ≈0.95\approx 0.95, supported but uncertain ≈0.75\approx 0.75, unresolved ≈0.50\approx 0.50). Running dual evaluations across 150 GSM8K tasks demonstrated high consistency:
LevelOriginal MeanEnhanced MeanΔ\Delta (Abs.)MAE (95% CI)Lin's CCC (95% CI)Change ≤0.05\le 0.05
Principles0.94720.9535+0.00630.0168 [0.0156, 0.0180]0.8296 [0.7857, 0.8627]90.50%
**Checklist **0.96730.9669−0.00030.0122 [0.0097, 0.0146]0.7172 [0.6081, 0.8082]93.67%

Over 90% of confidence scores change by ≤0.05\le 0.05, confirming that confidence outputs under the original prompt are naturally consistent with explicit guidelines.

  • Comparability & Match-Level Normalization: The match outcome depends solely on the sign of m(A,B)=∑kckvkm(A,B) = \sum_k c_k v_k. Across matches, confidence is used only to order candidates eliminated within the same tournament round. Normalizing confidence per match (c~k=ck/∑jcj\tilde c_k = c_k / \sum_j c_j) results in zero match flips across all 3,106 tournament matches. Final candidate rankings remain virtually identical:
BenchmarkMax Change in Model ScoreChange in Final RankingSpearman ρ\rho (Normalized vs. Original)
HumanEval0.0035Ranks 3 & 4 swap0.976
GSM8K0.0038Unchanged1.000
MMLU0.0084Unchanged1.000
BFCL-v20.0195Unchanged1.000
  • Manuscript Revision: To address this point, we updated Section 2.2 with formal definitions of ckc_k, summarized normalization results in Section 4.5, and detailed full prompt guidelines and normalization analyses in Appendix D.1.
AuthorsOpenReview ↗

Response (2/3)

2. W2 / Q2: Additional Validation & Reproducibility of HVR

To further validate HVR, we conducted a full double-blind re-annotation across all 12,236 pairs with a second independent reviewer. Inter-annotator agreement reaches 96.96% (Cohen's κ=0.927\kappa = 0.927), with direct preference inversions occurring in only 25 pairs (0.22%0.22\%).

  • Blinded Protocol: To establish reproducibility, every HVR pair was re-annotated by a reviewer under strict double-blind conditions. Reviewers saw no model identities, original HVR labels, or LLM judge verdicts/scores, and candidate response order was randomly shuffled per task. For HumanEval, objective code execution outputs were provided as evidence.
  • High Agreement Breakdown:
BenchmarkTotal PairsBoth DefiniteAgree / DisagreePair AgreementCohen's κ\kappaAgreement
HumanEval4,4104,1023,930 / 17295.81%0.87180.25%
GSM8K4,1853,8233,808 / 1599.61%0.98597.48%
MMLU3,0562,6592,524 / 13594.92%0.91890.38%
BFCL-v2585521505 / 1696.93%0.93991.49%
Overall12,23611,10510,767 / 33896.96%0.92788.94%
  • Task-level bootstrap resampling (2,000 iterations over whole tasks) yields tight 95% CIs for agreement ([95.99%,97.81%][95.99\%, 97.81\%]) and Cohen's κ\kappa ([0.905,0.947][0.905, 0.947]).
  • Of the 338 total disagreements, 313 were minor boundary differences between a tie and a preference; only 25 represented direct preference inversions, mostly arising from differing readings of underspecified task requirements (e.g., handling non-integers in HumanEval/110).
  • Manuscript Revision: To address this point, we updated Section 3 and Appendix C.4 with the complete re-annotation results and statistical bounds, detailed reviewer qualifications in Appendix C.3.

3. W3 / Q3: Robustness to the Choice of Meta-Judge

While the choice of meta-judge influences which specific checklist criteria are added first in early refinement rounds, the final candidate ranking is highly robust to the meta-judge model (Spearman ρ=0.929\rho = 0.929, maintaining identical relative order for 25 out of 28 model pairs when replacing both judge and meta-judge with GPT-6-sol).

  • Checklist Evolution Sensitivity: Replacing both judge and meta-judge with GPT-6-sol on HumanEval revealed that 57.69% of first-round refined criteria focused on the same or closely related semantic checks, while 42.31% explored alternative valid task dimensions (e.g., checking algorithm complexity vs. input boundary validation).
  • High Ranking Stability: Despite intermediate rubric differences, the final model rankings demonstrate exceptional alignment with the original SEAL tournament output:
Model CandidateOriginal SEAL RankRank with GPT-6-sol
GPT-511
o4-mini-high22
Gemini-3.1-Pro34
Grok-4.2043
Claude-Sonnet-4.656
DeepSeek-V3.265
Claude-Opus-4.778
Qwen3-Coder-Plus87

Overall Spearman ρ=0.929\rho = 0.929 (Kendall τ-b=0.786\tau\text{-b} = 0.786). The top two models remain identical, and all minor rank shifts occur between models with near-identical scores (e.g., score difference <0.0009< 0.0009).

  • Manuscript Revision: To address this point, we summarized meta-judge sensitivity findings in Section 4.5 and provided full rubric comparisons and ranking tables in Appendix D.2.

4. Minor 1: Relevance of Established vs. Newer Saturated Benchmarks

Standard established benchmarks represent the primary real-world instances of benchmark saturation on public leaderboards, making them ideal testbeds for SEAL.

  • Saturation is a widespread phenomenon across AI evaluation suites [Ott et al., 2022; Akhtar et al., 2026]. On standard benchmarks like GSM8K and HumanEval, top frontier models score within narrow bands of 0.960–0.978 and 0.963–1.000, creating dense ties under binary pass/fail metrics.
  • SEAL is designed to recover discriminative ranking signal from existing saturated suites without requiring synthetic prompt re-engineering. Additionally, our evaluation includes BFCL-v2, a recent function-calling benchmark suite, confirming that SEAL generalizes effectively across diverse task modalities.
AuthorsOpenReview ↗

Response (3/3)

5. Minor 2: Confidence Intervals for Task Subsampling Stability

Across 30 random task subsamples (80 tasks per benchmark), SEAL demonstrates tight performance stability relative to full-task reference rankings:

Benchmark (80-Task Subsamples)Mean ρ\rhoSD95% Empirical RangeMin / Max ρ\rhoTop-1 Model Recovery Rate
HumanEval0.9690.028[0.905, 1.000]0.905 / 1.00030 / 30 (100%)
GSM8K0.9500.023[0.929, 1.000]0.922 / 1.00030 / 30 (100%)
MMLU0.9750.026[0.922, 1.000]0.905 / 1.00030 / 30 (100%)
  • Manuscript Revision: To address this point, we updated Section 4.5 with the explicit 95% empirical ranges and standard deviations.

6. Q4: Rationale for Subset Selection and Scope of Subsampling Analysis

Subsets were selected via stratified sampling based purely on task metadata (independent of model outputs or native scores) to manage the quadratic O(N2)O(N^2) human reference auditing cost (12,40612,406 pairs evaluated).

  • Stratified Sampling Strata:

  • HumanEval (162 tasks): 162 of 164 tasks used (excluding 2 tasks with reference test mismatches).

  • GSM8K (150 tasks): Stratified by reference solution reasoning steps (2–3 / 4–5 / ≥6\ge 6).

  • MMLU (110 tasks): Stratified across 4 subject categories (STEM, Humanities, Social Sciences, Other).

  • BFCL-v2 live_simple (22 tasks): Stratified by target function parameter count (1 / 2 / ≥3\ge 3).

  • Scope Clarification: The task-sampling analysis in Section 4.5 evaluates ranking stability within the audited subsets.

  • Manuscript Revision: To address this point, we detailed the sampling strata in Appendix B and explicitly clarified the scope of subset sampling in Section 4.5.


We hope that these comprehensive experimental additions, formalizations, and manuscript revisions address all of your comments and provide confidence in the solidness of our work. Given these new results, we respectfully hope you might consider re-evaluating our paper and overall rating. Please let us know if you have any further questions!

References

[Akhtar et al., 2026] Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, et al. When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

[Etzine et al., 2025] Bryan Etzine, Masoud Hashemi, Nishanth Madhusudhan, Sagar Davasam, Roshnee Sharma, Sathwik Tejaswi Madhusudhan, and Vikas Yadav. Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performance. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP), pages 511–523, 2025.

[Kadavath et al., 2022] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, et al. Language Models (Mostly) Know What They Know. arXiv preprint arXiv:2207.05221, 2022.

[Ott et al., 2022] Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner, and Matthias Samwald. Mapping Global Dynamics of Benchmark Creation and Saturation in Artificial Intelligence. Nature Communications, 13:6793, 2022.

[Tian et al., 2023] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5433–5442, 2023.

Reviewer_XjgFOpenReview ↗

Official Comment by Reviewer XjgF

Thank you for addressing my concerns and clarifying the points I raised. I consider my main concerns sufficiently addressed and have increased my rating accordingly. Best wishes for your continued work.

Reviewer_qzyUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SEAL, which combines seeded elimination with progressively refined evaluation checklists to distinguish model outputs on saturated benchmarks. Across four domains, SEAL achieves the highest ranking agreement with a human-verified reference (HVR), averaging 0.733 versus 0.623 for complete pairwise judging.

Critical strengths

The paper presents a practical way to make evaluation more precise without changing the tasks or responses being assessed. It keeps the core principles fixed while refining the evaluation checklists, helping reveal differences that broader metrics can miss.

The results are encouraging across four task types, with SEAL matching or exceeding the baseline’s agreement with reference rankings. Reusing the discovered checklists without further changes also improves results, suggesting that the criteria are useful beyond the tournament itself, at least within the set of responses evaluated.

Critical weaknesses

My main concern is that the reliability of the ground-truth/reference ranking has not been independently established. HVR relies on a single human audit of each existing LLM judgment, with no blinded evaluation or subset assessed by two independent reviewers. Given that 76.54% of labels change during auditing, it is especially important to show that independent reviewers agree on the revised judgments. Without this evidence, it is difficult to assess how reliable the reference ranking is, which limits confidence in the main results.

Questions for authors

What evidence establishes that HVR judgments are reproducible across reviewers? An independently reviewed, blinded subset would materially strengthen confidence in the reported rankings.

1 comment

We sincerely thank the reviewer for the constructive feedback. Below, we address the core question regarding whether HVR judgments were verified under blinded conditions and whether they are reproducible across independent reviewers.


Evidence of HVR Reproducibility & Blinded Evaluation

To provide concrete evidence of reproducibility, we went beyond an independently reviewed subset and conducted a full double-blind cross-re-annotation across all 12,236 HVR pairs. When independently re-annotated under blinded conditions, agreement with the revised HVR labels reached 96.96% (Cohen's κ=0.927\kappa = 0.927, 95% CI [0.905,0.947][0.905, 0.947]). Out of 11,105 pairs where both annotations assigned a definite verdict, only 25 pairs (0.22%0.22\%) resulted in opposite preference inversions.

1. How the Blinded Re-annotation was Conducted

  • Strict Double-Blind Setup: During re-annotation, reviewers were completely blinded to:
  1. Candidate model identities;
  2. Original HVR labels;
  3. LLM judge scores, reasoning traces, and verdicts.
  • Randomized Order: Candidate response order (Left / Right) was randomly shuffled for every comparison. Thus, the new judgments were completely unanchored from prior LLM traces or previous reviewer notes.
  • Objective Rule-Based Derivation: Reviewers evaluated each anonymized response on task principles (0=major defect0 = \text{major defect}, 1=local defect1 = \text{local defect}, 2=satisfies requirement2 = \text{satisfies requirement}), and pairwise labels were derived algorithmically by a fixed rule. For HumanEval, code execution outputs were provided as objective ground-truth evidence.

2. High Inter-Annotator Consistency Results

BenchmarkTotal PairsBoth DefiniteAgree / DisagreePair AgreementCohen's κ\kappaAgreement
HumanEval4,4104,1023,930 / 17295.81%0.87180.25%
GSM8K4,1853,8233,808 / 1599.61%0.98597.48%
MMLU3,0562,6592,524 / 13594.92%0.91890.38%
BFCL-v2585521505 / 1696.93%0.93991.49%
Overall12,23611,10510,767 / 33896.96%0.92788.94%
  • Statistical Bounds: Task-level bootstrap resampling (2,000 resamples) yields tight 95% confidence intervals for agreement ([95.99%,97.81%][95.99\%, 97.81\%]) and Cohen's κ\kappa ([0.905,0.947][0.905, 0.947]).
  • Minimal Direct Contradictions: Of the 338 disagreements, 313 were minor boundary differences between a tie and a preference; only 25 (0.22%0.22\%) were direct preference inversions.
  • Control Baseline: On the 174 control pairs featuring identical candidate responses, neither annotation assigned a preference.

3. Clarification on the 76.54% Audit Modification Rate

  • The 76.54% modification rate during auditing represents pruning ungrounded LLM judge preferences (converting false wins into ties) rather than arbitrary preference flips.
  • Specifically, 96.5% (9,040 / 9,365) of modifications converted decisive LLM judge verdicts into ties, while only 325 pairs (2.66% of retained pairs) reversed preference direction.
  • Because the blinded second annotation reproduces the revised HVR labels with 96.96% agreement without seeing the original judge traces, the revised HVR reference is firmly established as reliable and reproducible.

Manuscript Revisions

To reflect these results in the revised paper, we have updated:

  • Section 3 & Appendix C.4: Added the full double-blind protocol, pair-level agreement table, confidence intervals, and disagreement breakdown.
  • Appendix C.3: Documented reviewer qualifications, qualification training, and time spent per pair.

We hope this blinded verification evidence addresses your core concern and establishes full confidence in HVR as a reliable ground-truth reference. Given these new double-blind experimental results and manuscript updates, we respectfully hope you might reconsider the experimental solidness and overall assessment of our work. Please let us know if you have any further questions or if any additional clarifications would be helpful!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

We thank the reviewers for their constructive and insightful feedback. In the revised version of our manuscript, we have addressed the common concerns raised across the reviews by incorporating new empirical analyses, formalizing definitions, and updating the paper structure accordingly.

Below, we detail our responses alongside the exact updates made to the manuscript.


1. Reliability, Inter-Annotator Agreement, and Protocol of HVR

  • Response & Findings: To address concerns regarding single-annotator auditing and label reproducibility, we conducted a full double-blind re-annotation of all 12,236 HVR pairs. Annotators were strictly blinded to model identities, original HVR labels, and LLM judge verdicts. On the 11,105 pairs where both annotations yielded a definite label, agreement reached 96.96% (Cohen's κ=0.927\kappa = 0.927, 95% CI [0.905,0.947][0.905, 0.947]). Out of 338 total disagreements, 313 were minor boundary differences between a tie and a preference, and only 25 pairs (0.22%0.22\%) resulted in direct preference inversions.
  • Manuscript Revision: To address this point, we updated Section 3 and Appendix C.4 with the complete experimental protocol, agreement statistics, confidence intervals, and disagreement analysis of the re-annotation study. Additionally, we detailed the annotators'backgrounds, qualification procedures, and time expenditures per pair in Appendix C.3.

2. Benchmark Selection and Subsampling Rationale

  • Response & Findings: The primary objective of SEAL is to resolve benchmark saturation on standard, widely used evaluation suites (GSM8K, HumanEval, MMLU, BFCL-v2), where frontier models frequently receive near-tied scores under traditional metrics. Rather than shifting to new synthetic distributions, SEAL demonstrates that refined criteria evolution can recover discriminative ranking signals from existing tasks. Stratified sampling was employed to manage the quadratic O(N2)O(N^2) pairwise reference auditing cost while preserving difficulty and topic distribution.
  • Manuscript Revision: To address this point, we updated Appendix B with explicit details on the stratified sampling design and candidate model selection for each benchmark. Correspondingly, we expanded Section 4.5 to clarify the scope of our task-sampling stability analysis and discuss generalizability to full-scale datasets.

3. Confidence Score Definition, Calibration, and Normalization

  • Response & Findings: We evaluated whether confidence score variations across matches affect overall tournament rankings. Empirical results show that per-match confidence normalization leaves all 3,106 tournament match outcomes unchanged. Final model rankings remain identical on GSM8K, MMLU, and BFCL-v2, and virtually unchanged on HumanEval (Spearman ρ=0.976\rho = 0.976).
  • Manuscript Revision: To address this point, we updated Section 2.2 to provide formal mathematical definitions for pairwise confidence scores and their vote-weighting mechanisms. Furthermore, we incorporated summary findings on confidence normalization into Section 4.5 and provided the complete dataset-level normalization tables in Appendix D.1.

4. Meta-Judge Sensitivity and Statistical Uncertainty

  • Response & Findings: To evaluate potential meta-judge bias, we replaced meta-judge with GPT-6-sol on HumanEval. The final candidate ranking remained highly consistent (Spearman ρ=0.929\rho = 0.929, maintaining the exact relative order for 25 out of 28 model pairs, with the top two models unchanged). Additionally, repeated run evaluations confirmed high statistical stability.
  • Manuscript Revision: To address this point, we updated Section 4.5 to include 95% confidence intervals across repeated tournament runs and task resamplings. We also summarized the meta-judge sensitivity findings in Section 4.5 and provided full ablation breakdown tables in Appendix D.2.

We believe these updates thoroughly address all raised points and strengthen the theoretical and empirical foundation of our work. Please let us know if any further clarifications would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.