- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes an approach to improve the informativeness of saturated benchmarks by using checklist-based tournaments grounded on human-verified dense references. This approach achieves high agreement with human references while using fewer judge calls across code generation, math reasoning, knowledge-based QA and tool calling.
Critical strengths
- The idea behind repurposing saturated benchmark is interesting but the authors should prove it on more recent benchmarks (that are also saturated) rather than those that are 5 years old and no longer relevant given how easy the prompts are.
Critical weaknesses
- The Human validated Reference dataset collection process was described poorly. The paper contains no information on the background on the seven reviewers, how much time they spend on the task, what guidance they were given. In addition, no subset of tasks were labelled by two annotators, which means it’s not possible to figure out how reliable the labels even are.
- The benchmarks chosen seem outdated and overly easy. For instance, GSM8k is a benchmark developed in 2021 that tests grade school problems when models in 2026 are already capable of contributing to research-level problem. Therefore, comparing how say PhD-level mathematicans (or LLMs) can do on grade school problem solution process is not that useful, even if some can describe the process of getting to the final answer more comprehensively.
Questions for authors
- Why did the authors choose dated evals over more recent evaluations (e.g. GSM8K rather than AIME or IMO)?
5 comments
Response (2/2)
3. W2 & Q2: Rationale for Benchmark Selection (GSM8K vs. AIME/IMO) & Substance of Evolved Criteria
Competition-level math benchmarks (AIME/IMO) are not saturated and do not exhibit ceiling effects. Standard suites like GSM8K and HumanEval are heavily saturated (models score 96%–100%), yet conceal subtle reasoning bugs and edge-case errors that SEAL successfully uncovers without rewarding superficial verbosity.
-
Why not AIME or IMO? AIME and IMO still provide non-trivial score separation among frontier models under standard metrics. They do not suffer from saturation. By contrast, on GSM8K and HumanEval, frontier candidate models score within narrow ranges of 0.960–0.978 and 0.963–1.000, creating near-tied ranks. Testing SEAL on these benchmarks directly targets the core research question: Can we recover discriminative signal from saturated benchmarks?
-
Saturated Benchmarks Conceal Genuine Errors: Prior work proves that even on "simple" benchmarks, removing evaluation artifacts reveals genuine model failures [Vendrow et al., 2025; Liu et al., 2023]. SEAL uncovers real technical defects:
-
HumanEval/124: Two generated programs both pass standard unit tests. However, one program incorrectly accepts
2-29-2020despite the prompt strictly mandatingmm-dd-yyyyformat. SEAL identifies this edge-case violation (Section 4.2). -
SEAL Does NOT Reward Cosmetic Verbosity: The review suggests that SEAL might merely reward models that describe the process "more comprehensively". This is explicitly prevented by our design (Section 2.2): judges strictly evaluate adherence to task requirements, and HVR converts ungrounded preferences into ties.
-
GSM8K/93: Winning model (50 words) correctly applies speed-time inverse proportionality (). Losing model (225 words) incorrectly treats a 10% speed increase as a 10% time reduction (). The longer response lost due to reasoning invalidity, proving SEAL prioritizes correctness over length.
-
GSM8K/73: Losing response arbitrarily assumes 2% interest is monthly without task support. SEAL flags the hallucinated premise and penalizes the output.
Direct Mapping of Revisions in Manuscript
| Point Raised | Response Summary & Action Taken | Revised Manuscript Location |
|---|---|---|
| Research Setting & HVR Role | Clarified that SEAL is fully automated and does not use HVR labels during evaluation. | Section 1, Section 3 |
| HVR Annotation Transparency | Added full details on annotator STEM qualifications, pilot training, and 310 person-hours budget. | Appendix C.3 |
| Label Reliability & Dual Review | Added full double-blind re-annotation results ( agreement, ). | Section 3, Appendix C.4 |
| Substantive vs. Cosmetic Separation | Provided case studies (GSM8K/93, HumanEval/124) showing SEAL penalizes real logical bugs. | Section 4.2, Section 4.5 |
We hope these clarifications resolve the fundamental misunderstandings regarding our research objectives, evaluation protocol, and experimental findings. Given these clarifications and new empirical evidence, we respectfully hope that you would reconsider the solidness and overall rating of our work. Please let us know if you have any further questions.
References
[Akhtar et al., 2026] Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, et al. When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.
[Etzine et al., 2025] Bryan Etzine, Masoud Hashemi, Nishanth Madhusudhan, Sagar Davasam, Roshnee Sharma, Sathwik Tejaswi Madhusudhan, and Vikas Yadav. Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performance. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP), pages 511–523, 2025.
[Kiela et al., 2021] Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, et al. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of NAACL-HLT, pages 4110–4124, 2021.
[Liu et al., 2023] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[Ott et al., 2022] Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner, and Matthias Samwald. Mapping Global Dynamics of Benchmark Creation and Saturation in Artificial Intelligence. Nature Communications, 13:6793, 2022.
[Vendrow et al., 2025] Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do Large Language Model Benchmarks Test Reliability? arXiv preprint arXiv:2502.03461, 2025.
Reviewer Response
Hi thanks for the response!
I will keep my original ratings for the following reasons:
W1: As a researcher with multiple years of experience doing data collection, the current evidence in the paper and author response makes me doubt the validity of the Human Verification dataset. Some suspicious pointers:
a. The authors claim to curate data on 12,236 datapoints in 310 person-hours with just seven reviewers over two weeks. This works out to be about 1.5min per datapoint. I think it's hard to believe that the data is of high quality when the reviewer only has 1.5 minutes to read/understand the question, think through the correct answer might be and then read through the answers of both models. Bearing in mind that some evals like MMLU tests knowledge across 57 categories (across science, social science and humanities) so it's unlikely people can grade all questions without doing internet search which will then take extra time (since the annotators have STEM/CS degrees).
b. Upon requesting for the agreement rate, the authors managed to arrange for 310 person-hours of annotation with 7 reviewers in around a week (reviews released on 5 Nov and author response by 11 Nov). This is basically 45 hours of work per reviewer done within a week (possible but unlikely to result in high quality data). Even under such urgency, the inter-annotator agreement rate seems too good to be true at ~95-100%. As a note, the error rate within MMLU itself is ~6.5% (see https://aclanthology.org/2025.naacl-long.262.pdf )
c. The authors claim "formal qualification guidelines and a pilot training round" but did not provide evidence of such.
W2: I'm not convinced that say outdated evals such as GSM8K (grade school problems) with recalibrated judges is a useful proxy to a model's math capabilities today.
Response (1/2)
We thank the reviewer for the feedback. However, we respectfully note that the review contains fundamental misunderstandings of our research setting, SEAL's core design, and our primary contributions.
Below, we clarify the misconceptions with corresponding manuscript updates.
1. Clarification on Core Research Setting & Role of HVR
SEAL is completely autonomous and does not use human-verified references (HVR) at any stage of execution or rubric evolution. HVR is strictly an external evaluation ground truth to benchmark SEAL against baseline evaluators.
2. W1 & Q1: HVR Annotation Protocol, Transparency, and Double-Blind Reliability
HVR annotation guidelines were fully established in Section 3 and detailed in Appendix C.3. To eliminate all doubts regarding label reliability, we conducted a full double-blind re-annotation of all 12,236 pairs, achieving 96.96% agreement (κ=0.927).
Detailed Protocol & Documentation: Contrary to the statement that the paper "contains no information", Section 3 specified the core audit principles: checking whether cited defects exist, excluding non-applicable criteria, and converting ungrounded LLM preferences into ties. In Appendix C.3, we have further documented:
Annotator Background: Professional annotators from a qualified vendor holding STEM/CS degrees (code generation tasks were evaluated by annotators with CS coursework).
Training & Qualification: All annotators completed formal qualification guidelines and a pilot training round.
Time Metrics: ~310 person-hours spent over two weeks.
Full Double-Blind Re-annotation Results: We re-annotate all 12,236 pairs (100% coverage, not a subset) under strict double-blind conditions (blind to model names, original labels, and LLM judge traces):
Out of 11,105 pairs with definite labels from both runs, only 25 pairs (0.22%) resulted in direct preference inversions. Detailed in Section 3 and Appendix C.4.