- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a simple way to identify potentially defective benchmark questions: expand the answer choices to include “none of the above” and “multiple answers are correct,” then flag questions when all three frontier models disagree with the keyed answer. On MMLU and Humanity’s Last Exam (HLE), the method achieves 72.4% and 84.4% precision, and 61.7% and 65.4% recall, respectively, against independently verified defects. It also recovers subject-level defect profiles and benefits from model diversity. The method is best viewed as a way to prioritize human review, not as an automatic correction system.
Critical strengths
- The paper tackles a practical and consequential problem: benchmark items can be ambiguous, mis-keyed, malformed, or have multiple or no correct answers, while expert verification is costly.
- The proposed signal is simple, and the evaluation reports both precision and recall against verified errors on two benchmarks. The paper also examines answer-space expansion, model diversity, panel size, and earlier model generations.
- The authors appropriately frame the approach as triage. In addition, the Introduction is generally easy to follow: it connects benchmark validity concerns to the proposed screening method and previews the main contributions.
Critical weaknesses
-
Model disagreement is a useful signal but does not necessarily indicate flawed items. The method assumes sufficient model capability and diversity, yet cross-provider models may still share knowledge gaps or correlated errors. Although verified labels support the screening results, the paper provides limited analysis of shared model failures.
-
Item-level screening shows notable false positives and false negatives. On MMLU, approximately 9% of items were verified as problematic, while 72% of flagged items were confirmed. However, many problematic items remained undetected, with similar issues on HLE. The method is therefore more suitable for prioritizing human review, but its cost-effectiveness under fixed review budgets remains unclear.
-
Comparisons with closely related screening methods are insufficient. Prior studies, including Fantastic Bugs and Where to Find Them in AI Benchmarks [1] and Auditing LLM Benchmarks with Item Response Theory [2], report screening precision of up to 84% across nine benchmarks and 95% among the top 200 items using 114 models, respectively. While these results are not directly comparable, the paper lacks evaluations under matched items, budgets, and metrics. Its main direct baseline is limited to an LLM-as-a-Judge panel using the same three models.
Year-over-year trends across unverified benchmarks require caution. Differences in question formats, subjects, difficulty, and model capabilities may affect flagging rates. Thus, higher rates in newer benchmarks do not necessarily imply more actual defects. Although the authors acknowledge this distinction, stronger controls are needed to support broader conclusions about benchmark quality over time.
References
[1] Fantastic Bugs and Where to Find Them in AI Benchmarks. NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/5b44b99d012922011f0be1031a98c4f9-Abstract-Datasets_and_Benchmarks_Track.html
[2] Auditing LLM Benchmarks with Item Response Theory. arXiv, 2026. https://arxiv.org/abs/2605.30504
Questions for authors
-
Could the authors directly compare their method with response-statistics-based screening approaches, such as Fantastic Bugs, and IRT-based methods on a shared question set under a fixed human review budget? Please report precision, recall, the number of items requiring manual review, and per-item inference costs across the full dataset, rather than focusing only on high-confidence top-k results.
-
What types of errors or questions account for most false positives and false negatives on MMLU and HLE? Could the authors report precision and recall by defect category, such as incorrect answer keys, ambiguity, multiple valid answers, and missing correct options, to clarify which benchmark issues the method detects most effectively?
-
The paper reports greater response diversity across models from different providers than from repeated sampling of a single model. Could the authors further examine how often all three models reject the correct answer on verified valid items, and whether this frequency varies across subjects or difficulty levels? Such an analysis would help identify settings where the method should be applied more cautiously.
2 comments
Question 2. Which defects are detected or missed?
We have added a breakdown using the published defect labels in Appendix D.1 (Table 12). On MMLU, the panel detects 72/74 incorrect keys, 18/28 questions with multiple correct answers, and 10/19 with no correct answer. It detects fewer questions labeled with unclear wording or options, at 34/87 and 2/18, respectively. Those two clarity categories account for 69 of the 90 missed defects. This may be partly because, even when the verifiers identify unclear wording or ambiguity in the answer options, the models can still answer the question in agreement with the original benchmark key. These questions are not flagged if at least one of the three models agrees with that key.
On HLE, we checked which types of defects the panel detects using HLE-Verified's labels for the original questions. The panel detects 330/463 questions with an incorrect answer, 8/12 with ambiguity or underspecification, and 10/21 with missing information. HLE-Verified labels problems in the question, answer and rationale separately, so a question can fall into more than one category. We report recall for each category because our method identifies questions for review but does not predict what type of defect they contain; reporting precision by category would require the method to make that additional distinction.
Claims about newer benchmarks.
Thank you for pointing this out. We have clarified the claims about newer benchmarks in the fifth contribution in the abstract, the corresponding contribution in the introduction, and the conclusion. The revised passages read as follows.
In the abstract:
Fifth, we find rising flag rates in newer benchmarks which, while requiring human verification, directly inform prioritization of that human review, and collectively suggest that the current trend to design harder benchmarks may have inadvertently produced rising rates of fallaciousness.
In the introduction:
(5) Applied across benchmarks released over the past six years, the rate of diagnostic flags rises with release date: newer, harder benchmarks may contain more likely-fallacious items, not fewer (Section 4.5). However, these flag rates indicate benchmarks and subjects that may need to be prioritized for human review, rather than verified fallaciousness in these benchmarks.
In the conclusion:
Our results on newer benchmarks, while not evidence of error without human verification, both unlock that prioritized review and directionally suggest that the high-pressure pursuit of benchmark difficulty has increasingly settled for the wrong kind---fallaciousness---which raises urgent questions about whether benchmarking may be leading us astray as a measure of progress in the field.
These clarifications complement the discussion in Section 4.5 of the paper, which states that the flag rates should not be interpreted as estimates of defect prevalence. We hope these clarifications, together with the additional analyses of shared model failures and defect types, address the Reviewer's concerns and will be taken into account in the assessment.
Official response to Reviewer vxi4
We thank the Reviewer for the detailed assessment and for acknowledging the effectiveness of the method, as well as how consequential the problem we are tackling in this work is. We have made many improvements and added many clarifications to the manuscript directly based on the feedback provided. We address the weaknesses and respond to the questions and comments raised below.
Weakness 1 and Question 3. Shared model failures.
The different models used by the method can still miss the same item, and while Table 2 shows extensive experiments on repeated-call and temperature variation, it does not clearly show shared model failures. We have now added a direct check in Appendix A.1 of how often all three models flag questions accepted as sound by independent verifiers. Among these questions scored by all three models, this was the case for 55 of 2,256 MMLU questions and 68 of 571 HLE questions, with 95% confidence intervals of 1.9–3.2% and 9.5–14.8%, respectively.
Weakness 2. False positives, false negatives, and review budgets.
While the method missed some fallacious items and flagged some sound ones, it still identified 136 verified defects among 191 MMLU flags and 376 among 444 HLE flags. This uses fewer models than related methods in the literature; furthermore, we report both precision and recall across the evaluated questions from the two benchmarks and provide the comparison with Fantastic Bugs and the IRT approach in our response to Weakness 3 and Question 1 below. We estimate answer-generation costs of about USD 0.009 per MMLU question and USD 0.049 per HLE question for all three models combined. The only additional cost is grading answers, which adds about USD 0.008. This is very cost-effective and does not require responses from the 40–91 models used by Fantastic Bugs or the 114 used by the IRT approach, or training a model on the dataset as required by other methods.
Weakness 3 and Question 1. Comparison with response-statistics methods.
Our method uses a much simpler review pannel compared with the 40–91 models used by Fantastic Bugs and the 114 used by the IRT approach, while achieving high precision across the evaluated MMLU and HLE questions. Fantastic Bugs reports its highest precision of 84% on GSM8K, while its average across the nine benchmark groups is 32.7%, this performance is based on reviewing the top 50 questions per benchmark. The IRT approach reports 95% precision among its top 200 questions against model-derived labels.
Looking at our highest per-subject precision, our method proposed in this work reaches 100% in MMLU College Chemistry (24/24 flags), Formal Logic (8/8) and Machine Learning (3/3). The main advantage is that we also report both precision and recall across the evaluated questions from both benchmarks, using independent verification labels to measure the defects identified and those missed. Across these questions, precision is 71.2% on MMLU and 84.7% on HLE, with recall of 60.2% and 66.2%, respectively. The table below summarizes the reported results and evaluation scope of each method.