acceptodds
Under review as a conference paper at ICLR 2027

The Wrong Kind of Benchmark Difficulty: Diagnosing Benchmark Errors Using Model Disagreement

Abstract

Current AI benchmarking practice has aimed for harder and harder questions. Yet increasingly, evidence suggests that hard questions may just be fallacious ones. We offer and test the reliability of a remarkably simple diagnostic approach for surfacing such questions: expanding answer choices to allow abstention or multiple correct answers and assessing model disagreement (from keyed answer and convergence on the same non-keyed answer) for frontier models. First, we show that a panel as small as , with full disagreement and any amount of convergence, recovers the per-subject quality profile established by previously published, independent expert audits of two popular benchmarks, Humanity's Last Exam (HLE) and Massive Multitask Language Understanding (MMLU), without per-benchmark tuning. Second, we demonstrate that this simple three-model disagreement signal outperforms prior disagreement-based screening and an LLM-as-judge panel, recovering independently verified errors with 72–84% precision and 61–65% recall. Third, we assess key assumptions, including independence, and provide evidence that model diversity enhances performance more so than panel size. Fourth, we highlight the recency of these diagnostic capabilities by running the same procedure with earlier model generations and tracing improvements, particularly in precision, across successive model generations. Fifth, we show that the current trend to design harder benchmarks may have produced rising rates of fallacious items in newer benchmarks. Our findings have considerable implications for current benchmarking practice, and offer a scalable method for detecting fallacious items.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, AI Evaluation, Benchmark Validity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vxi4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a simple way to identify potentially defective benchmark questions: expand the answer choices to include “none of the above” and “multiple answers are correct,” then flag questions when all three frontier models disagree with the keyed answer. On MMLU and Humanity’s Last Exam (HLE), the method achieves 72.4% and 84.4% precision, and 61.7% and 65.4% recall, respectively, against independently verified defects. It also recovers subject-level defect profiles and benefits from model diversity. The method is best viewed as a way to prioritize human review, not as an automatic correction system.

Critical strengths
  1. The paper tackles a practical and consequential problem: benchmark items can be ambiguous, mis-keyed, malformed, or have multiple or no correct answers, while expert verification is costly.
  2. The proposed signal is simple, and the evaluation reports both precision and recall against verified errors on two benchmarks. The paper also examines answer-space expansion, model diversity, panel size, and earlier model generations.
  3. The authors appropriately frame the approach as triage. In addition, the Introduction is generally easy to follow: it connects benchmark validity concerns to the proposed screening method and previews the main contributions.
Critical weaknesses
  1. Model disagreement is a useful signal but does not necessarily indicate flawed items. The method assumes sufficient model capability and diversity, yet cross-provider models may still share knowledge gaps or correlated errors. Although verified labels support the screening results, the paper provides limited analysis of shared model failures.

  2. Item-level screening shows notable false positives and false negatives. On MMLU, approximately 9% of items were verified as problematic, while 72% of flagged items were confirmed. However, many problematic items remained undetected, with similar issues on HLE. The method is therefore more suitable for prioritizing human review, but its cost-effectiveness under fixed review budgets remains unclear.

  3. Comparisons with closely related screening methods are insufficient. Prior studies, including Fantastic Bugs and Where to Find Them in AI Benchmarks [1] and Auditing LLM Benchmarks with Item Response Theory [2], report screening precision of up to 84% across nine benchmarks and 95% among the top 200 items using 114 models, respectively. While these results are not directly comparable, the paper lacks evaluations under matched items, budgets, and metrics. Its main direct baseline is limited to an LLM-as-a-Judge panel using the same three models.

Year-over-year trends across unverified benchmarks require caution. Differences in question formats, subjects, difficulty, and model capabilities may affect flagging rates. Thus, higher rates in newer benchmarks do not necessarily imply more actual defects. Although the authors acknowledge this distinction, stronger controls are needed to support broader conclusions about benchmark quality over time.

References

[1] Fantastic Bugs and Where to Find Them in AI Benchmarks. NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/5b44b99d012922011f0be1031a98c4f9-Abstract-Datasets_and_Benchmarks_Track.html

[2] Auditing LLM Benchmarks with Item Response Theory. arXiv, 2026. https://arxiv.org/abs/2605.30504

Questions for authors
  1. Could the authors directly compare their method with response-statistics-based screening approaches, such as Fantastic Bugs, and IRT-based methods on a shared question set under a fixed human review budget? Please report precision, recall, the number of items requiring manual review, and per-item inference costs across the full dataset, rather than focusing only on high-confidence top-k results.

  2. What types of errors or questions account for most false positives and false negatives on MMLU and HLE? Could the authors report precision and recall by defect category, such as incorrect answer keys, ambiguity, multiple valid answers, and missing correct options, to clarify which benchmark issues the method detects most effectively?

  3. The paper reports greater response diversity across models from different providers than from repeated sampling of a single model. Could the authors further examine how often all three models reject the correct answer on verified valid items, and whether this frequency varies across subjects or difficulty levels? Such an analysis would help identify settings where the method should be applied more cautiously.

2 comments
AuthorsOpenReview ↗

Official response to Reviewer vxi4

We thank the Reviewer for the detailed assessment and for acknowledging the effectiveness of the method, as well as how consequential the problem we are tackling in this work is. We have made many improvements and added many clarifications to the manuscript directly based on the feedback provided. We address the weaknesses and respond to the questions and comments raised below.

Weakness 1 and Question 3. Shared model failures.

The different models used by the method can still miss the same item, and while Table 2 shows extensive experiments on repeated-call and temperature variation, it does not clearly show shared model failures. We have now added a direct check in Appendix A.1 of how often all three models flag questions accepted as sound by independent verifiers. Among these questions scored by all three models, this was the case for 55 of 2,256 MMLU questions and 68 of 571 HLE questions, with 95% confidence intervals of 1.9–3.2% and 9.5–14.8%, respectively.

Weakness 2. False positives, false negatives, and review budgets.

While the method missed some fallacious items and flagged some sound ones, it still identified 136 verified defects among 191 MMLU flags and 376 among 444 HLE flags. This uses fewer models than related methods in the literature; furthermore, we report both precision and recall across the evaluated questions from the two benchmarks and provide the comparison with Fantastic Bugs and the IRT approach in our response to Weakness 3 and Question 1 below. We estimate answer-generation costs of about USD 0.009 per MMLU question and USD 0.049 per HLE question for all three models combined. The only additional cost is grading answers, which adds about USD 0.008. This is very cost-effective and does not require responses from the 40–91 models used by Fantastic Bugs or the 114 used by the IRT approach, or training a model on the dataset as required by other methods.

Weakness 3 and Question 1. Comparison with response-statistics methods.

Our method uses a much simpler review pannel compared with the 40–91 models used by Fantastic Bugs and the 114 used by the IRT approach, while achieving high precision across the evaluated MMLU and HLE questions. Fantastic Bugs reports its highest precision of 84% on GSM8K, while its average across the nine benchmark groups is 32.7%, this performance is based on reviewing the top 50 questions per benchmark. The IRT approach reports 95% precision among its top 200 questions against model-derived labels.

Looking at our highest per-subject precision, our method proposed in this work reaches 100% in MMLU College Chemistry (24/24 flags), Formal Logic (8/8) and Machine Learning (3/3). The main advantage is that we also report both precision and recall across the evaluated questions from both benchmarks, using independent verification labels to measure the defects identified and those missed. Across these questions, precision is 71.2% on MMLU and 84.7% on HLE, with recall of 60.2% and 66.2%, respectively. The table below summarizes the reported results and evaluation scope of each method.

MethodModelsReported precision
Fantastic Bugs40–9132.7% average across nine benchmark groups; highest 84% on GSM8K
IRT approach11495% among the top 200 questions
Our method371.2% on MMLU; 84.7% on HLE; 100% in several subjects
MethodEvaluation
Fantastic BugsHuman review of the top 50 questions per benchmark; additional recall analysis on GSM8K
IRT approachModel-aggregated reference labels
Our methodPrecision and recall against independent verification labels across 2,482 scored MMLU questions and 1,139 scored HLE questions

Question 2. Which defects are detected or missed?

We have added a breakdown using the published defect labels in Appendix D.1 (Table 12). On MMLU, the panel detects 72/74 incorrect keys, 18/28 questions with multiple correct answers, and 10/19 with no correct answer. It detects fewer questions labeled with unclear wording or options, at 34/87 and 2/18, respectively. Those two clarity categories account for 69 of the 90 missed defects. This may be partly because, even when the verifiers identify unclear wording or ambiguity in the answer options, the models can still answer the question in agreement with the original benchmark key. These questions are not flagged if at least one of the three models agrees with that key.

On HLE, we checked which types of defects the panel detects using HLE-Verified's labels for the original questions. The panel detects 330/463 questions with an incorrect answer, 8/12 with ambiguity or underspecification, and 10/21 with missing information. HLE-Verified labels problems in the question, answer and rationale separately, so a question can fall into more than one category. We report recall for each category because our method identifies questions for review but does not predict what type of defect they contain; reporting precision by category would require the method to make that additional distinction.

Claims about newer benchmarks.

Thank you for pointing this out. We have clarified the claims about newer benchmarks in the fifth contribution in the abstract, the corresponding contribution in the introduction, and the conclusion. The revised passages read as follows.

In the abstract:

Fifth, we find rising flag rates in newer benchmarks which, while requiring human verification, directly inform prioritization of that human review, and collectively suggest that the current trend to design harder benchmarks may have inadvertently produced rising rates of fallaciousness.

In the introduction:

(5) Applied across benchmarks released over the past six years, the rate of diagnostic flags rises with release date: newer, harder benchmarks may contain more likely-fallacious items, not fewer (Section 4.5). However, these flag rates indicate benchmarks and subjects that may need to be prioritized for human review, rather than verified fallaciousness in these benchmarks.

In the conclusion:

Our results on newer benchmarks, while not evidence of error without human verification, both unlock that prioritized review and directionally suggest that the high-pressure pursuit of benchmark difficulty has increasingly settled for the wrong kind---fallaciousness---which raises urgent questions about whether benchmarking may be leading us astray as a measure of progress in the field.

These clarifications complement the discussion in Section 4.5 of the paper, which states that the flag rates should not be interpreted as estimates of defect prevalence. We hope these clarifications, together with the additional analyses of shared model failures and defect types, address the Reviewer's concerns and will be taken into account in the assessment.

Reviewer_fhSEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a simple benchmark-error screening rule: query three strong models from different providers, allow NONE/MULTIPLE answers where appropriate, and flag items for which every model rejects the published key. It evaluates precision and recall against existing MMLU-Redux and HLE-Verified annotations, reporting 72.4%/61.7% and 84.4%/65.4%, respectively. Additional experiments vary disagreement/convergence thresholds, provider diversity, repeated sampling, model generation, and direct GOOD/BAD judging. The method is also applied to five benchmarks without complete independent verification to prioritize human review. The central empirical contribution is a small-panel triage procedure, with broader arguments about the relationship between benchmark difficulty, release date, and defective questions.

Critical strengths
  1. The method is simple, interpretable, and potentially practical for prioritizing expert review. Evaluating against independently published item-level annotations, and reporting recall as well as precision, is more informative than validating only a handful of top-ranked examples. The expanded answer-space ablation is particularly useful: the reported recall gains on multiple-choice MMLU and HLE indicate that forcing a choice among the original options can hide useful diagnostic information.

  2. The study examines several relevant alternatives and failure modes: single/two/three-provider panels, repeated calls and temperatures, older model panels, convergence thresholds, and a direct LLM-as-judge baseline. It also appropriately keeps the author-led re-review separate from the fixed-reference headline evaluation. These analyses support the practical value of the screening signal while showing that stricter convergence can sacrifice substantial recall and does not consistently improve precision.

Critical weaknesses
  1. The measurement pipeline underlying the main HLE result is not specified well enough to assess or reproduce it. Of the 1,147 HLE items, 854 are open-form. Appendix A.2 says disagreement with the reference and convergence across models are judged by semantic equivalence. The paper does not identify the equivalence judge, provide its complete prompt/rubric and reference-access policy, or validate its decisions against a blinded human sample. Those decisions directly determine whether an item is flagged. The main answer-selection prompt, decoding/reasoning settings, per-item outputs, and scoring implementation are also unavailable in the submission, despite the reproducibility statement's claim of complete prompts. This affects the comparison with the three-call GOOD/BAD baseline as well, any additional semantic-grading calls or human effort should be included in the budget. There is also an answer-expansion ambiguity, Figure 10 shows NONE but no MULTIPLE even though its caption says two original options are correct. The actual expansion/deduplication rule needs clarification.

  2. The stronger conclusions do not follow from the validated screening results. Figure 3 compares flag rates across benchmarks with different domains, answer formats, genuine difficulty, and public-subset selection. It can't establish that newer/harder benchmarks have increasing defect prevalence. Section 4.5 correctly acknowledges this, but the abstract's fifth contribution and conclusion return to rising fallaciousness. Representative expert audits of flagged and unflagged cases, or narrower claims, are needed. Similarly, above-chance marginal competence and lower cross-provider agreement do not establish that joint errors on sound items are negligible, shared item difficulty and correlated mistakes remain alternative explanations. The selected MMLU subject strata, exclusion of HLE's Uncertain subset, and pooled subject correlation further limit transport to new benchmarks. Finally, rejecting memorized original keys does not rule out exposure to publicly released corrections, so the historical gains cannot be attributed uniquely to improved reasoning capability.

Questions for authors
  1. Please provide the exact answer-selection and semantic-equivalence protocols, judge identity, inference settings, item-level predictions, and scoring code. How accurate is equivalence grading on a blinded human sample, and do the HLE results persist under an alternative grader? Was MULTIPLE actually available for the Figure 10 item, and what deduplication rule was used?

  2. Can you report complete-response coverage by benchmark/class/model, confidence intervals for the paired method comparisons, empirical joint error on verified sound items, and within-benchmark subject correlations? These would clarify the effect of missing responses, shared item difficulty, and between-benchmark differences.

  3. Can the conclusions about rising benchmark defects and contamination robustness be restricted to what is established, or supported by representative blinded audits and a test less exposed to public corrections? The practical triage contribution would remain useful with narrower claims.

4 comments
AuthorsOpenReview ↗

Official response to Reviewer fhSE

We thank the Reviewer for the careful assessment and for proposing these very helpful additional analyses. We address the weaknesses and questions below.

Weakness 1 and Question 1. Semantic grading and reproducibility.

The grading pipeline we implement is very simple, as we use the benchmark's original answer and compare it to the answer given by each model, and use Claude Sonnet 4.6 to judge whether the model's extracted final answer is equivalent to it. This comparison allows differences in wording and formatting; for cases where the final answer is $4, this allows us to catch variations like 4 or four dollars, which is especially important for open-form questions. We manually verified the grading on a random sample of 200 questions from each benchmark without seeing Sonnet's verdicts first, and found 100% agreement with the human judgments in both samples. Based on these checks, we found the semantic equivalence task to be simple enough for the model and did not see the need to add another grader.

We have included the exact grading and answer-selection prompts in Appendix A.3, and attached the results as JSON files together with a summary spreadsheet in the supplementary material. The JSON files provide results for each question, including model selections and correctness judgments for open-form answers.

For the comparison with the direct GOOD/BAD judges, we used the same number of calls to the three panel models, which is what is matched in this experiment. This does not mean that the total computation is the same, as we still need the additional semantic grading calls for the open-form answers, and further calls for the convergence comparisons, which need to be counted separately from the three answer-generation calls when estimating the total cost.

MULTIPLE was available to the models for the question in Figure 10, but the option was missed in the figure, which we have updated to show the full set of choices. The actual MMLU prompt included both E, “More than one of the above answers is correct,” and F, “None of the above answers is correct.” The original option “All of these” does not replace MULTIPLE, since more than one original option can be correct without all of them being correct. For HLE multiple-choice questions, the preprocessing checks for existing multiple-answer and no-answer options and appends only those it identifies as absent. We have clarified these procedures in Appendix A.3. The recorded model answers remain unchanged.

Weakness 2 and Question 3. Claims about newer benchmarks and contamination.

Thank you for pointing this out. We have made further clarifications in the fifth contribution in the abstract as follows:

Fifth, we find rising flag rates in newer benchmarks which, while requiring human verification, directly inform prioritization of that human review, and collectively suggest that the current trend to design harder benchmarks may have inadvertently produced rising rates of fallaciousness.

We have made corresponding edits in the introduction and the conclusion as well. These clarifications complement the discussion in Section 4.5, which states that the flag rates should not be interpreted as estimates of defect prevalence. Our conclusions about performance are based on the selected MMLU and HLE questions with independently verified labels, while the higher flag rates on the additional benchmarks indicate that there are more questions to review.

We agree with the Reviewer that a model may have seen a published correction, in which case it could reject the benchmark's original answer key for that reason. We have added this clarification to the “Training-data contamination” paragraph in Appendix C:

However, a model may also have seen a publicly released correction and could reject the original labeled answer for that reason.

We have also replaced the paragraph's final statement with:

So the higher precision we get with the newer panels in the historical comparison cannot be attributed just to better reasoning, and does not show that the method is unaffected by contamination.

Question 2. Coverage, shared errors, and subject correlations.

We are grateful for these suggestions. We have separated the coverage by benchmark, question format, and model in Table 3 in Appendix A.1, reproduced below. The table gives the number of questions with usable answers from each model in the main panel, alongside the total number of questions and the number with usable answers from all three models.

Benchmark and formatTotal questionsClaudeGeminiGPTAll three
MMLU, multiple choice2,4892,4832,4882,4892,482
HLE, multiple choice293292293293292
HLE, open form854847853854847

Only seven MMLU questions and eight HLE questions lack a usable answer from at least one model in the main panel. We exclude these questions from both precision and recall, and Table 2 is restricted further to the common set with all 15 responses scored.

We have also added paired bootstrap confidence intervals for the panel comparisons in Table 2 to Appendix A.1, using 100,000 resamples of the same questions for each panel. The three-model panel has higher precision than every other panel on both benchmarks after adjusting for multiple comparisons (all adjusted p<0.004p<0.004). Compared with the two-model pairs, its precision gains range from 4.25 to 12.27 percentage points on MMLU and 3.01 to 6.79 on HLE. The recall differences against the two-model pairs are not significant on MMLU; on HLE, the three-model panel has lower recall than Claude + GPT and Gemini + GPT after correction, while the difference from Claude + Gemini is not significant.

We have added the joint error results to Appendix A.1, where all three models rejected 55 of 2,256 MMLU questions (2.4%) and 68 of 571 HLE questions (11.9%) accepted as sound by independent verifiers and scored by all three models. These shared errors, especially on HLE, show why the competence and agreement measurements alone cannot establish that the models' errors are independent or that all three are unlikely to be wrong on the same question.

We also repeated the correlation analysis separately for MMLU and HLE, to check that the relationship in Figure 1 still holds when the two are not pooled. With the same weights for subject size, we get Pearson correlations of 0.92 for MMLU and 0.73 for HLE, and Spearman correlations of 0.85 and 0.75, respectively. So subjects with higher verified defect rates also tend to have higher flag rates within each benchmark, with a weaker relationship on HLE. We have added these results under “Within-benchmark subject correlations” in Appendix A.1.

We thank the Reviewer again for acknowledging the simplicity and interpretability of our method, as well as the thoroughness of our experiments. We have added several clarifications and additional analyses based on this feedback. We hope that these clarifications, together with the additional analysis and the corrected figure, address the Reviewer's concerns and will be taken into account in the assessment.

Thank you for clarifying the semantic-grading pipeline, adding prompts and outputs, and reporting the blinded grading check. These substantially address my reproducibility concern. However, higher flag rates still do not establish rising benchmark defect rates, and shared model errors remain a plausible explanation. I will maintain my current rating.

Thank you for acknowledging these additions. We would like to clarify that the main contribution is a simple three-model diagnostic approach that identifies benchmark errors with 71.2% precision on MMLU and 84.7% on HLE, while recovering 60.2% and 66.2% of independently verified defects, respectively. These results use only three frontier models, compared with the 40–91 models used by Fantastic Bugs [1], the 114 models used by the IRT approach [2], or the 50 answers per question generated by Beyond the Singular [3], without requiring training on the target dataset as in Dataset Cartography [4]. Our evaluation reports both false positives and false negatives against independently verified labels across the evaluated questions. This provides a broader assessment than precision among selected flags, such as the top 50 questions per benchmark reviewed in Fantastic Bugs [1] or the top 200 used for the IRT study's headline precision result, which is evaluated against model-derived reference labels [2].

We have also conducted extensive analyses of model diversity, repeated sampling, shared failures, and the types of defects detected or missed to show when this approach works and where its limitations lie. We appreciate your continued engagement and your acknowledgment of the improvements, and hope this evidence of the approach’s practical value and its advantages over prior methods will be taken into account in the final assessment, as this is the main contribution of the work, independently of whether defect rates are rising in newer benchmarks.

References

[1] Sang Truong, Yuheng Tu, Michael Hardy, Anka Reuel-Lamparth, Zeyu Tang, Jirayu Burapacheep, Jonathan Perera, Chibuike Uwakwe, Benjamin Domingue, Nick Haber, and Sanmi Koyejo. 2025. Fantastic Bugs and Where to Find Them in AI Benchmarks. Advances in Neural Information Processing Systems, 38.

[2] Sander Land and Daniel M. Bikel. 2026. Auditing LLM Benchmarks with Item Response Theory. Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pp. 18016–18039.

[3] Wenbo Zhang, Hengrui Cai, and Wenyu Chen. 2026. Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation. Findings of the Association for Computational Linguistics: ACL 2026, pp. 10033–10043.

[4] Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 9275–9293.

Reviewer_aXnUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for detecting benchmark errors. The method expands multiple-choice answer sets with NONE and MULTIPLE, then flags an error task when three frontier models disagree with its reference answer. The authors evaluate this method on MMLU-Redux and HLE-Verified.

Critical strengths

The method is simple and empirically effective in two benchmarks.

Critical weaknesses
  1. My main concern is the limited range of tasks studied. MMLU has fixed answer choices, while open-ended HLE questions are evaluated through semantic equivalence with a reference answer. It is unclear how the method would extend to long-horizon tasks such as coding or deep research, where comparing final answers alone may not capture correctness. Although the paper acknowledges this limitation, it still restricts the method’s practical impact, particularly as LLMs are increasingly used for agentic tasks that produce lengthy reports or changes to an entire codebase rather than a single answer.

  2. Another issue is that the related work omits relevant label-error detection studies. For example, Dataset Cartography [1] uses confidence and variability across training epochs to identify potentially mislabeled examples, while Beyond the Singular [2] uses repeated generations and semantic consistency to find benchmark errors. Discussing these studies would help clarify what cross-model disagreement adds to existing approaches.

[1] Swayamdipta, Swabha, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. “Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics.” EMNLP 2020.

[2] Zhang, Wenbo, Hengrui Cai, and Wenyu Chen. “Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation.” Findings of the Association for Computational Linguistics: ACL 2026.

Questions for authors
  1. How could the method be extended to coding or general search tasks, where comparing final answers with a reference may not be sufficient to determine correctness?

  2. In Appendix Table 2, why is HLE recall higher for the three-model panel (65.6%) than for Claude–Gemini (57.6%)? I think that if the same responses and scoring rules are used, requiring all three models to reject the reference should not increase recall.

2 comments
AuthorsOpenReview ↗

Official response to Reviewer aXnU

We thank the Reviewer for the insightful review and for highlighting the simplicity and empirical effectiveness of the method. Below, we address the weaknesses and questions raised in the review.

Weakness 1 and Question 1. Scope and extension to coding or research tasks.

As noted in the review, the method is effective on multiple-choice questions and open-form questions with answers that can be evaluated through semantic equivalence, which still account for a large portion of benchmarks used by model developers and AI evaluators. We agree that it does not apply directly to long-horizon tasks where the expected output includes changes to an entire codebase or more complex research reports for which the user might not have a ground truth.

We have explicitly highlighted this limitation in the Discussion and clarified that the problem this paper addresses is that many of the "difficult" benchmarks include questions with an incorrect key, no correct option, or multiple correct options. Tasks without a fixed answer key are unlikely to have these particular problems, although their specifications and evaluation criteria can also be flawed. Extending the method to identify such flaws would be an interesting direction for future work. We have added this clarification to the Limitations section.

Weakness 2. Related work.

Thank you for pointing out these very relevant, and closely related works. We have added both to the Related Work section and clarified how they differ from the proposed method.

For Dataset Cartography, we have added the following discussion.

Swayamdipta et al. (2020) check how a model learns each example during training and use low confidence in the assigned label to flag possible labeling errors. The proposed method trains a model on the target dataset and records its label probabilities across training epochs, which can be computationally expensive for large open-source models and is not feasible for proprietary models. Our method instead uses already-trained models and does not require access to training data or label probabilities.

For Beyond the Singular, we have added the following discussion.

Zhang et al. (2026) generate 50 answers per question from a single model and flag questions with high answer consistency but no more than 10% agreement with the reference. Their manual review covers 18 flagged questions, and their approach also relies heavily on the model used, whereas our approach uses three models from different providers.

Our method reports both precision and recall against independently verified labels and uses three models from different providers to mitigate reliance on any single model. Our detailed ablations show that this approach is much more effective than repeated runs from the same model, even with varying temperatures.

Question 2. HLE recall in Table 2.

Thank you for pointing this out. In the previous Table 2, the three-model result incorrectly included questions flagged in the repeated-response experiments shown in the second half of the table. The pair results did not include those additional responses, which is why the three-model recall appeared higher. This only affected the recall calculation in Appendix Table 2 and did not affect the main results. To correct this, we made sure that the single-model, pair, and three-model results all use only the first answer from each model for each item, with the repeated-response results kept separate. We have attached all results in the supplementary material.

We have also extended the experiment beyond the original 1,000-question sample to the questions evaluated on both benchmarks. The updated Table 2 compares all panels on the same 2,476 MMLU questions and 1,135 HLE questions with all 15 responses scored. The updated results are shown below.

PanelMMLU precisionMMLU recallHLE precisionHLE recall
Claude59.7%61.5%76.6%69.8%
Gemini53.4%61.9%69.3%79.3%
GPT41.9%63.7%69.0%78.6%
Claude + Gemini67.3%60.2%81.6%66.6%
Claude + GPT67.3%61.1%81.1%67.5%
Gemini + GPT59.3%60.6%77.8%70.7%
All three71.6%60.2%84.6%66.1%

Gemini alone has the highest HLE recall at 79.3%, but its precision is substantially lower at 69.3%, compared with 84.6% for the three-model panel. Using three models gives a more precise set of questions for review, while missing some defects that individual models identify.

We hope that clarifying the method's limitations for long-horizon tasks and directions for future work, adding the related work, and extending the experiments in Table 2 address the concerns raised by the Reviewer and will be taken into account in the final assessment of the paper. We would be happy to address any further issues.

Thank you for the detailed response. These responses address my concerns, and I will raise my score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Response

We thank the Reviewers for their thoughtful and detailed feedback. We have addressed the points raised in the reviews through additional experiments and clarifications, which we summarize below.

Reviewer aXnU

The Reviewer raised a good point about extending this work to long-horizon tasks such as coding or deep research. We focus on benchmarks with reference answers, either multiple-choice answers or open-form answers that can be checked through semantic equivalence, as we do for HLE. We have tested both formats and found the approach effective. Extending this to coding or deep research is a much more difficult and separate task, and none of the approaches in our related work covers both [1–4]. Many of the issues we study, including an incorrect key, no correct option or multiple correct options, are also unlikely to be present where there is no fixed answer key. We have made this scope clearer in the paper and discussed extending the approach to long-horizon coding and deep research as future work.

We have added Dataset Cartography [1] and Beyond the Singular [2] to the related work and outlined the differences from our approach. One requires training a model on the dataset, and the other generates 50 answers per question from the same model, whereas we use three models and report both precision and recall against independent labels. We have also corrected the recall calculation in Appendix Table 2 and extended this additional experiment from the 1,000-question sample to all questions selected for the study.

Reviewer fhSE

The Reviewer raised several helpful questions about the grading pipeline, for which we have provided the exact prompts in the appendix and the results in the supplementary material, reported the blinded human check on both benchmarks, and specified the information given to the grader. The response coverage is now in the appendix. The Reviewer acknowledged that these additions substantially address the reproducibility concern.

The remaining concern was that higher flag rates do not establish higher defect rates. We have clarified the abstract:

Fifth, we find rising flag rates in newer benchmarks which, while requiring human verification, directly inform prioritization of that human review, and collectively suggest that the current trend to design harder benchmarks may have inadvertently produced rising rates of fallaciousness.

We made similar clarifications in the introduction and conclusion, consistent with Section 4.5, and provided the flagged questions in newer benchmarks for expert review in the supplementary material. This analysis of newer benchmarks is an additional application and the fifth contribution of the paper, while the main contribution remains the diagnostic approach itself and its demonstrated performance against independently verified labels on MMLU and HLE.

Reviewer vxi4

The Reviewer asked about shared model failures, and we have now checked how often the three models all reject a question accepted as sound by independent verifiers. We have added this to Appendix A.1 with the uncertainty intervals, and provided a breakdown of detected and missed defects by type in Appendix D.1. For the questions about costs and the comparison with previous work, we have provided the estimates and a more detailed comparison with Fantastic Bugs [3] and the IRT approach [4]. Our approach uses only three models and achieves high precision while measuring both detected and missed defects against independently verified labels. This gives a more complete picture of performance than precision among the top 50 or 200 flagged questions, without requiring the 40–91 models used by Fantastic Bugs or the 114 used by the IRT approach.

For the panel comparisons, we have added the paired bootstrap confidence intervals to Appendix A.1. We have also repeated the correlation analysis for MMLU and HLE separately, since the Reviewer asked whether the relationship would hold without pooling the two. With the extended Table 2 experiments, we now have more evidence on the benefits of model diversity and the cases where the models still make the same mistake.

We remain available to answer any final questions before the discussion period closes. We thank all three Reviewers for their detailed feedback and engagement, and for taking these improvements into account in the final assessment.

References

[1] Swabha Swayamdipta et al. 2020. Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. Proceedings of EMNLP.

[2] Wenbo Zhang, Hengrui Cai, and Wenyu Chen. 2026. Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation. Findings of ACL.

[3] Sang Truong et al. 2025. Fantastic Bugs and Where to Find Them in AI Benchmarks. Advances in Neural Information Processing Systems, 38.

[4] Sander Land and Daniel M. Bikel. 2026. Auditing LLM Benchmarks with Item Response Theory. Proceedings of EMNLP.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.