acceptodds
Under review as a conference paper at ICLR 2027

Accepted but Invalid: When Reasoning Benchmarks Credit Wrong Answers

Abstract

When a reasoning benchmark marks an answer as correct, we usually assume that the submitted answer actually satisfies the task. We show that this assumption can break, even while the evaluator runs exactly as implemented. In MathConstruct, for instance, an archived model answer repeats an index in a required permutation, yet the released checker accepts it and awards credit. To see how far the problem extends, we craft small, explicit violations of the written task requirements and pass them through the released evaluation code. Across seven MathConstruct tasks, the released checkers accept all 127 applicable invalid constructions in our fixed cross-instance test, and the erroneous acceptance of five invalid archived model answers has already entered the task-weighted benchmark scores. DCP-Bench-Open reveals a further and subtler failure: omitted answer fields are left free for the evaluator’s solver to complete. In the tested release, the evaluator accepts an empty JSON object for all 164 problems in our controlled completeness test, and accepts 735/735 incomplete outputs overall. Together, these findings show two ways an evaluator can award credit for a task the model never completed, i.e., by checking too little of the submitted answer, or by completing part of the answer itself.

Primary area:
datasets and benchmarks
Keywords:
large language model evaluation, reasoning benchmarks, benchmark reliability, evaluator soundness, constructive reasoning, benchmark auditing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ekF9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper audits the evaluators of constructive reasoning benchmarks to test whether they credit answers that violate the task requirements. Starting from a valid answer, it makes a minimal change that breaks one explicitly stated requirement and passes the result through the officially released evaluator. On MathConstruct, the checkers of seven tasks miss stated conditions (e.g., distinctness, ordering, positivity); across further official instances, all 127 applicable invalid constructions are accepted and all are rejected once the missing checks are added; five archived model answers that violate requirements had entered published scores, each of which drops by about 0.2 percentage points after correction. On DCP-Bench-Open v0.2.2, the evaluator fixes only the variables present in the answer and lets the solver choose the rest, so an empty JSON object is accepted on all 164 problems and all 735 incomplete outputs are accepted; a field check rejects all of them. The authors recommend testing evaluators with answers that should fail.

Critical strengths

The counterexamples are concrete and reproducible, and one evaluator flaw has broad impact. Each MathConstruct counterexample is read back by the official parser, accepted by the official checker, rejected once the missing condition is added while valid answers remain accepted, and persists in the maintained revision. The DCP-Bench-Open evaluator completes incomplete answers itself, so an empty JSON object is credited on all 164 problems, which directly affects anyone evaluating or training with this benchmark.

Critical weaknesses
  1. The method is standard, and the contribution is mainly specific flaws in two benchmarks. Testing a program with inputs that should fail is standard practice in property-based and mutation testing, and EvalPlus and UTBoost, cited by the paper, have already shown in code evaluation that insufficient tests let wrong answers pass. This work applies the same check to constructive math and constraint-modeling benchmarks and yields a set of specific flaws rather than a new method or a generalizable conclusion.

This determines my assessment of the contribution: the findings are well suited as issue reports to the benchmark maintainers or as a short paper in a benchmarks and datasets track, but their information gain as a main-conference paper is limited.

  1. The coverage is narrow and the confirmed real-world impact is small. Only two benchmarks are audited, both of which are recent and of limited adoption (MathConstruct appeared at ICML 2025 and DCP-Bench-Open is currently a preprint), unlike the widely used HumanEval audited by EvalPlus; only 10 of the 121 MathConstruct checkers are manually inspected, and the 127/127 cross-instance result shows that known failures recur on other instances rather than how many other failures exist. Only five answers are confirmed to have entered historical scores, each lowering the corresponding score by about 0.2 percentage points; the authors acknowledge that this is insufficient to assess ranking changes and that the trigger frequency in general model outputs cannot be estimated.

This affects my assessment of significance: the paper shows that the flaws exist, but not how common such problems are across reasoning benchmarks or how much they affect existing conclusions.

Questions for authors
  1. Could the audit procedure be turned into a reusable tool and applied systematically to all 121 MathConstruct checkers and to further reasoning benchmarks with executable evaluators, reporting how many missing checks are found?

  2. Have these issues been reported to the benchmark maintainers, and has the field-completion behavior of DCP-Bench-Open been fixed in releases after v0.2.2?

2 comments

Thanks for the detailed review. In short: yes, we built the tool and ran it on all 121 checkers (38 of them, 31%, accept an answer that violates the task), and DCP-Bench-Open has since fixed the field-completion issue on its own. Details below.

Q1 (reusable tool, all 121 checkers, more benchmarks). The tool only needs a task's official solution, parser and checker, so it runs on every task without task-specific code. It makes 12 generic edits to the official solution (e.g. duplicating, deleting or swapping list elements, negating or zeroing numbers) and sends each edited answer through the official parser and checker. A finding counts only if the checker accepts a concrete answer and a separate check written from the task statement rejects it.

On all 121 MathConstruct checkers (62,924 edited answers) this gives 36 findings in 33 tasks, 31 of them new, so 38 of 121 including the paper's 7. On all 50 ComBench construction tasks it finds nothing beyond the case in App. I.

W1 (method is standard). We agree, and we don't claim a new testing method. The difference from EvalPlus/UTBoost is the ground truth: they test a program against a reference implementation, while we test the evaluator itself against the written task statement. Some bugs only exist in evaluators, e.g. float tolerance on exact tasks, counting after set(), or the evaluator filling in missing fields. What we add is the measurement: 31% of MathConstruct checkers vs. 2% of ComBench's accept a wrong answer.

W2 (impact). Re-checking all 24,875 archived model outputs, 30 wrong answers received credit (5 in the paper), and 14 main-collection scores drop by 0.2-0.6 points (5 in the paper). Rankings within the main collections don't change, and we will say so clearly.

We think the per-task view matters more in practice. If a checker is used as an RL reward or a data filter, a wrong answer that passes is what the model learns from, and the archive shows o1 and o3-mini-high already produce such answers.

Q2 (maintainers, DCP after v0.2.2). DCP-Bench-Open fixed field completion in v0.3.0 (released 2026-09-29) with an exact key/shape/type check, essentially our proposed fix. They did this independently; we had not contacted them. The diamond-free issue in App. G is still present. We will report the MathConstruct findings with patches after the anonymity period.

If this addresses your concerns, we would appreciate it if you could consider updating your score.

Thank you for the additional experiments. I appreciate the practical contribution of the reusable audit tool, the expanded coverage, and the proposed fixes. The audit relies on standard mutation testing, and the confirmed impact on historical scores remains small, with no changes to rankings within the evaluated collections. These considerations lead me to maintain my rating.

Reviewer_jcP4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether evaluators in constructive-reasoning benchmarks accept answers that violate written requirements. The authors define an acceptance contract across answer selection, parsing or binding, and predicate checking, and audit released evaluation paths using parser-reachable mutations and controlled deletions. In MathConstruct, seven released checkers accept all 127 applicable invalid constructions, and five invalid model outputs in the archive enter task-weighted scores. In DCP-Bench-Open v0.2.2, the released evaluator leaves omitted fields free for its solver to complete and accepts all 735 incomplete outputs, including an empty JSON object for all 164 problems, before repairs across both benchmarks reject these violations and retain the valid controls.

Critical strengths
  • The audit identifies two failure stages in constructive-reasoning benchmarks, as seven released MathConstruct checkers accept all 127 applicable invalid constructions because of missing checks and the released DCP-Bench-Open v0.2.2 evaluator accepts all 735 incomplete outputs, including an empty JSON object for all 164 problems, because its binding stage leaves omitted fields free for the solver to complete.
  • The historical MathConstruct analysis verifies five accepted model outputs across Flash-Thinking, GPT-4o-mini, QwQ, and o1-mini that violate task requirements, and correcting these false accepts lowers the task-weighted archive scores by 0.1969 to 0.1984 percentage points.
Critical weaknesses
  • The MathConstruct audit covers only a small subset of tasks without explaining how those tasks were selected. Section 4.1 manually inspects only 10 tasks out of 121 checkers in the MathConstruct snapshot without naming the 3 inspected tasks that had no confirmed failures.
  • The paper provides limited evidence that model outputs trigger the seven source-audit failures in MathConstruct. In Section 4.3, 1 of the 5 historical false accepts comes from the seven source-audit omissions, while the 4 EMC cases come from a separate distinct-number omission found in the historical records.
Questions for authors
  • How were the 10 manually inspected tasks out of the 121 checkers in MathConstruct chosen, which 3 inspected tasks had no confirmed failures, and is it possible to expand the manual inspection to a larger subset of the remaining 111 checkers?
  • Can authors provide more evidence that model outputs trigger the other six source-audit omissions in MathConstruct, for example by checking the raw-only archive records in Appendix H.1?
Reviewer_35yDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims that multiple reasoning benchmarks award correctness to answers that do not actually satisfy the task requirements. An audit of MathConstruct revealed that its released checkers accepted all 127 tested invalid constructions, meaning incorrect archived model answers have already falsely inflated benchmark scores. Testing on DCP-Bench-Open, another benchmark, showed that the evaluator's own solver automatically filled in omitted answer fields, accepting empty JSON objects for every problem. Together, these findings demonstrate that evaluators routinely compromise benchmark integrity by either failing to check critical parts of a submission or by completing the missing portions of the answer themselves.

Critical strengths
  • interesting finding, I wasnt aware of these specific benchmarks, but the flaws appear to be real and statistically significant.
Critical weaknesses
  • I would like to see methods suggested to fix the benchmark flaws and actually implement the methods and rebuild the benchmarks
  • do more (or all) reasoning benchmarks have this flaw? I wasnt aware of this issue prior to this paper, but it was tested on just two benchmarks.
  • this is really a simple workshop paper or just a blog/tweet. Interesting, please dont doubt the value, but not a "research" paper. Especially when ICLR is way way oversubscribed.
Questions for authors

Why not fix the benchmarks you identified faults with and release them at scale?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 35yD

Thanks for the review.

On fixing and releasing: for MathConstruct we built a stricter checker layer that adds all 38 missing checks, corrected leaderboards for all archived collections, and the audit tool itself so the maintainers can re-run it after changes. We will release all of these. DCP-Bench-Open has already fixed its issue in v0.3.0 in the way we proposed, and we will send the remaining fixes to the MathConstruct maintainers after the anonymity period.

On other benchmarks: we ran the audit on all 121 MathConstruct checkers and on all 50 construction tasks of a second benchmark, ComBench. In MathConstruct, 38 of 121 checkers (31%) accept an answer that violates the task; for example, USAMO 2002/5 asks for positive integers and accepts 22, 0, 15. In ComBench we found nothing beyond the case in App. I (1 of 50), and on the two problems that appear in both benchmarks, MathConstruct's checker is wrong while ComBench's is right. So the problem is not universal, which is why we think a reusable audit is more useful than a general claim about all benchmarks.

On real impact: rechecking all 24,875 archived model outputs, we find 30 wrong answers that received credit (5 in the paper), including answers from o1 and o3-mini-high.

If this addresses your concerns, we would appreciate it if you could consider updating your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.