- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper audits the evaluators of constructive reasoning benchmarks to test whether they credit answers that violate the task requirements. Starting from a valid answer, it makes a minimal change that breaks one explicitly stated requirement and passes the result through the officially released evaluator. On MathConstruct, the checkers of seven tasks miss stated conditions (e.g., distinctness, ordering, positivity); across further official instances, all 127 applicable invalid constructions are accepted and all are rejected once the missing checks are added; five archived model answers that violate requirements had entered published scores, each of which drops by about 0.2 percentage points after correction. On DCP-Bench-Open v0.2.2, the evaluator fixes only the variables present in the answer and lets the solver choose the rest, so an empty JSON object is accepted on all 164 problems and all 735 incomplete outputs are accepted; a field check rejects all of them. The authors recommend testing evaluators with answers that should fail.
Critical strengths
The counterexamples are concrete and reproducible, and one evaluator flaw has broad impact. Each MathConstruct counterexample is read back by the official parser, accepted by the official checker, rejected once the missing condition is added while valid answers remain accepted, and persists in the maintained revision. The DCP-Bench-Open evaluator completes incomplete answers itself, so an empty JSON object is credited on all 164 problems, which directly affects anyone evaluating or training with this benchmark.
Critical weaknesses
- The method is standard, and the contribution is mainly specific flaws in two benchmarks. Testing a program with inputs that should fail is standard practice in property-based and mutation testing, and EvalPlus and UTBoost, cited by the paper, have already shown in code evaluation that insufficient tests let wrong answers pass. This work applies the same check to constructive math and constraint-modeling benchmarks and yields a set of specific flaws rather than a new method or a generalizable conclusion.
This determines my assessment of the contribution: the findings are well suited as issue reports to the benchmark maintainers or as a short paper in a benchmarks and datasets track, but their information gain as a main-conference paper is limited.
- The coverage is narrow and the confirmed real-world impact is small. Only two benchmarks are audited, both of which are recent and of limited adoption (MathConstruct appeared at ICML 2025 and DCP-Bench-Open is currently a preprint), unlike the widely used HumanEval audited by EvalPlus; only 10 of the 121 MathConstruct checkers are manually inspected, and the 127/127 cross-instance result shows that known failures recur on other instances rather than how many other failures exist. Only five answers are confirmed to have entered historical scores, each lowering the corresponding score by about 0.2 percentage points; the authors acknowledge that this is insufficient to assess ranking changes and that the trigger frequency in general model outputs cannot be estimated.
This affects my assessment of significance: the paper shows that the flaws exist, but not how common such problems are across reasoning benchmarks or how much they affect existing conclusions.
Questions for authors
-
Could the audit procedure be turned into a reusable tool and applied systematically to all 121 MathConstruct checkers and to further reasoning benchmarks with executable evaluators, reporting how many missing checks are found?
-
Have these issues been reported to the benchmark maintainers, and has the field-completion behavior of DCP-Bench-Open been fixed in releases after v0.2.2?
2 comments
Thank you for the additional experiments. I appreciate the practical contribution of the reusable audit tool, the expanded coverage, and the proposed fixes. The audit relies on standard mutation testing, and the confirmed impact on historical scores remains small, with no changes to rankings within the evaluated collections. These considerations lead me to maintain my rating.
Thanks for the detailed review. In short: yes, we built the tool and ran it on all 121 checkers (38 of them, 31%, accept an answer that violates the task), and DCP-Bench-Open has since fixed the field-completion issue on its own. Details below.
Q1 (reusable tool, all 121 checkers, more benchmarks). The tool only needs a task's official solution, parser and checker, so it runs on every task without task-specific code. It makes 12 generic edits to the official solution (e.g. duplicating, deleting or swapping list elements, negating or zeroing numbers) and sends each edited answer through the official parser and checker. A finding counts only if the checker accepts a concrete answer and a separate check written from the task statement rejects it.
On all 121 MathConstruct checkers (62,924 edited answers) this gives 36 findings in 33 tasks, 31 of them new, so 38 of 121 including the paper's 7. On all 50 ComBench construction tasks it finds nothing beyond the case in App. I.
W1 (method is standard). We agree, and we don't claim a new testing method. The difference from EvalPlus/UTBoost is the ground truth: they test a program against a reference implementation, while we test the evaluator itself against the written task statement. Some bugs only exist in evaluators, e.g. float tolerance on exact tasks, counting after set(), or the evaluator filling in missing fields. What we add is the measurement: 31% of MathConstruct checkers vs. 2% of ComBench's accept a wrong answer.
W2 (impact). Re-checking all 24,875 archived model outputs, 30 wrong answers received credit (5 in the paper), and 14 main-collection scores drop by 0.2-0.6 points (5 in the paper). Rankings within the main collections don't change, and we will say so clearly.
We think the per-task view matters more in practice. If a checker is used as an RL reward or a data filter, a wrong answer that passes is what the model learns from, and the archive shows o1 and o3-mini-high already produce such answers.
Q2 (maintainers, DCP after v0.2.2). DCP-Bench-Open fixed field completion in v0.3.0 (released 2026-09-29) with an exact key/shape/type check, essentially our proposed fix. They did this independently; we had not contacted them. The diamond-free issue in App. G is still present. We will report the MathConstruct findings with patches after the anonymity period.
If this addresses your concerns, we would appreciate it if you could consider updating your score.