- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper exposes a fundamental flaw in evaluating mathematical reasoning: standard exact-match scoring against a single gold answer conflates a model’s true capability with its tendency to produce a specific answer format. The authors formalize this insight as the identity S=C⋅π_gold , where S is the observed benchmark score, C is the model’s true capability, and π_gold is the probability that a correct answer matches the gold form. This conflation can cause identical underlying capability to be reported as differing by up to 4.75× depending solely on training-data ordering. The paper proposes a practical remedy, a summation estimator that scores model generations against all valid answer forms. The paper validates this approach through pre-registered experiments on the Qwen model family, achieving near-perfect recovery of true capability without any model retraining.
Critical strengths
S-1 The summation estimator requires no retraining, no architectural changes, and no additional model inference — only extra scoring passes over existing generations. This makes adoption trivially easy for any evaluation pipeline.
Critical weaknesses
W-1 The most significant limitation is that all conflict corpora are synthetically constructed rather than discovered in naturally occurring benchmark data. The authors deliberately engineer problems where multiple valid forms exist and are mutually exclusive under exact-match scoring. While this provides clean experimental control, it raises questions -- how prevalent are such conflicts in real-world benchmarks?
W-2, The entire analysis is confined to competition mathematics with mutually exclusive answer conventions (e.g., decimal vs. fraction). The paper does not explore whether the S=C⋅π_gold framework extends to other domains where answer ambiguity is common — such as open-ended generation or code evaluation. Even within mathematics, the focus on clean numerical answers with discrete alternative forms may not capture the full spectrum of format ambiguity.
W-3 All experiments use models from the Qwen family (Qwen2.5-3B/7B/14B and Qwen3-8B). While the theoretical framework is model-agnostic, the empirical validation on a single model family limits confidence in generalizability.
Questions for authors
Clarification Question:
In experimental setup, the author claims that the capability floor should be larger than the contrast floor. It appears to assume independent errors in acc_A and acc_B. Since each generation matches at most one convention, allocation variation moves the two accuracies in opposite directions, so their errors are plausibly negatively correlated and C could be less noisy than either accuracy. Was the capability floor measured directly from the cross-seed spread of C?
This matters because the key claim (capability is unchanged across arms) is tested against the larger of the two floors, and a wider floor makes "no difference" easier to conclude.
Suggestions for future work:
The summation estimator recovers capability only for the conventions that are enumerated, and any correct form left out shows up as a downward support leak. In the current experiments, the conventions are fixed by construction of the conflict corpora. What is your plan for applying the estimator to existing benchmarks, where the set of valid conventions is unknown?