acceptodds
Withdrawn from ICLR 2027

The Underdetermined Stream: What a Single-Gold Score Actually Measures on Contested Material

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Benchmarks for mathematical reasoning compare a model's answer with a single gold string, and on material where several forms of the same answer are equally correct that comparison is measuring something other than capability. The field treats the mismatch as a data-quality bug for a better normaliser to repair, which presupposes that one of the two golds is wrong; when both are correct there is nothing to fix, and what the score measures instead has never been derived. The obvious remedy — publish a correction factor beside the score — presupposes that the multiplier is a property of the corpus, and latent-variable estimators of a hidden true label cannot price their own error. We derive the identity: an exact-match score on contested material is capability multiplied by the rate at which the model writes the one form this corpus shipped as gold, a product whose second factor belongs to the model-corpus pair rather than to either alone. Capability is nonetheless exactly recoverable without retraining, by scoring the same generations against each convention's gold in turn and summing, under two premises that are measured rather than assumed. On four arms of one corpus differing only in the order training data reached the optimiser, capability holds inside the 0.0270 measurement floor while the allocation moves 0.4621 — the policy at F=612.18 against the capability sum's 1.63 — so a single-gold benchmark reports the same capability 4.75x apart depending only on how the model was taught to write. The estimator returns an estimated capability of 0.4452 against a directly measured 0.4522, and that 0.0070 gap is exactly the support leak the identity predicts (0.00705), so its error is always downward and always priced by a quantity the same run reports. The correction-factor shortcut is registered, run and rejected: the model's policy correlates with per-item solvability at r=-0.257 on 24 of 24 arms, so the multiplier is a property of the model as well, and the effective enumeration it hides runs 1.149 to 1.811. The rule is to publish the summed score, which costs extra scoring passes over generations already produced and no new training.

Primary area:
datasets and benchmarks
Keywords:
evaluation methodology, benchmark evaluation, exact match, answer equivalence, construct validity, measurement theory, mathematical reasoning, capability estimation, pre-registration, underdetermination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bwqDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper exposes a fundamental flaw in evaluating mathematical reasoning: standard exact-match scoring against a single gold answer conflates a model’s true capability with its tendency to produce a specific answer format. The authors formalize this insight as the identity S=C⋅π_gold , where S is the observed benchmark score, C is the model’s true capability, and π_gold is the probability that a correct answer matches the gold form. This conflation can cause identical underlying capability to be reported as differing by up to 4.75× depending solely on training-data ordering. The paper proposes a practical remedy, a summation estimator that scores model generations against all valid answer forms. The paper validates this approach through pre-registered experiments on the Qwen model family, achieving near-perfect recovery of true capability without any model retraining.

Critical strengths

S-1 The summation estimator requires no retraining, no architectural changes, and no additional model inference — only extra scoring passes over existing generations. This makes adoption trivially easy for any evaluation pipeline.

Critical weaknesses

W-1 The most significant limitation is that all conflict corpora are synthetically constructed rather than discovered in naturally occurring benchmark data. The authors deliberately engineer problems where multiple valid forms exist and are mutually exclusive under exact-match scoring. While this provides clean experimental control, it raises questions -- how prevalent are such conflicts in real-world benchmarks?

W-2, The entire analysis is confined to competition mathematics with mutually exclusive answer conventions (e.g., decimal vs. fraction). The paper does not explore whether the S=C⋅π_gold framework extends to other domains where answer ambiguity is common — such as open-ended generation or code evaluation. Even within mathematics, the focus on clean numerical answers with discrete alternative forms may not capture the full spectrum of format ambiguity.

W-3 All experiments use models from the Qwen family (Qwen2.5-3B/7B/14B and Qwen3-8B). While the theoretical framework is model-agnostic, the empirical validation on a single model family limits confidence in generalizability.

Questions for authors

Clarification Question:

In experimental setup, the author claims that the capability floor should be larger than the contrast floor. It appears to assume independent errors in acc_A and acc_B. Since each generation matches at most one convention, allocation variation moves the two accuracies in opposite directions, so their errors are plausibly negatively correlated and C could be less noisy than either accuracy. Was the capability floor measured directly from the cross-seed spread of C?

This matters because the key claim (capability is unchanged across arms) is tested against the larger of the two floors, and a wider floor makes "no difference" easier to conclude.

Suggestions for future work:

The summation estimator recovers capability only for the conventions that are enumerated, and any correct form left out shows up as a downward support leak. In the current experiments, the conventions are fixed by construction of the conflict corpora. What is your plan for applying the estimator to existing benchmarks, where the set of valid conventions is unknown?

Reviewer_srRUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies exact-match scoring on mathematics problems where several surface forms of one answer are equally correct (0.5 vs 1/2, XIV vs 14). It writes the single-gold exact-match score as S = C * pi_gold (Prop. 1): capability C (probability of a correct value) times the capability-weighted rate pi_gold at which the model writes the one form the corpus chose. When k forms are equiprobable, the gold is drawn uniformly and the model's output is supported on the k forms, S = C/k for every policy (Prop. 2), a case the authors call limiting. Scoring one generation file against each form's gold and summing recovers C if the forms are mutually exclusive and cover the model's correct answers (Prop. 3). Experiments use Qwen2.5 (3B, 7B, 14B) and Qwen3-8B fine-tuned on constructed conflict corpora whose two halves use incompatible conventions. Four arms trained on the same data in different orders have similar summed accuracy C (range 0.0242, below the design's minimum detectable difference) but minority-convention shares from 0.407 to 0.869 (Table 1). On 1,988 generations with four renderings, the estimator gives 0.4452 against a directly measured 0.4522, and the gap is attributed to the support leak (Sec. 6.2). The paper argues that a corpus-level correction factor C/S cannot exist, because it varies with the model (Table 2, 1.149 to 1.811) and because per-problem solvability correlates negatively with the policy (mean r between -0.257 and -0.355 across three conditions, negative in all 24 seed-arms of one 7B setup; Table 9). It recommends reporting the summed score and the enumeration next to the single-gold score.

Critical strengths
  1. The paper states its scope honestly: the conflicts are constructed, the prevalence claim is borrowed, the C/k pin is called a limiting case (Sec. 1, Sec. 3, Fig. 6), and Appendix D records two self-corrections (the withdrawn independence hypothesis and the sigma-at-k=1 mislabel).

  2. The design reads every number off one fixed generation file, so differences between conditions are differences in the scorer only (Figs. 1, 4). Table 1 is a clean design for showing that a single-gold score can hide a large change in how a model writes at about constant union accuracy.

  3. The practical recipe (score once per accepted form, publish the enumeration and sigma) is cheap and easy to adopt.

  4. Remark 1 / Table 9 measure policy heterogeneity across problems against a simulated null band. That is a real, if narrow, empirical argument against a corpus-level correction factor, and the scale-sweep control (Table 8, inert to 0.08 floors) is a useful check on instrument drift.

Critical weaknesses
  1. The estimator validation is circular. Table 5 defines sigma = S(4)*4/C, and C-hat = S(4)*4, so sigma = C-hat/C and the "predicted shortfall" (1 - sigma)C equals C - C-hat by algebra. The agreement of 0.00705 and 0.0070 "to the last place" (Sec. 6.2, abstract) is therefore an identity, not a test. The definition of C is also inconsistent: Table 3 defines it as "the sum of its accuracies over the enumerated conventions" (which would make C = C-hat and the gap zero), Table 5 as "form-independent correctness", and Fig. 4 as "value-correct under any enumerated convention". If C comes from an independent value-equivalence check, that check, not the estimator, is the ground truth and should be described and validated. If not, sigma cannot be computed independently. With partial enumerations the estimator is far off (Table 2: errors +0.418, +0.826, +0.355 for k = 1, 2, 3). Finally, the paper itself says scorer-side fixes work exactly when forms are interchangeable downstream (Sec. 2, Prop. 5(iii)), and that otherwise the honest route is a paired contrast (Sec. 8). The sum's advantage over an existing canonicaliser therefore needs to be shown, not assumed.

  2. The evidence comes from a constructed setting, and the claim about real benchmarks is not measured. All models are Qwen fine-tuned on corpora built to contain two incompatible conventions, with allocation then set by data order. A policy that follows the most recent training block (Table 1, blocked arm, acc_A 0.0671) is the expected outcome. Sec. 7 reports that "An attempt to find one in the wild returned an empty set at the scale we could search". Prevalence and benchmark-level size come from a survey reviewers cannot read, and the figures quoted from it are small (1.96% of shared problems, 0.0240 accuracy points at benchmark level, Sec. 6.2). The abstract's "4.75x apart" (0.3183/0.0671) takes convention A as the single gold for four fine-tuned arms, while Sec. 6.1 says the reported score on the mixed corpus is C/2 and moves only 0.0121. The abstract's "depending only on how the model was taught to write" is also a causal claim that Sec. 1 says is borrowed and "never as evidence".

  3. The abstract claims more than the body supports. (a) "capability holds inside the 0.0270 measurement floor", while Sec. 6.1 says the minimum detectable difference is 0.0242, the observed range, so "capability is not resolved here rather than shown equal". (b) "two premises that are measured rather than assumed", while Sec. 4 says "Exclusivity holds by construction of the enumeration." (c) "r = -0.257 on 24 of 24 arms", while -0.257 is the mean of the eight unmarked seed-arms only, and the 24 are seeds of one 7B setup in three conditions (Table 9). (d) Sec. 6.2 says "Proposition 2 is not an idealisation", contradicting Sec. 3 ("Its domain is empty on real data") and Sec. 1.

  4. Judge-based equivalence is not run as a baseline. The paper places model-based graders as a scorer-side escape (Sec. 2, App. C) but never measures one. A single-reference LLM judge may anchor on the form of the reference it is shown and partly reproduce the single-gold effect; a judge shown the full accepted list needs the same enumeration as the sum. Comparing the sum, a canonicaliser and a list-conditioned judge on the same generations, against a small hand-labelled set, would show whether the sum adds anything and would provide the independent C that Weakness 1 asks for.

Notes on the three scores above. Presentation (2): The tables are clear and the authors are open about limits (Sec. 7, Appendices D, F, G). Readability is the main problem. The paper relies on five anonymous works of the same programme (a parent preprint and four concurrent manuscripts) and carries inherited or programme-level vocabulary (stream, walls, escapes, keys) and figures (Figs. 2, 5, 7) that the results do not use; Fig. 2's axes B and D are not defined in the main text. Several figure captions disagree with the tables (Fig. 3's displacement range and seed count; Fig. 7's "publish k" advice versus Table 2). Moving the programme material to an appendix and stating the main result, its evidence and its scope in the first two pages would help readers a great deal. Informativeness (1): The decomposition S = C * pi_gold and the fact that the sum of per-form accuracies is union accuracy follow directly from the chain rule and an indicator identity, as the authors acknowledge (Sec. 4). The new empirical content is narrower: a policy that can be moved by training order at roughly constant union accuracy, on constructed corpora, and a negative per-problem correlation between solvability and policy. That a model trained on two conventions in blocks favours the last block (Table 1) is expected. The size of the effect on public benchmarks with standard models is not measured here, and the authors report that their search for a firing conflict in the wild returned an empty set (Sec. 7). Soundness (2): The algebra is correct (I checked Props. 1 to 3 and the table arithmetic). The concerns are about what the experiments show: circular validation of the estimator, an abstract that claims more than the body supports, a control that is described two incompatible ways, three-seed designs, and heavy dependence on unreadable companion work. These are fixable but not by a rebuttal paragraph.

Questions for authors

Clarification (could change my assessment):

  1. How is "value-correct" decided for C (Sec. 6.2: 0.4522), and is it independent of the four-form enumeration? Tables 3, 5 and Fig. 4 define C differently. If C is not independent, what does the 0.4452 vs 0.4522 agreement test?

  2. Table 7's caption says the spelled-both-halves control "produces a displacement of the same size, which is why the tilt cannot be read as a measure of conflict" (tilt 0.0919, acc_A/acc_B 0.0979/0.2817). Appendix H says that control "returns the allocation to the corpus rate within the contrast floor", and Table 8's control sits at 0.4985-0.5000. Are these different controls? If the spelled-both control moves, what is the displacement evidence for conflict?

  3. The text near Fig. 3 says checkpoints scoring 0.20 and 0.18 "solve 0.385" (matching Table 8's three-seed 7B row, C = 0.3850), while Fig. 3, whose caption says "three seeds", shows C = 0.3727, the eight-seed value of Tables 2, 6 and 7. Which seed set does Fig. 3 use? Relatedly, the Fig. 3 caption gives the displacement range as 0.0195 to 0.1040, while Table 7 lists 0.1550 for Qwen3-8B.

  4. Remark 1 mentions k = 4 samples per problem. What are the decoding temperature and the number of evaluation problems behind Tables 1, 6 and 8?

  5. Which claims do the 5 registrations that returned an unexpected branch, and the 5 that declined at a guard, concern (Sec. 5, App. E)?

Suggestions:

  1. Report the effect on at least one public math benchmark with standard instruct models, where the gold form differs between corpora, and say how much model rankings change.

  2. Validate the estimator against a small hand-labelled set that includes forms the enumeration misses, and report how the support leak changes when a rendering is dropped or added.

  3. Add a canonicaliser baseline and a judge baseline (one reference, and the full accepted list) scored against the same labelled set.

  4. Replace "capability holds inside the floor" with an equivalence test (TOST) or "not distinguishable at n = 3", and align the abstract with the body on premises, the r-statistic and Prop. 2's scope.

Reviewer_wQqqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

On math benchmarks, if a question's answer is 1/2, then answering 0.5 should also be judged as correct. But some existing benchmarks ask for exact string match. This is a false premise. Standard practice is to extract the \boxed{} answer and compare it with a symbolic checker such as Math-Verify or SymPy, or with an LLM grader. So in most MATH benchmarks, 1/2 and 0.5 as the same answer.

Also the paper didn't use ICLR template.

Critical strengths

None

Critical weaknesses

The premise of the paper is simply wrong. Most math benchmarks will consider mathematically equivalent answers as all correct.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.