acceptodds
Under review as a conference paper at ICLR 2027

Scoring the Same Outputs: How Answer-Extractor Choice and Consistency Change Benchmark Comparisons

Abstract

Free-response benchmarks do not score model text directly: an answer extractor first decides which number counts as the answer, so two pipelines scoring the same outputs can report different gaps. We make that scoring step the object of study. For a frozen class of seven deterministic extraction rules, we compute every gap fixed outputs can yield when one shared rule scores every response, and how far the range expands when the rule may vary per question or per system—oracle stress tests of assignment consistency. A registered reanalysis of a disclosed development cohort (22 comparisons, four 1–2B families) and a preregistered held-out confirmation locate both robustness and fragility, and the pipeline falsifies its own development sign flip. On 193 harder ContextMATH pairs the audit separates two regimes: Qwen2.5's scenario effect has no unique value until the extraction rule is declared (a shared-rule range of −13.0 to +12.4 points at 7B, −14.5 to +8.8 at 32B—rule dependence, not a confidence interval), while four newer checkpoints from two families earn exact within-class robustness certificates: one identical negative gap under every rule and assignment. Bigger models alone do not remove scoring ambiguity. Declare one extractor before scoring, apply it to every system, and report the shared-rule range over a prespecified class.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, answer extraction, benchmark reproducibility, evaluation methodology, scoring robustness, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bzqWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

AI Slop.

Critical strengths

None.

Critical weaknesses

AI Slop.

Questions for authors

What is the point of this paper?

2 comments

Thank you for the review. Your question deserves a direct answer.

The point of the paper. Benchmark numbers for free-response tasks depend on an answer extractor that is almost never declared. Holding model outputs fixed, we compute every gap a prespecified class of extractors can produce under several assignment disciplines. When the gap is the same under every rule, the comparison earns a certificate; when it is not, the paper reports the range instead of a sign. Concretely: the same saved Qwen2.5-7B outputs on a hard benchmark yield a scenario effect anywhere from −13.0 to +12.4 points depending on which deterministic extractor scores them (we now also show which convention is responsible), while four newer checkpoints from two families yield one identical negative gap under every rule.

On the ethics flag. The paper's mandatory AI use statement discloses that generative AI assisted with study design, formal reasoning, implementation, analysis, literature review, and writing, and that the author reviewed all of it and takes full responsibility, as the ICLR 2027 policy requires. The two theorems carry proofs, the designs were preregistered, and every number in the submitted paper is produced by the supplementary code from pinned artifacts (the two discussion-period diagnostics by two further scripts under the same conventions, included in the revised supplementary material).

If there is a specific claim, proof, or number you believe is wrong, we will address it.

Thanks for the response. I don't have further questions.

Reviewer_a6LsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how answer-extraction choices affect benchmark comparisons when model outputs are held fixed. It introduces seven deterministic extraction policies and analyzes attainable performance gaps under three assignment schemes. Experiments on GSM8K, GSM1k, and ContextMATH reveal both extractor-sensitive and invariant comparisons, while the held-out study fails to reproduce the development cohort's sign reversal. The authors recommend consistent extractor assignment and reporting sensitivity across a predefined policy class.

Critical strengths
  1. The design separates evaluator variation from changes in generated outputs. Applying multiple deterministic policies to the same saved responses makes the effect of the scoring procedure directly observable. The distinction between globally shared policies and less constrained assignments provides a clear way to state the scope of an evaluation claim.

  2. The paper reports evidence that qualifies its own conclusions. It discloses the development-informed policy class, reports the failure to reproduce the sampled-budget sign reversal on GSM1k, and includes public evaluators that disagree with the registered class. These disclosures are valuable for assessing the scope of the findings.

Critical weaknesses
  1. The practical validity of the seven-policy class remains unclear. All policies share a restricted parsing grammar, and ten Qwen comparisons reverse direction under three public evaluators. This raises concerns about whether the observed robustness and sensitivity generalize beyond the chosen grammar.

  2. The practical relevance of the assignment-set analysis is insufficiently demonstrated. The free-assignment setting allows different extractors for every system–item pair, while real evaluation pipelines typically use fixed scripts. The paper does not systematically analyze the more realistic case where each system uses one fixed extractor.

Questions for authors
  1. If terminal-segment recognition is extended to ordinary trailing LaTeX display blocks while all seven acceptance sets and priority orders remain unchanged, do the two Qwen2.5 ContextMATH comparisons still exhibit sign reversals? This would directly test whether the central finding survives the identified format-coverage issue.

  2. What independent validation supports the observation grammar on answers for which the registered class and public evaluators disagree? In particular, can the authors distinguish genuinely ambiguous answers from answers whose intended result is explicit but is rejected by the shared grammar?

  3. What are the attainable effects when each system uses one fixed policy across all items, giving at most 49 policy pairs? How much additional variation arises only when policies may change independently at the item level, and what practical conclusion depends on that additional freedom?

2 comments

Thank you for a careful reading; your three questions are the right tests, and we ran two of them exactly as posed. We summarize here; details are in Appendix Q and the supplementary declarations.

Q1 — trailing display blocks. We froze one grammar variant: when a completion ends with a multi-line \[ … \] block, its lines are joined into the terminal line; nothing else changes. Only the two Qwen2.5 ContextMATH cells are touched, and the within-class sign reversal disappears: the shared-rule range moves from [−13.0, +12.4] to [−13.0, −11.4] points at 7B and from [−14.5, +8.8] to [−14.5, −14.0] at 32B, with every rule and every assignment discipline negative.

So the registered statement—that the Qwen2.5 sign is extractor-dependent within the class—does not survive this coverage extension in the form we gave it, and we now say so. The sensitivity traces to one convention: whether a trailing display block counts as the terminal segment. What survives is that two pipelines scoring the same outputs report opposite signs because of a convention neither would normally declare, and that declaring it makes the class agree with the public recovery readers (Math-Verify −10.9 / −14.5, within one item of the new range). This is revised in the abstract, Section 5, and Appendix Q, and the practitioner checklist gains a clause: declare the terminal-segment convention.

Q3 / W2 — one fixed rule per system. We computed the attainable set when each system is scored by one fixed rule (≤ 49 pairs) for all 37 registered comparisons (Definition 2; Appendix Q). On every one of the 32 comparisons with a non-degenerate free range, the fixed-pair range equals the free range exactly; item-level freedom adds interior values but never widens it. The five point certificates stay single values. On the ten Qwen comparisons that reverse under public readers, no fixed pair gives a positive gap (the largest is −0.5 points, versus +8.9 to +15.1 for the public readers). We were wrong to present the free set only as an oracle stress test; the revision presents the fixed-pair set as the realistic case alongside it.

Q2 — validating the grammar where the class and public readers disagree. The disagreements are structural, not semantic. On the historical Qwen chain-of-thought arm, of the 203 completions credited by Math-Verify and by no class rule, 142 end with the literal placeholder sentence "The answer is N.", 46 state the result only in the body, and 15 have a final line with eligible numerals that no class rule resolves to the gold value (14 are all-rule abstentions on a line with several unmarked numerals, 1 is a wrong selection). Appendix P gives the full decomposition and corrects the submitted split (56 and 5). Across all ten reversed comparisons, every public-reader credit on a completion without an eligible terminal candidate (2,291 unique completions) is a placeholder ending (1,460) or a body-only statement (831); none comes from a numeral format the grammar failed to recognize. The class scores whether a response delivers a terminal answer under a declared grammar; the public readers recover one from the trace. The paper reports both readings rather than adjudicating which is "right."

A blinded audit by one author of 45 sampled cases from this arm (20 placeholder-ending, 20 body-only, and 5 terminal-read-differently cases), reading each response without the gold value or any reader's verdict and recording the single number it commits to, found that number equal to the gold value in 20/20 placeholder cases, 20/20 body-only cases and 5/5 terminal cases; the response committed to a different number in 0 cases and to no single number in 0 cases. On this sample the public readers mostly credit a value the response does commit to; the disagreement with the class is about the terminal convention, not about whether an answer was given.

One of our three pre-declared predictions failed: we expected at most 2 of the 5 terminal cases to read as the gold value, and all 5 did (final lines such as "… is 86 + 42 = 128," where the class abstains for lack of a unique terminal candidate). That abstention is a cost of the uniqueness rule, and we report it as one. The protocol, sample, predictions, and wording were declared before sampling and are in the revised supplementary material; this is one annotator reading 45 cases, reported as counts.

W1 — the restricted grammar. Appendix H (three marker-grammar variants) and Appendix P (three public evaluators at pinned revisions) already test beyond the class; the certificates transfer and the reversals have the structural cause above. We agree the class is a compliance contract over one grammar: Section 4 says so, and Sections 5 and 7 now describe the Qwen2.5 sensitivity as a property of the grammar's final-line convention, not of the outputs alone.

We hope these additions address your concerns and are happy to clarify further.

Thank you for your response and clarifications, which address some of my questions. After considering the response, I have decided to maintain my original rating.

Reviewer_SPZnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper focuses on studying the scoring step of how to process the generated response. It formalizes seven deterministic extractors and three assignment disciplines and compute the range of effects fixed outputs can yield. The paper summarizes with practice of choosing the answer reader before scoring, apply it identically to every system, report the shared-reader range over a prespecified plausible class, and report abstention and unsupported-answer rates.

Critical strengths
  • The paper touches important problem of how the scoring rule affects performance.
  • The paper conduct broad experiment of 22 comparisons of four families on multiple benchmarks.
Critical weaknesses
  • How is the seven deterministic extractors and three assignment disciplines chosen? Is this choice broad and general enough?
  • While the analysis is broad, it is focused on Qwen family. Also, while the authors propose the practice to follow, I'm not sure how this changes on any real conclusion. Where there any papers or previous results where with the practice, you observe more robust performance?
Questions for authors
  • Appendix K is left empty.
  • It is mentioned that the same model families are used for drafting and reviewing in reproducibility statement. Could there be any bias from the work?
2 comments

Thank you for the review. Point by point:

How the seven rules and the assignment disciplines were chosen. The rules were not selected. The grammar defines four kinds of evidence a numeral can carry (terminal-marked, marked, terminal-unmarked, body-unmarked). A source-preserving rule is an upward-closed set of accepted kinds (a rule that accepts weaker evidence also accepts every stronger kind) plus a priority order, and there are exactly seven such rules (enumeration theorem, Section 2, proof in Appendix A). The design choice is the grammar, which we disclose as development-informed. Breadth beyond it is tested by three public evaluators (Appendix P), a LaTeX-aware reader (Appendix O), three marker-grammar variants (Appendix H), and, new in the revision, a display-math variant and a per-system fixed-rule discipline (general response). Section 2 now states the exhaustiveness in plain words and names the two endpoints of the class: a marked final-line answer alone, and every level down to an unmarked number in the body.

The disciplines are exhaustive in the same sense. An assignment maps each (system, item) pair to a rule, and a non-adaptive choice can depend on neither coordinate (global), on the system only (per-system, added in the revision), on the item only (item-coupled), or on both (free). Those are the four subsets of {system, item}, so the lattice of disciplines is complete.

"Focused on the Qwen family." The development cohort covers four families (Gemma 3, Llama 3.2, OLMo 2, Qwen2.5 at 1–2B); ContextMATH covers Qwen2.5 at 7B and 32B, Qwen3-4B, Qwen3.8-27B, Gemma-4-E4B, Gemma-3-4B, and Llama-3.2-3B. Thirteen of the 22 development comparisons and three of the seven ContextMATH cells are non-Qwen. Rule sensitivity is concentrated in the Qwen2.5 cells: every shared-rule range wider than four points, and every hull that contains both signs, belongs to a Qwen2.5 comparison. The other families keep one sign under every rule. In the development cohort their ranges are at most 3.8 points, and four of the thirteen comparisons are exact singletons. On the two ContextMATH cells that passed the clipping gate they give one identical gap under every rule; the Llama-3.2-3B cell is suppressed and reported descriptively. That asymmetry, and its mechanism (on ContextMATH, Qwen2.5 ends many responses inside a LaTeX display block; on GSM8K it often ends with the literal placeholder sentence "The answer is N."), is one of the findings.

Does the practice change any real conclusion? The recommendation is a reporting discipline, not a training method, so it does not make a model perform better. It changes what is concluded from the same outputs. The three public extractors we apply are the ones shipped with the lm-evaluation-harness (Biderman et al., 2024), the GSM8K script released with Qwen, and Math-Verify, applied to our own frozen outputs. Concretely:

  • Ten Qwen comparisons carry a negative gap under every class rule and a positive gap under those three evaluators. The historical Qwen protocol comparison moves from a class range of −20.8 to −2.5 points to +12.9 under the harness's flexible-extract filter (Appendix P). The mechanism is visible in the outputs: Qwen2.5 often works the problem out and then ends with the literal placeholder sentence "The answer is N.", which the class does not credit and the recovery readers read past.
  • On ContextMATH, a pipeline using one rule reports that scenario rewording hurts Qwen2.5-7B by 13 points and a pipeline using another reports that it helps by 12 (Table 1).
  • A development sign flip did not replicate on the held-out set.

Declaring the extractor, applying it to every system, and reporting the shared-rule range turns those into a stated range or a certificate instead of a sign. Appendix I provides the checklist.

Appendix K. You are right, and we apologize. The section's only content was the full 22-comparison table, which floated to the next page and left the heading empty. Fixed in the revision.

Same model families for drafting and AI review. Yes, shared blind spots are possible, which is why the statement discloses this as AI review and not as validation. The evidence rests on executed checks that do not depend on a reviewer's judgment: a 105-case adversarial test battery, unit-tested exact-set algorithms, a preregistered held-out confirmation with falsification rules, public evaluators at pinned revisions, and predictions registered before each analysis. In this response, four of those predictions failed and are reported as failed.

Thank you for the clarifications. I will consider them along with responses from other reviewers in my final decision.

Comments on the paper

1 comment

We thank the reviewers. Two of the questions pointed at analyses the paper should have contained. We ran both on the frozen outputs, with predictions registered before execution, and report them in full below and in the revised PDF (uploaded; main text within 10 pages).

What changed in the revision.

  1. Per-system fixed-extractor analysis (Reviewer a6Ls, Q3/W2). For every registered comparison we now also report the set of gaps attainable when each system is scored by one fixed rule across all items (at most 49 rule pairs). Its range coincides exactly with the free-assignment range on all 32 comparisons where that range is non-degenerate: item-level freedom adds interior values but never widens the range. The free set is therefore the realistic "two pipelines, two scripts" worst case, not an exotic adversary.
  2. Display-math terminal segment (Reviewer a6Ls, Q1). Extending terminal-segment recognition to a trailing LaTeX display block, with all seven acceptance sets and priorities unchanged, makes all seven rules agree on a negative effect for both Qwen2.5 ContextMATH comparisons (7B: −13.0 to −11.4 points; 32B: −14.5 to −14.0). The registered sign sensitivity traces to that one convention. We state this in the abstract, Section 5, and a new appendix section.
  3. Appendix K (Reviewer SPZn) had its only content, the full 22-comparison table, float to the following page, leaving an empty heading. Fixed.
  4. Section 2 now states plainly that the seven rules are not a subset we chose—given the grammar, they are the only source-preserving rules possible (Theorem 1)—and names the two endpoints of the class.

On the ethics flag. The paper's AI use statement discloses, as the ICLR 2027 policy requires, that generative AI assisted with study design, formal reasoning, implementation, analysis, literature review, and writing, and that the author reviewed all of it and takes full responsibility. Every number in the submitted paper is produced by code in the supplement from pinned artifacts. The two discussion-period diagnostics, and the sampling and scoring of the blinded audit (whose 45 labels the author wrote by hand), are produced by three further scripts under the same conventions (predictions registered before execution, byte-identical regeneration), included in the revised supplementary material. The two theorems carry proofs. We are glad to answer any specific technical concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.