acceptodds
Under review as a conference paper at ICLR 2027

Answer Extraction Changes Conclusions in Controlled Mathematical Reasoning Tests

Abstract

A mathematical answer score combines the response a model produces with the rule used to read that response. We separate these two components in a controlled study of six mathematical template families. The dataset contains 240 programmatically checked problems, each presented with and without a task-specific reasoning cue. Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct produce 960 responses under a common decoding protocol. A strict final-answer rule assigns correct scores to 217 of 480 Qwen responses and only 2 of 480 Llama responses. Yet the rule fails to extract an answer from 477 Llama responses, including texts that state a numerical conclusion in ordinary language. A post hoc, label-blind terminal-answer sensitivity recovers 242 additional Llama predictions, raising its correct count to 135 without changing any generated text. Under this sensitivity, the mean cue effect is percentage points for Qwen and points for Llama, with descriptive family-level intervals spanning zero. Individual family effects vary substantially. These results do not establish either a universal cue benefit or a model ranking in mathematical reasoning. They show why a benchmark must distinguish answer-format compliance, extractable correctness, and the effect of a reasoning intervention. We release paired prompts, exact labels, complete responses, and both scoring rules so that these quantities can be recomputed separately.

Primary area:
datasets and benchmarks
Keywords:
mathematical reasoning, answer extraction, benchmark evaluation, controlled diagnostics, scoring robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZXGqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Sometimes answers are marked incorrectly for math benchmarks because of formatting issues.

Critical strengths

It's obviously true.

Critical weaknesses

It's not obviously important.

Questions for authors

[I did this first pass with my own faulty eyes and brain, i haven't yet used AI to go through it, I want to have a discussion first]

  1. Could you elaborate a bit on why the reader should care? the methodology and whatever are obviously fine, but even if you did this for 100 models, I struggle to understand why this is important or surprising.

  2. I don't see a lot of discussion or analysis as to why this is happening (merely that it is happening), nor do any constructive insights pop out at me, perhaps you can tell me what you think the takeaways are.

  3. Please answer honestly and without AI; did you really need 12 pages to get the message across? Either I am missing something incredibly important somewhere in the article, or this really is a simple factual observation measured for two open source models inflated to be the length of a paper. Please educate me.

2 comments

Thank you for asking directly. The reason to care is that extraction does not merely move an accuracy number slightly; it can change the scientific conclusion drawn from the same model outputs. Under strict scoring, Qwen and Llama score 45.2% and 0.4%, a gap of 44.8 percentage points. Reading terminal answers from the identical responses gives 47.7% and 28.1%, reducing the apparent gap to 19.6 points without changing either model. The estimated cue effect also changes: for Qwen it moves from +1.25 points under strict scoring to -0.4 points under terminal scoring, while for Llama it moves from -0.83 to -2.9 points. A model comparison or prompting conclusion can therefore be partly a property of the evaluator rather than the generated reasoning.

We also checked that this is not peculiar to our two extractors. Math-Verify 0.8.0 extracts from 42.7% of the 960 frozen responses under an anchored configuration and from 100% under its default configuration. For the 160 percentage-family responses, raw-number and percentage-aware comparison changes its correct count from 1 to 103. The concrete mechanism is therefore not one isolated parser bug: answer anchoring, format compliance, unit interpretation, and truncation each determine what enters the final score.

The constructive takeaway is an evaluation procedure. Before generation, specify the response contract, extraction anchor, and unit semantics. After generation, report extraction availability, correctness conditional on extraction, and full-denominator accuracy separately. Finally, rerun the conclusion with an established alternative extractor on the same frozen outputs. This makes it possible to tell whether a claimed model or prompting improvement survives the measurement choice.

The central message can indeed be stated briefly. The value of the full study is the controlled evidence behind it: 240 programmatically checked problems, paired prompt interventions, 960 released responses, family-level uncertainty, token-cap diagnostics, and an independent Math-Verify audit. Together these show not only that extraction matters, but which conclusions change, why they change, and how an evaluator can detect the problem before reporting a benchmark result.

Reviewer_ZXGqOpenReview ↗

yea, no.

history will look poorly upon this era when we flooded the epistemic commons with such pollution.

Reviewer_WSf7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper used several group of program-generated math problems and compared LLMs' performance on them under either original prompt or prompt with hints, and the answers are graded under different standards. Under strict grading methods, such as Llama with requirements of matching answers with "Final:" prefix or being boxed, the LLMs performed much worse than under broader constraints (extracting natural language answers from the last line), indicating mathematical performance evaluation could be affected by the model's alignment with verification protocol. The rest of this paper focuses on discussing what this observation cannot imply.

Critical strengths

The evaluation conducted in the paper is clear and well explained.

The paper makes a valid empirical observation that answer-extraction rules can substantially affect measured accuracy on identical model outputs, highlighting the importance of distinguishing format compliance from mathematical correctness.

Based on my inspection, the released code is consistent with the methodology described in the paper.

Critical weaknesses

The paper establishes limited novelty and overlooks directly relevant prior work, such as Math-Verify and existing strict/flexible extraction protocols.

The observations provide limited practical guidance: the post hoc extractor is neither independently validated nor compared with established alternatives.

The experiments cover only two models and six handcrafted problem families, leaving the broader relevance of the findings unclear.

Questions for authors
  1. How do the results compare with Math-Verify or other existing extractors? It would help to apply them to the same saved responses and explain what this study adds beyond the already known issue of strict parsers missing answers.

  2. Have you manually checked a sample of the newly recovered answers and those still marked as missing? In particular, how often does the terminal extractor correctly identify the model’s final answer, and how often does it pick an intermediate number or overlook a contradictory conclusion?

  3. Have you checked whether these changes for llama mainly come from the reasoning itself, the answer format, or responses hitting the 512-token limit?

  4. How were the six problem families chosen? Have you checked whether similar effects appear on standard benchmarks or other models? This would help clarify how much of the result depends on this particular setup.

  5. Based on these findings, what should someone evaluating a model actually do differently? Reporting extraction rates helps reveal the problem, but how should they choose an extraction rule and check that its scores are reliable?

2 comments

Thank you for suggesting comparison with an established extractor. We applied Math-Verify 0.8.0 to all 960 saved responses without regenerating any output. The original strict and terminal rules extract answers from 32.5% and 60.9% of responses. Anchored Math-Verify extracts from 42.7%, while its default configuration produces an extraction for all 960 responses. This independently confirms that extraction configuration can change the reported result on an identical response corpus.

Unit semantics produce an additional measurable change. For the 160 percentage-family responses, default Math-Verify marks 1 correct under raw-number comparison and 103 correct under percentage-aware comparison; the corresponding anchored counts are 0 and 36. Extraction anchoring and unit interpretation therefore need to be specified separately.

The Llama diagnosis separates answer format from the generation cap. Of 343 responses below the 512-token cap, strict extraction succeeds on 3 and terminal extraction on 243. Of 137 capped responses, terminal extraction succeeds on 2. Under the same anchored Math-Verify rule, availability is 62.9% and 72.9% for Qwen's two prompt conditions, versus 15.4% and 19.6% for Llama. The extraction effect is therefore not reducible to the token cap and differs across models and prompts.

The six families provide exact programmatic labels and paired narrative/scaffolded prompts, allowing the cue intervention and scoring rule to be varied independently. Based on these results, evaluators should freeze the response contract, extraction anchor, and unit convention before scoring; preserve missing answers in the denominator; report availability and conditional correctness separately; and verify the conclusion with an established alternative extractor on the identical outputs.

I found Q1 as partially solved, but the authors didn't convince me that their design was scientific enough nor the findings are new. I keep my score.

Reviewer_mx64OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the sensitivity of mathematical reasoning evaluation to answer extraction rules. The authors construct a controlled dataset of 240 problems from six mathematical task families, each evaluated under a standard prompt and a task-specific reasoning cue. Using Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, the study collects 960 responses and compares a strict final-answer extractor with a post-hoc terminal-answer extraction heuristic. The results show substantial differences in measured accuracy under the two rules, particularly for Llama, whose number of correct predictions increases from 2 to 135 out of 480 responses. The authors further analyze extraction availability, conditional correctness, and family-level reasoning-cue effects, finding no consistent average benefit from the additional cues. Based on these observations, the paper argues that answer extraction and mathematical correctness should be distinguished when interpreting benchmark results. The study primarily provides an empirical analysis of scoring sensitivity under a controlled setting, rather than proposing a new evaluation method or establishing a generalizable measure of mathematical reasoning ability.

Critical strengths
  1. Clear identification and empirical illustration of an evaluation confound. The paper isolates an important distinction between mathematical correctness and compliance with a prescribed answer format. Applying different extraction rules to exactly the same generated responses provides direct evidence that evaluation outcomes can change substantially without any improvement in the underlying model. The particularly large discrepancy observed for Llama offers a concrete example of why strict answer-format scores should not automatically be interpreted as pure measures of reasoning competence. Although the underlying concern is not fundamentally new, the controlled comparison makes the issue explicit and empirically inspectable.
  2. Reasonably controlled experimental protocol and transparent reporting. The use of paired prompts, programmatically checked reference answers, fixed decoding settings, and identical response corpora across extraction procedures reduces several potential confounding factors. The paper also distinguishes extraction availability, correctness conditional on extraction, and overall accuracy, rather than relying exclusively on a single aggregate score. The inclusion of family-level results, uncertainty estimates, and abstention-completion bounds further improves the transparency of the analysis. These design choices support the paper's narrower measurement claims, even though they do not resolve the limitations of task representativeness or extractor validity.
  3. Appropriate acknowledgment of uncertainty and limitations. The authors generally avoid treating recovered answers as evidence of improved reasoning and explicitly acknowledge that the post-hoc extraction rule is not a validated semantic evaluator. They also refrain from claiming a universal benefit from reasoning cues or a definitive model ranking. This restraint is valuable because the observed results are sensitive to extraction decisions, task composition, and missing predictions. Nevertheless, such careful qualification should be viewed primarily as sound experimental reporting rather than a substitute for stronger methodological contributions or broader empirical validation.
Critical weaknesses
  1. The paper's central observation—that answer extraction rules can substantially influence benchmark accuracy—is valid but not particularly novel. The distinction between answer extractability and mathematical correctness is conceptually straightforward and closely related to existing concerns about evaluation validity in LLM benchmarks. The paper would benefit from a clearer explanation of what new methodological insight it offers beyond quantifying this distinction in a controlled setting. The paper primarily demonstrates this issue through a comparison of two relatively simple extraction rules, without introducing a fundamentally new evaluation framework or a validated solution to the underlying problem. The proposed terminal-answer extraction rule is a heuristic introduced post hoc, rather than a principled semantic evaluation method. Consequently, the work largely quantifies a known measurement artifact in a narrowly controlled setting. It remains unclear what substantive methodological insight the study provides beyond the general recommendation to report extraction coverage separately from accuracy.
  2. The experimental evidence is too narrow to support conclusions of broader relevance to mathematical reasoning evaluation. The study considers only two relatively small, older instruction-tuned models, six handcrafted elementary mathematical task families, and a single deterministic generation per prompt. More importantly, the dataset is deliberately constructed to exclude instances where designated shortcut strategies produce correct answers, introducing a selection bias toward particular failure modes. The unusually severe formatting failure of Llama, with only 3 out of 480 responses successfully extracted under the strict rule, raises concerns that the headline result is driven by an idiosyncratic interaction between the selected checkpoint, prompting protocol, and extraction implementation. Without experiments involving stronger reasoning models, established mathematical benchmarks, alternative prompting configurations, or independent datasets, the paper does not establish whether the observed sensitivity is a general evaluation problem or primarily a peculiarity of this experimental setup.
  3. The magnitude and interpretation of the reported accuracy differences depend heavily on a post-hoc extraction rule that has not been independently validated against human-annotated final answers. Although the authors acknowledge this limitation, the alternative extractor is not validated against independently annotated semantic judgments. As a result, the increase in Llama's measured accuracy from 0.4% to 28.1% cannot be interpreted as an estimate of its actual mathematical competence. The problem is further complicated by the 512-token generation limit, which is reached by 99 Qwen responses and 137 Llama responses, without any controlled experiment examining the effects of extending the generation budget. Moreover, the reported average reasoning-cue effects are small, heterogeneous across task families, and accompanied by wide uncertainty intervals. Thus, while the paper convincingly demonstrates that its scoring outcomes are sensitive to extraction rules, it does not establish which evaluation procedure is more valid, how much mathematical competence is actually being missed, or whether its findings generalize beyond the particular experimental configuration.
Questions for authors
  1. How does the proposed analysis extend beyond existing work on answer extraction and evaluation validity? What is the main methodological contribution beyond quantifying extraction sensitivity?
  2. Can the authors provide independent human validation of the terminal-answer extractor, particularly for responses that fail strict extraction? How frequently does the heuristic recover the model's intended final answer correctly?
  3. How sensitive are the main findings to the choice of models, prompting configurations, and generation budget? In particular, would the large accuracy discrepancy persist with stronger reasoning models and a larger token limit?
1 comment

Thank you for these questions. The methodological contribution is a paired measurement design that holds all 960 generated responses fixed while changing only the scoring layer. It separately identifies extraction availability, correctness conditional on extraction, full-denominator accuracy, and the estimated cue effect. This reveals when an apparent change in model performance is produced entirely by the evaluation rule.

For an established-extractor robustness check, we rescored the same corpus with Math-Verify 0.8.0 under two label-blind configurations. The original strict and terminal rules extract answers from 32.5% and 60.9% of responses. Anchored Math-Verify extracts from 42.7%, while its default configuration produces an extraction for all 960 responses. Unit interpretation also changes the result: among 160 percentage-family responses, default Math-Verify marks 1 correct under raw-number comparison and 103 correct under percentage-aware comparison. The Math-Verify comparison reproduces the paper's central finding that evaluation conventions can change reported conclusions without changing model outputs.

The token-cap analysis localizes the source of the Llama discrepancy. Among 343 responses below the 512-token cap, strict extraction succeeds on 3 and terminal extraction on 243. Among 137 capped responses, terminal extraction succeeds on 2. This separates answer-format mismatch from truncation rather than attributing the result to a single cause.

The result is model- and prompt-dependent: under anchored Math-Verify, availability is 62.9% and 72.9% for Qwen's narrative and scaffolded responses, and 15.4% and 19.6% for Llama. The resulting protocol is concrete: predeclare the response contract, extraction anchor, and unit convention; report availability, conditional correctness, and full-denominator accuracy separately; and check conclusions with an established extractor on the identical saved outputs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.