- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors investigated the impact of answer extraction methods on the evaluation of reasoning models. They found that even when applied to the same reasoning output, different extraction algorithms can significantly affect benchmark performance, the distribution of extracted answers, and model rankings. To address this issue, the authors propose an “answer regeneration” framework, which introduces an additional reasoning step that uses the original question, the previous reasoning output, and the “Answer:” prefix to generate concise final answers. The authors conducted experiments on various reasoning models and benchmarks, including MMLU, MMLU-Pro, GSM8K, and TriviaQA. The results show that, compared to traditional extraction methods, answer regeneration improves benchmark performance and achieves better alignment with human interpretations of sampling discrepancies.
Critical strengths
- This paper points out that answer extraction methods significantly impact the baseline performance and rankings of large language models. This observation is of great value for improving the reliability and reproducibility of large language model evaluations. The authors substantiate this observation through extensive experiments conducted across various reasoning models and benchmarks.
- The proposed “Answer Regeneration” framework is simple in structure and requires no additional model training or complex extraction rules. The authors also conducted a human evaluation on MMLU, comparing the extracted answers with the human-interpreted answers in the original outputs, thereby providing empirical support for the effectiveness of their method.
Critical weaknesses
- The evaluation objective of Answer Regeneration is not sufficiently clarified. Although the authors acknowledge the distinction between "faithfully extracting the model's original answer" and "improving the model's answer accuracy," this distinction is not sufficiently addressed in their evaluation. The existing human evaluation on MMLU provides limited evidence of faithful answer recovery, while most quantitative comparisons primarily measure benchmark correctness rather than preservation of the original intended answer. The human evaluation on MMLU also has limitations: (1) It evaluates only those samples where the two methods differ, which only indicates that, within the evaluated disagreement samples, Regeneration is closer to human interpretations. (2) The reported results exclude cases where the original output does not provide a clear, definitive answer or contains multiple possible answers. However, these are precisely the scenarios in which regeneration might choose a different answer or generate a new one. Moreover, the human evaluations on GSM8K and TriviaQA primarily focus on semantic equivalence with the gold answers rather than preservation of the original intent. Additionally, if the research goal is to obtain a more accurate assessment of model capability, it remains unclear why allowing an additional Answer Regeneration step would not change the capability being assessed.
- The experimental analysis lacks several important ablation studies. The authors do not compare Answer Regeneration with a second inference step that uses only the original question without the previous reasoning output. It would help determine whether the improvement comes from extracting information from the original reasoning process or simply giving the model another opportunity to answer the question. In addition, the stability of Answer Regeneration under different regeneration prompts and decoding settings is not systematically evaluated. These experiments would provide stronger evidence for the effectiveness and robustness of the proposed framework.
Questions for authors
- How does the proposed “Answer Regeneration” method distinguish between faithful extraction of expected answers and generation of new answers in the second inference step? Can you provide a systematic analysis of answer revisions relative to the original human-interpretable answer?
- Can you provide an ablation experiment where the second inference step uses only the original question and not the previous inference output? This will help determine whether the reported improvement results from recovering information from the original reasoning trace or simply from additional answering attempts.
- Can you design experiments to evaluate the stability of answers generated by the Regeneration method? Including the stability of generated options and the stability of original answer fidelity.
- Except for the human evaluation part, is there any experiment designed based on "answer faithfulness" other than "answer correctness"? How is the experiment evaluated?
- Assuming the same evaluation set and optimal rule baseline are used, can you clarify how the number of "helpful" and "harmful" flips in Table 6 reconcile with the accuracy gains reported in Table 2?
2 comments
(Cont.)
(2) Excluded "no definitive / multiple answers" cases. These are where regeneration may generate an answer; we now include them. Labeled disagreements, by the answer the response itself commits to (independent of gold):
| Human reading of the response | Q3-32B | Q3-8B | gpt20b | Phi4+ | R1-Ll |
|---|---|---|---|---|---|
| Labeled disagreements (n) | 300 | 300 | 300 | 300 | 300 |
| Recover: regeneration = human-read answer | 83.7% | 57.7% | 85.5% | 19.3% | 55.3% |
| Revise: rule = human-read answer, regeneration differs | 4.7% | 13.0% | 1.2% | 1.3% | 8.3% |
| Generate: no definitive answer, or several | 10.7% | 26.7% | 13.2% | 79.3% | 33.7% |
- Revise: regeneration is right in only 0-28% of these cases, the response's own answer in 56-79%, so revisions are errors, not covert corrections (at most 1.1% of MMLU items).
- Generate: regeneration must choose an option and is correct 43-53% of the time (gpt-oss-20b: 26%; chance: 25%) vs 8-26% for the rule, so part of the gain comes from these cases: 0.6-4.3% of MMLU items for the four models other than Phi-4, but 23.4% for Phi-4, yielding about half of its 19-point gain (counts from the run behind Fig. 6).
- Fig. 6 and Sec. 5 will report all three categories and state that regeneration selects an answer when none is committed.
GSM8K / TriviaQA human evaluations target equivalence to gold, not intent. We agree and will describe them (Sec. 6.2; Table 4, right) as correctness checks. The transplant test measures intent (W = the trace's final answer; no gold in its closing lines; 41-220 GSM8K and 2,930-7,553 TriviaQA traces per model). Reports W / reverts to own correct answer (%):
| Q3-32B | Q3-14B | Q3-8B | gpt20b | Phi4+ | R1-Ll | R1-Q3 | |
|---|---|---|---|---|---|---|---|
| GSM8K | 93.2 / 2.7 | 94.3 / 4.3 | 94.0 / 3.5 | 14.5 / 2.0† | 92.8 / 3.1 | 82.9 / 7.3 | 85.3 / 0.0 |
| TriviaQA | 77.3 / 3.3 | 82.4 / 3.6 | 82.1 / 3.8 | 19.8 / 1.6† | 78.6 / 5.8 | 85.0 / 4.7 | 76.4 / 6.3 |
TriviaQA outputs are matched to W or gold by normalized text (≥ 3 characters; the rest are mostly W phrased differently, or empty). Six models answer 26-55% of these items correctly from the question alone, yet none reverts to that answer in more than 6.3% of cases.
Q1: tests (a) and (c), and the recover / revise / generate breakdown in W1 (2) (revisions are errors). Q2: W1 (b). Q4: (a) scores agreement with a wrong conclusion, excluding gold by construction; (c) uses gold only to split by correctness.
Q3 (and W2): Stability under prompts and decoding.
- Prompts. Across five answer prefixes (e.g., "Answer:", "Therefore, the final answer is"), full-MMLU accuracy moves 0.1-0.9 points per model, 93.9-99.1% of items keep the same answer, and the model ranking is identical; extraction rules change accuracy by up to 78.7 points (Fig. 2).
- Fidelity. Across the five prompts, the share of transplant outputs reporting the trace's answer moves 0.4-6.4 points; 82.7-98.6% of traces get the identical answer.
- Sampled regeneration (five samples per item): within 1.4 points of greedy at T = 0.6 (all five agree on 88.5-98.5% of items); 0.2-5.2 points lower at T = 1.0. We keep greedy decoding.
- Reasoning-pass decoding. Re-running the reasoning pass with each model's recommended sampling (App. A.2) instead of greedy decoding changes AnswerRegeneration by at most 0.6 points (1.3 for Phi-4) and the rules by up to 4.3 points (12.2 for Phi-4).
Q5: Reconciling Table 6 with Table 2. Thank you for catching this. With the same items and best rule, gain = (helpful − harmful) / n. The submitted tables fail it because Table 6 came from an earlier diagnostic run, and Table 2's Rule (best) row, like Fig. 2, averaged the 57 subjects (0.8-3.3 points above per item), while flips are per item. Recomputed from one run, per item, they reconcile exactly:
| Q3-32B | Q3-14B | Q3-8B | gpt20b | Phi4+ | R1-Ll | R1-Q3 | |
|---|---|---|---|---|---|---|---|
| Rule (best) | 78.8 | 81.4 | 80.0 | 78.2 | 64.4 | 64.0 | 75.6 |
| AnswerRegeneration | 86.7 | 85.0 | 82.6 | 84.0 | 83.4 | 68.8 | 80.4 |
| Gain | +7.9 | +3.6 | +2.6 | +5.8 | +19.0 | +4.8 | +4.8 |
| Helpful (incorrect → correct) | 1,222 | 562 | 499 | 900 | 3,119 | 990 | 783 |
| Harmful (correct → incorrect) | 113 | 58 | 137 | 91 | 449 | 322 | 113 |
| (Helpful − harmful) / 14,042 | 7.9 | 3.6 | 2.6 | 5.8 | 19.0 | 4.8 | 4.8 |
In the revised Table 2, every gain is larger than submitted, and AnswerRegeneration still exceeds xFinder for all seven models; both are within 0.8 and 0.1 points of the submitted values for six models (from the run all new experiments share). Phi-4 changes more (AnswerRegeneration 77.4 -> 83.4, xFinder 65.3 -> 59.9); its submitted AnswerRegeneration value came from a second reasoning run, which gives 82.9 here. Revised Table 6 harmful-flip rates: 0.4-3.2% (submitted: 0.2-2.2%).
We thank the reviewer for a precise and constructive review.
Summary
W1: Faithful extraction vs. a more accurate answer. We agree, and Sec. 5 will state it: the target is the answer the response commits to, and improving accuracy is not the objective. For committed responses, the step thus does not change the capability assessed: it reports the trace's conclusion even against the model's own correct answer (a). For non-committing responses, every protocol makes a choice (rules mostly score them wrong; regeneration selects an answer), so we report their share per model (2). Tests (a) and (c) measure agreement with the response's conclusion, not gold.
(a) Trace transplant. On items a model answers correctly with its own reasoning, we replace that reasoning with another model's reasoning that concludes a wrong answer W. Reading the trace yields W; re-solving yields the model's own answer (MMLU, every such trace from the other six models):
The models can answer many of these items without reasoning, yet report what the reasoning concludes, even for traces from their own model family (89-98%).
(b) Input ablation (W2/Q2). Same regeneration step, only its input changed (MMLU accuracy, %):
The reasoning alone recovers the score within 1.9 points for six of seven models. Phi-4 is the exception (−5.6): its traces mostly stop before concluding (Table 1), so the question helps the second pass select an answer the trace often never states. Other tasks (points below question + reasoning, over the seven models):
(c) Commitment preservation. Where the best rule finds an option letter (over 90% of items), does regeneration keep it? Gold only splits the items by whether the letter is right:
A correctness-seeking step would move wrong letters toward gold; they are mostly kept. Phi-4 is the exception: its rule letter is usually an option mentioned mid-reasoning.
(1) Only disagreements were evaluated. Agreement cases are 87-94% of items for six models (67% for Phi-4), so we audited 100 each from Qwen3-32B and R1-Llama-8B, mixed with 40 disagreements. Two annotators labeled the option each response commits to (or none, or several), seeing only question, options and response (κ = 0.99). Both read the shared answer in 195 of 200 (97.5%); on the disagreements, their answer equals regeneration in 34 of 40 and the rule in 1.