acceptodds
Under review as a conference paper at ICLR 2027

Finding Answers in Thought Matters: Revisiting Evaluation on Large Language Models with Reasoning

Abstract

Evaluating generative models, such as large language models (LLMs), commonly involves multiple-choice question-answering tasks where the final answer is selected based on the probabilities of the answer choices. For reasoning models, however, the method of answer extraction plays a critical role. In this paper, we show that the benchmark performance of reasoning models and their extracted-answer distributions are highly sensitive to the answer-extraction algorithm employed. To mitigate this sensitivity, we propose a simple framework: Answer Regeneration. The method uses an additional inference step, providing the evaluated model with its original input and previous output, followed by the prefix "Answer:". The final answer is then selected using option probabilities or extracted from the regenerated output. We show that this reduces reliance on handcrafted extraction rules and achieves competitive performance with specialized extractors. Furthermore, it aligns more closely with human interpretation on sampled disagreements and provide reliable performance on incomplete reasoning responses. This framework also extends to mathematical reasoning and open-ended question answering. Our analysis and framework offer a practical approach to more reproducible evaluation of reasoning models.

Primary area:
datasets and benchmarks
Keywords:
reasoning model evaluation, evaluation sensitivity, model-based answer extraction, self-extraction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JYQ5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigated the impact of answer extraction methods on the evaluation of reasoning models. They found that even when applied to the same reasoning output, different extraction algorithms can significantly affect benchmark performance, the distribution of extracted answers, and model rankings. To address this issue, the authors propose an “answer regeneration” framework, which introduces an additional reasoning step that uses the original question, the previous reasoning output, and the “Answer:” prefix to generate concise final answers. The authors conducted experiments on various reasoning models and benchmarks, including MMLU, MMLU-Pro, GSM8K, and TriviaQA. The results show that, compared to traditional extraction methods, answer regeneration improves benchmark performance and achieves better alignment with human interpretations of sampling discrepancies.

Critical strengths
  1. This paper points out that answer extraction methods significantly impact the baseline performance and rankings of large language models. This observation is of great value for improving the reliability and reproducibility of large language model evaluations. The authors substantiate this observation through extensive experiments conducted across various reasoning models and benchmarks.
  2. The proposed “Answer Regeneration” framework is simple in structure and requires no additional model training or complex extraction rules. The authors also conducted a human evaluation on MMLU, comparing the extracted answers with the human-interpreted answers in the original outputs, thereby providing empirical support for the effectiveness of their method.
Critical weaknesses
  1. The evaluation objective of Answer Regeneration is not sufficiently clarified. Although the authors acknowledge the distinction between "faithfully extracting the model's original answer" and "improving the model's answer accuracy," this distinction is not sufficiently addressed in their evaluation. The existing human evaluation on MMLU provides limited evidence of faithful answer recovery, while most quantitative comparisons primarily measure benchmark correctness rather than preservation of the original intended answer. The human evaluation on MMLU also has limitations: (1) It evaluates only those samples where the two methods differ, which only indicates that, within the evaluated disagreement samples, Regeneration is closer to human interpretations. (2) The reported results exclude cases where the original output does not provide a clear, definitive answer or contains multiple possible answers. However, these are precisely the scenarios in which regeneration might choose a different answer or generate a new one. Moreover, the human evaluations on GSM8K and TriviaQA primarily focus on semantic equivalence with the gold answers rather than preservation of the original intent. Additionally, if the research goal is to obtain a more accurate assessment of model capability, it remains unclear why allowing an additional Answer Regeneration step would not change the capability being assessed.
  2. The experimental analysis lacks several important ablation studies. The authors do not compare Answer Regeneration with a second inference step that uses only the original question without the previous reasoning output. It would help determine whether the improvement comes from extracting information from the original reasoning process or simply giving the model another opportunity to answer the question. In addition, the stability of Answer Regeneration under different regeneration prompts and decoding settings is not systematically evaluated. These experiments would provide stronger evidence for the effectiveness and robustness of the proposed framework.
Questions for authors
  1. How does the proposed “Answer Regeneration” method distinguish between faithful extraction of expected answers and generation of new answers in the second inference step? Can you provide a systematic analysis of answer revisions relative to the original human-interpretable answer?
  2. Can you provide an ablation experiment where the second inference step uses only the original question and not the previous inference output? This will help determine whether the reported improvement results from recovering information from the original reasoning trace or simply from additional answering attempts.
  3. Can you design experiments to evaluate the stability of answers generated by the Regeneration method? Including the stability of generated options and the stability of original answer fidelity.
  4. Except for the human evaluation part, is there any experiment designed based on "answer faithfulness" other than "answer correctness"? How is the experiment evaluated?
  5. Assuming the same evaluation set and optimal rule baseline are used, can you clarify how the number of "helpful" and "harmful" flips in Table 6 reconcile with the accuracy gains reported in Table 2?
2 comments

We thank the reviewer for a precise and constructive review.

Summary

  • Objective (W1): the answer the response commits to; new tests measure agreement with it, not with gold.
  • The second pass reads the reasoning (W1, W2/Q2): given another model's reasoning that concludes a wrong answer, models report it 91-97% of the time; question only is 7.9-28.6 points lower.
  • Non-committing responses (W1 (2), Q1): 0.6-4.3% of MMLU items for four labeled models, 23.4% for Phi-4, now reported.
  • Agreement cases (W1 (1)): a blinded audit reads the shared answer in 97.5%. Stability (W2, Q3): accuracy moves 0.1-0.9 points across five prompts.
  • Tables 2 and 6 (Q5) reconcile exactly when recomputed from one run.

W1: Faithful extraction vs. a more accurate answer. We agree, and Sec. 5 will state it: the target is the answer the response commits to, and improving accuracy is not the objective. For committed responses, the step thus does not change the capability assessed: it reports the trace's conclusion even against the model's own correct answer (a). For non-committing responses, every protocol makes a choice (rules mostly score them wrong; regeneration selects an answer), so we report their share per model (2). Tests (a) and (c) measure agreement with the response's conclusion, not gold.

(a) Trace transplant. On items a model answers correctly with its own reasoning, we replace that reasoning with another model's reasoning that concludes a wrong answer W. Reading the trace yields W; re-solving yields the model's own answer (MMLU, every such trace from the other six models):

Q3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
Transplanted traces6,3975,4694,7015,6406,0193,1964,549
Reports the trace's wrong answer91.7%94.7%92.9%91.0%97.2%95.9%90.9%
Reverts to its own correct answer7.2%4.6%6.2%5.3%2.4%2.9%7.5%
Same model, question only: accuracy on these items70.1%66.8%68.0%43.2%68.6%42.7%54.3%

The models can answer many of these items without reasoning, yet report what the reasoning concludes, even for traces from their own model family (89-98%).

(b) Input ablation (W2/Q2). Same regeneration step, only its input changed (MMLU accuracy, %):

Regeneration inputQ3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
Best rule78.881.480.078.264.464.075.6
Question + reasoning (=AnswerRegeneration)86.785.082.684.083.468.880.4
Reasoning only (question and options removed)86.385.082.783.377.866.980.2
Question only (reasoning removed)78.876.272.455.472.050.965.3

The reasoning alone recovers the score within 1.9 points for six of seven models. Phi-4 is the exception (−5.6): its traces mostly stop before concluding (Table 1), so the question helps the second pass select an answer the trace often never states. Other tasks (points below question + reasoning, over the seven models):

MMLU-ProGSM8KTriviaQA
Question only21.8-55.663.5-91.97.3-56.3
Reasoning only1.4-4.8≤ 0.6≤ 1.8

(c) Commitment preservation. Where the best rule finds an option letter (over 90% of items), does regeneration keep it? Gold only splits the items by whether the letter is right:

Q3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
Kept, when the letter is correct99.0%99.5%98.8%99.2%95.0%96.4%98.9%
Kept, when the letter is wrong89.1%91.7%77.2%78.1%22.2%74.5%92.7%

A correctness-seeking step would move wrong letters toward gold; they are mostly kept. Phi-4 is the exception: its rule letter is usually an option mentioned mid-reasoning.

(1) Only disagreements were evaluated. Agreement cases are 87-94% of items for six models (67% for Phi-4), so we audited 100 each from Qwen3-32B and R1-Llama-8B, mixed with 40 disagreements. Two annotators labeled the option each response commits to (or none, or several), seeing only question, options and response (κ = 0.99). Both read the shared answer in 195 of 200 (97.5%); on the disagreements, their answer equals regeneration in 34 of 40 and the rule in 1.

(2) Excluded "no definitive / multiple answers" cases. These are where regeneration may generate an answer; we now include them. Labeled disagreements, by the answer the response itself commits to (independent of gold):

Human reading of the responseQ3-32BQ3-8Bgpt20bPhi4+R1-Ll
Labeled disagreements (n)300300300300300
Recover: regeneration = human-read answer83.7%57.7%85.5%19.3%55.3%
Revise: rule = human-read answer, regeneration differs4.7%13.0%1.2%1.3%8.3%
Generate: no definitive answer, or several10.7%26.7%13.2%79.3%33.7%
  • Revise: regeneration is right in only 0-28% of these cases, the response's own answer in 56-79%, so revisions are errors, not covert corrections (at most 1.1% of MMLU items).
  • Generate: regeneration must choose an option and is correct 43-53% of the time (gpt-oss-20b: 26%; chance: 25%) vs 8-26% for the rule, so part of the gain comes from these cases: 0.6-4.3% of MMLU items for the four models other than Phi-4, but 23.4% for Phi-4, yielding about half of its 19-point gain (counts from the run behind Fig. 6).
  • Fig. 6 and Sec. 5 will report all three categories and state that regeneration selects an answer when none is committed.

GSM8K / TriviaQA human evaluations target equivalence to gold, not intent. We agree and will describe them (Sec. 6.2; Table 4, right) as correctness checks. The transplant test measures intent (W = the trace's final answer; no gold in its closing lines; 41-220 GSM8K and 2,930-7,553 TriviaQA traces per model). Reports W / reverts to own correct answer (%):

Q3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
GSM8K93.2 / 2.794.3 / 4.394.0 / 3.514.5 / 2.0†92.8 / 3.182.9 / 7.385.3 / 0.0
TriviaQA77.3 / 3.382.4 / 3.682.1 / 3.819.8 / 1.6†78.6 / 5.885.0 / 4.776.4 / 6.3

TriviaQA outputs are matched to W or gold by normalized text (≥ 3 characters; the rest are mostly W phrased differently, or empty). Six models answer 26-55% of these items correctly from the question alone, yet none reverts to that answer in more than 6.3% of cases.

Q1: tests (a) and (c), and the recover / revise / generate breakdown in W1 (2) (revisions are errors). Q2: W1 (b). Q4: (a) scores agreement with a wrong conclusion, excluding gold by construction; (c) uses gold only to split by correctness.

Q3 (and W2): Stability under prompts and decoding.

  • Prompts. Across five answer prefixes (e.g., "Answer:", "Therefore, the final answer is"), full-MMLU accuracy moves 0.1-0.9 points per model, 93.9-99.1% of items keep the same answer, and the model ranking is identical; extraction rules change accuracy by up to 78.7 points (Fig. 2).
  • Fidelity. Across the five prompts, the share of transplant outputs reporting the trace's answer moves 0.4-6.4 points; 82.7-98.6% of traces get the identical answer.
  • Sampled regeneration (five samples per item): within 1.4 points of greedy at T = 0.6 (all five agree on 88.5-98.5% of items); 0.2-5.2 points lower at T = 1.0. We keep greedy decoding.
  • Reasoning-pass decoding. Re-running the reasoning pass with each model's recommended sampling (App. A.2) instead of greedy decoding changes AnswerRegeneration by at most 0.6 points (1.3 for Phi-4) and the rules by up to 4.3 points (12.2 for Phi-4).

Q5: Reconciling Table 6 with Table 2. Thank you for catching this. With the same items and best rule, gain = (helpful − harmful) / n. The submitted tables fail it because Table 6 came from an earlier diagnostic run, and Table 2's Rule (best) row, like Fig. 2, averaged the 57 subjects (0.8-3.3 points above per item), while flips are per item. Recomputed from one run, per item, they reconcile exactly:

Q3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
Rule (best)78.881.480.078.264.464.075.6
AnswerRegeneration86.785.082.684.083.468.880.4
Gain+7.9+3.6+2.6+5.8+19.0+4.8+4.8
Helpful (incorrect → correct)1,2225624999003,119990783
Harmful (correct → incorrect)1135813791449322113
(Helpful − harmful) / 14,0427.93.62.65.819.04.84.8

In the revised Table 2, every gain is larger than submitted, and AnswerRegeneration still exceeds xFinder for all seven models; both are within 0.8 and 0.1 points of the submitted values for six models (from the run all new experiments share). Phi-4 changes more (AnswerRegeneration 77.4 -> 83.4, xFinder 65.3 -> 59.9); its submitted AnswerRegeneration value came from a second reasoning run, which gives 82.9 here. Revised Table 6 harmful-flip rates: 0.4-3.2% (submitted: 0.2-2.2%).

Reviewer_X3jcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper focuses on finding answers from a long reasoning response in LLM. Specifically, it proposes AnswerRegeneration, which uses an additional inference step to generate answer based on the original question and reasoning response. The experimental results show that this reduces reliance on handcrafted extraction rules and achieves competitive performance with specialized extractors.

Critical strengths
  1. The proposed method is simple yet effective in identifying answers from long reasoning responses.
  2. The paper conducts extensive experiments comparing different answer extraction methods across multiple models and benchmarks.
Critical weaknesses
  1. The technical novelty of the proposed method is limited.
  2. The experimental insights are relatively limited despite the extensive evaluation.
    In particular, the distinction between answer recovery and answer revision, as discussed in Section 7, deserves further investigation through ablation studies or more fine-grained human evaluations, rather than merely being acknowledged as an insufficiently explored issue.
  3. The motivation for answer extraction is insufficiently justified.
    It remains unclear why failures to follow a prescribed answer format should be addressed through additional inference rather than considered part of the model's task performance.
Questions for authors

None

1 comment

We thank the reviewer for recognizing the method's simplicity and the breadth of the experiments.

W1: Limited novelty. The procedure is simple by design, as an evaluation protocol should be. Prior work documented extraction inconsistencies for instruction-tuned models (Molfese et al., 2025) and proposed trained extractors (xFinder; Yu et al., 2024). Our contribution is twofold:

(1) For reasoning models, whose long traces mention many candidate answers, the extraction step alone changes scores on identical outputs and reorders models, including within one family (Fig. 2), even when the prompt requests a format with a matching extractor (Table 8).

(2) A tuning-free protocol, now characterized by what it measures: when it recovers a committed answer, when it does not, and how often.

W2: Limited insights; recovery vs revision (Sec. 7). We agree this deserves investigation; it is now the main content of the revision, with four findings:

(1) The second pass reads the reasoning; it does not re-solve the question. Given another model's reasoning that concludes a wrong answer, models report it 91-97% of the time on MMLU items they answer correctly themselves, although they answer 43-70% of them correctly from the question alone.

Input ablation. Same regeneration step, only its input changed (MMLU accuracy, %):

Regeneration inputQ3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
Best rule78.881.480.078.264.464.075.6
Question + reasoning (=AnswerRegeneration)86.785.082.684.083.468.880.4
Reasoning only (question and options removed)86.385.082.783.377.866.980.2
Question only (reasoning removed)78.876.272.455.472.050.965.3

Phi-4 is the exception (−5.6 with reasoning only): its traces mostly stop before concluding (Table 1), so the question helps the second pass select an answer the trace often never states.

(2) Revisions are errors, not hidden corrections. Fine-grained human evaluation of rule-regeneration disagreements, now including previously excluded cases, by the answer the response commits to (independent of gold):

Human reading of the responseQ3-32BQ3-8Bgpt20bPhi4+R1-Ll
Labeled disagreements (n)300300300300300
Recover: regeneration = human-read answer83.7%57.7%85.5%19.3%55.3%
Revise: rule = human-read answer, regeneration differs4.7%13.0%1.2%1.3%8.3%
Generate: no definitive answer, or several10.7%26.7%13.2%79.3%33.7%

When regeneration departs from the human-read answer (Revise), it is usually wrong (0-28% correct, vs 56-79% for the rule). Where rule and regeneration agree, a blinded audit of 200 items (κ = 0.99) reads the shared answer in 97.5%.

(3) Part of the gain comes from responses that never commit (Generate), and we now quantify it: 0.6-4.3% of MMLU items for the four other labeled models, 23.4% for Phi-4. For these, regeneration selects an answer (correct 43-53% of the time; 26% for gpt-oss-20b).

(4) The protocol is stable where rules are not. Accuracy moves 0.1-0.9 points across five regeneration prompts, with the ranking unchanged, but up to 78.7 points across extraction rules, which reorder models.

W3: Why extract answers instead of treating format failures as part of task performance?

(1) No answer format was requested. Our prompts (lm-evaluation-harness thinking templates, App. A.2) contain only the question and options, so responses commit in many forms: a boxed letter, "Answer: (D)", or an option's text ("the answer is Merton", Fig. 1). Recognizing a commitment is the extractor's job, not the model's.

(2) Counting format deviations as failures gives implausible MMLU scores: 2.8% for gpt-oss-20b and 7.6-22.9% for the other models under the harness's strict-match pattern (Fig. 2). Option-likelihood scoring (Sec. 1) keeps formatting out of the score; reasoning models make it inapplicable, and AnswerRegeneration restores the separation.

(3) When a format is requested, the score depends on which format. Four prompt-aligned format/extractor pairs change a single model's score by up to 23 / 28 / 57 / 13 points (MMLU / MMLU-Pro / GSM8K / TriviaQA), and no format is best across models and tasks (Table 8).

(4) Rule failures are mostly misreadings, not missing answers (except for Phi-4). Annotators found a definitive answer in 64-88% of disagreements for the four models other than Phi-4 (21% for Phi-4), and it matched regeneration far more often than the rule.

Format compliance is a legitimate capability; we will recommend reporting it separately (the rule-based score alongside the regenerated one) rather than letting it decide a knowledge benchmark.

We hope these analyses address the reviewer's concerns.

Reviewer_Q5xsOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the impact of different answer extraction methods on the evaluation results of reasoning models and finds that extraction rules can significantly change the model scores and rankings. The author proposes Answer Regeneration, which obtains the final answer through an additional non-reasoning mode generation to reduce the dependence on manually designed extraction rules. The experiments cover MMLU, MMLU-Pro, GSM8K, and TriviaQA, and the effectiveness of the method is verified through human evaluation.

Critical strengths
  1. The research question is important and practically valuable. This system reveals how answer extraction strategies affect model evaluation results, helping to improve the reliability of reasoning model assessments.
Critical weaknesses
  1. It's hard to distinguish between answer extraction and secondary reasoning. Extra reasoning takes in both the original question and the reasoning process, and it might regenerate the answer instead of just extracting the original result. This affects how reliable this method is as an evaluation tool.

  2. There's a lack of sufficient efficiency and sensitivity analysis. The method requires extra processing of the full reasoning context, but the actual delay isn’t reported. Also, there isn’t enough testing on how robust it is to changes in the regenerated prompts.

  3. The numbers in Figure 2 are overlapping, which looks ugly and unprofessional. Figure 6 even overlaps Figure 5. I think this is a serious issue and could make people doubt the quality of the paper and the effort put into the work. A random image somehow got inserted into table 2, and I didn’t see a caption. Same with table 4.

Questions for authors
  • Is it possible to add ablation experiments that use only the original question or only the reasoning process, to distinguish the contributions of answer recovery versus re-reasoning?

  • Can you report the actual reasoning delay and analyze how different regeneration prompts affect the evaluation results?

  • Why are there so many formatting issues? Do you understand ICLR's submission requirements and guidelines, or the submission rules for any conference for that matter?

2 comments

We thank the reviewer for the clear, actionable questions.

Summary

  • The second pass reads the reasoning; it does not re-solve the question (W1/Q1). On MMLU, reasoning only is within 1.9 points of AnswerRegeneration for six of seven models; question only is 7.9-28.6 points lower.
  • The overhead is small (W2/Q2): +14% wall-clock over the reasoning pass, the same as xFinder, without a second model.
  • The prompt barely matters (W2/Q2): across five regeneration prompts, MMLU accuracy moves 0.1-0.9 points and the model ranking does not change.
  • Formatting (W3): we will fix the Fig. 5/6 overlap; the other elements are explained below.

W1/Q1: Answer extraction or secondary reasoning? Ablations with only the question or only the reasoning.

We ran exactly this ablation. Same regeneration step, only its input changed (MMLU accuracy, %):

Regeneration inputQ3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
Best rule78.881.480.078.264.464.075.6
Question + reasoning (=AnswerRegeneration)86.785.082.684.083.468.880.4
Reasoning only (question and options removed)86.385.082.783.377.866.980.2
Question only (reasoning removed)78.876.272.455.472.050.965.3
  • Reasoning only ≈ question + reasoning for six of seven models (within 1.9 points): the answer comes from the reasoning.
  • Question only is 7.9-28.6 points lower for every model: a second attempt at the question does not explain the scores.
  • Phi-4 is the exception (-5.6). Its traces mostly stop before concluding (Table 1), so the question helps the second pass select an answer the trace often never states.

The same holds on the other three tasks (points below question + reasoning, range over the seven models):

MMLU-ProGSM8KTriviaQA
Question only21.8-55.663.5-91.97.3-56.3
Reasoning only1.4-4.8≤ 0.6≤ 1.8

A direct test of re-solving. On items a model answers correctly with its own reasoning, we replace that reasoning with another model's reasoning that concludes a wrong answer. If the second pass reads the reasoning, it returns the wrong answer; if it re-solves the question, it should often return the model's own correct answer (MMLU):

Q3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
Transplanted traces6,3975,4694,7015,6406,0193,1964,549
Reports the trace's wrong answer91.7%94.7%92.9%91.0%97.2%95.9%90.9%
Reverts to its own correct answer7.2%4.6%6.2%5.3%2.4%2.9%7.5%
Same model, question only: accuracy on these items70.1%66.8%68.0%43.2%68.6%42.7%54.3%

Models report the wrong conclusion although they answer 43-70% of these items correctly from the question alone; the same holds for traces from the same model family (89-98%). On GSM8K and TriviaQA, six of seven models report the trace's answer 83-94% and 76-85% of the time. The second pass reports what the reasoning concludes.

W2/Q2: Actual delay, and sensitivity to the regeneration prompt.

Delay. Qwen3-8B on one NVIDIA B200 (vLLM 0.33.0, prefix caching on), all 14,042 MMLU items, 64 concurrent requests:

PhaseWall-clockRelative to the reasoning pass
Reasoning pass (always paid; mean 1,503 output tokens)2,005.5 s100%
AnswerRegeneration280.0 s+14.0%
AnswerRegeneration, no prefix cache287.3 s+14.3%
xFinder-llama38it (a second model to host)267.0 s+13.3%
Regex rule6.3 s~0%

Regeneration prompts. Five answer prefixes ("Answer:" as in the paper, "Answer is", "The answer is", "Final answer:", "Therefore, the final answer is"), full MMLU:

Q3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
Accuracy range across the 5 prompts (pts)0.30.10.10.20.60.90.5
Same answer under all 5 prompts98.5%99.1%98.4%98.6%95.5%93.9%96.4%
  • The model ranking is identical under all five; extraction rules change accuracy by up to 78.7 points on the same outputs and reorder the models (Fig. 2).
  • Sampled regeneration (five samples per item) stays within 1.4 points of greedy at T = 0.6 and loses 0.2-5.2 points at T = 1.0, so the protocol keeps greedy decoding.

W3/Q3: Formatting. The paper uses the official ICLR 2027 template within the page limit. We apologize for Fig. 6 overlapping the last line of Fig. 5's caption, a float-placement error we will fix. The right-hand panels of Tables 2 and 4 are part of those tables, and each caption describes them under "(right)". The value labels in Fig. 2 are legible.

We hope the new experiments address the reviewer's concerns.

Thank you for your reply. Please keep working hard to improve your work.

Comments on the paper

2 comments

Thank you for the follow-up.

As you said, the human labels are the direct evidence. These indirect tests add what sampled human annotations (done in the submission and this response) cannot: at full scale on all 7 models, the transplants show that re-solving and moving toward gold are rare.

(a-1) Why only wrong-answer traces?

(a) uses items the model gets right. There, only a wrong trace separates reading from re-solving: reading gives the trace's answer, re-solving gives the model's own. A correct trace gives the same answer either way.

(a-2) Reverts with correct traces.

As suggested, we also gave each model correct traces from other models on items it gets wrong on its own (2,840-14,692 traces per model). It rarely reverts to its own wrong answer:

Q3-32BQ3-14BQ3-8Bgpt20bPhi4+R1-LlR1-Q3
Reverts to its own wrong answer5.2%2.2%3.9%2.1%0.7%1.2%4.2%
Question only: gives that wrong answer50.0%54.4%61.5%36.2%63.8%39.8%47.6%

(a-3) Manual check on the real dataset.

  • Labeling every output from every model is not feasible, so we check samples manually.
    • The paper labels 300 disagreements per model for five models (Fig. 6), and this response adds agreements (87-89% of items) for Qwen3-32B and R1-Llama-8B.
    • Putting the two groups together in their real proportions, regeneration matches the annotators on 96.8% and 95.1% of all MMLU items, and the best rule on 86.5% and 85.5% (95.5% and 92.6% with the 300 disagreement labels of (2)).

(b) Parity with reasoning-only is the expected result.

  • It shows that the answer comes from the reasoning, not the question.
  • With and without the question, the step picks the same option on 99.0% (Q3-14B) and 98.1% (Q3-8B) of items; the 0.1-point gap is 4 of 14,042 items.
  • Removing the reasoning instead costs 8.8 and 10.2 points.
  • We keep the question because it costs at most 0.1 points, and it helps when a trace stops before concluding (Phi-4: 77.8 -> 83.4); those are the "generate" cases of (2).

(c-1) Is the best rule's letter faithful? "Kept" is agreement, not faithfulness.

We agree; the rule's letter is not ground truth, and (c) only shows that regeneration rarely moves wrong letters toward gold. Faithfulness comes from the human labels below.

(c-2) Does regeneration extract the answer when the rule does not?

Where the two methods disagree, annotators read regeneration's answer in 34 of 40 items and the rule's in 1 (in the 300 labeled disagreements per model of (2): 19-86% vs 1-13%).

(c-3) Does regeneration keep the trace's meaning?

Where the two methods agree, annotators read that answer in 195 of 200 items. Over all MMLU items, regeneration matches the annotators on 96.8% and 95.1% (a-3).

AuthorsOpenReview ↗

Summary of discussion and Revision

We thank the reviewers and the AC.

Summary of discussion

All three reviewers asked whether the second pass recovers the answer the reasoning commits to or solves the question again (Q5xs-W1, JYQ5-W1/W2, X3jc-W2). Our responses answered this with new tests. Given another model's reasoning that concludes a wrong answer, models report that answer 91-97% of the time. With the question alone, accuracy drops by 7.9-28.6 points. The human evaluation now also covers responses that commit to no answer. The responses also reported:

  • the cost: +14% wall-clock, about as much as xFinder (Q5xs);
  • stability across five regeneration prompts: 0.1-0.9 points (Q5xs, JYQ5);
  • the reconciliation of Tables 2 and 6, now Table 8 (JYQ5);
  • the contributions and why we extract answers (X3jc).

We answered JYQ5's follow-up questions with a reverse transplant and the human labels. In the reverse transplant, models revert to their own wrong answer in 0.7-5.2% of cases. Where rule and regeneration agree, annotators read the shared answer in 195 of 200 audited items. Q5xs and X3jc acknowledged our responses and kept their scores without raising a remaining concern.

We have revised the paper to include all of these results. The revision uses the one additional page. The changes highlighted in blue.

Main changes

  • Recovery vs. re-solving (Q5xs-W1/Q1, JYQ5-W1/W2/Q1/Q2/Q4, X3jc-W2). New Sec. 5.3 (Tables 3 and 14) tests this on MMLU with an input ablation (question only or reasoning only), a trace transplant, and commitment preservation. App. A.9 adds the other tasks, the reverse transplant with correct traces, and the agreement audit from the JYQ5 discussion. Sec. 1, 7 and 8 state the findings.
  • Human evaluation (JYQ5-W1, JYQ5-Q1, X3jc-W2). Sec. 5.2.1 and Fig. 6 split the labeled disagreements into recover, revise and uncommitted, so responses that commit to no answer are now included. Sec. 5.2.1 also reports an audit of 200 agreement cases (App. A.9). Sec. 3 states what each human evaluation checks: on MMLU, whether the extracted answer is the one the response commits to; on GSM8K and TriviaQA (Sec. 6.2, 6.3, Table 5), whether it matches gold.
  • Evaluation objective (JYQ5-W1). Sec. 5.1 states the target: the answer the response commits to, not a more accurate answer. When a response commits to none, regeneration selects one; Sec. 5.2.1 reports how often.
  • Stability (Q5xs-W2/Q2, JYQ5-W2/Q3). Sec. 7 and App. A.10 report five regeneration prompts (Table 15), sampled regeneration, a sampled reasoning pass, and an abstaining variant.
  • Cost (Q5xs-W2/Q2). App. A.7 (Table 7) reports wall-clock time: regeneration adds 14.0% to the reasoning pass, about as much as xFinder (13.3%). Sec. 5.1 and 7 summarize it.
  • Tables 2 and 8, Fig. 2 (JYQ5-Q5). Table 2 reports per-item accuracy over all 14,042 MMLU items, using each model's best rule on the same outputs. Fig. 2 is redrawn per item on these outputs, so its best bar matches Table 2; instructed-format uses its own prompt and is reported in Table 10. Table 8 (submitted Table 6) counts flips on the same run and rule, so helpful minus harmful equals the gain in Table 2 up to rounding. Values that repeat or depend on Table 2 (Table 1, Table 2 right, Fig. 6 left, Sec. 5.2.2 and 5.3, the AnsReg values in Table 10's MMLU rows, App. A.8 and A.9) are updated.
  • Novelty and motivation (X3jc-W1, X3jc-W3). Sec. 1 states the contributions more precisely. New App. A.11 explains why we extract answers instead of counting format failures.
  • Recommendations (JYQ5, X3jc-W3). Sec. 7 recommends reporting the rule-based score alongside the regenerated one, and Sec. 8 the share of responses that commit to no answer.
  • Layout (Q5xs-W3). Fig. 6 no longer overlaps the Fig. 5 caption.

Table numbers. Our responses use the submitted numbering. Submitted Tables 3, 4, 5, 6, 7 and 8 are now Tables 4, 5, 6, 8, 9 and 10. New tables: Table 3 (Sec. 5.3), Table 7 (latency) and Tables 11-15 (App. A.9-A.10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.