acceptodds
Under review as a conference paper at ICLR 2027

Learning Teacher Output Preferences from Final Responses for Reasoning Distillation

Abstract

Large reasoning models often expose only their final responses while keeping their native chain-of-thought (CoT) unavailable, limiting their use as teachers for smaller models. Training directly on final responses omits intermediate supervision, while generic synthetic rationales may overlook source-dependent output patterns. We investigate whether visible final responses provide signals for synthesizing reasoning supervision that reflects these patterns. We introduce Final-Output Preference Learning (FOPL), a final-response-conditioned preference-learning framework. In the first offline stage, we use auxiliary trace-accessible source teachers to pair each native CoT with answer-conditioned synthetic alternatives generated by strong reasoning models. Each preference pair shares the same question and visible final response, allowing a single reconstruction policy to learn preferences over reasoning realizations while holding the response fixed. At the second transfer stage, the policy generates a CoT from only a question and the target teacher's visible final response, without access to that teacher's identity or native CoT. Reconstructions conditioned on a teacher family's own responses resemble its native traces more closely than those conditioned on other families' responses, including for a family excluded from preference training. On several completed reasoning evaluations, students trained on reconstructed CoTs match or outperform students trained on native-CoT references. These results suggest that visible final responses can support useful reasoning supervision without requiring the recovery of a target teacher's hidden reasoning process.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, chain-of-thought

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5C37OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FOPL, a framework for synthesizing reasoning supervision when a target teacher exposes only its final response. FOPL trains a reconstruction policy with DPO using native CoTs as preferred traces and answer-conditioned synthetic CoTs as rejected traces, while keeping the question and visible final response fixed. At transfer time, the policy reconstructs a CoT using only the target teacher’s question and final response. Experiments show source-dependent reconstruction across several teacher families, including a held-out MiMo family, and improved downstream distillation into Qwen2.5-1.5B. The paper also evaluates judge robustness, final-response-only supervision, Trace Inversion, data scale, and transfer from a closed-source GPT-5.6-Luna teacher.

Critical strengths
  1. Practical and interesting problem setting. The paper targets a realistic distillation setting where strong teachers expose final responses but not native CoTs. The held-out MiMo experiment and the GPT-5.6-Luna transfer make the intended use case more convincing.
  2. Strong downstream results. FOPL consistently outperforms prompted synthetic-CoT baselines and often matches or exceeds native-CoT supervision on reasoning benchmarks. The controls against final-response-only training and Trace Inversion further support the usefulness of the reconstructed traces.
Critical weaknesses
  1. The value of preference learning is not cleanly isolated. FOPL is not compared with a matched SFT reconstructor trained on the same multi-family native traces. The Trace Inversion baseline uses different training data, so it is hard to attribute the gain specifically to DPO.
  2. Downstream validation uses only one small student. All distillation experiments use Qwen2.5-1.5B-Base. It is unclear whether the gains hold for larger students or other model families.
  3. The source-alignment evidence is narrow. The main cross-family similarity experiment is only conducted on MATH-500. It is unclear whether the same teacher-specific signal appears in other reasoning domains.
Questions for authors
  1. Can you compare FOPL with a matched SFT reconstructor trained on the same multi-family native traces, prompts, initialization, and training budget?

  2. Have you tested FOPL with larger or more recent student models, and on harder recent reasoning benchmarks?

  3. Does the source-matching effect still hold outside MATH-500, for example on code or science reasoning tasks?

4 comments

Thank you for your thorough comments and support!

W1&Q1: The value of preference learning is not cleanly isolated. FOPL is not compared with a matched SFT reconstructor trained on the same multi-family native traces. The Trace Inversion baseline uses different training data, so it is hard to attribute the gain specifically to DPO.

Response: Thank you for this suggestion. We agree that the previous comparison with Trace Inversion (TI) did not sufficiently isolate the contribution of preference learning, because TI used different training data from FOPL.

To address this concern, we trained an SFT reconstruction policy directly on the same multi-family native traces used as the positive examples in FOPL’s DPO pairs. We matched the reconstruction model, training data size, and training hyperparameters. Both reconstruction policies were then used to generate student supervision conditioned on the same DeepSeek final responses.

Student supervisionGSM8KMATH-500GPQAOpenBookQAMBPP
FOPL, SFT reconstructor48.6719.6013.1333.6029.80
FOPL, DPO reconstructor57.8533.2019.1950.3032.22
DPO − SFT, percentage points+9.18+13.60+6.06+16.70+2.42

Under this matched comparison, the DPO-trained reconstructor produces stronger downstream student supervision across all five benchmarks. This result supports the contribution of preference learning beyond direct supervised training on the same native traces. We will add the matched SFT baseline and this comparison to the revised paper.

W2&Q2: Downstream validation uses only one small student. All distillation experiments use Qwen2.5-1.5B-Base. It is unclear whether the gains hold for larger students or other model families.

Response: Thank you for this suggestion. To examine whether the gains extend to a larger student from a different model family, we additionally evaluated Llama-3.2-3B-Base on GSM8K.

Response sourceStudent supervisionGSM8K
Base model—1.52
DeepSeekNative CoT21.30
DeepSeekDeepSeek synthetic CoT14.18
DeepSeekKimi synthetic CoT13.42
DeepSeekGLM synthetic CoT14.10
DeepSeekFOPL, DPO reconstructor21.46

FOPL achieves 21.46 on GSM8K, outperforming the prompted synthetic-CoT baselines by 7.28–8.04 percentage points and obtaining a result comparable to native-CoT supervision. These results suggest that FOPL’s downstream benefits extend to a larger student from another model family.

Due to limited time and the use of only 5K training examples, gains on substantially larger models may be less apparent and may require more training data. We therefore focused on validating a different model family.

W3&Q3: The source-alignment evidence is narrow. The main cross-family similarity experiment is only conducted on MATH-500. It is unclear whether the same teacher-specific signal appears in other reasoning domains.

Response: Thank you for this suggestion. To examine source alignment outside the mathematical domain, we conducted an additional source-matching experiment on MBPP, using the FOPL-DPO reconstructor and Gemini as the judge.

Gemini preference rateDeepSeek synKimi synGLM synMiMo syn
DeepSeek ref78.73%25.52%22.66%15.62%
Kimi ref14.06%84.81%24.22%7.29%
GLM ref20.31%24.48%82.29%8.33%
MiMo ref11.72%11.98%11.98%88.11%

The same-source reconstruction has the highest preference rate for each of the four reference teachers, including the held-out MiMo family. These results provide additional evidence of source-specific alignment in the coding domain, beyond the MATH-500 analysis. We will include the MBPP experiment in the revised paper.

AuthorsOpenReview ↗

Follow-up on rebuttal

Dear Reviewer 5C37,

Thank you again for your valuable comments and supports! We have carefully considered your feedback, provided our responses, and submitted a revised version of the paper.

Please let us know if our responses and revisions have addressed your concerns. We would be happy to answer any further questions or provide additional clarification.

Thank you very much!

AuthorsOpenReview ↗

Further follow-up on rebuttal

Dear Reviewer 5C37,

Thank you very much for your time and helpful comments. We have addressed your concerns in our rebuttal. As the discussion period is drawing to a close, we would appreciate any further comments or suggestions you may have on our responses.

Thank you again for your time and consideration.

The Authors

Thank you for the additional experiments and clarifications. I appreciate the authors’ efforts to address the concerns raised in my review. After carefully considering the new evidence, I do not find sufficient reason to revise my initial assessment, and therefore maintain my original score.

Reviewer_Sou3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies reasoning distillation when a target teacher exposes its final responses but not its native reasoning traces. FOPL trains a response-conditioned reconstruction policy using preferences between native traces from accessible source teachers and synthetic alternatives conditioned on the same question and final response. The policy is then applied to target-teacher responses to generate reasoning traces for student supervised fine-tuning.

Critical strengths
  • The method addresses a practical limitation of distillation from teachers with inaccessible reasoning traces. Using observable cues in final responses to construct reasoning supervision offers a potentially useful approach in this setting.
Critical weaknesses
  • The contribution of preference training over direct supervised training of the reconstruction policy is not sufficiently isolated. Although Appendix C includes a supervised reconstruction baseline, TI, its training data differ from those used by FOPL, and the comparison does not establish the advantage of DPO under matched reconstruction models, data, and training budgets.

  • The source-matching analysis provides evidence that reconstruction is sensitive to information in the final response beyond the answer value. However, it does not distinguish solution-related cues from stylistic features or establish whether preference training exploits these cues better than supervised reconstruction, leaving their connection to the downstream gains unclear.

Questions for authors
  • How would the reconstruction policy perform if trained directly with SFT on the same native traces used as positive examples for FOPL? Could the authors compare this alternative with DPO while matching the model, training data, and budget?

  • Could the authors include this SFT reconstruction policy in the source-matching analysis and examine solution-related cues, such as strategy and verification, separately from wording and notation? This would help clarify what preference training adds and how that relates to the distillation gains.

5 comments

Thank you for your thorough comments.

W1: The contribution of preference training over direct supervised training of the reconstruction policy is not sufficiently isolated. Although Appendix C includes a supervised reconstruction baseline, TI, its training data differ from those used by FOPL, and the comparison does not establish the advantage of DPO under matched reconstruction models, data, and training budgets.

Response: Thank you for this suggestion. We agree that the previous comparison with Trace Inversion (TI) did not sufficiently isolate the contribution of preference training, because TI used different training data from FOPL.

To address this concern, we trained an SFT reconstruction policy directly on the same native traces used as the positive examples in FOPL’s DPO pairs. We matched the reconstruction model, training data size, and training hyperparameters. Both reconstruction policies were then used to generate student supervision conditioned on the same DeepSeek final responses.

Student supervisionGSM8KMATH-500GPQAOpenBookQAMBPP
FOPL, SFT reconstructor48.6719.6013.1333.6029.80
FOPL, DPO reconstructor57.8533.2019.1950.3032.22
DPO − SFT, percentage points+9.18+13.60+6.06+16.70+2.42

Under this matched comparison, the DPO-trained reconstructor produces stronger downstream student supervision across all five benchmarks. This result supports the contribution of preference training beyond direct supervised training on the same native traces. We will add the matched SFT baseline and this comparison to the revised paper.

W2: The source-matching analysis provides evidence that reconstruction is sensitive to information in the final response beyond the answer value. However, it does not distinguish solution-related cues from stylistic features or establish whether preference training exploits these cues better than supervised reconstruction, leaving their connection to the downstream gains unclear.

Response: Thank you for this suggestion. To distinguish solution-related cues from stylistic features, we repeated the MATH-500 source-matching analysis using two separate Gemini judge prompts. The first evaluates solution-pattern similarity, focusing on the reasoning approach and solution steps. The second evaluates expression-style similarity, focusing on wording, notation, and presentation. Both analyses use traces generated by the FOPL-DPO reconstructor.

Table 1: Source matching by solution-pattern similarity, using the DPO reconstructor.

Gemini preference rateDeepSeek synKimi synGLM synMiMo syn
DeepSeek ref55.71%47.88%43.36%41.65%
Kimi ref30.63%69.51%29.47%31.37%
GLM ref44.62%49.50%54.71%41.77%
MiMo ref39.78%41.40%43.14%58.56%

Table 2: Source matching by expression-style similarity, using the DPO reconstructor.

Gemini preference rateDeepSeek synKimi synGLM synMiMo syn
DeepSeek ref56.53%47.01%41.52%41.90%
Kimi ref32.54%68.95%29.07%31.55%
GLM ref45.25%52.99%52.83%43.27%
MiMo ref46.88%48.88%45.89%52.78%

In the solution-pattern analysis, same-source preference is above 50% for all four reference teachers. We also observe same-source preference above 50% in the expression-style analysis. These results suggest that the reconstructor captures both solution-related and stylistic information from the final response.

We also included the SFT reconstructor in the overall source-matching analysis. Table 3 reports its preference rates, with the difference from the corresponding DPO result in parentheses, in percentage points.

Table 3: Overall source matching using the SFT reconstructor; parentheses show SFT − DPO.

Gemini preference rateDeepSeek synKimi synGLM synMiMo syn
DeepSeek ref53.83% (−3.11)47.01% (+1.16)45.01% (+5.16)46.50% (+3.02)
Kimi ref39.32% (+6.91)61.68% (−3.11)35.38% (−0.14)40.27% (+2.58)
GLM ref42.00% (+0.77)48.49% (+5.26)54.88% (−2.68)44.88% (+2.02)
MiMo ref44.00% (+7.10)48.62% (+4.64)45.39% (+9.08)54.00% (−6.93)

The SFT reconstructor also exhibits same-source preference, while its same-source rates are lower than the corresponding DPO results for all four reference teachers. Together with the matched downstream comparison, these results are consistent with preference training improving both source alignment and the utility of reconstructed traces.

Q1:

Response: We trained an SFT reconstruction policy on the same native traces used as positive examples for FOPL’s DPO training, matching the model, data size, and training settings. The DPO reconstructor produced stronger downstream student performance across all five evaluated benchmarks. Please see our response to W1 for the detailed results.

Q2:

Response: We included the SFT reconstruction policy in the source-matching analysis and added separate evaluations of solution-pattern and expression-style similarity. The results show same-source preference under both criteria for the DPO reconstructor, while the SFT reconstructor has lower same-source preference rates in the overall comparison. Please see our response to W2 for the detailed results and discussion.

AuthorsOpenReview ↗

Follow-up on rebuttal

Dear Reviewer Sou3,

Thank you again for your valuable comments. We have carefully considered your feedback, provided our responses, and submitted a revised version of the paper.

Please let us know if our responses and revisions have addressed your concerns. We would be happy to answer any further questions or provide additional clarification.

Thank you very much!

Reviewer_YQyDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper applies DPO in a final-response-conditioned preference-learning framework (FOPL). The reconstruction policy learns to prefer a teacher's native CoT over synthetic CoTs written for the same question and visible final response. At transfer time, it generates CoTs from only a target teacher's question and final response, and these traces are used to fine-tune a smaller student. Experiments show that the reconstructions resemble the source teacher's native traces, including for a held-out family. They also provide better student supervision than prompted synthetic CoTs.

Critical strengths
  • Practical and well-motivated problem. Most frontier reasoning models expose only final responses, so constructing reasoning supervision from visible outputs is a useful setting for distillation.
  • Evidence of transfer. The source-alignment effect holds for a held-out family (MiMo) and is consistent across two judges. FOPL traces also outperform prompted synthetic CoTs as student supervision, and they help with a closed-source teacher (GPT-5.6-Luna).
Critical weaknesses
  • Missing key ablation on DPO. The method is standard DPO on a new type of pair, so the central question is what the rejected traces contribute. The paper should compare against SFT on the same native traces (p(x,y)→zp(x,y)\to z). The TI baseline is trained only on DeepSeek-R1 traces, so it does not separate the effect of preference learning from the effect of seeing multiple teachers' native CoTs.
  • Possible shortcut in the preference signal. All rejected traces come from one rationalization prompt and are likely shorter than native thinking traces. The policy may learn to avoid template artifacts or to favor longer outputs, rather than learning teacher-specific preferences. The paper reports no length statistics or controls.
  • Domain mismatch between training and evaluation. Preference data consists of 433 coding problems (HARDTESTS), while all similarity and downstream evaluations are on math and science QA. The paper does not discuss this mismatch.
  • Weak evaluation reliability. All results come from a single seed and a single generation per question.
Questions for authors

Please refer to the weakness section.

13 comments

Thank you for your thorough comments.

W1:Missing key ablation on DPO. The method is standard DPO on a new type of pair, so the central question is what the rejected traces contribute. The paper should compare against SFT on the same native traces (p(x,y)→zp(x,y)\to z). The TI baseline is trained only on DeepSeek-R1 traces, so it does not separate the effect of preference learning from the effect of seeing multiple teachers' native CoTs.

Response: Thank you for this suggestion. We agree that the previous comparison with Trace Inversion did not isolate preference learning, because TI was trained on DeepSeek-R1 traces rather than the same multi-teacher data used by FOPL.

To address this concern, we trained an SFT reconstruction policy directly on the same native traces used as the positive examples in FOPL’s DPO pairs. We matched the amount of training data and the training hyperparameters. Both reconstruction policies were then used to generate student supervision conditioned on the same DeepSeek final responses.

Student supervisionGSM8KMATH-500GPQAOpenBookQAMBPP
FOPL, SFT reconstructor48.6719.6013.1333.6029.80
FOPL, DPO reconstructor57.8533.2019.1950.3032.22
DPO − SFT, percentage points+9.18+13.60+6.06+16.70+2.42

The DPO-trained reconstructor produces stronger downstream student supervision across all five benchmarks under this matched comparison. This result supports the contribution of preference training beyond supervised training on the same native traces. We will add the matched SFT baseline and its comparison to the revised paper.

W2: Possible shortcut in the preference signal. All rejected traces come from one rationalization prompt and are likely shorter than native thinking traces. The policy may learn to avoid template artifacts or to favor longer outputs, rather than learning teacher-specific preferences. The paper reports no length statistics or controls.

Response: Thank you for raising this concern. We analyzed the lengths of the chosen and rejected traces. The median length is 1,070 tokens for chosen traces and 815 tokens for rejected traces. Chosen traces are longer in 62.29% of the pairs, while in the remaining approximately 38%, they are not longer than the rejected traces.

These statistics show that chosen traces are generally longer, but the rejected traces are also longer in a substantial proportion of pairs. Therefore, length alone should not determine the overall preference signal.

Moreover, we trained an SFT reconstruction policy using only the chosen native traces. The DPO-trained reconstruction policy achieved better downstream results than this SFT policy across all five benchmarks. This comparison suggests that the gains are not simply due to training on the chosen traces. We will include the length statistics and the matched SFT comparison in the revised paper.

W3: Domain mismatch between training and evaluation. Preference data consists of 433 coding problems (HARDTESTS), while all similarity and downstream evaluations are on math and science QA. The paper does not discuss this mismatch.

Response: Thank you for pointing out the domain mismatch. We used coding problems to train the reconstruction policy because we expect programming solutions to expose model-family-specific characteristics in their reasoning and problem-solving approaches. We view the difference between the preference-training and evaluation domains as an opportunity to examine the generalization of the learned reconstruction policy.

Although the preference data is constructed from coding problems, FOPL can use mathematical questions and final responses to construct reasoning supervision for student models that perform well on both mathematical and scientific reasoning benchmarks. This suggests that the learned response-conditioned reconstruction capability transfers beyond the domain used for preference training.

To complement these results, we also added MBPP as a downstream coding benchmark:

Student supervisionMBPP pass@1
Base model20.80
DeepSeek synthetic CoT31.00
Kimi synthetic CoT31.00
GLM synthetic CoT31.40
Native CoT33.80
FOPL, SFT reconstructor29.80
FOPL, DPO reconstructor32.22

On MBPP, the student trained on FOPL-DPO reconstructions achieves 32.22 pass@1, outperforming the three prompted synthetic-CoT baselines and the matched SFT reconstructor. Together, the math, science, and coding results provide evidence of FOPL’s generalization across reasoning domains.

W4: Weak evaluation reliability. All results come from a single seed and a single generation per question.

Response: Thank you for pointing out this limitation. To assess evaluation reliability, we conducted two additional experiments on GSM8K: we evaluated multiple generations per question and repeated pass@1 evaluation with three random seeds.

First, Table 1 reports pass@1 and pass@4 using seed 41. FOPL achieves 55.12 pass@1 and 83.02 pass@4, outperforming the compared supervision methods under both metrics.

Table 1: GSM8K results with multiple generations per question, seed 41.

Student supervisionPass@1Pass@4
Base model28.5362.62
DeepSeek native CoT52.1081.20
DeepSeek final output only48.7376.88
FOPL, DPO reconstructor55.1283.02
DeepSeek synthetic CoT46.6576.57
Kimi synthetic CoT34.1770.66
GLM synthetic CoT38.1073.24

Second, Table 2 reports pass@1 using three evaluation seeds. FOPL achieves an average of 55.80 ± 1.80 and obtains the highest score among the compared methods for each of the three seeds.

Table 2: GSM8K pass@1 across evaluation seeds. Mean ± SD uses the sample standard deviation.

Student supervisionSeed 41Seed 42Seed 43Mean ± SD
Base model28.5327.2228.5828.11 ± 0.77
DeepSeek native CoT52.1049.2849.9650.45 ± 1.47
FOPL, DPO reconstructor55.1257.8554.4455.80 ± 1.80
DeepSeek synthetic CoT46.6545.9449.8147.47 ± 2.06
Kimi synthetic CoT34.1733.6635.2534.36 ± 0.81
GLM synthetic CoT38.1037.0748.7541.31 ± 6.47

Together, these results show that FOPL’s advantage on GSM8K remains present when evaluating multiple generations per question and is consistent across the three tested evaluation seeds.

AuthorsOpenReview ↗

Follow-up on rebuttal

Dear Reviewer YQyD,

Thank you again for your valuable comments. We have carefully considered your feedback, provided our responses, and submitted a revised version of the paper.

Please let us know if our responses and revisions have addressed your concerns. We would be happy to answer any further questions or provide additional clarification.

Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.