- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes FOPL, a framework for synthesizing reasoning supervision when a target teacher exposes only its final response. FOPL trains a reconstruction policy with DPO using native CoTs as preferred traces and answer-conditioned synthetic CoTs as rejected traces, while keeping the question and visible final response fixed. At transfer time, the policy reconstructs a CoT using only the target teacher’s question and final response. Experiments show source-dependent reconstruction across several teacher families, including a held-out MiMo family, and improved downstream distillation into Qwen2.5-1.5B. The paper also evaluates judge robustness, final-response-only supervision, Trace Inversion, data scale, and transfer from a closed-source GPT-5.6-Luna teacher.
Critical strengths
- Practical and interesting problem setting. The paper targets a realistic distillation setting where strong teachers expose final responses but not native CoTs. The held-out MiMo experiment and the GPT-5.6-Luna transfer make the intended use case more convincing.
- Strong downstream results. FOPL consistently outperforms prompted synthetic-CoT baselines and often matches or exceeds native-CoT supervision on reasoning benchmarks. The controls against final-response-only training and Trace Inversion further support the usefulness of the reconstructed traces.
Critical weaknesses
- The value of preference learning is not cleanly isolated. FOPL is not compared with a matched SFT reconstructor trained on the same multi-family native traces. The Trace Inversion baseline uses different training data, so it is hard to attribute the gain specifically to DPO.
- Downstream validation uses only one small student. All distillation experiments use Qwen2.5-1.5B-Base. It is unclear whether the gains hold for larger students or other model families.
- The source-alignment evidence is narrow. The main cross-family similarity experiment is only conducted on MATH-500. It is unclear whether the same teacher-specific signal appears in other reasoning domains.
Questions for authors
-
Can you compare FOPL with a matched SFT reconstructor trained on the same multi-family native traces, prompts, initialization, and training budget?
-
Have you tested FOPL with larger or more recent student models, and on harder recent reasoning benchmarks?
-
Does the source-matching effect still hold outside MATH-500, for example on code or science reasoning tasks?
4 comments
Follow-up on rebuttal
Dear Reviewer 5C37,
Thank you again for your valuable comments and supports! We have carefully considered your feedback, provided our responses, and submitted a revised version of the paper.
Please let us know if our responses and revisions have addressed your concerns. We would be happy to answer any further questions or provide additional clarification.
Thank you very much!
Further follow-up on rebuttal
Dear Reviewer 5C37,
Thank you very much for your time and helpful comments. We have addressed your concerns in our rebuttal. As the discussion period is drawing to a close, we would appreciate any further comments or suggestions you may have on our responses.
Thank you again for your time and consideration.
The Authors
Thank you for the additional experiments and clarifications. I appreciate the authors’ efforts to address the concerns raised in my review. After carefully considering the new evidence, I do not find sufficient reason to revise my initial assessment, and therefore maintain my original score.
Thank you for your thorough comments and support!
W1&Q1: The value of preference learning is not cleanly isolated. FOPL is not compared with a matched SFT reconstructor trained on the same multi-family native traces. The Trace Inversion baseline uses different training data, so it is hard to attribute the gain specifically to DPO.
Response: Thank you for this suggestion. We agree that the previous comparison with Trace Inversion (TI) did not sufficiently isolate the contribution of preference learning, because TI used different training data from FOPL.
To address this concern, we trained an SFT reconstruction policy directly on the same multi-family native traces used as the positive examples in FOPL’s DPO pairs. We matched the reconstruction model, training data size, and training hyperparameters. Both reconstruction policies were then used to generate student supervision conditioned on the same DeepSeek final responses.
Under this matched comparison, the DPO-trained reconstructor produces stronger downstream student supervision across all five benchmarks. This result supports the contribution of preference learning beyond direct supervised training on the same native traces. We will add the matched SFT baseline and this comparison to the revised paper.
W2&Q2: Downstream validation uses only one small student. All distillation experiments use Qwen2.5-1.5B-Base. It is unclear whether the gains hold for larger students or other model families.
Response: Thank you for this suggestion. To examine whether the gains extend to a larger student from a different model family, we additionally evaluated Llama-3.2-3B-Base on GSM8K.
FOPL achieves 21.46 on GSM8K, outperforming the prompted synthetic-CoT baselines by 7.28–8.04 percentage points and obtaining a result comparable to native-CoT supervision. These results suggest that FOPL’s downstream benefits extend to a larger student from another model family.
Due to limited time and the use of only 5K training examples, gains on substantially larger models may be less apparent and may require more training data. We therefore focused on validating a different model family.
W3&Q3: The source-alignment evidence is narrow. The main cross-family similarity experiment is only conducted on MATH-500. It is unclear whether the same teacher-specific signal appears in other reasoning domains.
Response: Thank you for this suggestion. To examine source alignment outside the mathematical domain, we conducted an additional source-matching experiment on MBPP, using the FOPL-DPO reconstructor and Gemini as the judge.
The same-source reconstruction has the highest preference rate for each of the four reference teachers, including the held-out MiMo family. These results provide additional evidence of source-specific alignment in the coding domain, beyond the MATH-500 analysis. We will include the MBPP experiment in the revised paper.