- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper identifies three beneficial properties of a posterior teacher in the OPD setting, namely "correctness, "minimal shift" from the students' distribution, and "preference preservation". They show that conditioning the initial student’s policy on correctness satisfies these properties and uniquely minimizes over distributions supported only on correct responses. Motivated by this, they train a teacher conditioned on privileged information to approximate the correctness-conditioned distribution using a trajectory-balance objective. Practically speaking, the objective downweights incorrect responses while preserving the initial student's relative probabilities among correct responses. The trained teacher is then frozen and used for token-level distillatiuon on the students' rollouts. Empirically, FlowTD achieves the highest average accuracy across four math benchmarks on Qwen2.5-3B and Qwen3-8B-Base, outperforming the strongest distillation baselines.
Critical strengths
S1: The characterization of a good OPD teacher is well motivated, and its formalization through the correctness-conditioned posterior provides a principled foundation for further analysis.
S2: The trajectory-balance objective turns the theoretical posterior into a tractable teacher-training method, encouraging correctness and preference preservation without the need of inegrating over all correct responses.
Critical weaknesses
W1: Unclear teacher-input specifications + missing code limit reproducibility, particularly for the SFT baseline
W2: The experiments use only base models (Qwen2.5-3B and Qwen3-8B-Base). Can the benefits of FlowTD be shown for thinking and instruct models as well?
W3: The distributional analyses show associations with student performance but do not isolate which proposed teacher properties cause the improvement.
Questions for authors
Q1: How does FlowTD perform when the teacher receives only the problem, without a reference solution, under the same training budget? This would help separate the contribution of privileged information from that of the teacher-training objective.
Q2: Could you report pass@k for the teachers as well, to test whether FlowTD’s higher entropy translates into broader coverage of correct solutions?
Q3: Does the SFT teacher receive the reference solution as input during training, or only during distillation and evaluation? Are teacher input templates identical across SFT, RL, and FlowTD?
6 comments
Response by Authors [2/2]
Q1: Ablation of Privileged Information
We thank the reviewer for this question. We train a FlowTD teacher without privileged information (PI; Table R5).
Table R5. Greedy accuracy (%) on Qwen2.5-3B.
| Method | OlympiadBench | AMC23 | AIME24 | AIME25 | AVG |
|---|---|---|---|---|---|
| OPSD | 21.8 | 35.0 | 3.3 | 0.0 | 15.0 |
| RL teacher | 25.9 | 32.5 | 3.3 | 3.3 | 16.3 |
| FlowTD without PI | 23.5 | 42.5 | 6.7 | 3.3 | 19.0 |
| FlowTD | 25.4 | 42.5 | 10.0 | 6.7 | 21.1 |
Even without reference solutions, FlowTD outperforms all baselines in Table 2. This is consistent with our formulation. The target posterior is defined by the initial student's prior and the verifier, so removing PI leaves the target unchanged; it instead makes it harder for the teacher to reach correct trajectories and approximate this target. We have added this ablation to Section 4.3 (Table 5) of the revised manuscript.
Q2: Pass@k of the Teachers
We thank the reviewer for this suggestion. Table R6 reports teacher pass@k for Qwen3-8B-Base, estimated from 32 samples per problem at temperature 1.0 with the estimator in Appendix A.3.
Table R6. Teacher pass@k (%) for AIME 2025.
| Teacher | pass@1 | pass@8 | pass@16 | pass@32 |
|---|---|---|---|---|
| Initial model | 9.0 | 22.1 | 26.1 | 30.0 |
| RL teacher | 9.6 | 23.5 | 26.7 | 30.0 |
| FlowTD | 10.1 | 25.1 | 30.7 | 36.7 |
FlowTD achieves the highest pass@k at every k, and its margin widens as k grows: over the RL teacher, it increases from 0.5 points at k = 1 to 6.7 points at k = 32. This indicates that FlowTD's higher entropy translates into broader coverage of correct solutions. The distilled students show the same trend (Figure 5). We have added these results to Appendix D.1 of the revised manuscript.
Thank you once again for your constructive comments, which have strengthened our work. We have updated the manuscript to reflect these changes (highlighted in blue for easy identification). We hope that our responses and revisions address your concerns, and we welcome any further feedback you may have.
Thank you for providing the details on inputs and templates and for making the source code available. My concerns regarding W1 have been largely resolved.
Thank you for ablating the contribution of the distributional properties of the teacher. They resolve my concern in W3.
I remain concerned about the experimental setting for the thinking-model evaluation in W2. The 8k token response limit may be too restricive for Qwen3-1.7B (thinking) on hard mathematical reasoning tasks. The Qwen3 Technical Report allows up to 38,912 output tokens and reaches 48.3% and 36.8% accuracy on AIME24 and AIME25 whereas the rebuttal reports 23.3% and 10.0% for the initial model.
Could the authors report the mean response length and the fraction of truncated responses (responses that do not sample the end-of-sequence token) for the initial model, SDPO, and FlowTD?
A matched evaluation at longer response limits (e.g. 28k) would help determine whether FlowTD’s gains still persist when the models have enough room to finish their reasoning.
Response by Authors [1/2]
We sincerely thank you for the constructive review. We appreciate your recognition of our well-motivated teacher-distribution design and the principled posterior formulation, as well as the use of trajectory balance to make teacher learning tractable. We address each concern below.
W1 and Q3: Teacher Inputs, Templates, and Code Release
We thank the reviewer for highlighting the need for clearer implementation details. We clarify the three aspects below.
Table R1. Model inputs and prompt templates.
W2: Generalization to Instruct and Thinking Models
We thank the reviewer for this valuable suggestion. To examine whether FlowTD generalizes beyond base models, we extend our experiments to Qwen2.5-3B-Instruct and Qwen3-1.7B in thinking mode.
Table R2. Results on Qwen2.5-3B-Instruct, with a 6K-token response limit.
Table R3. Results on Qwen3-1.7B in thinking mode, with an 8K-token response limit.
As shown in Tables R2 and R3, FlowTD outperforms both the initial checkpoints and SDPO in average accuracy, supporting its effectiveness in both instruct and thinking settings. We have added the instruct and thinking results to Appendices B.3 and B.4, respectively.
W3: Contribution of Teacher Distributional Properties
We thank the reviewer for this insightful comment. To examine the contributions of different teacher properties, we apply three controlled perturbations to the learned FlowTD teacher q during the distillation stage, keeping the teacher checkpoint and the student training setup fixed:
Table R4. Controlled perturbations of the FlowTD teacher on Qwen2.5-3B.
All three perturbations reduce student accuracy by 1.6–2.4 percentage points, although the perturbed teachers still outperform the RL teacher (16.3%). This indicates that each property contributes to the distillation process. We have added this analysis to Section 4.3 (Table 4) of the revised manuscript.