acceptodds
Under review as a conference paper at ICLR 2027

Learning Posterior Teachers for On-Policy Self-Distillation

Abstract

On-policy self-distillation addresses the sparse supervision of reinforcement learning with verifiable rewards (RLVR) by providing dense token-level supervision from a privileged teacher. However, privileged information alone does not ensure reliable supervision, and teacher accuracy alone does not specify a suitable distribution for distillation. We identify three desirable teacher properties for distillation: correctness, minimal shift, and preference preservation. The correctness-conditioned posterior unifies these properties by restricting the initial student's prior to trajectories with correct final answers. We introduce Flow Teacher Distillation (FlowTD), which trains a privileged teacher to approximate this posterior using GFlowNet trajectory balance. Its probability-matching objective uses outcome verification to supervise correctness and explicitly trains the teacher to preserve the student's relative probabilities among correct trajectories. Our analyses show that FlowTD learns teachers with stronger correctness discrimination than prompt-only teachers, alongside smaller distributional shifts and better preference preservation than SFT- and RL-trained teachers. Across four mathematical reasoning benchmarks, FlowTD improves average accuracy over the strongest baselines by 4.7 and 3.9 points on Qwen2.5-3B and Qwen3-8B-Base, respectively. It also achieves the highest pass@ among the compared teacher constructions on AIME 2025.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, self-distillation, mathematical reasoning, posterior inference, large language models, reasoning models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_Kj2yOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper identifies three beneficial properties of a posterior teacher in the OPD setting, namely "correctness, "minimal shift" from the students' distribution, and "preference preservation". They show that conditioning the initial student’s policy on correctness satisfies these properties and uniquely minimizes KL(q∥p0)\mathrm{KL}(q\|p_0) over distributions supported only on correct responses. Motivated by this, they train a teacher conditioned on privileged information to approximate the correctness-conditioned distribution using a trajectory-balance objective. Practically speaking, the objective downweights incorrect responses while preserving the initial student's relative probabilities among correct responses. The trained teacher is then frozen and used for token-level distillatiuon on the students' rollouts. Empirically, FlowTD achieves the highest average accuracy across four math benchmarks on Qwen2.5-3B and Qwen3-8B-Base, outperforming the strongest distillation baselines.

Critical strengths

S1: The characterization of a good OPD teacher is well motivated, and its formalization through the correctness-conditioned posterior provides a principled foundation for further analysis.

S2: The trajectory-balance objective turns the theoretical posterior into a tractable teacher-training method, encouraging correctness and preference preservation without the need of inegrating over all correct responses.

Critical weaknesses

W1: Unclear teacher-input specifications + missing code limit reproducibility, particularly for the SFT baseline

W2: The experiments use only base models (Qwen2.5-3B and Qwen3-8B-Base). Can the benefits of FlowTD be shown for thinking and instruct models as well?

W3: The distributional analyses show associations with student performance but do not isolate which proposed teacher properties cause the improvement.

Questions for authors

Q1: How does FlowTD perform when the teacher receives only the problem, without a reference solution, under the same training budget? This would help separate the contribution of privileged information from that of the teacher-training objective.

Q2: Could you report pass@k for the teachers as well, to test whether FlowTD’s higher entropy translates into broader coverage of correct solutions?

Q3: Does the SFT teacher receive the reference solution as input during training, or only during distillation and evaluation? Are teacher input templates identical across SFT, RL, and FlowTD?

6 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

We sincerely thank you for the constructive review. We appreciate your recognition of our well-motivated teacher-distribution design and the principled posterior formulation, as well as the use of trajectory balance to make teacher learning tractable. We address each concern below.

W1 and Q3: Teacher Inputs, Templates, and Code Release

We thank the reviewer for highlighting the need for clearer implementation details. We clarify the three aspects below.

  • Teacher inputs. In the standard setup, all teachers (OPSD, SDPO, SFT, RL, and FlowTD) receive the problem and the same DeepSeek-R1 solution as privileged input during all stages. The SFT teacher is trained to generate a second, independently sampled DeepSeek-R1 solution to the same problem. All students receive only the problem (Table R1).
  • Templates. The teacher prompt is adapted from OPSD (Zhao et al., 2026), while the student uses a problem-only prompt. Reference solutions are provided only to teachers. We have included the complete templates in Appendix A.5.
  • Code release. We have made our code available in an anonymous repository: https://anonymous.4open.science/r/flowtd-code-D3EC.

Table R1. Model inputs and prompt templates.

ModelInputInput template
Teachers (OPSD, SDPO, SFT, RL, FlowTD)Problem + DeepSeek-R1 solutionOPSD template (with reference)
StudentProblemProblem-only prompt (Appendix A.5)

W2: Generalization to Instruct and Thinking Models

We thank the reviewer for this valuable suggestion. To examine whether FlowTD generalizes beyond base models, we extend our experiments to Qwen2.5-3B-Instruct and Qwen3-1.7B in thinking mode.

Table R2. Results on Qwen2.5-3B-Instruct, with a 6K-token response limit.

MethodOlympiadBenchAMC23AIME24AIME25AVG
Qwen2.5-3B-Instruct29.145.03.30.019.4
SDPO28.552.53.30.021.1
FlowTD29.150.010.06.724.0

Table R3. Results on Qwen3-1.7B in thinking mode, with an 8K-token response limit.

MethodOlympiadBenchAMC23AIME24AIME25AVG
Qwen3-1.7B (thinking)31.152.523.310.029.2
SDPO37.362.520.016.734.1
FlowTD38.265.026.720.037.5

As shown in Tables R2 and R3, FlowTD outperforms both the initial checkpoints and SDPO in average accuracy, supporting its effectiveness in both instruct and thinking settings. We have added the instruct and thinking results to Appendices B.3 and B.4, respectively.

W3: Contribution of Teacher Distributional Properties

We thank the reviewer for this insightful comment. To examine the contributions of different teacher properties, we apply three controlled perturbations to the learned FlowTD teacher qq during the distillation stage, keeping the teacher checkpoint and the student training setup fixed:

  • Weaker correctness guidance: interpolate the teacher toward the initial student, q~∝qλp01−λ\tilde q \propto q^{\lambda} p_0^{1-\lambda} with λ=0.7\lambda = 0.7, which weakens the correctness signal.
  • Entropy-matched teacher: lower the teacher's temperature until its token entropy matches that of the RL teacher (≈ 0.12 nats).
  • Perturbed prior preferences: reweight the teacher by the initial student's probabilities, q~∝q p0α−1\tilde q \propto q\, p_0^{\alpha-1} with α=0.6\alpha = 0.6, within its top-pp nucleus and with its entropy kept at FlowTD's level, which distorts the prior's relative preferences.

Table R4. Controlled perturbations of the FlowTD teacher on Qwen2.5-3B.

TeacherTeacher acc. (%)EntropyPref. SDStudent AVG (%)
FlowTD93.20.2498.221.1
+ Weaker correctness guidance86.50.2906.818.7
+ Entropy-matched teacher93.50.12512.619.1
+ Perturbed prior preferences93.30.25614.719.5
RL teacher93.90.12115.516.3

All three perturbations reduce student accuracy by 1.6–2.4 percentage points, although the perturbed teachers still outperform the RL teacher (16.3%). This indicates that each property contributes to the distillation process. We have added this analysis to Section 4.3 (Table 4) of the revised manuscript.

AuthorsOpenReview ↗

Response by Authors [2/2]

Q1: Ablation of Privileged Information

We thank the reviewer for this question. We train a FlowTD teacher without privileged information (PI; Table R5).

Table R5. Greedy accuracy (%) on Qwen2.5-3B.

MethodOlympiadBenchAMC23AIME24AIME25AVG
OPSD21.835.03.30.015.0
RL teacher25.932.53.33.316.3
FlowTD without PI23.542.56.73.319.0
FlowTD25.442.510.06.721.1

Even without reference solutions, FlowTD outperforms all baselines in Table 2. This is consistent with our formulation. The target posterior is defined by the initial student's prior and the verifier, so removing PI leaves the target unchanged; it instead makes it harder for the teacher to reach correct trajectories and approximate this target. We have added this ablation to Section 4.3 (Table 5) of the revised manuscript.

Q2: Pass@k of the Teachers

We thank the reviewer for this suggestion. Table R6 reports teacher pass@k for Qwen3-8B-Base, estimated from 32 samples per problem at temperature 1.0 with the estimator in Appendix A.3.

Table R6. Teacher pass@k (%) for AIME 2025.

Teacherpass@1pass@8pass@16pass@32
Initial model9.022.126.130.0
RL teacher9.623.526.730.0
FlowTD10.125.130.736.7

FlowTD achieves the highest pass@k at every k, and its margin widens as k grows: over the RL teacher, it increases from 0.5 points at k = 1 to 6.7 points at k = 32. This indicates that FlowTD's higher entropy translates into broader coverage of correct solutions. The distilled students show the same trend (Figure 5). We have added these results to Appendix D.1 of the revised manuscript.


Thank you once again for your constructive comments, which have strengthened our work. We have updated the manuscript to reflect these changes (highlighted in blue for easy identification). We hope that our responses and revisions address your concerns, and we welcome any further feedback you may have.

Thank you for providing the details on inputs and templates and for making the source code available. My concerns regarding W1 have been largely resolved.

Thank you for ablating the contribution of the distributional properties of the teacher. They resolve my concern in W3.

I remain concerned about the experimental setting for the thinking-model evaluation in W2. The 8k token response limit may be too restricive for Qwen3-1.7B (thinking) on hard mathematical reasoning tasks. The Qwen3 Technical Report allows up to 38,912 output tokens and reaches 48.3% and 36.8% accuracy on AIME24 and AIME25 whereas the rebuttal reports 23.3% and 10.0% for the initial model.

Could the authors report the mean response length and the fraction of truncated responses (responses that do not sample the end-of-sequence token) for the initial model, SDPO, and FlowTD?

A matched evaluation at longer response limits (e.g. 28k) would help determine whether FlowTD’s gains still persist when the models have enough room to finish their reasoning.

Reviewer_8HXHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies three desirable properties for teacher policy for on-policy self distillation: correctness, minimal shift and preference preservation under the initial student policy. The paper further introduces FlowTD, which trains a teacher additionally conditioned on privileged information to approximate the correctness conditioned posterior using FlowRL’s trajectory balance loss. Experiments across four mathematical reasoning tasks and two Qwen model families demonstrates improved performance against compared baselines.

Critical strengths
  • The analysis presented in Section 4.3 demonstrate that teachers of similar accuracy can produce different students despite being subjected to the same distillation loss highlighting the importance of the three properties of the posterior teacher. These provide useful insights for future self-distillation works.

  • FlowTD shows better average performance than compared baselines on two model families.

Critical weaknesses
  • Limited novelty. The teacher loss is a minor adaptation of FlowRL’s trajectory-balance objective. Table 2 and 3 compare against GRPO but not FlowRL, which is an important missing baseline.

  • Missing baselines. The paper cites pi-distill and VPD, but does not compare against these relevant methods.

Questions for authors
  1. The reported results use greedy decoding. Can the authors use recommended sampling for Qwen models ( Temperature=0.6, TopP=0.95, TopK=20, and MinP=0) and report mean and standard deviation?
8 comments
AuthorsOpenReview ↗

Response by Authors

We sincerely thank you for the constructive review. We are encouraged that you highlight the importance of the three teacher properties and find the supporting analysis useful for future self-distillation work. We are also glad that you recognize FlowTD's improved average performance over all compared baselines on both model families. We address each point below.

W1: Novelty and Comparison with FlowRL

We thank the reviewer for highlighting the connection to FlowRL. FlowTD and FlowRL share the trajectory-balance (TB) framework but address different research questions. FlowRL uses TB for direct policy optimization, whereas FlowTD investigates what makes an effective teacher distribution for on-policy self-distillation and how to learn it.

Our work is motivated by the observation that teachers with comparable accuracy can yield substantially different student performance under the same distillation loss (Section 4.3). An accurate reasoning model is therefore not necessarily an effective distillation teacher. Our central contribution is to identify the correctness-conditioned posterior as a principled teacher target and learn a teacher model that approximates it. This target enforces correctness while minimally shifting the initial student's distribution and preserving its relative preferences among correct solutions. Here, TB serves as the optimization tool for realizing this teacher-distribution design.

Table R1. Comparison with FlowRL on Qwen2.5-3B (greedy accuracy, %).

MethodGRPOFlowRLFlowTD
AVG15.817.521.1

To address the missing baseline, we additionally evaluate direct FlowRL (Table R1). FlowTD outperforms it by 3.6 percentage points, supporting the practical value of privileged posterior teacher learning followed by on-policy distillation. We have added this comparison to Appendix B.1 and clarified the conceptual distinction from FlowRL in Section 5 of the revised manuscript.

W2: Comparison with π-Distill and VPD

We thank the reviewer for this suggestion. We additionally evaluate π-Distill and VPD on Qwen2.5-3B, using the same backbone, training data, and evaluation protocol as FlowTD while retaining each method's original learning objective.

Table R2. Student accuracy (AVG, %) under greedy decoding.

MethodQwen2.5-3B
π-Distill16.7
VPD15.4
FlowTD21.1

FlowTD outperforms π-Distill and VPD by 4.4 and 5.7 percentage points, respectively, further supporting the effectiveness of our teacher-distribution design for on-policy distillation. We have incorporated these comparisons into Appendix B.1 of the revised manuscript.

Q1: Evaluation Variability Under Sampling

We additionally evaluate Qwen2.5-3B using the sampling protocol recommended by the Qwen (temperature = 0.6, top-p = 0.95, top-k = 20, min-p = 0), with 32 samples per problem on AMC23 and AIME and 4 samples per problem on OlympiadBench.

Table R3. Qwen2.5-3B Accuracy Under Sampling (%, Mean ± SD Across Sampling Runs).

MethodOlympiadBenchAMC23AIME24AIME25AVG
Base20.1 ± 1.626.8 ± 5.43.9 ± 2.71.6 ± 2.213.1
GRPO22.6 ± 0.828.4 ± 3.03.8 ± 1.81.4 ± 2.514.1
OPSD20.5 ± 1.027.3 ± 7.03.0 ± 2.11.1 ± 2.013.0
SDPO20.1 ± 1.425.7 ± 5.84.0 ± 2.71.7 ± 2.412.9
SFT teacher22.5 ± 0.329.5 ± 7.03.1 ± 2.51.5 ± 1.914.2
RL teacher23.4 ± 1.128.2 ± 4.64.0 ± 2.32.0 ± 2.214.4
FlowTD24.5 ± 1.335.0 ± 6.16.5 ± 3.03.0 ± 2.617.3

Note. SD reflects variation across sampling runs using fixed model checkpoints.

As shown in Table R3, FlowTD maintains the highest average accuracy under sampling, outperforming the strongest baseline by 2.9 percentage points. These results further support the effectiveness of FlowTD beyond greedy decoding. We have added these results to Appendix C.2 of the revised manuscript.


Thank you once again for your constructive comments, which have strengthened our work. We have updated the manuscript to reflect these changes (highlighted in blue for easy identification). We hope that our responses and revisions address your concerns, and we welcome any further feedback you may have.

I thank the authors for conducting additional experiments. My concerns are partially addressed. I listed my remaining concerns below:

  1. Novelty. The paper still under-attributes FlowRL and over claims its novelty. The teacher loss (equation 5) is similar to FlowRL's trajectory balance objective with the same components. The only changes are the privileged context in the teacher prompt and the binary reward. Section 3.1 should credit FlowRL, and contribution points should clearly state the additional novelty on top of this.

  2. Table R1 presents unweighted mean of greedy accuracy. Can the authors report per benchmark results using both greedy and recommended sampling parameters? Also provide the hyper-parameters used for FlowRL.

  3. The reported results for Pi-Distill and VPD appear under-tuned. VPD (avg 15.4 as reported in Table R2) is below GRPO (avg 15.8) and RL teacher (avg 16.3) reported in Table 2 of main paper. Also, VPD and Pi-Distill official code does not seem to be available. The reimplementation might be error-prone. Can the authors provide additional details on how they replicated these two baselines? Please also provide hyper-parameters used for training. Ideally, the authors should evaluate on the eval setup and benchmarks used in VPD and Pi-Distill to verify their reimplementation reproduce the reported results.

  4. The sampling results provided in Table R3 leaves out FlowRL. Can the authors include this baseline and also extend Table R3 to Qwen3-8B-Base?

AuthorsOpenReview ↗

Follow-up Response by Authors [1/2]

We sincerely thank the reviewer for the follow-up comments and for acknowledging our additional experiments. In response, we (1) credit FlowRL explicitly and clarify our contribution beyond it, (2) report per-benchmark FlowRL results under greedy decoding and sampling, together with its hyperparameters, (3) provide replication details and hyperparameters for π-Distill and VPD, and (4) extend the sampling evaluation to FlowRL and Qwen3-8B-Base. We address each point below.

1. Novelty

We thank the reviewer for this comment and agree that FlowRL should be credited explicitly.

TB objective: FlowRL [2] adapts the trajectory-balance (TB) loss [1] to LLM policy optimization, introducing components such as length normalization and importance weighting. Our teacher loss in Eq. (5) use the same TB objective. We do not claim a new TB objective, and have made this attribution explicit in Section 3.1.

Our contribution: We apply the same TB framework to a different research problem. FlowRL directly optimizes a reasoning policy, whereas FlowTD investigates what makes a good teacher distribution for on-policy self-distillation and how to learn it. Our contribution lies not in the TB loss, but in the teacher formulation and its investigation: (1) identifying the correctness-conditioned posterior of the initial student as a principled teacher distribution, characterized by correctness, minimal shift, and preference preservation; (2) learning a privileged teacher that approximates this target for on-policy distillation; and (3) empirically examining how these teacher properties affect student learning.

Importantly, this contribution is supported by evidence beyond teacher accuracy: similarly accurate teachers yield substantially different students, and controlled teacher-distribution perturbations reduce student performance (Section 4.3). Moreover, FlowTD without privileged information still outperforms FlowRL (19.0% vs. 17.5%; Table 5 and Appendix B.1), indicating that the performance advantage is not solely attributable to privileged conditioning.

References

[1] Malkin et al. Trajectory balance: Improved credit assignment in GFlowNets. NeurIPS 2022.

[2] Zhu et al. FlowRL: Matching reward distributions for LLM reasoning. ICLR 2026.

2. Per-Benchmark Results and Hyperparameters

We thank the reviewer for this suggestion. Tables R4 and R5 expand Table R1 with per-benchmark results under greedy decoding and the recommended sampling protocol, respectively. Table R6 summarizes FlowRL's key hyperparameters, with full details provided in Appendix B.1.

Table R4. Greedy accuracy (%) on Qwen2.5-3B.

MethodOlympiadBenchAMC23AIME24AIME25AVG
GRPO25.035.03.30.015.8
FlowRL25.737.53.33.317.5
FlowTD25.442.510.06.721.1

Table R5. Accuracy (%) on Qwen2.5-3B under recommended sampling (mean ± SD).

MethodOlympiadBenchAMC23AIME24AIME25AVG
GRPO22.6 ± 0.828.4 ± 3.03.8 ± 1.81.4 ± 2.514.1
FlowRL23.6 ± 0.931.0 ± 5.54.6 ± 2.62.1 ± 2.215.3
FlowTD24.5 ± 1.335.0 ± 6.16.5 ± 3.03.0 ± 2.617.3

Table R6. Key FlowRL hyperparameters.

HyperparameterValue
Optimizer / learning rateAdamW / 5×10−75\times10^{-7}
Prompts per step / rollouts128 / 8
Rollout temperature / top-p1.0 / 1.0
Epochs2
Reward scaling β\beta15
Importance weighting clip10

FlowTD outperforms FlowRL by 3.6 percentage points under greedy decoding and 2.0 points under sampling on average, demonstrating that its advantage persists under the recommended sampling protocol. We have incorporated these results and implementation details into the revised manuscript.

Reviewer_5YmgOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FlowTD, based on GFlowNet's trajectory balance, as a method to train LLM reasoning via a form of on-policy self-distillation.

Critical strengths

-- The extension of residual-quantized autoregressive modelling to full collider-event simulation is a potentially important contribution. Complete particle-level event modelling, rather than individual jets or detector components, addresses a practically important problem and provides a flexible representation for downstream physics analyses.

-- The approach is novel (as far as I know) and appears technically sound.

-- A natural connection to GFlowNet training, with a well-motivated use of trajectory balance to approximate the posterior. Rather than simply maximizing expected reward, the method attempts to match a distribution over correct reasoning trajectories which naturally addresses an issue observed with other on-policy self-distillation methods: sample diversity collapse.

-- The paper is generally well written and easy to follow.

-- The empirical analysis offers a good mechanistic analysis beyond benchmark accuracy.

Critical weaknesses

-- The teacher objective uses an alternative to their theory driven target posterior. This is to avoid instabilities in the computation of the teacher trajectory-balance loss. In doing so the authors have introduced a new temperature parameter whose value seems to have a significant impact on the resulting solution to the optimization of this objective. This warrants discussion that is currently absent.

-- The empirical results of the baseline approaches seems consistently lower than as reported it the literature (question on this below). Consequently, while the result presented for FlowTD seems significantly better to those of the presented baselines, these are not particularly impressive accuracies in the context of the current literature. This leads to a lack of empirical support for the claims of superior performance.

-- (minor) The theoretical formulation is elegant, but its novelty is limited (Proposition 1 is basically the standard information projection of a prior distribution onto a constrained support; and Proposition 2 is also a familiar consequence of conditioning on a terminal event).

Questions for authors

-- Why are these 3 listed properties of the posterior teacher the right ones? Correctness seems self-evident, but minimal shift or preference preservation require some defending. Together they suggest that the best answers are correct (sure!) but also those apriori preferred by the reference model. Why is that true? Why not the most concise answer or the easiest to understand? This requires some discussion.

-- Inconsistencies with existing literature (eg. the DCPO paper). In Table 2, the reported greedy-decoding accuracy of Qwen2.5-3B is 0% on both AIME 2024 and AIME 2025, while the GRPO baseline also achieves 0% on AIME 2025. These results appear on the low side compared to previously published evaluations of the same model, which report nonzero performance on AIME 2025 (e.g., 6.7% under greedy decoding). Could the authors clarify the evaluation protocol used to obtain these results, particularly the exact model checkpoint, prompting template, maximum generation length, treatment of truncated responses, and answer extraction and verification procedures? Given the small size of the AIME benchmarks, it would also be helpful to report variability across evaluation settings or runs.

5 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

We sincerely thank you for the constructive review. We are encouraged that you find our approach novel and technically sound, and the use of trajectory balance natural and well motivated for addressing the diversity collapse of on-policy self-distillation methods. We are also glad that you find the paper well written and the mechanistic analysis valuable. We address each point below.

W1: Analysis of the Penalty Coefficient β

We thank the reviewer for raising this point. The finite-penalty target is qβ(z)∝p0(z)exp⁡[−β(1−r(z))]q_\beta(z)\propto p_0(z)\exp[-\beta(1-r(z))]. For finite β, this target does not enforce zero probability on incorrect trajectories. However, it exactly preserves the prior's relative probabilities among correct trajectories, because they receive the same penalty. Increasing β suppresses incorrect probability mass and brings the target closer to the ideal correctness-conditioned posterior.

Larger β increases the log-reward gap between correct and incorrect trajectories and can make TB optimization harder. Thus, increasing β does not necessarily translate into better performance. We therefore evaluate β ∈ {3, 10, 15, 30}.

Table R1. Effect of β on Qwen2.5-3B.

β3101530
Student AVG (%)18.420.321.120.4

Consistent with this analysis, performance improves from 18.4% at β = 3 to 21.1% at β = 15 and remains relatively stable across β ∈ {10, 15, 30}. These results suggest that FlowTD does not require precise tuning within this range. We use β = 15 by default and have included the results in Appendix C.1.


W2 and Q2: Baseline Performance and Comparison With DCPO

We thank the reviewer for raising this concern. In short, the gap stems from the prompt template for the base model and from the training budget for trained baselines (Table R2). Under DCPO's evaluation protocol, our evaluation reproduces DCPO's reported base-model results, and under DCPO's training and evaluation setup, FlowTD outperforms the published GRPO and DCPO results by 4.00 and 4.35 points on average (Table R3).

Setup differences. Table R2 summarizes the key differences behind the performance gap. Our original training and evaluation setups are provided in Appendices A.1–A.3, with full prompt templates in Appendix A.5; the larger-data comparison is reported in Appendix B.2.

Table R2. Key Differences Between the Two Setups.

StageSettingOur 3B GRPO setupDCPO
EvaluationPrompt templateQwen2.5 chat template, default system promptQwen2.5 chat template, Qwen-Math system prompt
TrainingTraining problems~8.5K (MATH Levels 3–5)~25K (DAPO-Math-17K + MATH Levels 3–5)
TrainingSampled training trajectories~0.133M (130 × 128 × 8)~3.277M

Results under aligned setups. Table R3 reports the results when our evaluation and training are aligned with DCPO's:

  • Base model. The system prompt has a substantial impact on base-model accuracy: switching only to DCPO's system prompt already raises AIME accuracy above DCPO's reported values, and the fully aligned evaluation reproduces DCPO's reported results on all four benchmarks.
  • Trained models. Our original 3B GRPO baseline uses about 3× fewer problems and 25× fewer trajectories than DCPO's (Table R2). Trained and evaluated in DCPO's setup, FlowTD achieves higher average accuracy than the baselines (Table R3).

Table R3. Qwen2.5-3B Greedy Accuracy (%). † Benchmark scores reported by DCPO; AVG is recomputed as their unweighted mean.

MethodMATH500AMC23AIME24AIME25AVG
Base model under different evaluation settings
Ours (default system prompt)53.230.00.00.020.80
DCPO's system prompt only48.627.56.76.722.38
DCPO's full protocol (= DCPO's report)46.427.53.33.320.13
Trained under DCPO's training and evaluation setup
GRPO†69.262.510.06.737.10
DCPO†71.262.53.310.036.75
FlowTD73.667.513.310.041.10

Sampling-based results and variability are reported in Table R5. We have added the results to Appendix B.2 of the revised manuscript.


W3: Theoretical Novelty

We thank the reviewer for this observation and agree that our novelty does not lie in Propositions 1 and 2 themselves. They provide the theoretical motivation for our approach by connecting the correctness-conditioned posterior to minimal distributional shift, preference preservation, and outcome-relevant next-token supervision.

Building on these insights, our main contributions are to (1) characterize desirable properties of the teacher distribution for on-policy self-distillation, (2) develop FlowTD, which learns the posterior teacher distribution through trajectory balance, and (3) empirically demonstrate its advantages in both teacher-distribution quality and downstream student performance.

AuthorsOpenReview ↗

Response by Authors [2/2]

Q1: Motivation for the Three Teacher Properties

We thank the reviewer for this thoughtful question. In self-distillation, the initial student itself defines the prior. Correct reasoning paths preferred by this prior are not necessarily better in an absolute sense, but they are closer to the student's existing behavior, which makes them potentially more accessible targets for distillation.

Minimal shift (P2) reflects a conservative inductive bias: improve correctness (P1) while avoiding unnecessary changes to the student's distribution. As Equation 2 shows, among distributions supported on correct trajectories, any deviation from the correctness-conditioned posterior incurs KL divergence beyond what correctness requires. Empirically, the SFT teacher exhibits the largest distributional shift (Figure 4b) and yields the weakest student among the trained teachers (Table 2), despite comparable teacher accuracy (Figure 4a).

Preference preservation (P3) follows naturally from minimal shift: among correct reasoning paths, the posterior retains the student's relative preferences rather than imposing new ones. Empirically, FlowTD exhibits the lowest preference distortion among the trained teachers (Figure 6b), and its student maintains higher entropy during distillation (Figure 6c). Together with the improved student performance (Tables 2 and 3), these observations support preference preservation as a useful design principle.

Ablation of teacher properties. We further ablate the properties of the learned FlowTD teacher during distillation (Table R4; details in Appendix A.4). Each perturbation reduces student performance. These results strengthen the empirical support for prior-preserving teacher design beyond correctness or entropy alone.

Table R4. Ablation of Teacher Properties on Qwen2.5-3B.

TeacherTeacher acc. (%)EntropyPref. SDStudent AVG (%)
FlowTD93.20.2498.221.1
+ Weaker correctness guidance86.50.2906.818.7
+ Entropy-matched teacher93.50.12512.619.1
+ Perturbed prior preferences93.30.25614.719.5

Conciseness and readability are also valid objectives. Incorporating them through an explicit reward or preference signal would define a different target within the same framework. We have clarified the motivation in Section 2.2 and added the perturbation analysis in Section 4.3.


Q2: Evaluation Variability

To assess evaluation variability, we additionally evaluate Qwen2.5-3B using the recommended Qwen sampling protocol (temperature = 0.6, top-p = 0.95, top-k = 20, min-p = 0), with 32 samples per problem on AMC23 and AIME and four samples per problem on OlympiadBench.

Table R5. Qwen2.5-3B Accuracy (%) Under Sampling (Mean ± SD Across Runs).

MethodOlympiadBenchAMC23AIME24AIME25AVG
Base20.1 ± 1.626.8 ± 5.43.9 ± 2.71.6 ± 2.213.1
GRPO22.6 ± 0.828.4 ± 3.03.8 ± 1.81.4 ± 2.514.1
OPSD20.5 ± 1.027.3 ± 7.03.0 ± 2.11.1 ± 2.013.0
SDPO20.1 ± 1.425.7 ± 5.84.0 ± 2.71.7 ± 2.412.9
SFT teacher22.5 ± 0.329.5 ± 7.03.1 ± 2.51.5 ± 1.914.2
RL teacher23.4 ± 1.128.2 ± 4.64.0 ± 2.32.0 ± 2.214.4
FlowTD24.5 ± 1.335.0 ± 6.16.5 ± 3.03.0 ± 2.617.3

As shown in Table R5, FlowTD maintains the highest average accuracy under sampling, outperforming the strongest baseline (RL teacher) by 2.9 percentage points. We have added these results to Appendix C.2 of the revised manuscript.


Thank you once again for your insightful comments, which have strengthened our work. We have updated the manuscript to reflect these changes (highlighted in blue for easy identification). We hope that our responses and revisions address your concerns, and we welcome any further feedback you may have.

AuthorsOpenReview ↗

Brief Follow-up on Our Response

Dear Reviewer 5Ymg,

Thank you again for your insightful questions. As the discussion period ends soon, we briefly summarize our responses:

  • Baseline performance: We reproduce DCPO's reported base-model results under its evaluation protocol. Using DCPO's training data and evaluation protocol, FlowTD outperforms the published GRPO and DCPO results by 4.00 and 4.35 percentage points on average.
  • Penalty coefficient β: FlowTD is robust across β ∈ {10, 15, 30} (Appendix C.1).
  • Teacher properties and variability: Ablations show that each teacher property contributes to student performance (Section 4.3), and FlowTD remains the best method under sampling-based evaluation with mean ± SD (Appendix C.2).

Please let us know whether our responses address your concerns or if any further clarification would be helpful.

Best regards,
Authors

Reviewer_XCspOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

FlowTD uses GFlowNet trajectory balance to train a teacher with access to reference solutions, aiming to favor correct answers while preserving the initial student’s preferences among correct reasoning paths. The teacher is then frozen for on-policy JSD distillation, with experiments on Qwen2.5-3B and Qwen3-8B-Base showing higher average accuracy across four math benchmarks and better AIME 2025 pass@k than the compared teacher constructions.

Critical strengths

The teacher target specifies both which answers to favor and how to weight different correct solutions. Proposition 2 connects this target to the chance of completing a solution correctly, giving a clear rationale for the token-level guidance.

The comparisons use the same student loss, which helps assess the different teachers. On 3B, FlowTD produces a stronger student than the SFT and RL teachers despite their similar teacher accuracy.

The average gains over the strongest baseline are 4.7 points on 3B and 3.88 points on 8B. The AIME 2025 results also show an advantage across sampling budgets up to 32 responses, so the gains extend beyond greedy decoding.

Critical weaknesses

FlowTD’s teacher gives more diverse guidance, which may already explain part of the improvement. The experiments do not show whether preserving the student’s original preferences among correct solutions provides any additional benefit.

Figure 6a–b analyzes complete responses from the initial model, while the teacher actually guides a changing student during training. This leaves unclear how well its next-token feedback helps the student recover when its reasoning goes wrong.

For 8B, the privileged context uses DeepSeek-R1 solutions, while SFT targets come from ground_truth_solution (Appendix A.2). These references may differ in quality or style, so it would help to compare against SFT trained on the same DeepSeek-R1 solutions.

Questions for authors

Would an SFT or RL teacher with similarly diverse predictions still lag behind FlowTD, at comparable teacher accuracy?

On the student’s own rollouts, does FlowTD give better guidance after a reasoning mistake—for example, by favoring next steps that lead to a correct answer?

Does the SFT comparison change when it is trained on the same DeepSeek-R1 reference solutions used as privileged context?

The paper asks how teachers should distribute probability across correct solutions, which connects to broader KD work on loss design, probability-mass allocation, and hard/soft supervision [1–2]. A short discussion would help explain how learning the teacher complements these approaches.

[1] ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence. ICML, 2025.

[2] DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML, 2025.

4 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

We sincerely thank you for the thoughtful and constructive review. We appreciate your recognition that our teacher target specifies both which answers to favor and how to weight different correct solutions, and that Proposition 2 provides a clear rationale for token-level guidance. We are also glad that you note that the FlowTD teacher yields stronger students than SFT and RL teachers of similar accuracy, with gains extending beyond greedy decoding. We address each point below.

W1 & Q1: Diversity Versus Preference Preservation

We thank the reviewer for this insightful comment. To separate diversity from preference preservation, we perturb the learned FlowTD teacher qq during the distillation stage, keeping the teacher checkpoint and the student training setup fixed:

  • Entropy-matched teacher: lower the teacher's temperature until its token entropy closely matches that of the RL teacher (≈ 0.12 nats).
  • Perturbed prior preferences: reweight the teacher by the initial student's probabilities, q~∝q p0α−1\tilde q \propto q\, p_0^{\alpha-1} with α=0.6\alpha = 0.6, within its top-pp nucleus and with its entropy kept at FlowTD's level, which distorts the prior's relative preferences.

Table R1. Diversity and preference perturbations on Qwen2.5-3B.

TeacherTeacher acc. (%)EntropyPref. SDStudent AVG (%)
FlowTD93.20.2498.221.1
+ Entropy-matched teacher93.50.12512.619.1
+ Perturbed prior preferences93.30.25614.719.5
RL teacher93.90.12115.516.3

Two observations address the question (Table R1). First, diversity alone does not explain FlowTD's advantage: at closely matched entropy and similar teacher accuracy, the FlowTD teacher still yields a substantially stronger student than the RL teacher (19.1% vs. 16.3%). Second, preference preservation provides an additional benefit: with comparable teacher accuracy and entropy, perturbing the prior's preferences increases preference distortion from 8.2 to 14.7 and lowers student accuracy from 21.1% to 19.5%.

These results provide further evidence that preference preservation contributes beyond teacher diversity. We have added this analysis to Section 4.3 (Table 4) of the revised manuscript.

W2 & Q2: Next-Step Guidance on Student Rollouts

We thank the reviewer for this observation. To complement the complete-response analysis in Figure 6a–b, we evaluate next-step guidance on 100 prefixes from unsuccessful rollouts at intermediate and final student checkpoints.

For each prefix, each teacher selects one of eight student-generated candidate reasoning steps using cumulative teacher-to-student log-probability ratios, and the same frozen student continues from the selected step to the final answer. Full evaluation details are provided in Appendix D.2.

Table R2. Top-1 Continuation Accuracy (%) on Qwen2.5-3B.

Student checkpointRandom selectionRL teacherFlowTD
Intermediate31.937.640.4
Final33.539.342.2

At both stages, the step selected by FlowTD leads to a correct answer more often than a randomly selected step (by about 8.5 points) and the step selected by the RL teacher (by nearly 3 points). This indicates that FlowTD provides more accurate dense supervision, better aligned with eventual success, even on the student's own unsuccessful rollouts. We have added this analysis to Appendix D.2 of the revised manuscript.

W3 & Q3: SFT Reference Solutions

We thank the reviewer for raising this concern. We clarify that both the privileged reference and the SFT training target are generated by DeepSeek-R1. In the standard setup, all teachers receive the same DeepSeek-R1 solution as privileged input, while the SFT teacher is trained to generate a second, independently sampled DeepSeek-R1 solution for the same problem. This second solution was stored under the field name ground_truth_solution, which led to the ambiguity in Appendix A.2.

Thus, the original SFT comparison in Table 3 already uses DeepSeek-R1 solutions for both the privileged input and the training target. We apologize for the confusion and have corrected this description in Appendices A.2 and A.5 of the revised manuscript.

AuthorsOpenReview ↗

Response by Authors [2/2]

Q4: Relationship to Prior Knowledge Distillation Methods

We thank the reviewer for highlighting these connections. ABKD (Wang et al., 2025) and DistiLLM-2 (Ko et al., 2025) primarily improve distillation objectives through divergence design and contrastive loss design, respectively. These methods primarily focus on how knowledge is transferred through student-side distillation objectives, whereas FlowTD addresses a complementary question: what distribution the teacher should represent, including how it allocates probability among correct solutions. To this end, FlowTD designs and learns a teacher distribution before student distillation.

These directions are complementary: the teacher distribution learned by FlowTD could be combined with alternative distillation objectives, for example by replacing the JSD objective in Stage 2, which we fix across methods to isolate the effect of the teacher. We have added this discussion and the relevant references to Section 5 of the revised manuscript.


Thank you once again for your constructive comments, which have strengthened our work. We have updated the manuscript to reflect these changes (highlighted in blue for easy identification). We hope that our responses and revisions address your concerns, and we welcome any further feedback you may have.

Thank you for the detailed responses and revisions. They address most of my concerns, and I will maintain my positive score.

Thank you for your positive assessment and constructive feedback, which have helped us improve the paper. We appreciate your time and effort.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Reviewers,

We sincerely appreciate your time and expertise in reviewing our paper. We are pleased that the reviewers find our teacher-distribution formulation well motivated and principled (Kj2y, XCsp) and theoretically elegant (5Ymg). FlowTD is regarded as novel and technically sound, with trajectory balance making the posterior tractable and addressing diversity collapse (5Ymg, Kj2y). The reviewers also note that our analyses provide mechanistic insight, showing that teachers with similar accuracy can yield different students (5Ymg, 8HXH, XCsp), and offer useful insights for future self-distillation work (8HXH). Empirically, FlowTD achieves the best average performance on both model families (8HXH, Kj2y, XCsp). Finally, the paper is found clear and easy to follow (5Ymg, 8HXH, XCsp), providing a deeper understanding of the problem (5Ymg, Kj2y, XCsp).

We believe that all of the reviewers' concerns can be addressed. Below, we provide brief responses to the main concerns and suggestions raised in the reviews:

Ablations and Analysis of the FlowTD Teacher (Kj2y, XCsp). (1) Teacher-property ablations support our design principles for effective distillation (Section 4.3, Table 4). Even without privileged information, FlowTD outperforms all baselines (Section 4.3, Table 5). (2) On prefixes from unsuccessful student rollouts, FlowTD provides more effective next-step guidance than RL teacher (Appendix D.2). (3) It also achieves the highest teacher pass@k, supporting broader coverage of correct answers (Appendix D.1).

Additional Baselines, Models, and Settings (8HXH, 5Ymg, Kj2y). We have added FlowRL, π-Distill, and VPD as baselines and extended our experiments to a larger training set (DCPO's setup) and to instruct and thinking models. FlowTD consistently achieves higher performance than the baselines, demonstrating its effectiveness (Appendix B).

Evaluation Robustness and Reproducibility (5Ymg, 8HXH, Kj2y, XCsp). We have added sampling-based evaluation with mean ± SD (Appendix C.2) and a hyperparameter study (Appendix C.1). We have also clarified training and evaluation protocols (Appendix A) and released our code in an anonymous repository to facilitate reproducibility.

All of these revisions have been incorporated into the revised manuscript and are highlighted in blue for your convenience. We hope that our responses and revisions adequately address your concerns, and we would be very grateful if you could take them into account when making your final evaluation of our work.

Sincerely,

Authors

Dear Reviewers,

Thank you for your time and effort in reviewing this paper. Please take a moment to read the authors’ rebuttal and the other reviewers’ comments, and participate in the discussion. It would be helpful to indicate which concerns have been addressed and which remain unresolved, and to follow up with the authors where further clarification is needed. Please also update your review and score if warranted, or briefly explain why your assessment remains unchanged. Your engagement will be valuable in reaching a well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.