acceptodds
Under review as a conference paper at ICLR 2027

TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation

19 views

Abstract

On-policy self-distillation (OPSD) enables a language model to learn from a Teacher instantiated from the same model but given access to privileged information, such as a reference solution. We find that access to sufficient privileged information allows the Teacher to generate highly accurate trajectories; however, accuracy alone does not guarantee that these trajectories are suitable for Student learning. Direct access to a complete solution may encourage shortened, reference-conditioned derivations that omit the exploration, verification, and self-correction required to solve problems without privileged information. We introduce TeachBack, a bidirectional preference-learning framework for on-policy self-distillation that learns not only from the Teacher but also how the Teacher should teach. TeachBack shapes the Teacher’s generation behavior by controlling the length of the reference-reasoning suffix injected into the prompt and by varying the prompt style. While maintaining comparable Teacher accuracy, it encourages the Teacher to explore and re-solve problems, thereby reducing its reliance on the reference solution. At each rollout refresh, the shared policy generates both a privileged Teacher trajectory and a problem-only Student trajectory. A pairwise preference objective uses privileged Teacher trajectories to improve Student learning, while the reversed preference uses the Student’s behavior to shape future Teacher supervision under the privileged condition. Experiments on mathematical and logical reasoning tasks show that TeachBack consistently improves Student performance over existing OPSD methods and related baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy self-distillation, preference optimization, language model reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8T3iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TeachBack targets “thinking collapse” in on-policy self-distillation (OPSD), where a privileged-information (PI) teacher produces short, reference-shaped solutions that lack the exploration and self-correction a problem-only student needs. It makes two changes. (1) Teacher construction: the teacher prompt presents a suffix (50% in main runs) of the reference reasoning as optional “working notes” and asks the model to re-solve the problem. A prompt audit on Qwen3-4B compares prompt styles and suffix lengths using accuracy, length, and lexical exploration and reference-dependence metrics. (2) Bidirectional preference learning: one shared policy generates a teacher-conditioned and a student-conditioned response for each problem (G=4 pairs, ≤4K tokens). Both are scored under both prompts against a frozen reference. A DPO-style Student loss prefers the teacher-origin response under the problem-only prompt. A reversed Teacher loss prefers the student-origin response under the PI prompt. The two are mixed with weight λ (0.75 for math, 0.25 for Knights-and-Knaves). No verifier is used. On Qwen3-1.7B/4B/8B, TeachBack reaches the highest Math and KK averages (e.g., 77.6 vs 76.0 for OPSD and 76.5 for GRPO/SPIN at 4B).

Critical strengths
  1. The paper moves attention from how the student consumes the teacher signal to how the PI-conditioned teacher is constructed, and it backs the diagnosis with a measured audit (Table 2, Fig. 3). In the audit, the problem-only student produces ~7.5K-token rollouts, while PI-conditioned teachers produce ~1.3–4.2K tokens with much lower exploration density. The prompt-style/suffix-length sweep gives a usable recipe.
  2. Three Qwen3 scales, math plus in-domain and OOD Knights-and-Knaves, two additional architectures, a λ sweep, and per-direction margin/entropy/length tracking. TeachBack improves over Base in nearly all settings, including cases where OPSD/GRPO fall below Base (e.g., Llama-8B). The authors also acknowledge that their lexical metrics are proxies.
Critical weaknesses
  1. λ ablation (Fig. 5): λ=1 (only the Teacher-side loss, which prefers Student-origin responses under the PI prompt) gives the same held-out Math as λ=0.75 (54.02 vs 53.99), and λ=0 (Student-side transfer only, which is the analogue of PBSD/SPIN-style distillation) is far worse (45.44). Fig. 4 also shows the Student-side margin barely moves at λ=0.75. So the “distillation from the teacher” component contributes little or nothing, and the improvement comes from the reversed loss. The paper acknowledges that this “may” transfer through shared parameters but never tests it. By Proposition F.1, the combined objective reduces the privileged-condition advantage of teacher-origin responses, i.e., it makes the PI condition matter less. That is a regularizer against shortening, not evidence that the student learns from privileged information.
  2. No verifier or correctness signal is used anywhere, and labels depend only on role. There is no placebo run (e.g., a suffix from a different problem, or empty notes), so we cannot tell whether the privileged information matters at all. The method also trains on 4,096-token prefixes while student rollouts average ~6.6–7.5K tokens. Student-origin samples are therefore mostly truncated prefixes compared against mostly complete teacher responses. Length and “do not terminate early” are therefore plausible confounds, and the paper’s own evidence shows gains tracking response length (Fig. 5b). No length-matched or length-regularized baseline is given.
  3. All results are single training runs. AIME has 30 problems per year, and KK is greedy pass@1 on what appear to be ~100-problem test sets. Differences of 1–2 points are therefore inside plausible run-to-run noise. Per-benchmark swings in Table 7 (e.g., AIME25 from 29.1 to 39.1 on Llama-8B) show how noisy this setting is. TeachBack even loses on KK 11-roles at 1.7B (-1.0) and AMC23 for Qwen-7B.
  4. TeachBack samples G=4 pairs (eight rollouts) per prompt with a 4,096-token cap, plus four scoring passes per pair. OPSD and PBSD use one rollout at 1,024 tokens, SPIN uses 8,192, and GRPO uses 16,384. For thinking models whose natural responses are several thousand tokens, a 1,024-token cap plausibly handicaps OPSD/PBSD. PBSD at 4B scores 75.0, below the untrained Base (75.4), which contradicts that method’s own paper and suggests a reproduction/tuning gap (learning rate 1e-6 vs the 5e-6 used there, 300 vs 500 steps). No compute- or token-matched comparison is given.
Questions for authors
  1. Does the benefit survive without genuine privileged information, e.g., a Notes prompt with a suffix from a different problem, or with empty notes, for both λ=0.75 and λ=1? (Determines whether the effect is “learning from PI” or a regularizer.)
  2. How do gains compare against a length-matched or length-regularized baseline? What fraction of Student-origin and Teacher-origin samples are truncated at K=4,096 during training, and what are the truncation rates behind Fig. 5a?
  3. Can you add (a) Notes prompt + OPSD/PBSD loss and (b) TeachBack loss + Reference prompt under the Table 1 protocol, to separate prompt from loss? Why does λ=1 alone match λ=0.75 and λ=0 fall far below it, and what role does the Student-side loss play?
  4. Can you report ≥3 training seeds with confidence intervals (at least for 4B), and the size of the KK test sets?
  5. How were baseline settings chosen? Why does PBSD fall below Base at 4B (contrary to its original report), and why do OPSD/PBSD use a 1,024-token cap while TeachBack uses 4,096? Can you provide a token- or compute-matched comparison?
  6. How does TeachBack compare to RLCSD, AD-OPSD, Purified OPSD, and PAINT under the same data/splits?
Reviewer_kf5rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TeachBack uses the same model as a teacher and a student. The teacher gets part of the reference reasoning as working notes. The student gets only the problem. The two responses form opposite preference pairs under these prompts. The authors test the resulting model on math and Knight-and-Knaves problems, without reference reasoning at evaluation.

Critical strengths

In Section 4.5, training privileged role helps shared models answer problems without a reference. Tables 2 and 4 give some context for this result through reference framing and teacher transfer experiment.

Qwen3-4B goes from 75.4 to 77.6 on math and from 61.8 to 65.7 on logic in Table 1.

Critical weaknesses

OPSD and PBSD each get one generation capped at 1,024 tokens. TeachBack gets four generations capped at 4,096. With both settings changed, I cannot tell how much of the gain comes from preference objective.

Notes lowers the combined reference in Table 2, yet content overlap does not much move: 21.42 versus 21.26. Teacher-only training finishes at 54.02, compared with 53.99 for the mixed objective. The mixed version has better checkpoint-average scores, but I would want repeated runs before drawing conclusion about what student branch adds.

Questions for authors
  1. What gain remains over OPSD and PBSD when rollout length, generation count, and compute are matched?
  2. How does mixed training compare with teacher-only training across seeds? How were the weights for each domain chosen using validation data?
3 comments

We thank Reviewer kf5r for taking the time to provide these valuable comments on our paper. We first address the four points below.

1. Training cost and rollout settings

We agree that the main experiments in the original paper do not use exactly the same compute budget. In Table 6, we use the settings recommended in the original papers for each method. The training cost also differs across methods. OPSD uses logit-level supervision, so it keeps the top-kk and tail logits for each token when computing the gradient. Its loss signal has complexity O(BT(K+1))O(BT(K+1)) where BB is the batch size, TT is the number of training tokens, and KK is the number of retained top-kk logits; in our experiments, K=100K=100. PBSD uses a sequence-level preference loss only on the Student branch, with loss-signal complexity O(BT)O(BT). TeachBack computes and optimizes sequence-level losses for both the Student and Teacher branches, with complexity O(2BT)O(2BT). OPSD therefore gives a denser gradient update, while PBSD and TeachBack give sparser sequence-level updates. For this reason, it is difficult to align the total end-to-end compute exactly.

To better control the effect of rollout length, we also ran a controlled experiment using the same maximum generation length (per-rollout token budget) for all methods. Because PBSD and TeachBack need to generate a Student-Teacher pair, we used G=1G=1 for OPSD and G=2G=2 for PBSD and TeachBack. The results on Qwen3-4B Math are reported as mean@4:

Method1K2K4K
OPSD76.2875.9375.93
PBSD75.1574.6772.02
TeachBack77.1477.3277.60

TeachBack achieves the best result at all three maximum generation lengths. We will add this comparison to the revised paper.

2. Reference overlap in Table 2

We agree that the content-overlap values 21.42 and 21.26 are very close. Notes is designed to reduce the Teacher’s direct reliance on and explicit references to the provided solution, rather than to eliminate all content overlap. It also provides a controlled way to introduce the latter part of the reasoning chain of thought (CoT). Instead of using a reference-style prompt that forces the model to follow the injected CoT, Notes presents it as optional working guidance. Therefore, a lower combined-reference score can occur even when the content-overlap value changes only slightly.

3. Teacher-side weight ablation

We apologize that the wording in Section 4.5 (lines 467--470) was unclear. Our intended point was that training with a higher λ\lambda is generally better than training with a lower λ\lambda. The final Math performance of Teacher-only training and bidirectional training is very similar: 54.02% versus 53.99%. At step 300, the difference is small. As training continues, the room for further improvement on the dataset becomes limited. We therefore usually compare the best checkpoint. In Figure 5(d), the best checkpoint for λ=0.75\lambda=0.75 reaches 56.0% at step 100, compared with 54.3% for λ=1.0\lambda=1.0 at step 50. Our conclusion focuses on the overall training trend and the performance at the best checkpoint.

To further show the importance of training the Teacher, we ran the λ\lambda ablation on three Qwen3 model sizes with three random seeds. The results below report Math mean@4 as mean ±\pm standard deviation:

Modelλ=0.25\lambda=0.25λ=0.5\lambda=0.5λ=0.75\lambda=0.75λ=1.0\lambda=1.0
Qwen3-1.7B51.34±0.3251.34 \pm 0.3254.77±0.1854.77 \pm 0.1857.09±0.4657.09 \pm 0.4656.01±0.3156.01 \pm 0.31
Qwen3-4B74.63±0.2674.63 \pm 0.2675.88±0.2675.88 \pm 0.2677.37±0.3777.37 \pm 0.3776.20±0.3376.20 \pm 0.33
Qwen3-8B79.72±0.2479.72 \pm 0.2479.46±0.2879.46 \pm 0.2880.06±0.3280.06 \pm 0.3278.43±0.2478.43 \pm 0.24

Across all three model sizes, λ=0.75\lambda=0.75 gives the highest Math mean@4. We will revise Section 4.5 to make clear that larger Teacher-side weights generally perform better than smaller ones.

4. How the domain weights were chosen

The domain weights were fixed by task characteristics. For Math, we use λ=0.75\lambda=0.75 because mathematical problems have diverse solution paths and benefit from stronger Teacher-side feedback. For Knight-and-Knaves, we use λ=0.25\lambda=0.25 because the reasoning is more structured and repeatable, and a smaller Teacher-side weight helps limit adaptation to repeated surface patterns. We will clarify this selection rule in the revised paper.

Thanks for adding the rollout-cap comparison and the three-seed weight sweep. The results show that TeachBack leads on Qwen3-4B Math at the same per-rollout cap, and the weight sweep makes the case for including the student-side loss stronger. Could you report total generated training tokens and end-to-end accelerator time for the cap-matched runs, including rollout generation and updates? For the λ = 0.75 versus λ = 1 comparison, could you clarify how checkpoints were selected and whether the gain holds at a common training step?

Thank you for your questions. We provide additional details on computational cost and checkpoint performance below.

The first table reports the computational cost of 300 training steps with a rollout cap of 1K tokens. All time measurements are in GPU-hours. OPSD uses G = 1, whereas PBSD and TeachBack use G = 2, giving OPSD a smaller rollout token budget per prompt. TeachBack’s additional cost primarily comes from the forward and backward passes during parameter updates, as it jointly trains the teacher and student models.

MethodRollout GenerationParameter UpdatesOtherTotalRollout Tokens per step
OPSD1.502.081.044.621,024
PBSD2.502.420.965.882,048
TeachBack2.334.171.207.702,048

The following table presents Qwen-8B’s Mean@4 (%) at each evaluated checkpoint, averaged equally across the three benchmarks. The final “Mean” column averages the six checkpoint scores, and bold values indicate the best checkpoint for each λ.

λ \ Step50100150200250300Mean
0.2576.2475.4277.3075.8679.7476.5276.85
0.576.1276.3177.2279.4477.3777.2477.28
0.7577.3675.2779.0980.2077.6275.8177.56
176.5278.3475.8376.3977.4277.5077.00

Performance fluctuates across checkpoints, and the relative ranking of λ settings varies across training steps. Among the evaluated settings, λ = 0.75 achieves both the highest checkpoint score (80.20%) and the highest average across checkpoints (77.56%). Our reported result is taken from the best-performing checkpoint.

We hope these additional results and clarifications address your concerns. If they do, we would be grateful if you would consider raising your score to 3.

Reviewer_CN9xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TeachBack uses one shared model as a problem-only Student and a Teacher given partial reference reasoning. Bidirectional preference optimization transfers Teacher responses to the Student while encouraging the Teacher to retain Student-like reasoning behavior. The intended purpose is to improve the usefulness of supervision. Only the Student is used at inference. Experiments report improved average mathematical and logical reasoning performance across several model sizes.

Critical strengths
  • The paper addresses an interesting issue: accurate privileged Teacher responses may still provide poor supervision for a Student without that information. Adapting the Teacher’s behavior is a promising approach.

  • Improvements appear across multiple model sizes and both mathematical and logical reasoning tasks, providing encouraging evidence beyond a single configuration.

Critical weaknesses
  • The practical advantage is difficult to assess fairly. Table 6 uses different training budgets: TeachBack has four rollout pairs and a 4,096-token cap, whereas OPSD and PBSD use one rollout/pair and a 1,024-token cap. Without comparisons at matched compute or token budgets, the contribution of the proposed method remains uncertain.

  • The added value of bidirectional training is not clearly established. In Section 4.5, Teacher-only training achieves almost identical final Math performance to bidirectional training: 54.02% versus 53.99%. Although bidirectional training has a higher peak and checkpoint average, these results leave uncertainty about whether both directions are necessary.

Questions for authors
  • Do the gains over OPSD and PBSD persist under matched training compute and generated-token budgets? What is the additional training cost?
1 comment

We thank Reviewer CN9x for taking the time to provide these valuable comments on our paper. We first address the two points below.

1. Training cost and rollout settings

We agree that the main experiments in the original paper do not use exactly the same compute budget. In Table 6, we use the settings recommended in the original papers for each method. The training cost also differs across methods. OPSD uses logit-level supervision, so it keeps the top-kk and tail logits for each token when computing the gradient. Its loss signal has complexity$O(BT(K+1)),$where BB is the batch size, TT is the number of training tokens, and KK is the number of retained top-kk logits; in our experiments, K=100K=100. PBSD uses a sequence-level preference loss only on the Student branch, with loss-signal complexity O(BT)O(BT). TeachBack computes and optimizes sequence-level losses for both the Student and Teacher branches, with complexity O(2BT)O(2BT). OPSD therefore gives a denser gradient update, while PBSD and TeachBack give sparser sequence-level updates. For this reason, it is difficult to align the total end-to-end compute exactly.

To better control the effect of rollout length, we also ran a controlled experiment using the same maximum generation length (per-rollout token budget) for all methods. Because PBSD and TeachBack need to generate a Student-Teacher pair, we used G=1G=1 for OPSD and G=2G=2 for PBSD and TeachBack. The results on Qwen3-4B Math are reported as mean@4:

Method1K2K4K
OPSD76.2875.9375.93
PBSD75.1574.6772.02
TeachBack77.1477.3277.60

TeachBack achieves the best result at all three maximum generation lengths. We will add this comparison to the revised paper.

2. Teacher-side weight ablation

We apologize that the wording in Section 4.5 (lines 467--470) was unclear. Our intended point was that training with a higher λ\lambda is generally better than training with a lower λ\lambda. The final Math performance of Teacher-only training and bidirectional training is very similar: 54.02% versus 53.99%. At step 300, the difference is small. As training continues, the room for further improvement on the dataset becomes limited. We therefore usually compare the best checkpoint. In Figure 5(d), the best checkpoint for λ=0.75\lambda=0.75 reaches 56.0% at step 100, compared with 54.3% for λ=1.0\lambda=1.0 at step 50. Our conclusion focuses on the overall training trend and the performance at the best checkpoint.

To further show the importance of training the Teacher, we ran the λ\lambda ablation on three Qwen3 model sizes with three random seeds. The results below report Math mean@4 as mean ±\pm standard deviation:

Modelλ=0.25\lambda=0.25λ=0.5\lambda=0.5λ=0.75\lambda=0.75λ=1.0\lambda=1.0
Qwen3-1.7B51.34±0.3251.34 \pm 0.3254.77±0.1854.77 \pm 0.1857.09±0.4657.09 \pm 0.4656.01±0.3156.01 \pm 0.31
Qwen3-4B74.63±0.2674.63 \pm 0.2675.88±0.2675.88 \pm 0.2677.37±0.3777.37 \pm 0.3776.20±0.3376.20 \pm 0.33
Qwen3-8B79.72±0.2479.72 \pm 0.2479.46±0.2879.46 \pm 0.2880.06±0.3280.06 \pm 0.3278.43±0.2478.43 \pm 0.24

Across all three model sizes, λ=0.75\lambda=0.75 gives the highest Math mean@4. We will revise Section 4.5 to make clear that larger Teacher-side weights generally perform better than smaller ones.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.