- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TeachBack targets “thinking collapse” in on-policy self-distillation (OPSD), where a privileged-information (PI) teacher produces short, reference-shaped solutions that lack the exploration and self-correction a problem-only student needs. It makes two changes. (1) Teacher construction: the teacher prompt presents a suffix (50% in main runs) of the reference reasoning as optional “working notes” and asks the model to re-solve the problem. A prompt audit on Qwen3-4B compares prompt styles and suffix lengths using accuracy, length, and lexical exploration and reference-dependence metrics. (2) Bidirectional preference learning: one shared policy generates a teacher-conditioned and a student-conditioned response for each problem (G=4 pairs, ≤4K tokens). Both are scored under both prompts against a frozen reference. A DPO-style Student loss prefers the teacher-origin response under the problem-only prompt. A reversed Teacher loss prefers the student-origin response under the PI prompt. The two are mixed with weight λ (0.75 for math, 0.25 for Knights-and-Knaves). No verifier is used. On Qwen3-1.7B/4B/8B, TeachBack reaches the highest Math and KK averages (e.g., 77.6 vs 76.0 for OPSD and 76.5 for GRPO/SPIN at 4B).
Critical strengths
- The paper moves attention from how the student consumes the teacher signal to how the PI-conditioned teacher is constructed, and it backs the diagnosis with a measured audit (Table 2, Fig. 3). In the audit, the problem-only student produces ~7.5K-token rollouts, while PI-conditioned teachers produce ~1.3–4.2K tokens with much lower exploration density. The prompt-style/suffix-length sweep gives a usable recipe.
- Three Qwen3 scales, math plus in-domain and OOD Knights-and-Knaves, two additional architectures, a λ sweep, and per-direction margin/entropy/length tracking. TeachBack improves over Base in nearly all settings, including cases where OPSD/GRPO fall below Base (e.g., Llama-8B). The authors also acknowledge that their lexical metrics are proxies.
Critical weaknesses
- λ ablation (Fig. 5): λ=1 (only the Teacher-side loss, which prefers Student-origin responses under the PI prompt) gives the same held-out Math as λ=0.75 (54.02 vs 53.99), and λ=0 (Student-side transfer only, which is the analogue of PBSD/SPIN-style distillation) is far worse (45.44). Fig. 4 also shows the Student-side margin barely moves at λ=0.75. So the “distillation from the teacher” component contributes little or nothing, and the improvement comes from the reversed loss. The paper acknowledges that this “may” transfer through shared parameters but never tests it. By Proposition F.1, the combined objective reduces the privileged-condition advantage of teacher-origin responses, i.e., it makes the PI condition matter less. That is a regularizer against shortening, not evidence that the student learns from privileged information.
- No verifier or correctness signal is used anywhere, and labels depend only on role. There is no placebo run (e.g., a suffix from a different problem, or empty notes), so we cannot tell whether the privileged information matters at all. The method also trains on 4,096-token prefixes while student rollouts average ~6.6–7.5K tokens. Student-origin samples are therefore mostly truncated prefixes compared against mostly complete teacher responses. Length and “do not terminate early” are therefore plausible confounds, and the paper’s own evidence shows gains tracking response length (Fig. 5b). No length-matched or length-regularized baseline is given.
- All results are single training runs. AIME has 30 problems per year, and KK is greedy pass@1 on what appear to be ~100-problem test sets. Differences of 1–2 points are therefore inside plausible run-to-run noise. Per-benchmark swings in Table 7 (e.g., AIME25 from 29.1 to 39.1 on Llama-8B) show how noisy this setting is. TeachBack even loses on KK 11-roles at 1.7B (-1.0) and AMC23 for Qwen-7B.
- TeachBack samples G=4 pairs (eight rollouts) per prompt with a 4,096-token cap, plus four scoring passes per pair. OPSD and PBSD use one rollout at 1,024 tokens, SPIN uses 8,192, and GRPO uses 16,384. For thinking models whose natural responses are several thousand tokens, a 1,024-token cap plausibly handicaps OPSD/PBSD. PBSD at 4B scores 75.0, below the untrained Base (75.4), which contradicts that method’s own paper and suggests a reproduction/tuning gap (learning rate 1e-6 vs the 5e-6 used there, 300 vs 500 steps). No compute- or token-matched comparison is given.
Questions for authors
- Does the benefit survive without genuine privileged information, e.g., a Notes prompt with a suffix from a different problem, or with empty notes, for both λ=0.75 and λ=1? (Determines whether the effect is “learning from PI” or a regularizer.)
- How do gains compare against a length-matched or length-regularized baseline? What fraction of Student-origin and Teacher-origin samples are truncated at K=4,096 during training, and what are the truncation rates behind Fig. 5a?
- Can you add (a) Notes prompt + OPSD/PBSD loss and (b) TeachBack loss + Reference prompt under the Table 1 protocol, to separate prompt from loss? Why does λ=1 alone match λ=0.75 and λ=0 fall far below it, and what role does the Student-side loss play?
- Can you report ≥3 training seeds with confidence intervals (at least for 4B), and the size of the KK test sets?
- How were baseline settings chosen? Why does PBSD fall below Base at 4B (contrary to its original report), and why do OPSD/PBSD use a 1,024-token cap while TeachBack uses 4,096? Can you provide a token- or compute-matched comparison?
- How does TeachBack compare to RLCSD, AD-OPSD, Purified OPSD, and PAINT under the same data/splits?
We thank Reviewer kf5r for taking the time to provide these valuable comments on our paper. We first address the four points below.
1. Training cost and rollout settings
We agree that the main experiments in the original paper do not use exactly the same compute budget. In Table 6, we use the settings recommended in the original papers for each method. The training cost also differs across methods. OPSD uses logit-level supervision, so it keeps the top-k and tail logits for each token when computing the gradient. Its loss signal has complexity O(BT(K+1)) where B is the batch size, T is the number of training tokens, and K is the number of retained top-k logits; in our experiments, K=100. PBSD uses a sequence-level preference loss only on the Student branch, with loss-signal complexity O(BT). TeachBack computes and optimizes sequence-level losses for both the Student and Teacher branches, with complexity O(2BT). OPSD therefore gives a denser gradient update, while PBSD and TeachBack give sparser sequence-level updates. For this reason, it is difficult to align the total end-to-end compute exactly.
To better control the effect of rollout length, we also ran a controlled experiment using the same maximum generation length (per-rollout token budget) for all methods. Because PBSD and TeachBack need to generate a Student-Teacher pair, we used G=1 for OPSD and G=2 for PBSD and TeachBack. The results on Qwen3-4B Math are reported as mean@4:
TeachBack achieves the best result at all three maximum generation lengths. We will add this comparison to the revised paper.
2. Reference overlap in Table 2
We agree that the content-overlap values 21.42 and 21.26 are very close. Notes is designed to reduce the Teacher’s direct reliance on and explicit references to the provided solution, rather than to eliminate all content overlap. It also provides a controlled way to introduce the latter part of the reasoning chain of thought (CoT). Instead of using a reference-style prompt that forces the model to follow the injected CoT, Notes presents it as optional working guidance. Therefore, a lower combined-reference score can occur even when the content-overlap value changes only slightly.
3. Teacher-side weight ablation
We apologize that the wording in Section 4.5 (lines 467--470) was unclear. Our intended point was that training with a higher λ is generally better than training with a lower λ. The final Math performance of Teacher-only training and bidirectional training is very similar: 54.02% versus 53.99%. At step 300, the difference is small. As training continues, the room for further improvement on the dataset becomes limited. We therefore usually compare the best checkpoint. In Figure 5(d), the best checkpoint for λ=0.75 reaches 56.0% at step 100, compared with 54.3% for λ=1.0 at step 50. Our conclusion focuses on the overall training trend and the performance at the best checkpoint.
To further show the importance of training the Teacher, we ran the λ ablation on three Qwen3 model sizes with three random seeds. The results below report Math mean@4 as mean ± standard deviation:
Across all three model sizes, λ=0.75 gives the highest Math mean@4. We will revise Section 4.5 to make clear that larger Teacher-side weights generally perform better than smaller ones.
4. How the domain weights were chosen
The domain weights were fixed by task characteristics. For Math, we use λ=0.75 because mathematical problems have diverse solution paths and benefit from stronger Teacher-side feedback. For Knight-and-Knaves, we use λ=0.25 because the reasoning is more structured and repeatable, and a smaller Teacher-side weight helps limit adaptation to repeated surface patterns. We will clarify this selection rule in the revised paper.