acceptodds
Under review as a conference paper at ICLR 2027

In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners

Abstract

Stronger learning signals do not necessarily produce a stronger small reasoning model. Off-policy supervised fine-tuning (SFT) supplies stronger traces that may be incompatible with the student. On-policy post-training instead remains constrained by the quality of the student's own trajectories: reinforcement learning (RL) provides little learning signal because a small student rarely produces a correct rollout. Consequently, effective post-training must supply stronger reasoning in a form the student can learn from. We hypothesize that a small number of tokens with very low probability under the student can make an otherwise strong reasoning trace difficult to learn. To test this hypothesis, we introduce Interleaved-Policy Distillation (IPD), which generates SFT data balancing teacher guidance and student compatibility. During generation, the student replaces teacher proposals that have low probability under its own policy, allowing the teacher to continue reasoning from a prefix the student can support. IPD improves reasoning across model families, datasets, and varying degrees of student intervention; for example, it raises the seven-benchmark average from 24.55 to 29.72 on Qwen3-0.6B/s1K. It outperforms baselines with comparable training budgets and broadens problem-solving coverage, while ablations show that filtering traces or masking losses cannot match the gains from revising trajectories during generation. Analyses link transfer failures to the rare tokens the student finds least likely and show that the source of the tokens also matters. Small models reason better when they learn strong solutions in a form they can reach in their own words.

Primary area:
generative models
Keywords:
large language models, reasoning models, supervised fine-tuning, reinforcement learning, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_z76VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why is the distillation from larger teacher models to smaller models produce traces that are not fully compatible with smaller reasoning models, where the paper argues that this is because the stronger models traces might contain a few tokens that have low probability under the student model. To test this hypothesis, they propose Interleaved-Policy Distillation (IPD): During the generation of teacher tokens if any token has a probability below a specfic threshold, the student samples a replacement token, and the teacher continues reasoning. Then the resulting trajectory is used directly for SFT. A couple of experiments across multiple model families (Qwen3, EXAONE), sizes (0.6B to 1.7B), and datasets (s1K, LIMO) demonstrate that teacher-propose IPD consistently outperforms direct distillation, reinforcement learning (GRPO), on-policy distillation (OPD), and various baseline ablations.

Critical strengths

1- The paper presents a simple approach to improve reasoning distillation from larger teacher models to smaller student models. The approach is easy to understand and does not require changes to the standard SFT training procedure.

2- The paper provides extensive experiments that include four settings, two training datasets, three thresholds, and seven evaluation benchmarks, showing consistent improvements over several baselines. The analysis also provides interesting insights into reasoning distillation, showing that neither mean surprisal nor the number of correct traces alone explains knowledge transfer. The results also highlight the importance of proposal direction and preserving teacher guidance.

Critical weaknesses

1- The paper clearly highlights a strong link between extreme-tail surprisal and distillation performance. However, it remains not very clear whether lowering that surprisal is what actually drives the performance gains. As shown in Figure 4, methods with very similar extreme-tail surprisal values still end up with different performance levels, so surprisal reduction alone isn't the only cause of this improvement.

2-  The evaluation is focused on math reasoning benchmarks and scientific Q&A. Extending this to multi-turn tasks or code-generation tasks would help verify the generalizability of the findings.

Questions for authors

1- The results in Figures 3 and 4 suggest a link between extreme-tail surprisal and distillation performance. Can the authors provide further evidence that reducing extreme-tail surprisal directly contributes to performance improvements, rather than the gains being driven by other changes in the generated reasoning trajectories?

2- Do the authors expect IPD to provide similar improvements for larger students or more open-ended reasoning tasks?

2 comments
AuthorsOpenReview ↗

Initial Response

Reviewer z76V

Thank you for taking the time to review our work. Your questions about the interpretation of extreme-tail surprisal helped us distinguish more clearly between an observed association and an explanation of the performance gains. We appreciate the opportunity to clarify this distinction.

Extreme-tail surprisal and the source of improvement (W1, Q1)

Our analysis shows that lower extreme-tail student surprisal is associated with better transfer and serves as a primary indicator of learning compatibility, but it does not fully explain downstream performance. T-P produces stronger students than S-P even when their training datasets have comparable extreme-tail mean surprisal (Figure 4). This suggests that the source of the reasoning tokens matters alongside their compatibility with the student.

We agree, however, that the current experiments do not isolate the causal contribution of surprisal reduction from other changes in the reasoning trajectories. IPD selectively revises teacher proposals during generation, changing both student support and subsequent reasoning. Our experiments support the practical benefit of adapting teacher-led reasoning to student support, while disentangling the contributions of improved compatibility and changes in reasoning content remains an important direction for future work.

Model scale and task scope (W2, Q2)

Our goal was to test whether adapting teacher supervision can improve reasoning transfer to students smaller than 3.8B. This focus is motivated by successful reasoning distillation with roughly 800–1,000 traces in Qwen2.5-32B-Instruct [1, 2], alongside reported degradation when the 3.8B Phi-4-Mini model is fine-tuned on s1K [3].

The current comparisons show that IPD benefits both 0.6B and 1.7B students under the same 8B teacher, but they do not establish how the gain changes for larger students. We expect the value of adaptation to depend on how well the student supports the teacher’s reasoning. IPD should therefore help most when a student struggles to learn directly from teacher traces, and less when it already learns well from them, as the 32B models above do.

Other verifiable tasks, such as code reasoning with executable tests, offer a natural extension, because our current data-construction pipeline uses a correctness verifier to select full training traces. Multi-turn tasks, open-ended generation, and interactive agents would require suitable alternatives for assessing and selecting supervision; we leave these extensions for future work. In the revised version, we have added both limitations to a new Limitations and Extensions section (Section 6).

References

[1] Muennighoff et al. s1: Simple test-time scaling. 2025.

[2] Ye et al. LIMO: Less is more for reasoning. 2025.

[3] Xu et al. Phi-4-Mini-Reasoning: Exploring the limits of small reasoning language models in math. 2025.

Dear Reviewer z76V, Thank you again for your review. Your feedback helped us clarify our interpretation of extreme-tail surprisal. As the discussion period nears its end, could you let us know whether our initial response above addresses your concerns? We would be glad to clarify any remaining points.

Reviewer_k7JWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why direct reasoning distillation from strong teachers can fail for small student models. The authors hypothesize that a small number of tokens with very low probability under the student can make otherwise strong reasoning traces difficult to learn. They propose Interleaved-Policy Distillation (IPD), especially a teacher-propose variant: the teacher proposes each token, the student rejects and replaces proposals whose probability under the student falls below a threshold, and the teacher continues from the revised prefix. The resulting interleaved traces are used as ordinary SFT data. The paper evaluates IPD on Qwen3 and EXAONE student/teacher pairs, s1K and LIMO data, and seven reasoning benchmarks, comparing against Direct Distillation, Student RFT, Teacher RFT, GRPO, OPD, trace filtering, loss masking, distribution interpolation, and SKD. The paper further analyzes extreme-tail student surprisal, proposal direction, fallback-token categories, LLM-judge trace quality, and Pass@k coverage.

Critical strengths
  1. The method is simple and practical. IPD only requires frozen teacher and student models during data generation, token-level student scoring, and standard SFT afterward.
  2. The empirical evaluation is fairly broad, covering four model/data settings, seven benchmarks, multiple thresholds, and three training seeds.
  3. The paper includes many relevant controls, including Direct Distillation, Student RFT, Teacher RFT, increased student sampling, GRPO, OPD, trace filtering, loss masking, distribution interpolation, and SKD.
Critical weaknesses
  1. The originality is moderate rather than very high. The work is closely related to student-aligned distillation, likelihood-based data filtering, student-in-the-loop generation, and SKD. The main novelty is a specific and effective token-level gating and proposal-direction design.
  2. Some strong controls are numerically very close to IPD. In the primary setting, distribution interpolation reaches an average of 29.51 while IPD reaches 29.72. It is unclear whether this difference is statistically significant given the reported variability.
  3. The evaluation scope is limited. The largest student model is 1.7B, and the tasks are primarily mathematical reasoning plus GPQA-D. It remains unclear whether the conclusions transfer to larger students, open-ended generation, code reasoning, or interactive agents.
  4. The method assumes that teacher and student share a tokenizer so that each model can score the other's token proposals. The paper does not sufficiently discuss how IPD would work across model families with incompatible tokenizers.
Questions for authors
  1. Are the differences between IPD and the strongest controls statistically significant? In particular, in the primary setting, IPD averages 29.72 while distribution interpolation averages 29.51.
  2. What is the actual wall-clock data-generation cost of IPD compared with Teacher RFT, Student RFT, Direct Distillation, and OPD on the same hardware?
  3. How can IPD be extended when the teacher and student do not share a tokenizer? Would span-level gating, retokenization, or approximate log-probability alignment work?
  4. Does IPD remain effective for larger students, such as 3B or 7B models distilled from 30B or 70B teachers? Are rare low-probability tokens still the main bottleneck at that scale?
  5. Can the authors run a direct follow-up experiment such as Vanilla-to-RL, Teacher-RFT-to-RL, and IPD-to-RL to test whether the improved Pass@k coverage actually improves subsequent on-policy learning?
4 comments
AuthorsOpenReview ↗

Initial Response (1/3)

Reviewer k7JW

Thank you for your thoughtful and constructive review. Your questions about IPD’s contribution and its role in subsequent on-policy learning were particularly helpful in identifying where clearer explanations and more direct evidence were needed. Following your suggestions, we conducted additional experiments and analyses that strengthen the empirical basis of our work.

Contribution and proposal direction (W1)

Our main contribution lies in how student feedback is used to actively construct reasoning supervision through token-level revision of teacher-generated trajectories, rather than merely filtering, weighting, or selecting supervision as in prior student-aligned distillation methods. While methods that filter completed traces as a whole using student likelihood [1], mask token losses without revising the trajectory, or select multi-token chunks during student-in-the-loop generation [2] share the broad goal of adapting distillation to student capabilities, IPD instead selectively revises teacher proposals during generation and trains directly on the resulting interleaved traces. SKD [3] also uses interleaved prefixes, but it trains against teacher distributions rather than on the revised trajectory.

Our second contribution is an empirical analysis of what makes such student-adapted reasoning traces effective, beyond the proposed generation mechanism itself. Lower extreme-tail student surprisal is associated with better transfer but does not fully explain it: T-P produces stronger students than S-P even when their training datasets have comparable extreme-tail mean surprisal (Figure 4). This suggests that the source of the reasoning tokens matters alongside their compatibility with the student. The analysis thus identifies teacher-led proposal as the more effective direction, along with the range of thresholds over which its gains hold.

References

[1] Zhang et al. The Best Instruction-Tuning Data are Those That Fit. NeurIPS, 2025.

[2] He et al. Student-in-the-Loop Chain-of-Thought Distillation via Generation-Time Selection. 2026.

[3] Xu et al. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling. ICLR, 2025.

AuthorsOpenReview ↗

Initial Response (2/3)

Comparison with distribution interpolation (W2, Q1)

T-P ranks above distribution interpolation at every threshold in both settings where we compared them, although the margins are small and fall within statistical uncertainty. More importantly, we observe a consistent performance ordering (S-P < distribution interpolation < T-P) across these settings, which is particularly informative because the three methods differ in how they balance teacher guidance and student adaptation. Thus, while we do not claim a statistically significant advantage over distribution interpolation, the consistent ordering provides empirical support for the importance of proposal direction in student-adapted reasoning distillation.

We estimated 95% percentile confidence intervals for the accuracy difference between each T-P configuration and distribution interpolation in the Qwen3-0.6B/8B setting, using a paired two-way bootstrap over training seeds and questions within benchmarks (10,000 replicates). Both dimensions were resampled with replacement using identical indices across methods. All three intervals include zero.

Comparison95% CI for the accuracy difference (percentage points)
T-P (τ = 1%) − Distribution Interpolation[−1.22, +1.32]
T-P (τ = 0.1%) − Distribution Interpolation[−1.25, +1.72]
T-P (τ = 0.01%) − Distribution Interpolation[−1.12, +1.88]

Ranked by teacher influence, the three methods are student-led generation (S-P) < symmetric distribution mixing < teacher-led generation with selective student intervention (T-P). In all four model/dataset settings, even the lowest-performing T-P configuration exceeds the highest-performing S-P configuration in mean accuracy. In the two Qwen3/s1K settings, including the newly evaluated 1.7B setting, distribution interpolation falls between S-P and T-P at every threshold.

Student/Teacher on DatasetS-P: mean (min–max)Distribution interpolationT-P: mean (min–max)
Qwen3-0.6B/8B on s1K28.72 (28.34–29.10)29.5129.74 (29.58–29.91)
Qwen3-1.7B/8B on s1K46.58 (46.44–46.76)47.3748.20 (47.95–48.54)

This ordering is consistent with the benefit of preserving teacher-led reasoning while adapting it to student support, and it implies that what matters is not only whether student support is incorporated, but how. Both distribution interpolation and T-P improve on Teacher RFT, so student support helps even without gating; among the methods that incorporate it, however, the gain grows with teacher influence. Selectively revising unsupported teacher proposals may therefore be more effective than student-led generation or symmetric distribution mixing, and IPD provides a simple token-level formulation of this principle.

Subsequent on-policy learning (Q5)

IPD initialization yields the highest final accuracy, reward rate, and non-zero advantage group rate under the same GRPO budget.

To directly test whether IPD provides a useful initialization for subsequent on-policy learning, we compared GRPO training from three initializations: Vanilla, Teacher RFT, and IPD T-P (τ = 0.1%). The two SFT initializations used their seed-42 checkpoints. All runs followed the GRPO recipe in Appendix B.1, including the training data, number of optimization steps, and rollout budget.

InitializationAccuracy before GRPO (%)Accuracy after GRPO (%)Accuracy change (percentage points)Reward rate during GRPO (%)Non-zero advantage group rate during GRPO (%)
Vanilla25.0726.07+1.006.9524.12
Teacher RFT27.7728.36+0.597.4024.75
IPD, T-P (τ = 0.1%)31.0131.48+0.478.4026.00

The reward rate is the share of correct rollouts available to reinforce. The non-zero advantage group rate is the share of groups containing both correct and incorrect rollouts, the only groups that yield a non-zero group-relative reward advantage.

The higher reward rate and non-zero advantage group rate are consistent with our motivation that broader initial coverage can provide more opportunities for subsequent on-policy learning. Although Vanilla shows a larger incremental gain from GRPO, IPD retains the highest final accuracy.

AuthorsOpenReview ↗

Initial Response (3/3)

Computational cost (Q2)

IPD’s total wall-clock time is lower than that of GRPO and OPD under the budgets of Table 2. It takes approximately 2.2 times as long as Teacher RFT because both the teacher and student participate in generation.

We report measured wall-clock time on the same GPU configuration, using one NVIDIA B200 GPU per run. All times below are in hours.

MethodRecorded wall-clock time in hours (generation + SFT where applicable)
IPD, T-P (τ = 0.1%)25.56 (24.64 + 0.92)
Teacher RFT11.67 (10.71 + 0.96)
Student RFT-87.16 (6.73 + 0.43)
Student RFT-10061.51 (60.84 + 0.67)
Direct Distillation1.66 (SFT only; public traces reused)
Distribution interpolation20.33 (19.46 + 0.87)
GRPO136.93
OPD31.16

Model scale and task scope (W3, Q4)

Our goal was to test whether adapting teacher supervision can improve reasoning transfer to students smaller than 3.8B. This focus is motivated by successful reasoning distillation with roughly 800–1,000 traces in Qwen2.5-32B-Instruct [4, 5], alongside reported degradation when the 3.8B Phi-4-Mini model is fine-tuned on s1K [6].

The current comparisons show that IPD benefits both 0.6B and 1.7B students under the same 8B teacher, but they do not establish how the gain changes for larger students. We expect the value of adaptation to depend on how well the student supports the teacher’s reasoning. IPD should therefore help most when a student struggles to learn directly from teacher traces, and less when it already learns well from them, as the 32B models above do.

Other verifiable tasks, such as code reasoning with executable tests, offer a natural extension, because our current data-construction pipeline uses a correctness verifier to select full training traces. Open-ended generation and interactive agents would require suitable alternatives for assessing and selecting supervision; we leave these extensions for future work. In the revised version, we have added both limitations to a new Limitations and Extensions section (Section 6).

Incompatible tokenizers (W4, Q3)

The current token-level gate requires compatible tokenizers. Your suggestion of span-level gating provides a concrete direction for extending IPD to model pairs that do not share a tokenizer. In the revised version, Section 6 states this requirement explicitly and outlines the extension below.

One possible design would have the proposer generate short spans ending at boundaries shared by both tokenizers. The gate model would accept a span if the minimum token probability under its own tokenizer meets the threshold, and generate a replacement otherwise. Both models would continue from the resulting text.

This would preserve the principle of adapting teacher reasoning according to student support. Boundary-aware retokenization and the effectiveness of this proposed extension remain to be implemented and evaluated.

References

[4] Muennighoff et al. s1: Simple test-time scaling. 2025.

[5] Ye et al. LIMO: Less is more for reasoning. 2025.

[6] Xu et al. Phi-4-Mini-Reasoning: Exploring the limits of small reasoning language models in math. 2025.

Dear Reviewer k7JW, Thank you again for your review. Your questions led us to add the GRPO follow-up and the wall-clock cost comparison. As the discussion period nears its end, could you let us know whether our initial responses above address your concerns? We would be glad to clarify any remaining points.

Reviewer_3MyXOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Interleaved-Policy Distillation (IPD), a knowledge distillation method designed to improve the reasoning capabilities of small language models. Unlike conventional direct distillation approaches, IPD argues that directly applying supervised fine-tuning (SFT) on complete reasoning traces generated by a teacher model may not be optimal because of the limited capabilities of the student model. In particular, some tokens generated by the teacher may have extremely low probabilities under the student’s distribution, potentially limiting the effectiveness of distillation. To address this issue, IPD incorporates both the teacher and student models into the reasoning trace generation process through two variants: Teacher-Propose (T-P) and Student-Propose (S-P). In T-P, the teacher proposes tokens, and the student replaces those assigned probabilities below a predefined threshold under the student’s distribution. In S-P, the roles are reversed: the student proposes tokens, and the teacher replaces those assigned probabilities below the threshold under the teacher’s distribution. In both cases, generation continues from the modified context. In this way, IPD aims to preserve the reasoning capabilities of the teacher while improving the compatibility of the training data with the student’s own generation distribution. Experimental results demonstrate promising performance improvements across multiple mathematical reasoning benchmarks.

Critical strengths
  1. Simple and intuitive motivation

The core idea of this paper is straightforward and insightful: high-quality reasoning traces generated by a larger teacher model are not necessarily the most suitable training data for a smaller student model. By considering the distributional differences between teacher and student models, the authors propose modifying generated reasoning traces through a small number of token-level interventions. This provides an interesting direction for improving reasoning distillation.

  1. Promising empirical results

IPD does not require introducing a complicated training objective. Instead, it modifies the training data generation process while retaining standard SFT for student training. The experimental results show promising improvements across multiple models and reasoning tasks. In the primary experimental setting, IPD outperforms several baselines, including Direct Distillation, RFT, GRPO, and OPD.

Critical weaknesses

Insufficient ablation studies on the token replacement strategy

The central hypothesis of this paper is that a very small number of teacher-generated tokens that are incompatible with the student model may significantly affect distillation performance. Based on this hypothesis, the authors propose a threshold-based token replacement strategy using the probabilities assigned by the student model.

However, the actual proportion of replaced tokens is extremely small. For example, in the primary experimental setting, IPD T-P ($\tau $= 0.1%) has a fallback rate of only 0.241%.

I believe an important missing ablation is random token replacement, where token positions are randomly selected for replacement while maintaining the same replacement rate as the proposed threshold-based strategy.

Such an experiment would help clarify whether the performance improvements of IPD specifically come from identifying and replacing low-probability tokens, or whether they could also be achieved by introducing a small number of student-generated tokens into teacher-led reasoning trajectories, thereby modifying the resulting training traces.

Questions for authors
  1. Computational Budget Control

In Section 4.3, the authors estimate relative computational budgets using the maximum number of rollouts weighted by model parameter counts. However, this approximation does not fully account for differences in generated token counts, forward and backward computation, and training steps across methods.

Could the authors clarify why stricter computational budget controls, such as measured GPU hours or FLOPs, were not adopted? Are there specific challenges in matching computational budgets across IPD, RFT, GRPO, and OPD?

  1. Teacher–Student Model Scaling

Although the paper evaluates IPD across multiple model sizes, the impact of teacher and student model sizes is not systematically analyzed.

Could the authors elaborate on how teacher model size and the teacher–student capability gap affect IPD performance? In particular, would changing the teacher model size lead to different performance gains, fallback rates, or optimal thresholds?

Additionally, does the effectiveness of IPD change systematically as the student model size increases?

2 comments
AuthorsOpenReview ↗

Initial Response

We appreciate your comments and suggestions for improving the paper. Your suggestion of a random-replacement control helped us make the question of selective intervention more precise, and your comments on computational budgets highlighted the value of direct measurements. We appreciate these concrete suggestions and have added experiments and analyses to address them.

Selective versus random token replacement (W1)

At a nearly identical student-sampling rate, IPD T-P achieves higher accuracy than random replacement despite producing fewer correct full traces. This suggests that the choice of intervention positions provides additional value beyond sparse student participation alone.

To examine whether the selected positions matter, we added a control in which the student supplies tokens at randomly selected positions and the teacher continues from the resulting prefix. This comparison tests whether targeting low-support teacher proposals provides an advantage over introducing student tokens at random positions.

We report the student-sampling rate, defined as the proportion of generated tokens sampled from the student, alongside the number of correct full traces, extreme-tail student surprisal, and downstream accuracy.

Training dataStudent-sampling rate (%)Correct full tracesMean of top 0.01% token surprisals (nats)Accuracy (%)
Teacher RFT057213.1427.65
Random replacement0.09053413.8127.98
IPD, T-P (τ = 0.1%)0.0895167.0929.72

Random replacement raises accuracy over Teacher RFT by only 0.33 points, compared with 2.07 points for IPD.

Computational budget and measured cost (Q1)

Measured wall-clock time confirms that IPD costs less than GRPO and OPD under the budgets of Table 2.

Our budget matching in Table 2 uses maximum rollout counts weighted by model parameter counts. We chose this criterion because rollout generation accounts for most of the computation in the on-policy methods considered here. In our view, this provides a fairer basis for comparison by fixing the generation allowance independently of realized token counts and each method’s forward and backward computation.

To complement this budget-matched comparison, we report measured wall-clock times on the same hardware, using one NVIDIA B200 GPU per run. IPD takes approximately 2.2 times as long as Teacher RFT because both the teacher and student participate in generation. All times below are in hours.

MethodRecorded wall-clock time in hours (generation + SFT where applicable)
IPD, T-P (τ = 0.1%)25.56 (24.64 + 0.92)
Teacher RFT11.67 (10.71 + 0.96)
Student RFT-87.16 (6.73 + 0.43)
Student RFT-10061.51 (60.84 + 0.67)
Direct Distillation1.66 (SFT only; public traces reused)
Distribution interpolation20.33 (19.46 + 0.87)
GRPO136.93
OPD31.16

Teacher–student scale and capability gap (Q2)

Our results suggest that capability gap can help explain gains within a model family, but the value of adaptation across families must be assessed for the specific teacher–student pair rather than predicted from the accuracy gap alone.

Student/Teacher on DatasetCapability gap (teacher’s Vanilla accuracy − student’s Vanilla accuracy) (pp)Parameter gap (B)Vanilla accuracy (%)IPD T-P (τ = 0.1%) accuracy (%)Gain over Vanilla (pp)
Qwen3-0.6B/8B on s1K39.697.5925.0729.72+4.65
Qwen3-1.7B/8B on s1K20.286.4744.4848.12+3.64
EXAONE-1.2B/32B on s1K18.4630.7251.8458.16+6.32

Within the Qwen3 family, the smaller capability gap (39.69 → 20.28 points) is accompanied by a smaller gain over Vanilla (4.65 → 3.64 points). This relationship does not appear to extend across model families: EXAONE has a smaller capability gap (18.46 points) but a larger gain (6.32 points).

More generally, we expect a trade-off in the teacher–student capability gap. If the gap is too small, the teacher may offer limited additional supervision; if it is too large, frequent student fallback may make it difficult to generate reasoning traces that retain useful teacher guidance and remain suitable for student learning. For threshold selection, a smaller gap may favor a lower probability threshold to accept more teacher proposals, whereas a larger gap may call for a higher threshold to filter proposals that the student cannot adequately support.

I thank the authors for addressing my concerns, especially the random-replacement ablation and the computational cost comparison.

The new experiments show that the gains come from selecting which tokens to replace, which further strengthens the paper's claims.

I am happy to increase my score.

Reviewer_RkV2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that off-policy SFT on strong teacher CoTs can fail for small reasoners because a few extreme low-probability tokens break transfer, while pure on-policy RL / OPD is stuck with weak student rollouts. They propose Interleaved-Policy Distillation (IPD). In the main teacher-propose (T-P) setup, the teacher proposes tokens, the student replaces any token it assigns very low probability to, and the teacher continues from the revised prefix. Both committed tokens are used as SFT targets. Student-propose (S-P) flips the roles as a control. Experiments cover several small student / larger teacher pairs on math and science benches, with budget-matched RFT, GRPO, and OPD baselines, plus construction ablations and surprisal-tail diagnostics. They report consistent improvements over vanilla SFT, direct distillation, and student / teacher RFT. Gains vs the closest generative controls look more marginal.

Critical strengths
  • The core idea is simple and easy to understand. Selectively revise unsupported teacher tokens during generation, then train on the realized mix. It is well separated from just filtering bad traces or masking the loss.

  • The T-P vs S-P split is useful. Compatibility alone is not enough; who writes most of the tokens matters. That comparison actually teaches something.

  • Experiments cover multiple student / teacher settings and several benches, which is convincing. Improvements over vanilla, direct distillation, and RFT look consistent. Budget-matched GRPO and OPD also lose in the reported comparisons.

  • Surprisal-tail analysis supports the story that T-P keeps teacher signal while cutting extreme mismatch. S-P can look more "compatible" yet train worse students, which matches the hypothesis.

Critical weaknesses
  • The margin vs nearest generative baselines is thin. Distribution interpolation and SKD nearly catch T-P on the primary setting, and SKD even wins on some individual benches. Filtering and masking clearly fail, but the claim that you must selectively revise during generation is less decisive once those close controls are in the mix. Also, the heavy ablations mostly live in one setting.

  • The downstream claim outruns the evidence. The paper argues IPD should initialize RL / OPD because Pass@k is better, but there is no IPD then GRPO or OPD run. For a paper partly sold as enabling later on-policy learning, that missing experiment matters.

  • Scope is narrow. Students stay small (around 1.7B and below), and most evaluation is math plus one science bench. I am less sure how far this travels.

Questions for authors
  • On the primary setting, is distribution interpolation vs IPD T-P distinguishable across seeds? If not, how should we read the selective-revision story relative to geometric mixing?

  • Have you tried GRPO or OPD starting from a T-P checkpoint under a matched remaining budget? A positive or null result would change how much weight I give the Pass@k motivation.

  • Please run the close generative controls (interpolation, SKD) on at least one more student / teacher setting, not only the primary one.

3 comments
AuthorsOpenReview ↗

Initial Response (1/2)

Thank you for your careful review and constructive comments. Your questions about the closest generative controls and subsequent on-policy learning were particularly helpful in identifying where our interpretation needed clarification and where direct experimental evidence was needed. Following your suggestions, we extended our evaluation to address these points.

Comparison with the closest generative controls (W1, Q1)

T-P ranks above distribution interpolation at every threshold in both settings where we compared them, although the margins are small and fall within statistical uncertainty. More importantly, we observe a consistent performance ordering (S-P < distribution interpolation < T-P) across these settings, which is particularly informative because the three methods differ in how they balance teacher guidance and student adaptation. Thus, while we do not claim a statistically significant advantage over distribution interpolation, the consistent ordering provides empirical support for the importance of proposal direction in student-adapted reasoning distillation.

We estimated 95% percentile confidence intervals for the accuracy difference between each T-P configuration and distribution interpolation in the Qwen3-0.6B/8B setting, using a paired two-way bootstrap over training seeds and questions within benchmarks (10,000 replicates). Both dimensions were resampled with replacement using identical indices across methods. All three intervals include zero.

Comparison95% CI for the accuracy difference (percentage points)
T-P (τ = 1%) − Distribution Interpolation[−1.22, +1.32]
T-P (τ = 0.1%) − Distribution Interpolation[−1.25, +1.72]
T-P (τ = 0.01%) − Distribution Interpolation[−1.12, +1.88]

Ranked by teacher influence, the three methods are student-led generation (S-P) < symmetric distribution mixing < teacher-led generation with selective student intervention (T-P). In all four model/dataset settings, even the lowest-performing T-P configuration exceeds the highest-performing S-P configuration in mean accuracy. In the two Qwen3/s1K settings, including the newly evaluated 1.7B setting, distribution interpolation falls between S-P and T-P at every threshold.

Student/Teacher on DatasetS-P: mean (min–max)Distribution interpolationT-P: mean (min–max)
Qwen3-0.6B/8B on s1K28.72 (28.34–29.10)29.5129.74 (29.58–29.91)
Qwen3-1.7B/8B on s1K46.58 (46.44–46.76)47.3748.20 (47.95–48.54)

This ordering is consistent with the benefit of preserving teacher-led reasoning while adapting it to student support, and it implies that what matters is not only whether student support is incorporated, but how. Both distribution interpolation and T-P improve on Teacher RFT, so student support helps even without gating; among the methods that incorporate it, however, the gain grows with teacher influence. Selectively revising unsupported teacher proposals may therefore be more effective than student-led generation or symmetric distribution mixing, and IPD provides a simple token-level formulation of this principle.

Additional student–teacher setting (W1, Q3)

In the additional Qwen3-1.7B/8B setting, T-P still achieves higher accuracy than both distribution interpolation and SKD.

To examine the comparison beyond the primary setting, we extended the distribution-interpolation and SKD comparisons to Qwen3-1.7B/8B on s1K.

Training dataSeven-benchmark average accuracy (%)
Teacher RFT45.90
Distribution interpolation47.37
SKD47.77
T-P (τ = 1%)47.95
T-P (τ = 0.1%)48.12
T-P (τ = 0.01%)48.54
AuthorsOpenReview ↗

Initial Response (2/2)

Subsequent on-policy learning (W2, Q2)

IPD initialization yields the highest final accuracy, reward rate, and non-zero advantage group rate under the same GRPO budget.

To directly test whether IPD provides a useful initialization for subsequent on-policy learning, we compared GRPO training from three initializations: Vanilla, Teacher RFT, and IPD T-P (τ = 0.1%). The two SFT initializations used their seed-42 checkpoints. All runs followed the GRPO recipe in Appendix B.1, including the training data, number of optimization steps, and rollout budget.

InitializationAccuracy before GRPO (%)Accuracy after GRPO (%)Accuracy change (percentage points)Reward rate during GRPO (%)Non-zero advantage group rate during GRPO (%)
Vanilla25.0726.07+1.006.9524.12
Teacher RFT27.7728.36+0.597.4024.75
IPD, T-P (τ = 0.1%)31.0131.48+0.478.4026.00

The reward rate is the share of correct rollouts available to reinforce. The non-zero advantage group rate is the share of groups containing both correct and incorrect rollouts, the only groups that yield a non-zero group-relative reward advantage.

The higher reward rate and non-zero advantage group rate are consistent with our motivation that broader initial coverage can provide more opportunities for subsequent on-policy learning. Although Vanilla shows a larger incremental gain from GRPO, IPD retains the highest final accuracy.

Model scale and task scope (W3)

Our goal was to test whether adapting teacher supervision can improve reasoning transfer to students smaller than 3.8B. This focus is motivated by successful reasoning distillation with roughly 800–1,000 traces in Qwen2.5-32B-Instruct [1, 2], alongside reported degradation when the 3.8B Phi-4-Mini model is fine-tuned on s1K [3].

Other verifiable tasks, such as code reasoning with executable tests, offer a natural extension, because our current data-construction pipeline uses a correctness verifier to select full training traces. Open-ended generation and interactive agents would require suitable alternatives for assessing and selecting supervision; we leave these extensions for future work. In the revised version, we have added both limitations to a new Limitations and Extensions section (Section 6).

References

[1] Muennighoff et al. s1: Simple test-time scaling. 2025.

[2] Ye et al. LIMO: Less is more for reasoning. 2025.

[3] Xu et al. Phi-4-Mini-Reasoning: Exploring the limits of small reasoning language models in math. 2025.

Thanks for the detailed response, it address most of my concerns.

I maintain my score since it is already positive.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Brief Summary of Reviewer Concerns and Our Responses for the Area Chair

The reviewers raised five main concerns: the small margin over distribution interpolation and SKD (RkV2, k7JW), whether IPD helps subsequent on-policy learning (RkV2, k7JW), whether the selected positions matter (3MyX), whether lower extreme-tail surprisal drives the gains (z76V), and computational cost and scope (all reviewers). We responded with five new experiments or analyses, clarifications of the contribution and surprisal analysis, and a new Limitations and Extensions section (Section 6). As a result, the concerns about on-policy initialization, intervention positions, and cost are addressed with new evidence, the role of surprisal is clarified, the margin over distribution interpolation proves consistent but not significant, and scope limitations are now in Section 6.

Reviewer RkV2

Concerns raised.

  • The margin over distribution interpolation and SKD is thin, and most ablations use one setting.
  • The Pass@k argument for IPD as an RL initialization is not tested.
  • Students are 1.7B or smaller, and evaluation is mostly mathematical.

Responses provided.

  • Distinguishability. Student support helps even without gating, but among the methods that incorporate it, the gain grows with teacher influence. With CIs including zero, we read selective revision as the best way to use student support, not a significant margin.
  • Second setting. T-P (47.95–48.54) exceeds distribution interpolation (47.37) and SKD (47.77) at every threshold on Qwen3-1.7B/8B, with both controls newly run.
  • On-policy learning. Under GRPO, IPD yields the highest final accuracy (31.48% vs 28.36% and 26.07% for Teacher RFT and Vanilla), reward rate, and non-zero advantage group rate, though Vanilla shows the largest incremental gain.
  • Scope. We target students below 3.8B, since fine-tuning on s1K helps a 32B model but degrades a 3.8B one. Code reasoning and other verifiable tasks are natural extensions.

Reviewer 3MyX

Concerns raised.

  • No random-replacement control tests whether positions matter.
  • Rollout-count budget matching ignores token counts and training computation.
  • Teacher and student size effects are not analyzed.

Responses provided.

  • Random replacement. At a nearly identical student-sampling rate, random replacement improves accuracy over Teacher RFT by 0.33 points, versus 2.07 for IPD.
  • Budget. Measured wall-clock times show that IPD costs less than OPD (31.16 h) and GRPO (136.93 h), at 25.56 h on one B200 GPU. We used rollout-based matching because rollout generation dominates on-policy cost.
  • Scale. Capability gap helps explain the gain within a model family but not across families. Within Qwen3, a smaller gap (39.69 → 20.28) comes with a smaller gain (4.65 → 3.64), but EXAONE has a smaller gap (18.46) and a larger gain (6.32).

Reviewer k7JW

Concerns raised.

  • Originality is moderate relative to student-aligned distillation and SKD.
  • The margin over distribution interpolation (29.72 vs 29.51) may not be significant.
  • IPD is untested as an initialization for on-policy learning.
  • Data-generation cost is not reported.
  • Scope is limited to students up to 1.7B, mostly mathematical tasks, and a shared tokenizer.

Responses provided.

  • Originality. IPD constructs supervision by revising teacher trajectories token by token, rather than merely filtering, weighting, or selecting supervision. Unlike SKD, which also interleaves the two models but trains against teacher distributions, IPD trains on the revised trajectory itself.
  • Second contribution. Extreme-tail surprisal is a primary indicator of learnability, but the source of tokens also matters. Where T-P and S-P have comparable surprisal, T-P, with stronger teacher influence, performs better in all four settings (Figure 4).
  • Significance. The ordering S-P < distribution interpolation < T-P holds at every threshold in both settings, although the CIs include zero.
  • On-policy learning. IPD yields the highest final accuracy after GRPO in the requested comparison.
  • Cost. IPD costs less than OPD and GRPO, though about 2.2 times Teacher RFT.
  • Scope and tokenizer. Section 6 states these limits and outlines span-level gating across tokenizers.

Reviewer z76V

Concerns raised.

  • It is unclear whether lower extreme-tail surprisal drives the gains, as similar-surprisal methods differ in accuracy (Figure 4).
  • Evaluation is limited to mathematical reasoning and scientific QA.

Responses provided.

  • Surprisal. We agree that the causal contribution of surprisal reduction is not isolated, as each revision changes both student support and later reasoning. Surprisal remains a primary indicator of learnability, but token source also matters (Figure 4).
  • Scope. Verifiable tasks such as code generation suit our verifier-based pipeline; multi-turn and open-ended tasks would need other ways to select supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.