acceptodds
Under review as a conference paper at ICLR 2027

EvoOPD:Evolving Privileged Information Strategies for Black-Box On-Policy Distillation

Abstract

Proprietary language models exhibit strong reasoning and coding capabilities, but their output distributions are inaccessible, making fine-grained distillation methods that rely on teacher logits difficult to apply directly to black-box models. We observe that rich textual feedback from a black-box teacher can be treated as privileged information (PI) and converted, through a PI-conditioned privileged teacher, into dense supervision signals on the student’s own trajectories. However, what PI should be acquired from the teacher to most effectively support student learning remains unclear. PI strategies that differ in where they focus, what information they request, and how much detail they provide can have substantially different effects on the current student, and their relative effectiveness can further change as training progresses. Based on this observation, we propose Evolving Privileged Information Strategies for Black-Box On-Policy Distillation (EvoOPD), which treats PI acquisition itself as an optimization problem and maintains a PI Strategy Bank that evolves with the student state. EvoOPD uses the measured repair utility of PI on the current student to adapt the focus, information type, and granularity of teacher feedback, allowing the acquired information to continually match the student’s learning needs. We systematically evaluate EvoOPD on mathematical reasoning and code generation with Qwen3-1.7B, 4B, and 8B students. EvoOPD achieves the best overall performance in all six model-task settings, improving over the strongest baseline in each setting by an average of 2.14 percentage points in mathematics and 1.30 points in code. Further analysis shows that the relative effectiveness of PI acquisition strategies changes with student state, making a fixed strategy difficult to sustain as the best choice throughout distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Distillation, On-Policy Distillation, Black-Box Distillation, Privileged Information

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g6EHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EvoOPD presents a system for black-box distillation using a closed-weight teacher model. The system has two key parts. First, a system that generates privileged information (PI) from the teacher that acts as hints for the student. During RL, when the student generates an incorrect answer the teacher is prompted using strategies to provide hints. These hints localize the fault and provide information for producing a correct answer. The strategy bank is evolved throughout the training process using a strategy bank update procedure. The second part is a localization aware reward that applies additional negative advantage to the erroneous student span (identified by the teacher). The system is evaluated on two tasks (math and coding) using 1 family of student models (3 sizes) and 1 teacher model (gemini flash). EvoOPD is shown to be better on average than the other systems tested.

Critical strengths
  1. The paper presents a new end-to-end blackbox distillation approach that appears to work better than the other systems evaluated. The design is clearly explained and ablations demonstrates that each component of the system add improvements (fault localization, strategy evolution). I especially appreciated the result that evaluated different bank snapshots (early, middle, late) and different points in training.

  2. The paper was, on the whole, easy to follow. Figure 2 was more style over substance, and did not help me understand the fault localization component. From the diagram it was not clear that the teacher was required to extract the e (the erroneous step) via prompt. Looking at the appendix resolved this. Otherwise, the paper and its results were easy to follow, and the problem was well motivated.

Critical weaknesses
  1. A core part of this paper is the implementation of the strategy bank and generation of PI. The strategy bank is a form of prompt optimization for the teacher model. Only one teacher model is used, therefore it is unclear whether the techniques generalize to other teachers, or whether they provide additional benefit when even stronger teachers with better intrinsic error location and repair abilities are used. My key concern here is that a central part of this paper is prompt optimization, which itself can be hard to position as a robust research contribution, but additionally it is only evaluated on one model.

  2. The novelty of the system is modest, with memory based distillation tackling similar problems. The limited novelty caused be offset with a comprehensive evaluation that demonstrates that this is robust and generalizable technique, however the evaluation in the paper is limited to 1 teacher, 1 family of students, and two tasks (math and coding). Showing that the system scales to stronger teachers, domains, or more complex tasks (like agentic tool calling) would strengthen the work.

Questions for authors
  • What is teacher performance on the benchmarks?

  • OPSD uses frozen teacher. Which teacher is this? How do we know that the improvement of EvoOPD, or weakness of OPSD is due to the strengths of the relative teachers (not the techniques).

  • What is the teacher for ROPD?

  • Table 4. You bold EvoOPD as if it performs 'best', in a similar notation as the benchmark results. But these are token numbers and it isn't clear whether higher or lower is better. (And you bold cases where EvoOPD is both most and least). Please improve the clarity of this table. I think tokens / per accuracy points gained would be a more direct and informative presentation, is that correct?

  • The Fixed Strategy Bank token total is unavailable and remains a dash rather than zero.

    Why?

  • The gains on coding are slim. How do the gains change between training runs? How significant is variance between trained checkpoints?

4 comments

We thank the reviewer for the careful review and specific suggestions. In the revised manuscript, we have added analyses of the PI supervision mechanism, extended our evaluations across teachers and student model families, and clarified the baseline configurations and computational costs. We address each concern below.

W1 & W2: Novelty and Generalization

We would like to clarify that our main contribution is not simply prompt optimization, but a distinct approach to black-box on-policy distillation. Specifically, we investigate how textual privileged information (PI) from a black-box teacher can be converted into localized token-level supervision on the student's own trajectories, without access to teacher logits, rather than using teacher-generated text only for SeqKD or converting it into scalar rewards.

Our second contribution is the continual evolution of PI acquisition strategies. Rather than merely optimizing prompts, we adapt the type and granularity of acquired PI based on the current student's actual error-recovery performance. Figure 4 in the original submission already showed that the relative effectiveness of different PI strategies changes with the student's training state.

We also recognize that the original experimental analysis placed greater emphasis on PI strategy evolution, without sufficiently highlighting our first contribution. Therefore, we have added token-level mechanism analyses and PI-on/PI-off controlled experiments in Section 3.3 (Figure 3) to investigate which tokens are affected by PI supervision and how these changes influence unassisted learning. The results show that PI generally has a stronger effect on Planning and Calculation tokens than on Style tokens, and that Original PI induces larger probability shifts than Shuffled PI. Moreover, PI-on improves unassisted continuation correctness over PI-off by 1.15, 1.61, and 2.98 percentage points across the three training stages, providing further evidence that localized PI supervision translates into actual learning gains.

Regarding our second contribution, we further examine the relationship between PI acquisition strategies and student states in Section 3.4 (Figure 4). Students at different stages achieve the highest error-recovery utility and subsequent unassisted training gains with their corresponding stage-specific PI Strategy Banks. These measures are also significantly positively correlated (Pearson r = 0.814). With the same number of strategy evaluation rounds, EvoOPD outperforms Offline-9 by 1.16 and 1.69 percentage points on mathematics and code generation, respectively (Table 3), supporting the effectiveness of continual PI strategy evolution.

To address concerns about generalization, we conducted additional cross-teacher and cross-family experiments. We use GPT-6.1-Sol as an alternative black-box teacher and introduce Gemma 4 E4B IT and Ministral 3 8B Instruct as students from two additional model families. The results across six mathematical reasoning benchmarks are shown below.

Student / TeacherMethodAIME24AIME25AIME26HMMT25-FebHMMT25-NovHMMT26-FebOverall
Qwen3-1.7B / GPT-6.1-SolROPD31.0423.7521.6716.0413.3317.8020.56
EvoOPD35.0027.0825.0018.5416.2521.0223.77
Gemma 4 E4B IT / GeminiROPD61.2551.6756.4638.5445.0039.2048.53
EvoOPD64.1753.1358.3340.6347.0841.8650.72
Ministral 3 8B / GeminiROPD44.3830.4236.2522.5026.2526.5230.98
EvoOPD46.4631.2537.2924.5829.1729.1732.92

Across these three additional settings, EvoOPD outperforms ROPD by 3.21, 2.19, and 1.94 percentage points, respectively, with improvements in all 18 model–benchmark comparisons. These results indicate that EvoOPD's advantages are not limited to the original teacher or the Qwen model family. Full results have been added to Section 3.5 (Table 5).

Q1: Teacher Performance Across Benchmarks

We have added the independent problem-solving performance of both black-box teachers on all six mathematical reasoning benchmarks to provide a clearer picture of their capabilities. The results are shown below.

TeacherAIME24AIME25AIME26HMMT25-FebHMMT25-NovHMMT26-FebOverall
Gemini-3.6-Flash95.0094.5895.4298.7492.5090.1594.33
GPT-6.1-Sol100.0099.5899.5899.1799.1797.7399.18

The results show that Gemini-3.6-Flash, used in our original experiments, already demonstrates strong mathematical reasoning capabilities, achieving 94.33% Overall accuracy. The alternative teacher, GPT-6.1-Sol, achieves an even higher Overall accuracy of 99.18%. These results have been added to Appendix B.1 (Table 7).

Q2 & Q3: Teacher Configurations of Baselines and OPSD Performance

We clarify that SeqKD, OPSD, ROPD, and EvoOPD all use the same external black-box teacher, Gemini-3.6-Flash, in the original experiments, controlling for differences in external teacher capability. Specifically, SeqKD performs SFT on verified Gemini-generated responses; ROPD uses evaluative feedback from Gemini for reinforcement learning; OPSD obtains PI from Gemini and, following its official implementation, uses a frozen copy of the student as the probability teacher for self-distillation; and EvoOPD converts Gemini's textual PI into localized token-level advantage corrections.

Regarding OPSD's performance, prior studies have shown that privileged-information-based self-distillation can be sensitive to PI construction and training configurations (Zhu et al., 2026a; Yang et al., 2026a; Li et al., 2026d). In our non-thinking mathematics setting, OPSD does not outperform Base in Overall accuracy at any of the three student scales. In one training run, we also observed that early validation gains gradually disappeared as training progressed.

In contrast, EvoOPD does not directly match the full teacher probability distribution conditioned on PI. Instead, it uses changes in the student's own token probabilities to construct bounded advantage corrections within localized windows of incorrect trajectories. This difference in supervision construction may partly explain the different outcomes observed in our experiments.

We have further discussed OPSD's performance in Section 3.2 and clarified the baseline teacher configurations and training details in Appendix B.1.

Q4 & Q5: Cost Reporting in Table 4

We have incorporated the reviewer's suggestions regarding the presentation of Table 4. In the revised manuscript, we have adjusted the bold formatting of token usage and added the Tokens per Accuracy Point Gained metric, measured relative to No PI, to better illustrate the trade-off between performance gains and teacher interaction costs.

Regarding the missing token cost for Fixed Strategy Bank, we have supplemented the complete statistics. This configuration uses 150.16M teacher tokens, corresponding to 34.05M tokens/pp. In comparison, EvoOPD uses 187.05M teacher tokens and achieves 21.98M tokens/pp, demonstrating higher teacher-token efficiency.

These results have been updated in Section 3.4 (Table 4), with the complete breakdown of input, output, and thinking tokens provided in Appendix B.4 (Table 9).

Q6: Stability of Code Generation Performance

To address concerns about the relatively modest gains on code generation and their stability, we conducted three independent training runs of EvoOPD with different random seeds on Qwen3-1.7B. We also analyzed performance variation across checkpoints near the best-performing model.

First, regarding stability across training runs, we independently trained EvoOPD with three random seeds and evaluated each run on five code generation benchmarks.

Table R1: Multi-Run Stability on Code Generation (Pass@1, %).

Method / RunHumanEval+MBPP+BCB-HardLCB v5LCB v6Overall
GRPO66.8851.328.9915.3418.1136.27
EvoOPD (Seed 42)68.1050.9811.1917.5520.2137.37
EvoOPD (Seed 7)70.0851.1911.4418.8622.0438.32
EvoOPD (Seed 2026)65.6654.1213.1319.4622.3939.09
EvoOPD (Mean ± Std)67.95 ± 2.2152.10 ± 1.7611.92 ± 1.0618.62 ± 0.9821.55 ± 1.1738.26 ± 0.86

EvoOPD achieves Overall accuracies of 37.37%, 38.32%, and 39.09% across three independent runs, all exceeding the original GRPO baseline of 36.27%, with a mean performance of 38.26 ± 0.86%. Moreover, the average performance of EvoOPD across the three runs exceeds GRPO on all five code benchmarks, indicating that the improvements are not concentrated on a single dataset.

Second, regarding performance variation across checkpoints, we examined the validation performance of three checkpoints near the best-performing model for both GRPO and EvoOPD.

Table R2: Checkpoint Stability Around the Best Model.

MethodBest Val (%) ↑Nearby Mean ± Std (%) ↑Nearby Range (pp) ↓Selected Test Overall (%) ↑
GRPO28.7328.12 ± 0.561.0936.27
EvoOPD29.4528.85 ± 0.551.0937.37

EvoOPD achieves a higher mean validation accuracy across nearby checkpoints (28.85% vs. 28.12%), while the standard deviations are 0.55 and 0.56 percentage points, respectively, and both methods have the same range of 1.09 percentage points. These results show that EvoOPD achieves higher validation accuracy while exhibiting checkpoint-to-checkpoint variation comparable to GRPO.

Furthermore, Table 2 evaluates three student scales across five code generation benchmarks. EvoOPD outperforms GRPO by 1.10, 1.46, and 1.35 percentage points on Qwen3-1.7B, 4B, and 8B, respectively, with an average improvement of 1.30 percentage points. EvoOPD also achieves better results in 12 out of 15 model–benchmark comparisons.

Together, these additional results provide further evidence of the consistency of EvoOPD's code generation gains across independent training runs, nearby checkpoints, and different model scales and benchmarks.

We thank the reviewer again for the constructive feedback and hope that these additional experiments and clarifications have addressed the main concerns. We would be happy to clarify any remaining questions.

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our paper, especially given your busy schedule.

As the discussion deadline approaches, we would appreciate knowing whether our responses and additional experiments have adequately addressed your concerns. We would be happy to clarify any remaining questions or engage in further discussion.

Thank you again for your time and valuable feedback.

Reviewer_teAKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoOPD, a black-box on-policy distillation framework that leverages textual privileged information (PI) from a teacher without requiring access to its logits. The method converts PI-induced changes in student token probabilities into localized negative-advantage corrections. It further introduces an evolving PI Strategy Bank that adapts the type and granularity of teacher feedback to the student's learning state. Experiments on mathematical reasoning and code generation demonstrate consistent improvements over the evaluated baselines.

Critical strengths
  • Optimizing what feedback to request from a black-box teacher, rather than relying on fixed feedback strategies, is an interesting direction.
  • EvoOPD outperforms the evaluated baselines across three student sizes on both mathematics and code generation. The ablation studies also provide evidence for the benefits of PI Strategy Bank optimization and continued adaptation.
Critical weaknesses

[Main weaknesses]

  • Questionable reliability of PI-induced supervision: Prior studies [ref1-ref4] have shown that privileged-context supervision can alter response style, suppress uncertainty, or compress reasoning trajectories rather than reliably improve reasoning. Although EvoOPD restricts its PI-induced corrections to localized errors in incorrect responses, it does not analyze whether the affected tokens correspond to meaningful reasoning errors or stylistic changes. Further token-level and response-length analyses would help establish the reliability of the proposed supervision.
  • Concerns about training rollout length: While final evaluation allows 32,768 tokens, training rollouts and checkpoint selection are limited to 8,192 tokens. Given prior findings on reasoning compression in self-distillation [ref1-ref4], it remains unclear how this training length constraint affects the learned reasoning behavior. Reporting training response-length distributions and truncation rates would help clarify this issue.
  • Weak SeqKD results: SeqKD (or teacher-generated SFT), one of the simplest and most widely used alternatives for black-box distillation, shows almost no improvement over the base models across all six settings. Given its effectiveness in prior work, this limited improvement warrants further investigation. The paper would benefit from clarifying the SeqKD training configuration, including target lengths, truncation, and training epochs.

[Additional weaknesses]

  • Limited novelty relative to adaptive PI methods: Prior work such as AVSD [ref5] has already explored the importance of adaptively utilizing different privileged-information views in on-policy distillation. Although EvoOPD differs by optimizing PI acquisition strategies as the student evolves, its conceptual and empirical advantages over existing adaptive PI approaches could be better established.
  • Computational cost and limited generalization: EvoOPD requires substantial additional black-box teacher interactions for PI acquisition and strategy evolution. The experiments also cover only one teacher and one student-model family, leaving the cost-effectiveness and generalizability of the approach insufficiently explored.

 

References

[ref1] Kim, Jeonghye, et al. "Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?." COLM 2026.

[ref2] Kaur, Simran, et al. "Rethinking On-Policy Self-Distillation for Thinking Models." arXiv preprint arXiv:2607.05184 (2026).

[ref3] Pan, Leyi, et al. "RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation." arXiv preprint arXiv:2606.11709 (2026).

[ref4] Kim, Jaehoon, and Dongha Lee. "OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Models." arXiv preprint arXiv:2605.06188 (2026).

[ref5] Nguyen, Duy, et al. "AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals." arXiv preprint arXiv:2605.20643 (2026).

Questions for authors
  • How can the authors verify that PI-induced supervision really improves reasoning rather than changing response style or length?

  • Why does SeqKD show almost no improvement over Base?

  • What are the key advantages of EvoOPD over existing adaptive PI methods such as AVSD?

4 comments

We thank the reviewer for the thoughtful and detailed feedback. In the revised manuscript, we have added mechanism analyses, clarified baseline configurations, and expanded our evaluations of computational cost and generalization. We address each concern below.

W1 & Q1: Reliability of PI Supervision and Its Effect on Reasoning

We agree with the reviewer that PI-induced changes in token probabilities do not necessarily indicate improved reasoning ability, as they may also reflect changes in response style or generation behavior. To further examine the effectiveness of PI supervision, we added two sets of mechanism experiments in Section 3.3 (Figure 3) of the revised manuscript, investigating which tokens are affected by PI and whether these changes translate into actual learning gains.

First, we analyze PI-induced token probability changes (T1). The results show that PI generally has a stronger effect on Planning and Calculation tokens than on Style tokens. Moreover, when keeping student trajectories fixed and matching feedback type and length as closely as possible, Original PI produces larger probability shifts than Shuffled PI. For example, at S8, the Negative Shift for Planning tokens is 0.338 with Original PI, compared with 0.050 with Shuffled PI. This suggests that the effect of PI depends not only on the presence of additional text, but also on how well the feedback matches the student's current erroneous trajectory.

Second, we examine whether these changes lead to actual learning gains (T2). After conducting PI-on/PI-off training from identical initial states, we find that the additional PI supervision leads to different parameter updates, as reflected in token probability changes measured without PI. These differences are more pronounced for Planning and Calculation tokens, while Style tokens show relatively small differences. Furthermore, on independent evaluation prefixes without PI, PI-on improves continuation accuracy over PI-off by 1.17, 1.61, and 2.98 percentage points at the three training stages, respectively.

Together, these results show that PI not only induces content-dependent token probability changes, but also affects the student's learning process and improves reasoning performance without PI.

W2: Training Rollout Length and Reasoning Compression

We agree that generation length limits may affect the reasoning trajectories a model can explore. We therefore provide further clarification of this setting and its effect on training behavior.

In our training runs, student responses tend to be relatively short at the beginning and gradually become longer as training progresses. To examine this, we further analyzed the response-length distributions and truncation rates of Qwen3-1.7B at different training stages:

StageMean LengthMedianP90Truncation Rate
Early1,422.161,1532,870.60%
Middle2,473.531,9794,914.30.59%
Late3,655.672,9067,430.77.03%

The results show that the average response length increases from 1,422 to 3,656 tokens, while P90 increases from 2,871 to 7,431 tokens. We therefore do not observe an overall trend of reasoning-length compression during training. The truncation rates are 0% and 0.59% at the Early and Middle stages, respectively, increasing to 7.03% at the Late stage. This indicates that the generation length limit mainly affects a subset of longer trajectories in later training.

Our training length configuration is also comparable to protocols used in prior on-policy distillation work. For example, ROPD likewise uses an 8,192-token limit for training rollouts and a 32,768-token limit for final evaluation.

Our primary focus is whether textual PI from a black-box teacher can provide additional localized learning signals. Under the current training protocol, EvoOPD achieves performance improvements on both mathematical reasoning and code generation. We provide these additional length statistics here to clarify the effect of the training length setting on our results.

W3 & Q2: Understanding the Limited Improvement of SeqKD

We agree with the reviewer that SeqKD, as a widely used approach to black-box knowledge distillation, deserves a clearer explanation of its training configuration and performance.

In our experiments, SeqKD uses Gemini-generated responses that pass correctness verification as supervision targets and performs supervised fine-tuning on the same training question pool as the other methods. Specifically, we use a learning rate of 1e-6, a global batch size of 128, at most one training epoch, and a maximum training sequence length of 10,240 tokens. We validate every 10 steps and select the checkpoint with the highest validation accuracy. We also tested a learning rate of 1e-5, which performed worse than 1e-6.

We further analyzed the lengths of teacher-generated trajectories. Their average length is approximately 614 tokens, with a maximum of 4,255 tokens, both below the training sequence length limit. This suggests that excessively long teacher responses may not be the main reason for SeqKD's limited improvement.

One possible explanation is the reasoning capability gap between a strong teacher and a weaker student. Even when the teacher produces correct reasoning trajectories, the student may struggle to directly learn the complex reasoning processes they contain. Previous research has shown that when the capability gap between teacher and student is large, smaller models do not necessarily benefit more from stronger teachers' reasoning trajectories (Li et al., 2025). This phenomenon, termed the Small Model Learnability Gap, offers a possible explanation for the limited gains we observe with SeqKD.

We have provided the SeqKD training configuration in Appendix B.1 (Table 6) of the revised manuscript and further clarified its limited improvement in our response above.

W4 & Q3: Differences from Adaptive PI Methods such as AVSD

We agree with the reviewer that adaptive utilization of different forms of PI has already been explored in works such as AVSD. However, EvoOPD differs from AVSD in its distillation setting, supervision construction, and the object of adaptive optimization.

First, the two methods study different distillation settings. AVSD investigates on-policy self-distillation, constructing supervision from probability distributions produced by the same model under different privileged-information views. In contrast, EvoOPD studies knowledge distillation from an independent black-box teacher to a student model. The external teacher provides only textual information, without access to its logits or token probability distributions. Our focus is therefore on how to convert textual PI from a black-box teacher into localized token-level supervision on the student's own trajectories.

Second, the methods adapt different components. AVSD primarily investigates how to combine supervision signals from multiple PI views, improving their reliability through the selective integration of cross-view consensus and residual signals. EvoOPD instead treats the PI acquisition strategy itself as an optimization target. Based on the current student's error-repair outcomes, it continually adjusts what information to request from the black-box teacher, where the feedback should focus, and how detailed it should be, allowing the acquired PI to adapt to the student's evolving learning needs.

Thus, although both methods aim to make better use of PI, they address different questions: AVSD focuses on integrating existing PI views to construct reliable supervision, whereas EvoOPD focuses on acquiring useful textual PI from a black-box teacher and converting it into localized supervision for the student's on-policy learning.

W5: Computational Cost and Generalization

Computational cost. We agree that EvoOPD requires additional black-box teacher interactions for PI acquisition and strategy optimization. It is therefore important to evaluate the performance gains relative to these costs.

To this end, we added the teacher-token usage of ROPD in the revised manuscript and further compared the performance and cost of different strategies. The results for Qwen3-1.7B mathematical reasoning are summarized below:

MethodTeacher Tokens (M) ↓Overall (%) ↑
No PI0.0016.46
Offline-9184.2123.81
ROPD599.4622.64
EvoOPD187.0524.97

EvoOPD uses only approximately 31.2% of the teacher tokens required by ROPD, while achieving higher Overall accuracy (24.97% vs. 22.64%). Moreover, EvoOPD and Offline-9 use the same number of strategy evaluation rounds and have similar teacher-token costs (187.05M vs. 184.21M), yet EvoOPD further improves accuracy by 1.16 percentage points. This suggests that the benefits of continual strategy evolution are not simply due to more teacher calls.

The cost analysis and the complete breakdown of input, output, and thinking tokens are provided in Section 3.4 (Table 4) and Appendix B.4 (Table 9) of the revised manuscript.

Generalization. We further evaluated EvoOPD with different black-box teachers and student model families on the original six mathematical reasoning benchmarks.

Specifically, we used GPT-6.1-Sol as an alternative teacher for Qwen3-1.7B and introduced two additional student model families, Gemma 4 E4B IT and Ministral 3 8B Instruct, both using Gemini-3.6-Flash as the teacher.

StudentTeacherROPDEvoOPD
Qwen3-1.7BGPT-6.1-Sol20.5623.77
Gemma 4 E4B ITGemini-3.6-Flash48.5350.72
Ministral 3 8BGemini-3.6-Flash30.9832.92

In these three additional settings, EvoOPD outperforms ROPD by 3.21, 2.19, and 1.94 percentage points in Overall accuracy, respectively, and achieves higher accuracy on all six mathematical reasoning benchmarks in each setting.

These results indicate that EvoOPD's performance gains are not limited to the original teacher or the Qwen student family. The complete experimental results are included in Section 3.5 (Table 5) of the revised manuscript.

We thank the reviewer again for the constructive feedback and hope that these additional experiments and clarifications have addressed the main concerns. We would be happy to clarify any remaining questions.

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our paper, especially given your busy schedule.

As the discussion deadline approaches, we would appreciate knowing whether our responses and additional experiments have adequately addressed your concerns. We would be happy to clarify any remaining questions or engage in further discussion.

Thank you again for your time and valuable feedback.

Reviewer_qC1gOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

In standard on-policy distillation, training requires the teacher's logits, which black-box access does not provide; the paper studies distillation from a teacher for which the trainer only has such access to in the self-distillation setting. The paper hypothesizes that textual feedback from the teacher may be insufficient for the student to learn effectively. Hence, they propose EvoOPD, where the central idea is to combine GRPO with a set of instructions that evolve over training, which steer the teacher's feedback (e.g., what it should focus on, etc.). The set is updated (content is added) only if a feedback by the teacher improves the student's performance.

Experimental setup uses Qwen3-1.7B, 4B, and 8B as students and Gemini-3.6-Flash as the (black-box) teacher and DAPO-Math-17K and OpenThoughts-114k code subset as training data; evaluates on AIME 24~26, HMMT, HumanEval+, MBPP+, BigCodeBench Hard, and LiveCodeBench v5/v6. Shows that on average, EvoOPD outperforms other baselines.

Critical strengths
  • The experiments are run with multiple students, albeit all are in the Qwen family. The teacher used is also a strong teacher, which reflects a realistic setup where one may care about such forms of distillation.
Critical weaknesses
  • (Major) The paper is largely proposing a new method to do RL. But since we are prompting the teacher during training anyways, why not simply use Gemini-Flash as the verifier/reward signal---i.e., have it provide a score of 0 or 1? If this does comparably to EvoOPD, then the method's contribution is little.
  • The most interesting finding is that the best feedback strategy depends on the student and not simply that more detailed or better-optimized feedback is universally better. But this idea is well-established in the distillation literature: a strong model does not mean a strong teacher.
Questions for authors
  • Please run the baseline mentioned in Weaknesses (W1). Does this perform worse than EvoOPD (even worse sample efficiency is acceptable)?
  • Can you try running EvoOPD with different student families, and different teachers?
4 comments

We thank the reviewer for the careful review and constructive feedback. In the revised manuscript, we have further clarified our main contributions and added analyses of the PI supervision mechanism (Section 3.3, Figure 3), a direct teacher-reward baseline (Appendix C.4, Table 12), and cross-teacher and cross-family evaluations (Section 3.5, Table 5). We address the concerns below.

W1 & W2: Main Contributions and Novelty

We recognize that the original manuscript placed considerable emphasis on the self-evolution of PI acquisition strategies, without sufficiently highlighting our main contribution to black-box distillation. Our work makes two contributions at different levels, and we have clarified this distinction in the revision.

First, we investigate a different approach to black-box on-policy distillation. Without access to teacher logits, existing black-box approaches typically rely on teacher-generated responses for sequence-level distillation (SeqKD) or convert teacher evaluations into reinforcement learning rewards. In contrast, we study how textual feedback from a black-box teacher can be converted into localized token-level supervision on the student's own trajectories. Specifically, EvoOPD uses PI-induced changes in student token probabilities to selectively strengthen negative advantages within localized error regions, while retaining the original binary verifier rewards (Section 2.2, Eqs. 2–4).

To examine this mechanism, we added token-level analyses and PI-on/PI-off comparisons in Section 3.3 (Figure 3). The results show that Original PI produces substantially larger content-related probability shifts than Shuffled PI. For example, at S8, the Negative Shift for Planning tokens is 0.338 with Original PI versus 0.050 with Shuffled PI. Furthermore, PI supervision not only changes parameter updates measured without PI, but also improves unassisted continuation accuracy by 1.17, 1.61, and 2.98 percentage points at the three training stages, respectively. These findings provide empirical evidence that textual PI can supply additional fine-grained learning signals.

Second, we study how to dynamically optimize PI acquisition strategies according to the student state. We agree with the reviewer that the observation that a stronger model is not necessarily a better teacher is not new. However, our focus is a different question: even with the teacher model held fixed, the same student may benefit from different PI acquisition strategies at different stages of training.

As shown in Figure 1(b) and Figure 4, the relative effectiveness of different strategies changes with the student state. Furthermore, continual strategy evolution outperforms optimizing the strategy only before training on both mathematics and code tasks (Table 3). These results suggest that PI acquisition should not only be optimized, but also adapted to the student's evolving learning needs.

W1 & Q1: Direct Teacher-Scored Reward Baseline

Our original experiments already included ROPD, a strong baseline that uses teacher-generated evaluation rubrics to guide policy optimization. Nevertheless, we agree that directly using binary correctness rewards from a black-box teacher is an important additional comparison.

We therefore added Gemini-as-Judge, in which Gemini directly scores student responses as correct (1) or incorrect (0), and these scores are used as reinforcement learning rewards.

Additional results: Qwen3-1.7B mathematical reasoning (%).

MethodAIME24AIME25AIME26HMMT25-FebHMMT25-NovHMMT26-FebOverall
Base12.7110.217.505.633.756.827.75
GRPO24.7918.9617.0811.888.7513.4515.78
Gemini-as-Judge21.6715.2114.1712.297.7114.2014.21
ROPD35.6324.7922.0816.2516.2521.0222.64
EvoOPD38.7527.9228.1316.6715.8322.7324.97

Gemini-as-Judge achieves an Overall accuracy of 14.21%, slightly below verifier-based GRPO (15.78%) and substantially below EvoOPD (24.97%).

To better understand this result, we examined Gemini's binary judgments and found 94.65% agreement with the original verifier's 0/1 labels. This indicates that the teacher's direct binary evaluations largely overlap with the existing correctness verification signal and may therefore provide limited additional training information in this setting.

In contrast, EvoOPD goes beyond whether a response is correct or incorrect. It further uses textual PI from the teacher to identify local error regions and adjust the learning signals of the corresponding tokens.

These results support our motivation: rather than using a black-box teacher solely as a reward model, converting its textual feedback into finer-grained supervision can provide additional value for student learning.

Q2: Generalization Across Teachers and Student Model Families

We thank the reviewer for this suggestion. To further evaluate the generalization of EvoOPD, we conducted additional experiments with different black-box teachers and student model families.

Specifically, we introduced GPT-6.1-Sol as an alternative black-box teacher for Qwen3-1.7B to evaluate whether our approach transfers across teachers. We also evaluated Gemma 4 E4B IT and Ministral 3 8B Instruct as additional student model families, retaining Gemini-3.6-Flash as the teacher.

All experiments use the same six mathematical reasoning benchmarks as the original evaluation, with comparisons against Base, GRPO, and ROPD.

Table 1: Cross-teacher generalization (Qwen3-1.7B; Teacher: GPT-6.1-Sol).

MethodAIME24AIME25AIME26HMMT25-FebHMMT25-NovHMMT26-FebOverall
Base12.7110.217.505.633.756.827.75
GRPO24.7918.9617.0811.888.7513.4515.78
ROPD31.0423.7521.6716.0413.3317.8020.56
EvoOPD35.0027.0825.0018.5416.2521.0223.77

Table 2: Cross-family generalization (Gemma 4 E4B IT; Teacher: Gemini-3.6-Flash).

MethodAIME24AIME25AIME26HMMT25-FebHMMT25-NovHMMT26-FebOverall
Base47.7134.1743.5419.5827.5029.3633.57
GRPO59.7943.5454.1731.4638.3337.1243.95
ROPD61.2551.6756.4638.5445.0039.2048.53
EvoOPD64.1753.1358.3340.6347.0841.8650.72

Table 3: Cross-family generalization (Ministral 3 8B Instruct; Teacher: Gemini-3.6-Flash).

MethodAIME24AIME25AIME26HMMT25-FebHMMT25-NovHMMT26-FebOverall
Base33.5425.0027.9215.6318.7518.9423.22
GRPO40.6332.5034.3819.5824.1723.3029.00
ROPD44.3830.4236.2522.5026.2526.5230.98
EvoOPD46.4631.2537.2924.5829.1729.1732.92

These results show that EvoOPD's performance gains are not limited to the original Qwen student family or Gemini teacher. EvoOPD improves over ROPD by 3.21, 2.19, and 1.94 percentage points in the three additional settings, respectively, suggesting that the approach generalizes across different teachers and student architectures.

We have included these results in Section 3.5 (Table 5) of the revised manuscript to provide a more comprehensive evaluation of the method's generalization.

We thank the reviewer again for the constructive feedback and hope that the additional experiments and clarifications have addressed the main concerns. We would be happy to clarify any remaining questions.

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our paper, especially given your busy schedule.

As the discussion deadline approaches, we would appreciate knowing whether our responses and additional experiments have adequately addressed your concerns. We would be happy to clarify any remaining questions or engage in further discussion.

Thank you again for your time and valuable feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experimental Evidence

We thank all three reviewers for their careful evaluations and constructive feedback. The main concerns focused on novelty, the effectiveness of PI supervision, generalization, baseline configurations, computational costs, and training stability. We have conducted additional experiments and revised the manuscript accordingly.

Core Contributions

We would like to clarify the two main contributions of EvoOPD.

1. Black-box token-level distillation. EvoOPD explores how textual privileged information (PI) from a black-box teacher can be converted into token-level supervision on student-generated trajectories, without requiring teacher logits. Unlike SeqKD or methods that convert teacher feedback into scalar rewards, EvoOPD constructs supervision from PI-induced changes in the student's token probabilities (Section 2.2).

2. Continually evolving PI acquisition strategies. EvoOPD adapts the type and granularity of acquired PI based on the student's error-recovery performance, allowing PI acquisition to evolve with the student's training state rather than merely optimizing fixed prompts (Section 2.3).

Key Additional Experiments and Revisions

1. PI Supervision Mechanism (Reviewers teAK, qC1g)

We added token-level analyses and PI-on/PI-off controlled experiments. PI has stronger effects on Planning and Calculation tokens and improves unassisted continuation correctness by 1.15, 1.61, and 2.98 percentage points across three training stages. Additionally, Gemini-as-Judge achieves 14.21% Overall accuracy, compared with 24.97% for EvoOPD, further demonstrating the value of textual PI supervision.

Locations: Section 3.3 (Figure 3); Appendices C.3–C.4 (Table 12).

2. Effectiveness of Continual PI Strategy Evolution (Reviewers teAK, g6EH)

Students at different training stages benefit most from their corresponding PI Strategy Banks. PI recovery utility is positively correlated with subsequent unassisted learning gains (r = 0.814). With the same number of strategy evaluation rounds, EvoOPD outperforms Offline-9 by 1.16 and 1.69 percentage points on mathematics and code generation, respectively.

Locations: Section 3.4 (Figure 4, Table 3).

3. Cross-Teacher and Cross-Family Generalization (All Three Reviewers)

We extended our evaluation to an alternative teacher, GPT-6.1-Sol, and two additional student families, Gemma 4 E4B IT and Ministral 3 8B Instruct. Across these three new settings, EvoOPD outperforms ROPD by 3.21, 2.19, and 1.94 percentage points, respectively, with improvements in all 18 model–benchmark comparisons.

Locations: Section 3.5 (Table 5); Appendix B.1 (Table 7, teacher performance).

4. Baseline Configurations and Computational Costs (Reviewers teAK, g6EH)

We clarified the teacher configurations and training details of SeqKD, OPSD, and ROPD, further discussed their observed performance, and supplemented training rollout length and truncation statistics.

For computational costs, EvoOPD uses 187.05M teacher tokens, compared with 599.46M for ROPD, and achieves 21.98M tokens per accuracy point gained. We also completed the missing Fixed Strategy Bank cost statistics.

Locations: Sections 3.2 and 3.4 (Table 4); Appendices B.1 and B.4 (Tables 6 and 9).

5. Code Generation Stability (Reviewer g6EH)

We conducted three independent training runs with different random seeds. EvoOPD achieves 38.26 ± 0.86% Overall accuracy, with all three runs outperforming the original GRPO result of 36.27%. Analysis of nearby checkpoints also shows higher validation accuracy for EvoOPD, with performance variation comparable to GRPO.

Locations: Section 3.2 (Table 2, original code results); response to Reviewer g6EH (Tables R1–R2, additional stability analyses).

Conclusion

These additional experiments further support EvoOPD's two core contributions: constructing token-level supervision from black-box textual PI and continually evolving PI acquisition strategies to adapt to student learning. The new mechanism analyses, generalization experiments, and stability evaluations also strengthen the empirical evidence.

We thank all three reviewers again for their constructive feedback and hope these revisions and additional experiments have addressed the main concerns. We would be happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.