acceptodds
Withdrawn from ICLR 2027

MetaOPSD: Adapting Privileged Teachers with Post-Update Student Feedback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Access to reference solutions can make a language model a better predictor without making it a better teacher. On-policy self-distillation (OPSD) trains a problem-only student to match a reference-conditioned teacher on student-generated prefixes, but distributional agreement does not directly assess the learning gains induced by that supervision. We introduce MetaOPSD, which adapts the privileged teacher using verified student outcomes after a pilot distillation step. The pilot student solves separate feedback problems without privileged information, and a policy-gradient surrogate propagates outcome feedback through the pilot update to train the teacher. The adapted teacher then supervises an OPSD update from the original student state, retaining dense distillation as the student's training objective. MetaOPSD outperforms all evaluated baselines on each of three mathematical reasoning benchmarks at both model sizes after 200 student updates, with macro avg@12 gains of 2.16–2.96 percentage points over the OPSD baseline. On Qwen3-1.7B, checkpoint cross-evaluation further shows that the final teacher induces larger learning gains than the initial teacher at all evaluated student states.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
language model reasoning, on-policy self-distillation, teacher adaptation, meta-learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gKsvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Traditional on-policy self-distillation relies on distributional matching that does not directly assess a student's actual learning gains. To address this, MetaOPSD adapts the privileged teacher by backpropagating verified student outcomes from a pilot distillation step. Consequently, MetaOPSD consistently outperforms evaluated baselines on mathematical reasoning benchmarks and demonstrates superior teaching utility across student states.

Critical strengths
  1. The paper distinguishes predictive power from teaching utility by framing teacher adaptation around post-update student performance, which better aligns with the true goal of OPSD than static distribution or likelihood matching.

  2. It provides comprehensive specifications—from teacher adapters and differentiation through AdamW/clipping to RLOO surrogates and seed schedules—which are essential for reproducible optimizer unrolling.

Critical weaknesses
  1. Fixed-step comparisons ignore compute fairness: While MetaOPSD leads at 200 updates, Table 3 reveals its RLOO variant consumes ~7.2× the GPU-hours of OPSD and ~5.9× that of VISTA. Under a shared 24 GPU-hour budget, MetaOPSD-RLOO (38.04%) is actually outperformed by both OPSD (38.24%) and VISTA (39.66%), suggesting gains stem from heavy compute consumption rather than algorithmic efficiency.

  2. Teaching utility evidence is narrow and marginal: The cross-evaluation is limited to Qwen3-1.7B on just 128 MATH-500 questions over 20 updates, yielding an average gain of only ~0.48 pp over the initial teacher. Such tiny margins on a small probe set are insufficient to prove the teacher has acquired a meaningfully superior teaching strategy.

Questions for authors
  1. Can you provide accuracy-vs-GPU-hour Pareto curves for OPSD, VISTA, Reference-NLL, and MetaOPSD-RLOO, rather than only fixed 200-update endpoints? My recommendation would change if MetaOPSD is competitive on this curve.

  2. Under the same 24 GPU-hour budget, why does MetaOPSD-RLOO underperform VISTA so clearly? Is the bottleneck feedback generation, verifier cost, second-order graph construction, or teacher update frequency?

  3. Does the conclusion hold beyond Qwen3 and three short math benchmarks? Results on a larger held-out math set, non-math verifiable tasks, or at least more than 90 final benchmark questions would materially strengthen the paper.

Reviewer_tCb4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

While standard on-policy self-distillation can cause teachers to rely on reference-dependent shortcuts, MetaOPSD introduces a trainable teacher optimized via policy-gradient feedback evaluated on separate pilot problems. By updating a low-rank adapter based on student performance on independent tasks, the framework outperforms (by a very small margin) baseline methods on challenging math benchmarks. Additional analyses show that the teacher improves progressively through training and that alternative feedback signals remain effective, despite requiring a higher computational overhead.

Critical strengths
  1. The problem setting makes sense and is significant.
  2. The method appears novel and interesting.
Critical weaknesses
  1. The evaluation is limited to a single domain: mathematics.
  2. It is unclear why an arbitrary average@12 metric is reported instead of more standard metrics, which may suggest potential cherry-picking.
  3. The writing is generally very difficult to follow, even for someone with a background and prior experience in OPSD.
  4. OPSD is known to perform poorly at small model scales (see https://arxiv.org/pdf/2601.19897 and https://arxiv.org/pdf/2610.13113); the authors need to test whether this method remains beneficial on larger models where OPSD typically performs well.
  5. The method outperforms the strongest baseline by only one to two percentage points, utilizing a very small aggregate evaluation set and lacking multi-seed experiments, which raises concerns that the results may be due to random noise.
  6. The evaluation lacks insight into where each method saturates over additional compute. Given that OPSD is a lightweight method, longer training runs should be demonstrated.
Questions for authors
  1. The abstract is difficult to parse and understand, even for readers with prior knowledge and background in OPSD. The authors are encouraged to make it more accessible to first-time readers.
  2. Terminology such as "macro gains" and "macro average" makes the text difficult to parse. Phrasing of this nature often resembles how claude code speaks.
2 comments

We thank the reviewer for recognizing the importance of the research problem and the novelty of our proposed method. In response to your concerns, we provide the following clarifications and conduct additional experiments to further validate the effectiveness of our method.

1. Experimental Reliability and Random Seeds

First, we would like to clarify that our main experimental results are not based on a single training run, but are averaged over three independent runs with different random seeds (42, 43, and 44), as detailed in Table A.1 and Appendix C.3. MetaOPSD consistently outperforms VISTA across all three benchmarks at both model scales, achieving average improvements of 1.98 and 1.36 percentage points, respectively. To further address concerns regarding randomness and experimental variability, we additionally report the results for each individual seed, along with the corresponding standard deviations.

ModelMethodSeed 42Seed 43Seed 44Mean ± Std
Qwen3-1.7BOPSD39.2038.6139.6039.14 ± 0.50
Qwen3-1.7BVISTA40.8339.3840.1540.12 ± 0.73
Qwen3-1.7BMetaOPSD42.3142.8741.1142.10 ± 0.90
Qwen3-4BOPSD63.0964.1462.5063.24 ± 0.83
Qwen3-4BVISTA64.4163.6764.0464.04 ± 0.37
Qwen3-4BMetaOPSD65.4964.2966.4265.40 ± 1.07

2. Regarding the avg@12 Evaluation Metric

We adopt 12 samples to maintain consistency with the evaluation protocols used in related work. Both OPSD (https://arxiv.org/abs/2601.18734) and VISTA (https://arxiv.org/abs/2608.28306) report mathematical reasoning performance using avg@12, which measures the average proportion of correct responses across 12 independent samples for each problem. All methods in our paper are evaluated using identical sampling and scoring settings.

“Macro Avg.” refers solely to the arithmetic mean of the accuracies on the three benchmarks: AIME24, AIME25, and HMMT25. We will replace this term with “Average Accuracy” and revise “macro gains” to “improvements in average accuracy” to avoid ambiguity.

3. Regarding Test Set Size and Result Stability

We acknowledge that a test set of 90 problems may not be sufficient to fully characterize the generalization performance of our method. To address this concern, we conduct additional evaluations on Omni-MATH using the existing training checkpoints, without further training.

The supplementary evaluation results are summarized below (%):

Model and evaluation settingOPSDVISTAMetaOPSD
1.7B,Omni-MATH27.4228.2029.55
4B,Omni-MATH49.8650.5751.74

4. Regarding Model Scale and Computational Cost

We agree that validation on larger models is important. MetaOPSD currently demonstrates consistent performance improvements at both the 1.7B and 4B scales. Furthermore, cross-evaluation of teacher checkpoints provides additional evidence that teacher adaptation improves the effectiveness of the supervision provided to the student. Nevertheless, these results do not establish that the same benefits will extend to larger models or other task domains.

Regarding training cost, the primary objective of this work is to investigate how feedback from student learning can improve teacher supervision, rather than to minimize training compute. Table 3 explicitly reports the actual training costs of different methods and provides comparisons under a fixed computational budget. Our current results support the conclusion that MetaOPSD achieves higher accuracy with the same number of student update steps, but do not imply superior computational efficiency.

5. Regarding Paper Clarity and Presentation

We thank the reviewer for the suggestions on improving the readability of our paper. We will simplify the abstract and method description and refine the overall presentation to highlight the core idea more clearly: Rather than merely training the student to imitate the teacher, MetaOPSD optimizes the teacher based on the student's actual learning outcomes after receiving supervision, enabling the teacher to provide supervisory signals that better support the student's ability to solve problems independently.

  1. Experimental Reliability and Random Seeds

Are the p-vals significant? and what are the sizes of all the evaluations being done?

  1. Regarding the avg@12 Evaluation Metric

When we deploy a model to the real-world, we are also interested in pass@1 and other lower pass@k metrics. These should not be omitted, especially as not all user queries will have a verifier.

  1. Regarding Test Set Size and Result Stability

What is the sample size of this? What are the p-values of multi-seed runs? P-vals would not be required if the gains are consistent and large in magnitude but these are very tight gains.

  1. Regarding Model Scale and Computational Cost

This response does not alleviate my two concerns on larger model scale and larger compute budget. Notably, regarding the second point, we need to see the learning curve saturating to be sure that this is a relevant method in the real-world.

Thank you authors.

Reviewer_anv8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MetaOPSD, which adapts a reference-conditioned teacher using verified student outcomes after a pilot distillation step. A policy-gradient surrogate propagates this feedback through the pilot update to train the teacher, while the actual student retains dense on-policy distillation. Across three mathematical reasoning benchmarks and two model sizes, MetaOPSD consistently outperforms the evaluated baselines after 200 student updates, with macro accuracy gains of 2.16–2.96 percentage points over OPSD.

Critical strengths
  1. The method addresses a meaningful mismatch between privileged prediction and useful supervision. The contribution builds on the pilot-update framework of MetaDistil (Zhou et al., 2022), extending it to privileged autoregressive reasoning with verified outcome feedback. The derivation in Appendix B connects the RLOO meta-gradient to expected post-update correctness under explicit assumptions, while the actual student retains dense distillation training. I find this a coherent and useful methodological contribution.

  2. The empirical results consistently support improved learning at a fixed number of student updates. MetaOPSD outperforms the evaluated baselines on all three benchmarks at both model sizes. The checkpoint cross-evaluation is particularly useful: branches start from matched student parameters and optimizer states, and the final teacher improves teaching utility over the initial teacher at all five student checkpoints. This provides evidence that adaptation improves the supervision supplied by the teacher, beyond simply observing a better final student.

Critical weaknesses
  1. The accuracy gains have a substantial computational cost, and a favorable quality–compute tradeoff has not been demonstrated. On Qwen3-1.7B, 200 MetaOPSD updates require approximately 200 GPU-hours, compared with 27.7 for OPSD and 33.8 for VISTA—roughly 7.2× and 5.9× more computation. The paper appropriately reports a compute-matched comparison, but that comparison favors the baselines: at 24 GPU-hours, MetaOPSD achieves 38.04 macro accuracy versus 39.66 for VISTA. Its time to reach the OPSD target is also substantially longer. These results limit the practical significance of the fixed-update improvements.

  2. The experiments do not sufficiently isolate the benefit of routing outcome feedback through the teacher. The reference-NLL ablation is informative, but it already recovers 2.10 of the 2.96-point improvement over frozen OPSD. Outcome feedback adds 0.86 points while increasing training time from 7.095 to 24.994 hours. This supports an incremental benefit from outcomes within the proposed framework, but leaves open whether the expensive bilevel route is the best way to use those outcomes.

Questions for authors

What teacher parameterizations, learning rates, update frequencies, and feedback budgets were used for VISTA and DualOPSD, and how were their adaptation hyperparameters selected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.