acceptodds
Under review as a conference paper at ICLR 2027

MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student on its own generated responses using token-level teacher supervision. However, uniform weighting overlooks differences in token learning value, while existing weighting methods rely on predefined mappings from prediction signals to token weights. These mappings are not learned from the effectiveness of the resulting student updates, limiting their ability to adapt to evolving learning needs. In this paper, we propose MetaOPD, a bilevel optimization framework that jointly learns the student model and a lightweight token-weighting network. The inner objective updates the student through weighted OPD, while the outer objective optimizes the weighting network using validation loss on reference solutions after a virtual student update. Differentiating through this update connects weighting decisions to their effects on post-update performance, allowing the mapping from prediction signals to token weights to evolve alongside the student. Experiments on six mathematical reasoning and three out-of-domain datasets, covering two student scales and seven baselines, demonstrate the effectiveness of MetaOPD, with Avg@8/Pass@8 gains over OPD of 1.99/5.97 percentage points for the 0.6B student and 2.25/6.41 points for the 1.7B student.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation (OPD), Static proxy, A bilevel optimization framework

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uKudOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a token-weighting framework for on-policy distillation (OPD). While existing OPD methods typically assign equal weights to all tokens or rely on predefined rules, the proposed framework learns token weights through bi-level optimization. Specifically, a lightweight MLP predicts the token weights, which are used to perform a virtual update of the student model. The weighting network is then optimized based on the validation loss of the updated student.

Critical strengths
  1. The paper addresses an important limitation of existing OPD methods by exploring how to learn more effective token-level supervision.

  2. The proposed bi-level optimization method offers a novel approach to learning token weights based on student updates.

  3. The proposed method consistently improves over standard OPD and achieves competitive results across serveral benchmarks.

Critical weaknesses
  1. The paper does not sufficiently justify the additional optimization complexity. In particular, it remains unclear whether the virtual student update is necessary. For example, why not directly train the weighting MLP using the weighted OPD objective without virtual updates?

  2. The presentation is unnecessarily complicated relative to the core idea, which is quite simple. For example, Section 3.2, Algorithm 1, and Figure 2 largely repeat the same optimization procedure. Why not keep either Algorithm 1 or Figure 2? Moreover, Eqn. (21)-(24) provide detailed derivations of standard gradient clipping and AdamW operations. Although relevant to the meta-gradient computation, these details are not the main novelty and could be presented more concisely.

  3. The notation is sometimes inconsistent, making the paper difficult to follow. For example, the weighted inner loss is denoted as LinnerL_{inner} in Figure 2 but LinL_{in} in Eq. (5). Similarly, Figure 2 uses wtw_t, wt+w_t^{+}, and ℓt\ell_t, whereas the main text uses wη,i,tw_{\eta,i,t}, wη+,i,tw_{\eta^{+},i,t}, and ℓi,tOPD\ell_{i,t}^{OPD}.

  4. The figures are also not clear. Figure 1(a) does not clearly highlight the main difference between MetaOPD and existing approaches. Also, what is the purpose of the large green region in Figure 1(b)? Figure 2 is visually overloaded with icons that provide limited technical information. For example, what do the two different icons under "Optimization" represent, and what is the purpose of the icon for "top-32 distribution statistics"?

Overall, I find the problem meaningful, the approach reasonably novel, and the results promising. However, the paper is often difficult to follow due to unclear explanations and an insufficiently polished presentation. My current assessment is between Borderline and Weak Accept. If my concerns are not addressed well, I may change my rating to Weak Reject.

Questions for authors

Does the paper consider directly optimizing the token-weighting MLP using the weighted teacher-supervised OPD loss?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uKud (Part 1/2): Why Virtual Updates?

Part 1/2

Thank you for your careful review and constructive feedback. We hope our clarifications and additional experiments address your concerns, and we welcome any further questions.

W1 & Q1: Why use virtual updates instead of directly optimizing the weighting MLP with the OPD loss?

MetaOPD uses meta-learning to learn token weights from their effect on the student’s subsequent learning. Directly optimizing the weighting MLP with the current weighted OPD loss does not explicitly assess this effect. Let θ\theta and η\eta denote the student and weighting-MLP parameters, respectively. The two objectives are: Jdirect(η)=1NB∑i,tmi,twη,i,tℓi,tOPD(θ)+λRw(η),\mathcal J_{\mathrm{direct}}(\eta) = \frac{1}{N_B}\sum_{i,t}m_{i,t}w_{\eta,i,t}\ell_{i,t}^{\mathrm{OPD}}(\theta)+\lambda\mathcal R_w(\eta),

Jmeta(η)=Lout(θ′(η))+λRw(η),\mathcal J_{\mathrm{meta}}(\eta) = \mathcal L_{\mathrm{out}}\bigl(\theta'(\eta)\bigr)+\lambda\mathcal R_w(\eta),

where mi,tm_{i,t} masks valid tokens, NB=∑i,tmi,tN_B=\sum_{i,t}m_{i,t} counts valid tokens in the minibatch, θ′(η)\theta'(\eta) is the student after the virtual weighted update, and Lout\mathcal L_{\mathrm{out}} is the reference-solution NLL. Both objectives use the same weight regularizer.

With the current student fixed, the direct objective can reduce its loss term by shifting weight toward tokens with smaller current surrogate-loss values, even under response-level weight normalization. However, smaller current loss values do not necessarily indicate more beneficial student updates. MetaOPD instead evaluates the weights through the reference-solution loss after a virtual student update. Differentiating through this update allows the MLP to learn which token weights produce updates that improve reference-solution performance. The virtual update therefore provides the connection between supervision allocation and its learning effect; this is a local, one-step signal rather than a guarantee of long-term improvement.

To test this distinction empirically, we evaluated Direct-MLP, retaining the same architecture, input features, weight constraints, and regularizer, while replacing the meta objective with the current weighted OPD objective. Both variants use Qwen3-8B as the teacher and Qwen3-1.7B-Base as the student.

Method/Avg@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
Direct-MLP68.6541.5626.931.0410.006.6725.81
MetaOPD69.9543.7527.712.0812.088.3327.32
Method/Pass@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
Direct-MLP87.0065.0044.855.0030.0026.6743.09
MetaOPD88.2077.5048.5310.0033.3323.3346.82

MetaOPD achieves six-benchmark averages of 27.32% Avg@8 and 46.82% Pass@8, compared with 25.81% and 43.09% for Direct-MLP, improving by 1.51 and 3.73 percentage points, respectively. These results support learning token weights from the student’s post-update performance rather than directly minimizing the current weighted OPD loss.

AuthorsOpenReview ↗

Response to Reviewer uKud (Part 2/2): Presentation, Notation, and Figures

Part 2/2

W2: Repetition across the method description, pseudocode, and flowchart.

Thank you for pointing out the overlap in the presentation. Section 3.2, Algorithm 1, and Figure 2 are intended to serve complementary roles: Section 3.2 defines the objectives and derives the meta-gradient; Algorithm 1 specifies the execution order and handling of virtual and actual optimizer states; and Figure 2 illustrates the information flow. We will streamline the repeated procedural descriptions and make these roles clearer, keeping the mathematical details in Section 3.2, implementation steps in Algorithm 1, and a concise visual overview in Figure 2.

We will also condense Equations (21)–(24) in Appendix A.3. These equations detail standard gradient clipping and AdamW operations to make the meta-gradient derivation self-contained. The revision will retain only the dependencies needed to follow differentiation through the virtual update, with standard optimizer details summarized more briefly.

W3: The notation is sometimes inconsistent, making the paper difficult to follow.

We acknowledge that the notation differences between Figure 2 and the main text make the method harder to follow. In Figure 2, Linner\mathcal L_{\mathrm{inner}} denotes the same weighted OPD loss as Lin\mathcal L_{\mathrm{in}} in Eq. (5). Using different names for the same loss was an oversight, and we will use Lin\mathcal L_{\mathrm{in}} consistently throughout the paper.

For the token weights and per-token OPD loss, Figure 2 uses the shortened symbols wtw_t, wt+w_t^{+}, and ℓt\ell_t. These symbols retain the token-position index tt to highlight token-level weighting and keep the diagram readable. The main text uses the full notation to make the response identity and weighting-network parameter dependence explicit in the mathematical derivation. The correspondence is:

w_t^{+} \equiv w_{\eta^{+},i,t},\qquad \ell_t \equiv \ell_{i,t}^{\mathrm{OPD}}.$$ Here, $i$ indexes responses, and $\eta$ and $\eta^{+}$ denote the weighting-network parameters before and after the meta-update, respectively. Thus, $w_t$ and $w_t^{+}$ refer to the token weights computed before and after that update. We did not explain this correspondence in the original figure caption. We will add the mapping and definitions so that readers can directly relate the figure to the mathematical description. > **W4: Unclear presentation and visual clutter in the figures.** Figure 1(a) is intended to highlight how feedback is used to determine token weights. The proxy-based pipeline assigns weights through predefined rules, updates the student, and then evaluates its performance. MetaOPD uses the reference-solution loss after a virtual student update as feedback to learn the weighting rule. We will make this feedback path more explicit in the figure and caption. The green region in Figure 1(b) corresponds to the “→ better” label and marks positive loss reduction relative to Uniform OPD. We will clarify this meaning in the caption. In Figure 2, the lock icons indicate frozen parameters. The two icons under “Optimization” distinguish the weighting MLP's meta-gradient update, obtained by differentiating through the virtual student update, from the student's actual weighted-OPD update. The icon beside “top-32 distribution statistics” was intended as a visual aid and does not denote an additional computational operation. We recognize that the number of icons makes the figure harder to read. We will remove nonessential decorative icons and add clear labels for the two updates, retaining visual elements that convey a specific technical meaning.

The rebuttal addresses my concerns, and I am happy to maintain my original rating. If accepted, please incorporate the updates into the camera-ready version.

AuthorsOpenReview ↗

Thank you for your constructive feedback!

We are glad that our rebuttal addressed your concerns. Thank you for your constructive feedback, which helped us clarify the presentation, notation, and figures. If the paper is accepted, we will incorporate the revisions discussed in our rebuttal into the camera-ready version. Thanks again!

Reviewer_WMvTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MetaOPD learns token weights for on-policy distillation using a one-step bilevel objective. A small network maps detached student/teacher prediction statistics to response-normalized weights. A virtual student update uses the current AdamW state and global gradient clipping; reference-solution loss after that update trains the weighting network. The actual student update then uses recomputed weights. Experiments with 0.6B and 1.7B Qwen3 students report improved mathematical reasoning, selected transfer results, direct-supervision controls, and local update diagnostics.

Critical strengths
  1. The method makes update-dependent supervision allocation concrete for OPD, rather than relying only on fixed uncertainty or disagreement rules. Its treatment of response normalization, detached descriptors, clipping, and the stateful AdamW virtual update is unusually explicit. The local derivative interpretation is appropriately separated from a long-term improvement guarantee.

  2. The reported aggregate gains over uniform OPD occur at both student scales. Importantly, the paper includes SFT, OPD+SFT with a calibrated coefficient sweep, and MetaOPD+SFT controls, rather than attributing gains to meta-learning without considering the extra reference-solution supervision. The one-step comparison also uses separate reference sets for weight learning and evaluation.

Critical weaknesses
  1. The reliability and practical size of the advantage remain uncertain. Main training and the primary ablations use a single training seed. At 1.7B, the advantage over the strongest reported allocation baseline, TIP, is 1.13/1.56 points in macro-average Avg@8/Pass@8, while MetaOPD costs 1.97 times as much as uniform OPD (Table 10). The diagnostic bootstrap intervals condition on fixed checkpoints or rollouts and do not establish robustness of the main benchmark gains. An accuracy–cost comparison would clarify whether the extra optimization is worthwhile; this is distinct from claiming that the paper promises lower runtime.

  2. The baseline comparison supports a narrower conclusion than superiority to the original complete methods. Appendix B.2.1 implements FiRe-OPD, REOPOLD, and TIP within the shared sampled-OPD surrogate; TIP's original full-vocabulary reverse-KL training loss is replaced. This is useful for isolating allocation rules, but leaves open whether the gains persist against the original objectives and their tuned implementations. The out-of-domain advantage is also uneven, with essentially unchanged AlpacaEval LC-WR relative to OPD and lower scores than TIP.

Overall assessment: 3 — Weak acceptance, close to the boundary. The concrete OPD adaptation, detailed derivation, consistent gains over uniform OPD, and direct-supervision controls provide useful evidence for learning token weights from update outcomes. My support is tempered by single-seed main results, substantial overhead, and the scope of the adapted-baseline comparison.

Questions for authors
  1. Can you quantify uncertainty in the main MetaOPD–TIP and MetaOPD–OPD differences, distinguishing evaluation sampling uncertainty from variation across complete training runs? Are any additional training-seed results available?

  2. What is the attainable accuracy–cost trade-off against OPD/TIP at MetaOPD's approximately doubled budget? Please report any existing budget checkpoints or learning curves without treating equal update counts as equal compute.

  3. How sensitive is the ranking to using the original full-vocabulary training objective for TIP? Please clarify which claims concern controlled allocation-rule variants and which concern the original algorithms.

3 comments
AuthorsOpenReview ↗

Response to Reviewer WMvT (Part 1/3): Training and Evaluation Uncertainty

Part 1/3

Thank you for your careful review and constructive feedback. We hope our clarifications and additional experiments address your concerns, and we welcome any further questions.

W1-1 & Q1: How robust are the benchmark gains across training runs, and how do training and evaluation uncertainty differ?

To complement the original bootstrap analysis, we conducted additional independent training runs to directly assess the stability of MetaOPD’s gains across training seeds. Following the three-training-seed setup used in DecisionRank, we supplemented the original seed-42 experiments with complete independent training runs using seeds 43 and 44 for OPD, TIP, and MetaOPD, keeping all other experimental settings unchanged. Using Qwen3-8B as the teacher and Qwen3-1.7B-Base as the student, we report six-benchmark averages and their mean ± sample standard deviation (SD) across the three training seeds.

Method/Avg@8 (%)Seed 42Seed 43Seed 44Mean ± SD
OPD25.0725.7826.0125.62 ± 0.49
TIP26.1925.7626.1326.03 ± 0.23
MetaOPD27.3227.0127.7327.35 ± 0.36
Method/Pass@8 (%)Seed 42Seed 43Seed 44Mean ± SD
OPD40.4142.1942.2341.61 ± 1.04
TIP45.2642.6244.6344.17 ± 1.38
MetaOPD46.8245.8347.6746.77 ± 0.92

MetaOPD outperforms both baselines on both six-benchmark averages for each of the three training seeds. To quantify the variability of the improvements directly, we compute the MetaOPD–baseline difference within each matched seed. The resulting mean gains ± sample SD are 1.73 ± 0.51 / 5.16 ± 1.41 percentage points over OPD and 1.33 ± 0.24 / 2.60 ± 0.91 points over TIP, for Avg@8 / Pass@8, respectively.

These results complement the original bootstrap analysis by addressing a different source of uncertainty. The statistics above capture variability across complete training-and-evaluation runs, including evaluation sampling noise. In contrast, the original diagnostic bootstrap intervals condition on fixed checkpoints or rollouts and quantify uncertainty within those diagnostic settings; they do not measure variation across independently trained models.

Together, the additional runs provide evidence that MetaOPD’s aggregate gains persist across independent training seeds. We will incorporate these results and clarify the distinction between training-run variability and conditional evaluation uncertainty in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer WMvT (Part 2/3): Accuracy-Cost Trade-off

Part 2/3

W1-2 & Q2: What accuracy–cost trade-off is achievable at matched training-time budgets?

To directly assess the accuracy–cost trade-off, we supplemented the original equal-update comparison with evaluations under two matched training-time budgets: the time required by OPD to complete three epochs and the time required by MetaOPD to complete three epochs. Both comparisons use Qwen3-8B as the teacher and Qwen3-1.7B-Base as the student. The tables report per-benchmark results and unweighted averages across the six benchmarks.

Training-time budget: time required by OPD to complete three epochs.

Method/Avg@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
OPD67.2338.7527.801.678.336.6725.07
TIP68.6541.2527.481.678.336.6725.67
MetaOPD68.8540.6327.532.0810.007.0826.03
Method/Pass@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
OPD84.2067.5047.436.6720.0016.6740.41
TIP86.8072.5045.5910.0020.0016.6741.93
MetaOPD87.0070.0048.538.3320.0023.3342.87

At this budget, MetaOPD achieves the highest six-benchmark averages on both metrics, exceeding OPD by 0.96 percentage points in Avg@8 and 2.46 points in Pass@8, and TIP by 0.36 and 0.94 points, respectively. Thus, MetaOPD retains an aggregate advantage even when constrained to OPD’s original training-time budget.

Training-time budget: time required by MetaOPD to complete three epochs.

Method/Avg@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
OPD68.7836.8828.582.0810.005.8325.36
TIP70.3840.3128.262.089.587.9226.42
MetaOPD69.9543.7527.712.0812.088.3327.32
Method/Pass@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
OPD86.4067.5047.065.0023.3316.6740.99
TIP87.8062.5046.6910.0023.3316.6741.17
MetaOPD88.2077.5048.5310.0033.3323.3346.82

At this budget, OPD and TIP receive extended training. MetaOPD still achieves the highest six-benchmark averages, exceeding OPD by 1.96 percentage points in Avg@8 and 5.83 points in Pass@8, and TIP by 0.90 and 5.65 points, respectively.

These comparisons show that MetaOPD’s aggregate gains persist at both matched training-time budgets, including against baselines trained for longer than in the original comparison. They provide direct evidence of a favorable accuracy–training-time trade-off at the two evaluated budgets, beyond the original equal-update results.

AuthorsOpenReview ↗

Response to Reviewer WMvT (Part 3/3): Original TIP and OOD Gains

Part 3/3

W2-1 & Q3: Does restoring TIP's original objective change the ranking, and what do the baseline comparisons establish?

As detailed in Appendix B.2.1, our experiments follow EOPD’s setup and compare token-level supervision allocation rules using a common sampled-OPD objective. FiRe-OPD and REOPOLD already use sampled-token policy optimization objectives, so we retained their original training mechanisms, including filtering, weighting, and reward processing, while adapting the datasets and teacher–student configuration. For TIP, we kept its original token selection rule and used the shared sampled-OPD loss in place of the full-vocabulary reverse-KL loss at selected positions. This design controls the training objective when comparing allocation rules.

The reviewer raises a valid concern that replacing TIP’s objective may change gradient variance and its interaction with token selection. We therefore conducted an additional comparison that restores TIP’s original full-vocabulary reverse-KL training loss at each selected position, using the same data setup, with Qwen3-8B as the teacher and Qwen3-1.7B-Base as the student. As shown below, MetaOPD retains the highest six-benchmark averages on both Avg@8 and Pass@8, exceeding TIP with its original objective by 0.74 and 1.04 percentage points, respectively.

Method/Avg@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
TIP (sampled-token)68.7541.2527.531.6711.256.6726.19
TIP (full-vocabulary)70.7340.6327.713.339.177.9226.58
MetaOPD69.9543.7527.712.0812.088.3327.32
Method/Pass@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
TIP (sampled-token)86.8072.5045.5910.0030.0026.6745.26
TIP (full-vocabulary)89.0077.5044.8510.0030.0023.3345.78
MetaOPD88.2077.5048.5310.0033.3323.3346.82

These results complement the controlled allocation-rule comparison: MetaOPD’s aggregate advantage over TIP persists when TIP’s original training objective is restored. We will clarify the scope of these two comparisons in the revised manuscript.

W2-2: How should the uneven OOD gains be interpreted?

MetaOPD’s primary contribution is learning how to allocate token-level supervision in on-policy distillation. The OOD experiments provide a complementary assessment of its effects beyond mathematical reasoning. The original results show task-dependent gains: MetaOPD improves over OPD on GPQA-Diamond and MMLU-Pro, while its AlpacaEval LC-WR remains nearly unchanged (15.98% vs. 15.96%) and below TIP (16.70%). We therefore interpret these results as evidence of selective OOD benefits rather than uniformly stronger performance across tasks.

Following the reviewer’s feedback, we broadened this assessment with additional evaluations on LogiQA2, IFEval, WinoGrande, and BoolQ, covering natural-language logical reasoning, instruction following, commonsense reasoning, and yes/no reading comprehension, respectively. We evaluated the Qwen3-1.7B-Base students trained with Qwen3-8B as the teacher.

OOD BenchmarkMetricOPD (%)TIP (%)MetaOPD (%)
LogiQA2Accuracy26.7826.9727.74
IFEvalPrompt-level strict31.9831.7932.72
IFEvalInstruction-level strict45.9245.5646.76
WinoGrandeAccuracy63.9364.0164.33
BoolQAccuracy80.6179.6680.92

MetaOPD achieves the highest scores among the three compared methods on all five additional metrics, exceeding OPD by 0.31–0.96 percentage points and TIP by 0.32–1.26 points. These results broaden the evidence for modest OOD benefits across additional tasks, while the original AlpacaEval results highlight that these benefits remain task-dependent. We will revise the discussion to reflect this distinction.

Reviewer_8MSJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This Paper addresses Standard on-policy distillation (OPD) weights all generated tokens equally or relies on heuristic rules (like entropy or disagreement), failing to ensure that uncertain tokens actually deserve more supervision.

It proposes a novel approach called MetaOPD, which Introduces a small weighting network (~35.7K parameters) trained via meta-gradients. It simulates a virtual student update, measures its loss on reference solutions, and learns the optimal token-weighting policy based on actual student performance effects.

Outperforms standard OPD across mathematical benchmarks for both 0.6B and 1.7B student models (using a Qwen3-8B teacher), yielding significant improvements in both average response accuracy (Avg@8) and pass rates (Pass@8).

Critical strengths

This paper give a well-motivated distinction between prediction signals and learning value:It evaluates supervision allocation based on actual student improvement (via post-update loss) rather than relying on proxy signals like entropy or disagreement. And it shows that the methods directionally consistent improvements over standard OPD

The experiment was quite rigorous. They include:

  • SFT, OPD + SFT, and MetaOPD + SFT comparisons;
  • experiments with different input feature groups;
  • a one-step experiment in Appendix C that learns weights using reference set A and evaluates them using reference set B;
  • training-cost measurements;
  • bootstrap confidence intervals for some diagnostic results.
Critical weaknesses
  1. This Paper implement the allocation rules of FiRe-OPD, REOPOLD, and TIP within a shared sampled-OPD loss framework. But It makes some baselines not evaluated in their original full form,for example:TIP's original full-vocabulary reverse-KL training loss is replaced with the shared sampled surrogate. changing the underlying objective can alter gradient variance and the compatibility between an allocation rule and its training loss.

  2. Main results rely on a single training seed (42), while the multiple seeds in the appendix vary response sampling rather than representing independent full training runs. The absence of robustness testing across independent training seeds leaves the stability and generalizability of the reported improvements unverified. Furthermore I think the method’s generalization is unclear, and the reported out-of-domain gains are Little. The current experiments are constrained to a single model family (Qwen3) and small student models. the performance improvements on out-of-domain benchmarks (GPQA, MMLU-Pro, AlpacaEval) are minimal, failing to provide strong proof of broad applicability.

  3. Nearly twice the training cost, without an equal-compute comparison A small weighting network does not make the overall procedure inexpensive. The virtual update, outer loss, and meta-gradient account for substantial additional work. The reported comparison matches the number of student updates, not the total computational budget。 As you show in Table 10:

  • OPD uses 45.0 GPU-hours;
  • MetaOPD uses 88.5 GPU-hours;
  • the cost ratio is approximately 1.97×.
Questions for authors
  1. Reliability of the main results across independent training seeds.
  2. Comparisons with original strong baselines and under equal computational budgets.
  3. Sufficiently tuned controls with matched access to supervisory information.
  4. Experiments, such as freezing the weighting network, that establish the necessity of online meta-adaptation.
  5. More Out-of-Domain test sets / More difficult In-Domain test sets

If you answer these questions, I will consider raising my score.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 8MSJ (Part 1/5): TIP and Training Seeds

Part 1/5

Thank you for your careful review and constructive feedback. We hope our clarifications and additional experiments address your concerns, and we welcome any further questions.

W1 & Q2-1: Comparison with TIP’s original training objective.

As described in Appendix B.2.1, we followed EOPD’s experimental setup and used a shared sampled-OPD objective to compare token-level supervision allocation rules while holding the training loss fixed. FiRe-OPD and REOPOLD already use sampled-token policy optimization objectives. We therefore changed only the datasets and teacher–student configuration, retaining the training setups described in their original papers, including their respective filtering, weighting, and reward-processing mechanisms. To evaluate TIP under the same loss, we retained its original token selection rule and replaced the full-vocabulary reverse-KL loss at the selected positions with the shared sampled-OPD loss.

We agree with the reviewer that this substitution can affect gradient variance and how the token selection rule interacts with the training loss. To test whether MetaOPD’s advantage persists when TIP uses its original loss, we retrained TIP with the same data setup and model configuration (Qwen3-8B as the teacher and Qwen3-1.7B-Base as the student), computing reverse KL over the full vocabulary at each selected position. The results are shown below. After restoring TIP’s original loss, MetaOPD still achieves the highest six-benchmark average Avg@8 and Pass@8, outperforming TIP by 0.74 and 1.04 percentage points, respectively.

Method/Avg@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
TIP (sampled-token)68.7541.2527.531.6711.256.6726.19
TIP (full-vocabulary)70.7340.6327.713.339.177.9226.58
MetaOPD69.9543.7527.712.0812.088.3327.32
Method/Pass@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
TIP (sampled-token)86.8072.5045.5910.0030.0026.6745.26
TIP (full-vocabulary)89.0077.5044.8510.0030.0023.3345.78
MetaOPD88.2077.5048.5310.0033.3323.3346.82

W2-1 & Q1: Are the improvements reliable across independent training seeds?

Following EOPD, we used seed 42 for the original experiments. The reviewer raises an important question about whether the gains remain stable when the training seed changes. To examine this, we followed the three-training-seed setting used in DecisionRank and supplemented seed 42 with complete independent training runs using seeds 43 and 44 for OPD, TIP, and MetaOPD. All experimental settings were kept unchanged except for the training seed. For Qwen3-8B → Qwen3-1.7B-Base, we report six-benchmark averages and their mean ± sample standard deviation (SD) across the three seeds.

For each of the three training seeds, MetaOPD outperforms both baselines in Avg@8 and Pass@8 averaged across the six benchmarks, showing consistent gains across independent training runs.

Method/Avg@8 (%)Seed 42Seed 43Seed 44Mean ± SD
OPD25.0725.7826.0125.62 ± 0.49
TIP26.1925.7626.1326.03 ± 0.23
MetaOPD27.3227.0127.7327.35 ± 0.36
Method/Pass@8 (%)Seed 42Seed 43Seed 44Mean ± SD
OPD40.4142.1942.2341.61 ± 1.04
TIP45.2642.6244.6344.17 ± 1.38
MetaOPD46.8245.8347.6746.77 ± 0.92
AuthorsOpenReview ↗

Response to Reviewer 8MSJ (Part 2/5): Cross-Family and OOD Generalization

Part 2/5

W2-2: Is evaluation on small Qwen3 student models sufficient?

To address the model-family concern, we conducted additional experiments with Llama-3.1-8B-Instruct as the teacher and Llama-3.2-1B-Instruct as the student, comparing MetaOPD with all seven baselines on the same six mathematical reasoning benchmarks. MetaOPD achieves the highest unweighted six-benchmark averages on both metrics: 7.68% Avg@8 and 19.79% Pass@8. These exceed KD, the strongest baseline by both averages in this setting, by 0.74 and 2.05 percentage points, respectively, and exceed uniform OPD by 2.62 and 3.86 points. The table below reports per-benchmark results, providing additional evidence of MetaOPD’s effectiveness across the evaluated Qwen3 and Llama model families.

Method/Avg@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
KD26.109.065.650.420.420.006.94
GRPO12.902.811.560.631.250.003.19
OPD20.655.313.580.000.830.005.06
EOPD20.907.193.540.630.830.005.51
FiRe-OPD20.736.563.630.830.420.005.36
REOPOLD14.403.132.440.210.000.423.43
TIP20.757.503.540.630.420.425.54
MetaOPD29.189.385.650.211.250.427.68
Method/Pass@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
KD56.6025.0019.851.673.330.0017.74
GRPO37.0017.505.153.3310.000.0012.16
OPD48.2027.5013.240.006.670.0015.93
EOPD50.0027.5012.135.006.670.0016.88
FiRe-OPD52.4025.0013.605.003.330.0016.56
REOPOLD41.0020.0010.291.670.003.3312.72
TIP48.8025.0012.505.003.333.3316.33
MetaOPD58.6027.5017.651.6710.003.3319.79

W2-3 & Q5-2: What do the modest OOD gains show, and do they persist on additional tasks?

MetaOPD’s primary contribution is learning how to allocate token-level supervision in on-policy distillation, with OOD evaluation providing a complementary assessment of its effects beyond mathematical reasoning. We appreciate the reviewer’s suggestion to strengthen this assessment and have therefore added experiments on LogiQA2, IFEval, WinoGrande, and BoolQ, covering natural-language logical reasoning, instruction following, commonsense reasoning, and yes/no reading comprehension, respectively. The evaluated students are Qwen3-1.7B-Base models trained with Qwen3-8B as the teacher.

OOD BenchmarkMetricOPD (%)TIP (%)MetaOPD (%)
LogiQA2Accuracy26.7826.9727.74
IFEvalPrompt-level strict31.9831.7932.72
IFEvalInstruction-level strict45.9245.5646.76
WinoGrandeAccuracy63.9364.0164.33
BoolQAccuracy80.6179.6680.92

MetaOPD achieves the highest scores among the three compared methods on all metrics, improving over OPD by 0.31–0.96 percentage points and TIP by 0.32–1.26 points. These additional evaluations address the reviewer’s concern by showing modest but consistent OOD gains across the newly evaluated tasks, broadening the evidence beyond the original OOD benchmarks.

AuthorsOpenReview ↗

Response to Reviewer 8MSJ (Part 3/5): Matched Training-Time Budgets

Part 3/5

W3 & Q2-2: Do the gains persist when training time, rather than student-update count, is matched?

To address the reviewer’s concern, we conducted comparisons under two matched training-time budgets: the time required by OPD to complete three epochs and the time required by MetaOPD to complete three epochs. Both comparisons use Qwen3-8B as the teacher and Qwen3-1.7B-Base as the student.

Training-time budget: time required by OPD to complete three epochs.

Method/Avg@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
OPD67.2338.7527.801.678.336.6725.07
TIP68.6541.2527.481.678.336.6725.67
MetaOPD68.8540.6327.532.0810.007.0826.03
Method/Pass@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
OPD84.2067.5047.436.6720.0016.6740.41
TIP86.8072.5045.5910.0020.0016.6741.93
MetaOPD87.0070.0048.538.3320.0023.3342.87

At this budget, MetaOPD achieves the highest six-benchmark averages on both metrics, exceeding OPD by 0.96 percentage points in Avg@8 and 2.46 points in Pass@8, and TIP by 0.36 and 0.94 points, respectively.

Training-time budget: time required by MetaOPD to complete three epochs.

Method/Avg@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
OPD68.7836.8828.582.0810.005.8325.36
TIP70.3840.3128.262.089.587.9226.42
MetaOPD69.9543.7527.712.0812.088.3327.32
Method/Pass@8 (%)MATH500AMC23MinervaHMMTAIME24AIME25Avg.
OPD86.4067.5047.065.0023.3316.6740.99
TIP87.8062.5046.6910.0023.3316.6741.17
MetaOPD88.2077.5048.5310.0033.3323.3346.82

At this budget, OPD and TIP receive additional training. MetaOPD still achieves the highest six-benchmark averages, exceeding OPD by 1.96 percentage points in Avg@8 and 5.83 points in Pass@8, and TIP by 0.90 and 5.65 points, respectively.

These results address the reviewer’s concern by showing that MetaOPD’s aggregate gains persist under matched training-time budgets, including when the baselines receive extended training. Thus, its advantage is not explained solely by the additional training time used in the original equal-update comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.