- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a token-weighting framework for on-policy distillation (OPD). While existing OPD methods typically assign equal weights to all tokens or rely on predefined rules, the proposed framework learns token weights through bi-level optimization. Specifically, a lightweight MLP predicts the token weights, which are used to perform a virtual update of the student model. The weighting network is then optimized based on the validation loss of the updated student.
Critical strengths
-
The paper addresses an important limitation of existing OPD methods by exploring how to learn more effective token-level supervision.
-
The proposed bi-level optimization method offers a novel approach to learning token weights based on student updates.
-
The proposed method consistently improves over standard OPD and achieves competitive results across serveral benchmarks.
Critical weaknesses
-
The paper does not sufficiently justify the additional optimization complexity. In particular, it remains unclear whether the virtual student update is necessary. For example, why not directly train the weighting MLP using the weighted OPD objective without virtual updates?
-
The presentation is unnecessarily complicated relative to the core idea, which is quite simple. For example, Section 3.2, Algorithm 1, and Figure 2 largely repeat the same optimization procedure. Why not keep either Algorithm 1 or Figure 2? Moreover, Eqn. (21)-(24) provide detailed derivations of standard gradient clipping and AdamW operations. Although relevant to the meta-gradient computation, these details are not the main novelty and could be presented more concisely.
-
The notation is sometimes inconsistent, making the paper difficult to follow. For example, the weighted inner loss is denoted as in Figure 2 but in Eq. (5). Similarly, Figure 2 uses , , and , whereas the main text uses , , and .
-
The figures are also not clear. Figure 1(a) does not clearly highlight the main difference between MetaOPD and existing approaches. Also, what is the purpose of the large green region in Figure 1(b)? Figure 2 is visually overloaded with icons that provide limited technical information. For example, what do the two different icons under "Optimization" represent, and what is the purpose of the icon for "top-32 distribution statistics"?
Overall, I find the problem meaningful, the approach reasonably novel, and the results promising. However, the paper is often difficult to follow due to unclear explanations and an insufficiently polished presentation. My current assessment is between Borderline and Weak Accept. If my concerns are not addressed well, I may change my rating to Weak Reject.
Questions for authors
Does the paper consider directly optimizing the token-weighting MLP using the weighted teacher-supervised OPD loss?
4 comments
Response to Reviewer uKud (Part 2/2): Presentation, Notation, and Figures
Part 2/2
W2: Repetition across the method description, pseudocode, and flowchart.
Thank you for pointing out the overlap in the presentation. Section 3.2, Algorithm 1, and Figure 2 are intended to serve complementary roles: Section 3.2 defines the objectives and derives the meta-gradient; Algorithm 1 specifies the execution order and handling of virtual and actual optimizer states; and Figure 2 illustrates the information flow. We will streamline the repeated procedural descriptions and make these roles clearer, keeping the mathematical details in Section 3.2, implementation steps in Algorithm 1, and a concise visual overview in Figure 2.
We will also condense Equations (21)–(24) in Appendix A.3. These equations detail standard gradient clipping and AdamW operations to make the meta-gradient derivation self-contained. The revision will retain only the dependencies needed to follow differentiation through the virtual update, with standard optimizer details summarized more briefly.
W3: The notation is sometimes inconsistent, making the paper difficult to follow.
We acknowledge that the notation differences between Figure 2 and the main text make the method harder to follow. In Figure 2, denotes the same weighted OPD loss as in Eq. (5). Using different names for the same loss was an oversight, and we will use consistently throughout the paper.
For the token weights and per-token OPD loss, Figure 2 uses the shortened symbols , , and . These symbols retain the token-position index to highlight token-level weighting and keep the diagram readable. The main text uses the full notation to make the response identity and weighting-network parameter dependence explicit in the mathematical derivation. The correspondence is:
w_t^{+} \equiv w_{\eta^{+},i,t},\qquad \ell_t \equiv \ell_{i,t}^{\mathrm{OPD}}.$$ Here, $i$ indexes responses, and $\eta$ and $\eta^{+}$ denote the weighting-network parameters before and after the meta-update, respectively. Thus, $w_t$ and $w_t^{+}$ refer to the token weights computed before and after that update. We did not explain this correspondence in the original figure caption. We will add the mapping and definitions so that readers can directly relate the figure to the mathematical description. > **W4: Unclear presentation and visual clutter in the figures.** Figure 1(a) is intended to highlight how feedback is used to determine token weights. The proxy-based pipeline assigns weights through predefined rules, updates the student, and then evaluates its performance. MetaOPD uses the reference-solution loss after a virtual student update as feedback to learn the weighting rule. We will make this feedback path more explicit in the figure and caption. The green region in Figure 1(b) corresponds to the “→ better” label and marks positive loss reduction relative to Uniform OPD. We will clarify this meaning in the caption. In Figure 2, the lock icons indicate frozen parameters. The two icons under “Optimization” distinguish the weighting MLP's meta-gradient update, obtained by differentiating through the virtual student update, from the student's actual weighted-OPD update. The icon beside “top-32 distribution statistics” was intended as a visual aid and does not denote an additional computational operation. We recognize that the number of icons makes the figure harder to read. We will remove nonessential decorative icons and add clear labels for the two updates, retaining visual elements that convey a specific technical meaning.The rebuttal addresses my concerns, and I am happy to maintain my original rating. If accepted, please incorporate the updates into the camera-ready version.
Thank you for your constructive feedback!
We are glad that our rebuttal addressed your concerns. Thank you for your constructive feedback, which helped us clarify the presentation, notation, and figures. If the paper is accepted, we will incorporate the revisions discussed in our rebuttal into the camera-ready version. Thanks again!
Response to Reviewer uKud (Part 1/2): Why Virtual Updates?
Part 1/2
Thank you for your careful review and constructive feedback. We hope our clarifications and additional experiments address your concerns, and we welcome any further questions.
MetaOPD uses meta-learning to learn token weights from their effect on the student’s subsequent learning. Directly optimizing the weighting MLP with the current weighted OPD loss does not explicitly assess this effect. Let θ and η denote the student and weighting-MLP parameters, respectively. The two objectives are: Jdirect(η)=NB1∑i,tmi,twη,i,tℓi,tOPD(θ)+λRw(η),
Jmeta(η)=Lout(θ′(η))+λRw(η),
where mi,t masks valid tokens, NB=∑i,tmi,t counts valid tokens in the minibatch, θ′(η) is the student after the virtual weighted update, and Lout is the reference-solution NLL. Both objectives use the same weight regularizer.
With the current student fixed, the direct objective can reduce its loss term by shifting weight toward tokens with smaller current surrogate-loss values, even under response-level weight normalization. However, smaller current loss values do not necessarily indicate more beneficial student updates. MetaOPD instead evaluates the weights through the reference-solution loss after a virtual student update. Differentiating through this update allows the MLP to learn which token weights produce updates that improve reference-solution performance. The virtual update therefore provides the connection between supervision allocation and its learning effect; this is a local, one-step signal rather than a guarantee of long-term improvement.
To test this distinction empirically, we evaluated Direct-MLP, retaining the same architecture, input features, weight constraints, and regularizer, while replacing the meta objective with the current weighted OPD objective. Both variants use Qwen3-8B as the teacher and Qwen3-1.7B-Base as the student.
MetaOPD achieves six-benchmark averages of 27.32% Avg@8 and 46.82% Pass@8, compared with 25.81% and 43.09% for Direct-MLP, improving by 1.51 and 3.73 percentage points, respectively. These results support learning token weights from the student’s post-update performance rather than directly minimizing the current weighted OPD loss.