- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DrOPD retains reverse-KL supervision at all response positions and adds forward KL at the 20% of positions with the highest student-teacher JS divergence. Cross-sampling experiments motivate the routing rule, and a local gradient analysis explains the added corrective signal. The method is evaluated on six math benchmarks for two Qwen3 student-teacher pairs, with entropy-routing controls and ablations of the loss allocation and candidate set.
Critical strengths
-
The gradient analysis gives a clear motivation for adding forward KL when the student assigns little probability to a teacher candidate. The identities are valid for the candidate-normalized distributions used in the derivation (Section 3 and supplementary Section 2.2). The cross-sampling study also compares divergence with entropy and random intervention.
-
The entropy-routing baselines use the same all-position reverse KL and top-20% forward-KL allocation, making them useful comparisons. Mean accuracy rises from 31.57 to 33.41 for the smaller pair and from 44.64 to 45.23 for the larger pair (Table 1). The supplement reports training cost as well as accuracy; the smaller-pair run takes about 8% longer than OPD.
Critical weaknesses
-
The supplied implementation appears to optimize a different surrogate from the one derived in the paper. Equation 8 and supplementary Section 2.3 use candidate-normalized log probabilities. In
dp_actor.py,_forward_micro_batchinstead gathers full-vocabulary log-softmax values,kl_valuses the raw student-teacher log-probability difference, and these gathered values enter the policy loss without candidate renormalization. For forward KL, this gives a different local logit gradient and can affect logits outside the candidate set. The code comments describe this behavior. This need not be a bad update, but the theory and implementation need to agree about which method is being evaluated. This is the main reason for my current weak-reject recommendation. -
The advantage specifically due to JS routing is less clear than the advantage over pure reverse KL. The entropy controls help, but a training control that applies forward KL to randomly chosen 20% of positions is missing; random intervention during decoding tests a different question. Such a control would be useful given the relatively small margins over the strongest entropy baseline and over OPD for the larger pair.
Questions for authors
-
Which probability convention and code version were used for Table 1? Please reconcile the released reward/policy-loss path with Equation 8 and supplementary Section 2.3. A small fixed-context gradient check would help resolve whether the difference is intentional.
-
Do you have a random-20% routing training result under the same loss coefficient and budget? Please also clarify the checkpoint-selection rule and report any existing repeat runs for the larger-pair result.