- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VG-OPD combines specialist teachers to train a student in scientific reasoning. A verifier admits a teacher when its answer meets a criterion that the student fails; that teacher then supervises selected tokens according to its likelihood of the student's response. The method combines this supervision with GRPO and is evaluated on seven benchmarks at two model sizes.
Critical strengths
S1. A specialist may give correct guidance on one part of an answer and poor guidance on another. The paper takes this limitation seriously in designing how teachers supervise the student.
S2. The science-benchmark gains in Table 1 are consistent at both scales, with general reasoning performance largely preserved.
S3. Table 3 examines gating, localization, and weighting separately. I found the coverage-matched random mask particularly helpful as a control.
Critical weaknesses
W1. I do not see a justification for the distillation gradient. Appendix A.1 proves that the unclamped estimator in Eq. 9 has the correct expected KL value, but this result does not justify detaching the estimator and using it as an advantage in Eq. 10. The update can increase KL even with a fixed teacher, unit weights, and no clipping. For a binary student distribution p=(0.8,0.2) and teacher q=(0.6,0.4), it increases the first token's probability, whereas the reverse-KL gradient decreases it. This counterexample calls the claimed connection to OPD into question.
W2. Eq. 6 assigns the same mask to every criterion associated with a teacher. The mask is therefore teacher-specific, not criterion-specific: passing one criterion permits supervision on tokens that may concern another criterion the teacher fails. This does not enforce the exclusion of refused claims described in Figure 6. Figure 3a shows that selected tokens are suppressed after training, but does not establish that those tokens correspond to the verified criterion.
W3. The baseline comparison changes more than routing and localization. Appendix B.7 replaces MOPD's signed log-ratio advantage with the non-positive estimator in Eq. 9, changing its policy-gradient update. The signed advantage is part of the original MOPD method. For CriPO, the description covers advantage flipping but leaves unclear whether the behavior-injection loss from the original CriPO method was included.
W4. Some reported accuracies are difficult to reconcile with a single greedy evaluation. For example, 46.3% on 198 GPQA questions cannot come from an integer number of correct answers, even after rounding; the same holds for 80.9% on 500 MATH500 questions. Averaging could explain these values, but the paper does not describe an averaging procedure. It also omits training seeds and uncertainty estimates.
Questions for authors
Q1. What is the expected gradient of the detached estimator in Eq. 10, and how does it account for the binary counterexample in W1?
Q2. What prevents a licensed teacher from supervising tokens associated with criteria it fails? Could the authors check this directly on a small annotated sample?
Q3. Were MOPD's original signed advantage and CriPO's full behavior-injection loss evaluated? If so, please report those results separately.
Q4. How were Table 1 scores aggregated? Please provide the number of runs, effective test sizes, and uncertainty estimates.