- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents On-policy Verbal Distillation (OVD), a framework designed for black-box LLM teachers that provide only text outputs rather than full-vocabulary token distributions. OVD operates by having the teacher score student-generated sub-trajectories using discrete verbal score tokens (e.g., on a 0–9 scale). High-scoring sub-trajectories are retained, while low-scoring ones are rejected and replaced with teacher continuations, after which the student resumes rollout.
Critical strengths
Black-Box Logit-Free Formulation: OVD circumvents the massive memory and access bottlenecks of token-level reverse-KL distillation by relying strictly on discrete verbal feedback, enabling distillation from black-box teacher interfaces.
Prefix Preservation and Exploration: By replacing only low-scoring suffixes or sub-trajectories rather than entire responses, OVD preserves student-initiated reasoning paths, demonstrating empirically improved solution coverage (Pass@16) and reduced rollout compute.
Critical weaknesses
Weak Justification of Density-Ratio Assumptions: The distribution-matching guarantees require verbal acceptance probabilities to approximate density ratios (). In practice, verbal scores from an LLM critic are subjective heuristics rather than calibrated importance weights, making the theoretical total variation bounds more aspirational than strictly operational.
Benchmark Selection and Stability: Acknowledging post-hoc selection of math benchmarks in Appendix A.5 weakens confidence in general mathematical reasoning gains. Furthermore, Table 6 shows that random-suffix resampling (OVD-RS) occasionally matches or exceeds high-entropy suffix selection (OVD-HES), making the value of the entropy-split heuristic somewhat ambiguous.
Questions for authors
Post-Hoc Selection: Could you provide the aggregate results across the entire, uncurated pool of 23 math benchmarks for both OVD-HES and OVD-FR to confirm that gains are consistent without subset filtering?
How sensitive is the framework if the verbal score prompt or scale is modified, and does the observed correlation with the true density ratio hold across non-math domains?
What is the overall wall-clock training overhead when accounting for teacher verbal-scoring inferences compared to standard RLVR or offline SFT distillation?
No rebuttal received
No rebuttal received. Score remains the same.