- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper looks at on-policy distillation (OPD), where a teacher gives token-level feedback on whatever the student writes, and points out a hidden assumption nobody really questioned: that every state the student wanders into is still worth teaching at. The authors argue this is often false — once a student's partial answer has drifted so far that the original goal is basically unrecoverable, teacher feedback there is useless or even teaches bad habits. They formalize this with "prefix viability": roughly, the chance the task can still be solved if you continue from the student's current prefix, estimated as the teacher's success rate conditioned on that prefix divided by its success rate from scratch (which controls for task difficulty). Since estimating this online is too expensive, they build a small offline-trained "viability judge" that runs during distillation.
Critical strengths
The core observation: "visited does not mean viable" — is clean, intuitive, and genuinely underexplored.
I really liked the diagnostic experiments in Section 3: showing that viability decays along rollouts (38% of rollouts enter a low-viability zone, ~46% of tokens are generated after that point) is a compelling motivation.
The method itself is sensible and each design choice gets its own ablation (random vs overlap-based selection at matched budget, uniform vs shuffled token weights — the shuffled-weights control is a nice touch).
Critical weaknesses
My main concerns. First, the viability ground-truth labels used to train the judge are themselves Monte-Carlo estimates of teacher success probability — the paper never says how many continuations are sampled per prefix, or how noisy these labels are, and a noisy judge trained on noisy labels could compound errors in subtle ways.
Second, the selection is done at the rollout level, but viability declines within a rollout, so a rollout deemed viable at the end can still contain late tokens that already crossed into the hopeless zone — this seems like an inconsistency with the paper's own framing that is only hand-waved away ("requires viability estimation over multiple prefixes").
Questions for authors
2 comments
-
Thanks for you response. i keep my score unchaged.
Response to Reviewer GUXd
We sincerely thank Reviewer GUXd for the thoughtful and insightful comments. The reviewer raises two important questions concerning the reliability of Monte Carlo-based viability labels and the consistency between prefix-level viability and rollout-level selection. We appreciate the opportunity to further clarify these methodological aspects of ViableOPD and discuss the relevant design considerations and limitations.
1. Reliability of Monte Carlo-Based Viability Labels
Sampling Configuration. As reported in Table 4 of Appendix A, we sample 32 teacher-generated scratch completions per problem and 32 teacher continuations per student prefix to estimate the corresponding task success probabilities. The resulting empirical viability scores are converted into binary labels using the threshold τ_v = 0.75.
Since finite Monte Carlo sampling introduces estimation uncertainty, we further examine whether sampling perturbations change these binary labels by causing empirical viability scores to cross the decision threshold.
Additional Statistical Analysis. We conduct three complementary analyses:
Bootstrap Stability. We perform 1,000 bootstrap replicates, independently resampling the 32 continuations of each prefix and the 32 scratch completions per problem. Prefixes from the same problem share the resampled scratch-completion denominator. The average label stability is 97.31%. Based on bootstrap classification frequencies, 52.72% of prefixes have stable positive labels (>95%), 37.12% have stable negative labels (<5%), and 10.16% are ambiguous.
Leave-One-Out Sensitivity. Removing each continuation in turn yields an average label-flip rate of only 0.459%, with 98.66% of prefixes retaining their original labels across all 32 evaluations. Label changes are concentrated near the threshold: prefixes within 0.05 of the threshold have an average flip rate of 21.47%, compared with only 0.018% for those at least 0.20 away.
Sampling Budget Sensitivity. We evaluate sampling budgets N in {8,16,24,32} by randomly downsampling scratch completions and prefix continuations without replacement, using 10 fixed random seeds. Prefixes from the same problem share the same downsampled denominator. Compared with the full 32-sample labels, the agreement rates for budgets of 8, 16, and 24 are 95.51%, 97.20%, and 98.44%, respectively.
Implications for Viability Judge. Overall, these analyses show that Monte Carlo-based labels are highly stable. The Viability Judge further achieves an AUROC of 0.943 against these empirical labels (Figure 4), while downstream results in Tables 1 and 2 demonstrate the practical effectiveness of Judge-based supervision selection. Although local label noise cannot be entirely ruled out, these findings support the reliability of the proposed supervision signal. We will incorporate the statistical analyses, and relevant discussions into the revised manuscript.
2. Consistency between Prefix-Level Viability and Rollout-Level Selection
Clarification of Prefix Viability. We would like to clarify that prefix viability does not measure whether a student-generated prefix is itself correct, but whether the teacher can still provide meaningful, task-aligned supervision conditioned on that prefix. Thus, even if a student prefix contains reasoning errors, it can remain valuable for distillation as long as the teacher can provide effective corrective guidance. Our goal is to exclude states where accumulated errors have severely compromised the teacher's ability to provide useful supervision, while preserving those that still allow effective correction.
Rationale for Rollout-Level Selection. Our rollout-level selection is primarily motivated by computational efficiency. For a rollout classified as non-viable at its final position, earlier prefixes may still retain substantial supervision value. However, identifying the precise point at which supervision becomes ineffective requires evaluating multiple prefixes in reverse order, from the end of the rollout toward its beginning, thereby introducing additional Viability Judge inference overhead. To avoid this cost, we directly discard the entire non-viable rollout, accepting the loss of potentially useful supervision from its earlier portions. Conversely, we retain the entire rollout when its final prefix is viable, since the teacher can still provide meaningful supervision conditioned on the resulting context. Intermediate reasoning errors do not necessarily invalidate this supervision, as they may still admit effective corrective guidance.
Limitations and Future Work. We acknowledge that rollout-level selection does not explicitly identify the supervision boundary at each intermediate prefix. Exploring finer-grained prefix-level supervision allocation to more precisely identify effective supervision regions is an important direction for future work.