- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a post-training distillation algorithm, which given a prompt and teacher trajectory, selectively chooses whether to perform on-policy rollout or to use teacher trajectory, selectively choose whether to perform on-policy rollout or to use teacher trajectory for training. The method looks at directional support mismatch between student and teacher to make this decision.
Critical strengths
- The motivation is intuitive, and it asks an important question of how to know whether on-policy benefits more than off-policy or vice versa.
Critical weaknesses
[W1] Insufficient empirical evidence. Experiments are conducted on only one teacher-student pair and one task (GSM8K). It is unclear whether the method generalizes across model scales and tasks.
[W2] It is hard to see the effect of support-aware routing due to the lack of simple selective-routing baselines. Comparison with KL-only or entropy-only baselines under matched rollout budgets will be helpful.
[W3] The central hypothesis that support escape favors on-policy supervision is insufficiently validated. The paper does not directly demonstrate that the proposed support-statistic-based diagnosis successfully predicts the relative learning benefits of on/off-policy data.
[W4] The reported compute savings are difficult to attribute solely to the proposed routing strategy. It seems like the naive OPD and CAR use different batch sizes, number of steps, and learning rate schedules. Can the authors clarify this?
Questions for authors
[Q1] Could this approach be extended to token-level or segment-level, where the choice between on-policy and off-policy rollout is made at a finer granularity?
[Q2] Can the authors compare CAR against random routing and simpler KL- or entropy-based selection strategies under the same on-policy rollout budget?