- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes FReDraft, an on-policy training method that directly maximizes the expected accepted length of drafts sampled from the current draft model. By analyzing the gradients of the expected accepted length objective, the authors introduce two techniques: Token-level Expected Acceptance Rate Marginalization and Over-sampled Token Penalty. These methods aim to reduce gradient variance and improve the efficiency of on-policy training, respectively. Experiments compare FReDraft with recent methods, including D-PACE, LK Loss, and Draft-OPD, demonstrating competitive performance in terms of accepted length.
Critical strengths
1 Insightful gradient analysis. By analyzing the on-policy sampling process, the authors identify an important issue: tokens that are frequently sampled by the draft model may provide no direct gradient signal, leading to inefficient training. To address this issue, they propose an over-sampled token penalty that improves gradient utilization without changing the expected optimization direction. I find the theoretical analysis of the gradient particularly insightful.
2 Comprehensive ablation studies. The paper provides relatively thorough ablation studies that evaluate the contributions of individual components, helping to demonstrate the effectiveness of the proposed techniques.
Critical weaknesses
1 Missing comparison with E2E TV Loss [1]. The experimental evaluation lacks a comparison with E2E TV Loss, which also aims to optimize the expected accepted length. Including this baseline would provide a more comprehensive evaluation of the proposed method.
2 Fairness of comparison with Draft-OPD. It would also be valuable to evaluate Draft-OPD combined with E2E TV Loss. Such a comparison would help clarify whether the improvements of FReDraft come from its proposed gradient optimization techniques or from the choice of training objective, making the experimental evaluation more complete and fair.
[1] Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling
Questions for authors
Please refer to the questions raised in the Weaknesses section.