- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EOPSA, an efficient on-policy self-distillation framework for safety alignment. It introduces two mechanisms: Adaptive Rollout Scheduling (ARS), which adjusts rollout lengths based on Teacher Rescue Rate (TRR), and Token Filtering (TF), which selectively distills safety-critical positions using a rubric-guided semantic classifier. Experiments show improvements in safety alignment, reasoning preservation, and training efficiency.
Critical strengths
- Clear motivation. The paper identifies unreliable supervision over long rollouts and gradient dilution caused by safety-irrelevant advantage signals.
- Simple and effective methods. Both TRR-based scheduling and rule-based semantic filtering are intuitive and practical.
- Strong experimental results. The paper provides extensive evaluations across model scales and benchmarks.
Critical weaknesses
1. Robustness to distribution drift. The semantic lexicons are constructed from the initial student's top-16 high-KL positions and remain fixed throughout training. As the on-policy distribution evolves, do these rules remain effective? An analysis of classification quality across training stages would strengthen this assumption.
2. Validation of semantic token selection. Appendix F.9 evaluates human agreement on held-out high-KL positions. Human annotation on actual training rollouts, including both retained and filtered positions, would more directly validate the classifier. It would also be helpful to further connect the identified safety-critical semantics with the claimed effectiveness of selective supervision.
3. Fair comparison of rollout lengths. EOPSA uses a maximum rollout length of 1024, while OPSD allows 4096. Although Table 3 includes fixed-length ablations without TF, a comparison between fixed-1024 selective distillation and full EOPSA would better isolate the benefit of adaptive scheduling.
4. Token efficiency versus computational efficiency. The paper highlights a 98–99% reduction in optimized tokens. However, sparse loss masking may not reduce Transformer backward FLOPs proportionally. Could the authors clarify whether TF provides actual computational savings or primarily reduces the number of supervised positions?
Questions for authors
Please address the concerns and questions in points 1–4 under Critical Weaknesses.
3 comments
Official Review of Submission16521 by Reviewer YzfJ
Thank you for the response. I think my concerns are well addressed.
Dear Reviewer,
We sincerely thank you for your positive follow-up and thoughtful feedback. We are glad that our responses have addressed your concerns. Your constructive suggestions have been valuable in strengthening our work, and we will carefully incorporate the additional analyses and clarifications into the revised manuscript.
Thank you again for your time and consideration.
Best regards,
The EOPSA Authors
We thank the reviewer for the constructive feedback. We have clarified our research motivation and contributions in the Global Response and provide additional evidence addressing the concerns below.
W1: Robustness to Distribution Drift
To evaluate the stability of Token Filtering (TF) under distribution drift, we independently constructed semantic rules from the base model and checkpoints at Steps 50, 150, and 200, and applied them to the same 1,600 candidate token positions.
The initial rules cover 92.18% of positions selected by later checkpoints on average. One possible explanation is that safety alignment reinforces response patterns already present in the base model, while later checkpoints increasingly reuse a narrower set of recurring safety-related expressions. Consistent with this interpretation, the initial rules select more positions and cover most later selections. These findings support the stability of model-family-aware semantic rules across the evaluated training stages, without frequent reconstruction.
W2: Validation of Semantic Token Selection
TF employs lightweight semantic rules to filter safety-irrelevant supervision without expensive per-token contextual judgments. To evaluate their reliability, we conducted context-aware human validation on 640 token positions from 40 actual training rollouts. As shown in Table G1 of our Global Response, TF achieves strong agreement with human judgments (94.7% precision and 97.3% F1), indicating that most retained positions are safety-relevant, with few relevant positions missed.
Beyond semantic relevance, Table G2 shows substantially higher normalized safety gains for Pivot, Intent, and Risk categories. Under the same 2% token budget, Table G3 further demonstrates that TF outperforms Top-KL and Random Mask in both safety performance and reasoning retention. Together, these results support both the semantic reliability and empirical training value of our lightweight selection strategy.
W3: Fair Comparison of Rollout Lengths
We clarify that w/o ARS in the original Table 2 already corresponds to Fixed-1024 + TF, retaining TF while disabling Adaptive Rollout Scheduling (ARS). We also identified an inconsistent maximum rollout length in the original w/o ARS & TF configuration. To ensure a fair comparison, we reran this baseline and updated the ablation results under the same 1024-token maximum.
With TF enabled, ARS improves average safety from 89.20% to 91.15% (+1.95 points) under the same maximum rollout length. Similarly, TF improves safety from 86.11% to 91.15% when ARS is enabled. Together with the fixed-length comparisons in the original Table 3, these results support the independent contribution of ARS beyond simply reducing the rollout length. We will update Table 2 accordingly.
W4: Token Efficiency versus Computational Efficiency
We agree that sparse loss masking does not proportionally reduce Transformer backward FLOPs. The reported 98–99% reduction refers to supervised token positions, rather than equivalent computational savings.
As clarified in the Global Response (Section 1), EOPSA targets two complementary dimensions of efficiency: ARS improves computational efficiency by reducing rollout generation, while TF improves supervision efficiency by filtering safety-irrelevant distillation signals. The measured optimization time decreases from 0.25 h to 0.21 h (16%) in the matched OPSD-1024 comparison, although this reflects the combined EOPSA system rather than an isolated TF speedup.
Importantly, Tables G2–G3 of our Global Response demonstrate that TF achieves better safety alignment and reasoning retention by concentrating supervision on more valuable token positions, rather than merely reducing their number. We will clarify the distinction between supervision sparsity and actual computational savings in the revised paper.