acceptodds
Under review as a conference paper at ICLR 2027

EOPSA: Efficient On-Policy Self-Distilled Safety Alignment

10 views

Abstract

On-Policy Self-Distillation (OPSD) has emerged as a promising paradigm for safety alignment, delivering dense, token-level supervision by distilling from a teacher conditioned on refusal-oriented privileged prompts. However, we reveal that this paradigm suffers from critical inefficiencies that degrade both training efficiency and general reasoning capabilities. Specifically, we diagnose two fundamental bottlenecks: (1) supervisory collapse over extended rollouts, where the teacher's corrective efficacy degrades precipitously as the student's generation prefix lengthens, injecting noisy gradients into late-stage tokens; and (2) gradient dilution from stylistic shifts, where the distillation objective is dominated by safety-irrelevant stylistic discrepancies induced by privileged prompting, washing out genuine safety signals and impairing base reasoning. To resolve these issues, we propose Efficient On-Policy Self-Distilled Safety Alignment (EOPSA), which concentrates computational and gradient budgets exclusively on reliably supervised, safety-critical tokens. EOPSA incorporates two coordinated mechanisms: (i) Adaptive Rollout Scheduling, which dynamically bounds the generation horizon guided by a novel Teacher Rescue Rate (TRR) metric to operate strictly within reliable supervision regimes; and (ii) Selective Distillation, which filters out safety-neutral tokens to restrict gradient updates exclusively to safety-pivotal transitions. Extensive evaluations across reasoning models up to 32B parameters demonstrate that EOPSA slashes rollout computation by and backpropagates through merely of tokens, consistently outperforming full-token distillation baselines in both safety compliance and reasoning retention.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety alignment, on policy distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YzfJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EOPSA, an efficient on-policy self-distillation framework for safety alignment. It introduces two mechanisms: Adaptive Rollout Scheduling (ARS), which adjusts rollout lengths based on Teacher Rescue Rate (TRR), and Token Filtering (TF), which selectively distills safety-critical positions using a rubric-guided semantic classifier. Experiments show improvements in safety alignment, reasoning preservation, and training efficiency.

Critical strengths
  1. Clear motivation. The paper identifies unreliable supervision over long rollouts and gradient dilution caused by safety-irrelevant advantage signals.
  2. Simple and effective methods. Both TRR-based scheduling and rule-based semantic filtering are intuitive and practical.
  3. Strong experimental results. The paper provides extensive evaluations across model scales and benchmarks.
Critical weaknesses

1. Robustness to distribution drift. The semantic lexicons are constructed from the initial student's top-16 high-KL positions and remain fixed throughout training. As the on-policy distribution evolves, do these rules remain effective? An analysis of classification quality across training stages would strengthen this assumption.

2. Validation of semantic token selection. Appendix F.9 evaluates human agreement on held-out high-KL positions. Human annotation on actual training rollouts, including both retained and filtered positions, would more directly validate the classifier. It would also be helpful to further connect the identified safety-critical semantics with the claimed effectiveness of selective supervision.

3. Fair comparison of rollout lengths. EOPSA uses a maximum rollout length of 1024, while OPSD allows 4096. Although Table 3 includes fixed-length ablations without TF, a comparison between fixed-1024 selective distillation and full EOPSA would better isolate the benefit of adaptive scheduling.

4. Token efficiency versus computational efficiency. The paper highlights a 98–99% reduction in optimized tokens. However, sparse loss masking may not reduce Transformer backward FLOPs proportionally. Could the authors clarify whether TF provides actual computational savings or primarily reduces the number of supervised positions?

Questions for authors

Please address the concerns and questions in points 1–4 under Critical Weaknesses.

3 comments

We thank the reviewer for the constructive feedback. We have clarified our research motivation and contributions in the Global Response and provide additional evidence addressing the concerns below.

W1: Robustness to Distribution Drift

To evaluate the stability of Token Filtering (TF) under distribution drift, we independently constructed semantic rules from the base model and checkpoints at Steps 50, 150, and 200, and applied them to the same 1,600 candidate token positions.

Rule SourceSelected PositionsOverlap with InitialInitial Coverage ↑
Base (Step 0)626626100.00%
Step 5058655795.05%
Step 15041836587.32%
Step 20053250194.17%
Average (later steps)——92.18%

The initial rules cover 92.18% of positions selected by later checkpoints on average. One possible explanation is that safety alignment reinforces response patterns already present in the base model, while later checkpoints increasingly reuse a narrower set of recurring safety-related expressions. Consistent with this interpretation, the initial rules select more positions and cover most later selections. These findings support the stability of model-family-aware semantic rules across the evaluated training stages, without frequent reconstruction.

W2: Validation of Semantic Token Selection

TF employs lightweight semantic rules to filter safety-irrelevant supervision without expensive per-token contextual judgments. To evaluate their reliability, we conducted context-aware human validation on 640 token positions from 40 actual training rollouts. As shown in Table G1 of our Global Response, TF achieves strong agreement with human judgments (94.7% precision and 97.3% F1), indicating that most retained positions are safety-relevant, with few relevant positions missed.

Beyond semantic relevance, Table G2 shows substantially higher normalized safety gains for Pivot, Intent, and Risk categories. Under the same 2% token budget, Table G3 further demonstrates that TF outperforms Top-KL and Random Mask in both safety performance and reasoning retention. Together, these results support both the semantic reliability and empirical training value of our lightweight selection strategy.

W3: Fair Comparison of Rollout Lengths

We clarify that w/o ARS in the original Table 2 already corresponds to Fixed-1024 + TF, retaining TF while disabling Adaptive Rollout Scheduling (ARS). We also identified an inconsistent maximum rollout length in the original w/o ARS & TF configuration. To ensure a fair comparison, we reran this baseline and updated the ablation results under the same 1024-token maximum.

MethodARSTFMax LengthSafety Avg. ↑
w/o ARS & TF××102484.07
w/o TF✓×102486.11
w/o ARS×✓102489.20
EOPSA✓✓102491.15

With TF enabled, ARS improves average safety from 89.20% to 91.15% (+1.95 points) under the same maximum rollout length. Similarly, TF improves safety from 86.11% to 91.15% when ARS is enabled. Together with the fixed-length comparisons in the original Table 3, these results support the independent contribution of ARS beyond simply reducing the rollout length. We will update Table 2 accordingly.

W4: Token Efficiency versus Computational Efficiency

We agree that sparse loss masking does not proportionally reduce Transformer backward FLOPs. The reported 98–99% reduction refers to supervised token positions, rather than equivalent computational savings.

As clarified in the Global Response (Section 1), EOPSA targets two complementary dimensions of efficiency: ARS improves computational efficiency by reducing rollout generation, while TF improves supervision efficiency by filtering safety-irrelevant distillation signals. The measured optimization time decreases from 0.25 h to 0.21 h (16%) in the matched OPSD-1024 comparison, although this reflects the combined EOPSA system rather than an isolated TF speedup.

Importantly, Tables G2–G3 of our Global Response demonstrate that TF achieves better safety alignment and reasoning retention by concentrating supervision on more valuable token positions, rather than merely reducing their number. We will clarify the distinction between supervision sparsity and actual computational savings in the revised paper.

Reviewer_YzfJOpenReview ↗

Official Review of Submission16521 by Reviewer YzfJ

Thank you for the response. I think my concerns are well addressed.

Dear Reviewer,

We sincerely thank you for your positive follow-up and thoughtful feedback. We are glad that our responses have addressed your concerns. Your constructive suggestions have been valuable in strengthening our work, and we will carefully incorporate the additional analyses and clarifications into the revised manuscript.

Thank you again for your time and consideration.

Best regards,
The EOPSA Authors

Reviewer_irHYOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EOPSA, an efficient on-policy self-distillation framework for safety alignment in large reasoning models. It combines Adaptive Rollout Scheduling (ARS), guided by Teacher Rescue Rate (TRR), with selective KL distillation on safety-relevant tokens to reduce training costs and preserve reasoning capabilities. Experiments across multiple model scales demonstrate improved safety performance and efficiency over existing baselines.

Critical strengths
  • The paper tackles the important challenge of improving safety alignment in large reasoning models while minimizing the degradation of general reasoning capabilities and reducing training costs.

  • EOPSA achieves strong safety improvements while largely preserving reasoning performance and substantially reducing the number of optimized tokens. The experiments across multiple model scales and ablation studies further demonstrate the effectiveness of the proposed approach.

Critical weaknesses
  • The proposed TRR metric evaluates the safety of the entire response, including the student-generated prefix. Thus, a decreasing TRR may reflect accumulated unsafe content rather than genuinely unreliable teacher supervision. It remains unclear whether low-TRR prefixes actually produce harmful or noisy distillation gradients. Moreover, the standard OPSD baseline uses a maximum rollout length of 4k tokens, whereas EOPSA is capped at 1k, potentially confounding the effects of adaptive scheduling with simple truncation. More direct analysis of supervision quality and comparisons under matched rollout ceilings would strengthen the justification for ARS.

  • The proposed token classifier relies on context-free lookup rules, which may classify common words such as "maybe" or "which" as safety-relevant regardless of their context. This raises questions about how reliably the selected tokens capture meaningful safety-related transitions. There is also some ambiguity in how benign queries are handled during training. While the objective is formulated over harmful queries, Appendix includes a separate teacher prompt for benign data without clarifying whether or how it is used in EOPSA.

Questions for authors
  • Figure 5 compares methods with substantially different training data budgets. Could the efficiency comparison be conducted under matched computational or data budgets, while also accounting for the overhead of TRR validation?

  • On R1-7B, EOPSA substantially increases over-refusal compared to OPSD. What explains this degradation?

5 comments

We thank the reviewer for the constructive feedback. We have clarified our motivation and contributions in the Global Response and provide additional evidence addressing the specific concerns below.

W1: TRR Reliability and Effectiveness of ARS

W1-1: Does TRR Reflect Teacher Supervision Quality?

(1) TRR definition and human validation.

The Teacher Rescue Rate (TRR) measures whether the privileged teacher can recover the safety of a complete response given an on-policy student prefix, rather than evaluating the teacher suffix in isolation. This distinction matters because an unsafe prefix may establish harmful intentions that the teacher continues without generating additional explicit harmful content. To validate TRR, we sampled 250 cases and compared human judgments with Llama Guard evaluations of teacher rescue success at different prefix lengths, considering the complete response context.

Prefix LengthAccuracy ↑Precision ↑Recall ↑F1 ↑
12891.297.491.594.4
25686.897.286.091.2
102494.8100.093.596.6

The strong agreement with human judgments (91.2–96.6% F1) supports the reliability of TRR in measuring teacher rescue success across different prefix lengths.

(2) Effectiveness of Adaptive Rollout Scheduling.

To isolate the contribution of Adaptive Rollout Scheduling (ARS), we compare ARS alone against fixed rollout horizons with Token Filtering (TF) disabled (original Table 3).

StrategyHarmB. ↑WildC. ↑WildJ. ↑StrongR. ↑Safety Avg. ↑Gen Time ↓
Fixed-12897.5069.7374.4095.2184.210.26 h
Fixed-51299.0070.8175.2095.2185.060.31 h
Fixed-102496.5071.3570.8092.6582.830.42 h
ARS only97.0075.6877.2094.5786.110.28 h

ARS achieves the highest average safety (86.11%), outperforming Fixed-1024 by 3.28 points while reducing generation time by 33.3%. Notably, increasing the fixed horizon from 512 to 1024 reduces safety despite additional computation, supporting the benefit of dynamically adjusting the supervision horizon.

(3) Training utility of ARS-excluded suffixes.

To further examine the value of late-stage supervision, we additionally trained using only the suffix tokens beyond the ARS cutoff (maximum generation length: 4096) and compared their normalized training gains with safety-related token categories.

Supervision RegionSafety Avg. ↑Toks/Smp ↓ΔS/Tok ↑
ARS-excluded suffix69.79396.800.039
Pivot77.970.3371.39
Intent89.320.4577.58
Risk83.941.1425.90

ΔS/Tok = (Safety Avg. − Base Safety Avg.) / Toks/Smp, where Base Safety Avg. = 54.41.

Although the excluded suffixes provide some safety improvement, they require 396.8 supervised positions per sample and yield a normalized gain of only 0.039. In contrast, Pivot, Intent, and Risk achieve substantially higher normalized gains with far fewer supervised positions. While not a matched-budget comparison, these results suggest that late-stage supervision has relatively low training utility, further supporting TRR-guided rollout scheduling.

W1-2: Fairness of Maximum Rollout Length

We acknowledge that the original maximum rollout lengths differed between OPSD (4096) and EOPSA (1024). To ensure a fair comparison, we reran OPSD-1024, matching the maximum length while keeping training data, optimization steps, and other configurations unchanged.

MethodHarmB. ↑WildC. ↑WildJ. ↑StrongR. ↑Safety Avg. ↑Toks/Smp ↓Gen Time ↓Train Time ↓
OPSD-102498.0070.2772.8095.2184.07608.640.46 h0.25 h
EOPSA99.5080.2788.0096.8191.157.410.26 h0.21 h

In practice, the average rollout length in our training logs is approximately 700 tokens, suggesting that the difference between the 1024- and 4096-token limits primarily affects longer trajectories. More importantly, under the matched 1024-token setting, EOPSA continues to achieve superior safety performance, supervision efficiency, and training-time efficiency, confirming that its advantages are not simply due to different maximum rollout lengths.

W2: Token Filtering Reliability and Benign Data Handling

W2-1: Why Does Context-free Token Filtering Work?

TF uses lightweight semantic rules to filter safety-irrelevant supervision without costly per-token contextual analysis. Importantly, context-free selection does not imply context-free supervision: TF determines where to distill, while the context-conditioned teacher determines what to learn. Thus, general words such as maybe or which may occasionally be retained, but they are not automatically optimized toward refusal. Our context-aware human validation (Table G1 in the Global Response) shows that most retained positions exhibit safety-related semantics, with few relevant positions missed. Furthermore, Tables G2–G3 demonstrate higher normalized safety gains for safety-related categories and superior performance over Random Mask and Top-KL under the same 2% token budget. These findings support both the semantic reliability and training value of TF.

W2-2: How Are Benign Queries Used in Training?

EOPSA uses mixed harmful and benign queries, applying the same TF rules but different privileged teacher guidance. The teacher encourages appropriate refusal for harmful queries and helpful responses for benign queries, correcting unnecessary refusals. For example, if the student tends to refuse a benign request, TF can retain refusal-related positions while the teacher's KL supervision shifts the distribution toward a helpful response. Thus, the same rules support both appropriate refusal and over-refusal correction. We will clarify the mixed-training protocol and revise Equation (4) accordingly; both teacher prompts are provided in Appendix H.

Q1: Fair Efficiency Comparison and TRR Validation Overhead

We acknowledge that Figure 5 includes methods with different training data budgets. For a fair comparison, we focus on OPSD, which uses the same training data and optimization steps as EOPSA, and additionally match its maximum rollout length to 1024 tokens.

MethodRollout Time ↓Train Time ↓Validation Time ↓
OPSD-10240.46 h0.25 h~0.098 h
EOPSA0.26 h0.21 h~0.098 h + 0.049 h (TRR)

As shown above, EOPSA maintains lower rollout and training costs under matched settings. TRR is evaluated only every 50 steps and reuses results from regular validation, introducing just 176.37 s (0.049 h) of additional computation across four evaluations. We will clarify these costs and the comparison settings in the revision.

Q2: Safety–Utility Trade-off on R1-7B

We thank the reviewer for highlighting the increased over-refusal on R1-7B. While EOPSA increases the False Refusal Rate (FRR) from 5.40% to 14.60%, it substantially improves safety from 75.63% to 96.14% (+20.51 points) while preserving general reasoning ability (66.07 → 66.71). This motivates a broader evaluation of the Safety–Utility trade-off, considering both benign-query responsiveness and reasoning ability.

Following the harmonic-mean approach in SAVeR²[1], we evaluate Safety and Benign Non-refusal, and further extend it to include Reasoning.

ModelMethodSafety ↑FRR ↓Reason ↑H(S,B) ↑H3 ↑
Qwen3-1.7BOPSD83.486.8060.4988.0776.45
EOPSA91.156.2061.7392.4679.30
Qwen3-4BOPSD94.184.0273.9295.0786.79
EOPSA98.086.2274.2995.8887.41
R1-7BOPSD75.635.4066.0784.0677.06
EOPSA96.1414.6066.7190.4580.86

Here, S = Safety, B = 100 − FRR (Benign Non-refusal), and R = Reasoning. We compute H(S,B) = 2SB/(S+B) and H3 = 3/(1/S + 1/B + 1/R).

EOPSA achieves higher composite scores across all three models. Notably, on R1-7B, H(S,B) increases from 84.06 to 90.45 and H3 from 77.06 to 80.86, indicating a better overall safety–utility balance under equal weighting, despite the increased over-refusal.

[1] Yichen Sun, Qirui Liu, Yan Wang, Jianan Lin, Linbo Jiang, Shiyu Wang, Zhibo Wang, and Zhixuan Chu. SAVeR²: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards. 2026. https://arxiv.org/abs/2611.07482

Dear Reviewer,

Thank you again for your thoughtful and constructive feedback, which we have carefully addressed in our author response. As the discussion deadline is approaching, we would greatly appreciate any further feedback you may have. If our responses have resolved your main concerns, we would also be grateful if you could update your overall assessment accordingly.

Sincerely,

The Authors

Reviewer_erSCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies two key bottlenecks in LLM alignment with OPSD: (1) supervisory collapse over extended rollouts and (2) gradient dilution from stylistic shifts. To tackle the problem, the authors propose EOPSA, an efficient on-policy self-distillation method for LLM safety alignment. EOPSA introduces two modules: (1) Adaptive Rollout Scheduling, which adjusts rollout length based on the Teacher Rescue Rate, and (2) Selective Distillation, which calculates loss only on safety-critical tokens. Experiments on multiple reasoning models and across multiple benchmarks demonstrate the effectiveness of the method.

Critical strengths
  • The studied problem is very important, and the presentation of the paper is very nice
  • The experiments in the paper are comprehensive, covering the main results, ablation studies, and efficiency evaluation
Critical weaknesses
  • The definition of safety-relevant tokens is not very clear. For example, even if a model has generated words such as "refuse" and "harmful", these tokens might be task-relevant but are not involved in any safety decisions. Although Appendix F.9 reports a high agreement between humans and the classifier, it can only show that the classification of these words is semantically correct, but does not mean they (causally) determine the behavior of the models.
Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response: Semantic Relevance and Effectiveness of Token Filtering

We thank the reviewer for highlighting the distinction between semantic relevance and causal importance. We clarify the objective of Token Filtering (TF) and provide further evidence of its effectiveness.

(1) Design objective and lightweight selection.

TF aims to coarsely filter out safety-irrelevant positions, rather than precisely identify the causal importance of every token. Performing context-aware semantic or causal assessment at every position would incur substantial computational overhead. Instead, TF uses lightweight, model-family-aware semantic rules constructed offline. Importantly, TF determines where to distill, while the context-conditioned teacher determines what to learn. Therefore, even if a few safety-irrelevant positions are retained, they are not automatically optimized toward refusal, as the teacher's distribution, rather than the filtering rules, determines the optimization target.

(2) Semantic reliability on actual training rollouts.

To assess whether selected positions remain semantically safety-relevant in context, we manually evaluated 640 token positions from 40 actual training rollouts, comparing TF's selections with full-context human judgments of safety relevance (Pivot, Intent, or Risk).

Training StepPrecision ↑Recall ↑F1 ↑Accuracy ↑
Step 0100.0100.0100.0100.0
Step 5091.9100.095.898.1
Step 15088.2100.093.898.8
Step 20093.8100.096.899.4
Overall94.7100.097.399.1

These results indicate that most retained positions carry safety-related semantics, with few relevant positions missed. Consistently, Appendix F.9 reports 94.2% F1 on 600 held-out human-annotated positions.

(3) Empirical training utility of selected positions.

Beyond semantic relevance, the category-level ablation (Table G2 in our Global Response) shows substantially higher normalized safety gains for Pivot, Intent, and Risk than for safety-neutral categories, although their supervision budgets differ. More importantly, under the same 2% token budget and fixed rollout horizon (Table G3), TF achieves 90.02% average safety, outperforming Top-KL (85.27%) and Random Mask (80.36%), while achieving the best reasoning retention (62.77%). Furthermore, Figure 4(b) shows that TF retains only 1.5% of token positions and 7.3% of KL mass, yet outperforms full-token distillation. Together, these results support lightweight semantic selection as an effective and scalable approximation that improves supervision efficiency without requiring expensive token-level causal attribution.

Thanks for the authors' rebuttal. My concerns are addressed. I will keep my positive score.

Thank you very much for your valuable comments and suggestions, which have helped us improve our manuscript. We greatly appreciate your time and support!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response: Clarifications and Additional Evidence

Global Response: Clarifications and Additional Evidence

We thank the reviewers for their constructive feedback. Here, we clarify our research motivation, address shared concerns, and summarize the additional experimental evidence. Reviewer-specific concerns are addressed in the individual responses.

1. Research Motivation and Contributions

EOPSA addresses two sources of inefficient supervision in existing On-Policy Self-Distillation (OPSD):

  • Safety-irrelevant distribution shifts. Privileged prompting introduces substantial teacher–student distribution differences unrelated to safety decisions, including stylistic variations. Thus, high KL divergence does not necessarily indicate valuable safety supervision, and full-token distillation may dilute safety-critical signals.
  • Degraded teacher recoverability. As unsafe student prefixes accumulate, the privileged teacher becomes less capable of steering the complete response back toward safety, as reflected by the declining Teacher Rescue Rate (TRR). Thus, longer rollouts do not necessarily provide more effective supervision.

EOPSA addresses these issues through two complementary mechanisms. Adaptive Rollout Scheduling (ARS) dynamically adjusts the rollout horizon based on TRR, reducing unnecessary generation and less reliable late-stage supervision. Token Filtering (TF) concentrates distillation on safety-relevant positions, reducing safety-irrelevant updates while better preserving general reasoning capabilities.

Therefore, EOPSA improves both computational efficiency through ARS and supervision efficiency through TF.

2. Semantic Rule Design and Empirical Validation

Rule construction and motivation. TF aims to filter safety-irrelevant supervision rather than precisely identify the causal importance of every token. Performing context-aware semantic judgments at every token position would require substantial additional inference, limiting scalability. Instead, we extract recurring safety-related patterns from sampled student–teacher top-1 prediction pairs and construct lightweight, model-family-aware rules covering three categories: Pivot, Intent, and Risk.

Importantly, context-free selection does not imply context-free supervision. TF determines which positions receive supervision, while the teacher, conditioned on the full context and privileged guidance, determines the optimization target. Thus, even if some safety-irrelevant positions are retained, the rules do not themselves enforce refusal behavior.

(1) Context-aware human validation on actual training rollouts.

To assess whether context-free TF rules identify safety-relevant positions in actual training contexts, we manually evaluated 640 sampled token positions from 40 training rollouts using full-context human annotations of safety relevance (Pivot, Intent, or Risk).

Table G1. Context-aware human validation of TF on actual training rollouts.

Training StepPrecision ↑Recall ↑F1 ↑Accuracy ↑
Step 0100.0100.0100.0100.0
Step 5091.9100.095.898.1
Step 15088.2100.093.898.8
Step 20093.8100.096.899.4
Overall94.7100.097.399.1

The results show strong agreement with contextual human judgments (94.7% precision and 97.3% F1). Consistently, Appendix F.9 reports 94.2% F1 on 600 held-out human-annotated positions.

(2) Supervision efficiency across semantic categories.

To examine whether the semantic categories also differ in training utility, Table G2 summarizes the category-level ablation from Table 4, reporting safety gains and supervision budgets:

Table G2. Safety supervision efficiency across token categories.

Token CategorySafety Avg. ↑Tokens/Sample ↓ΔSafety/Token ↑
Consistent79.27115.170.22
Function75.631.7911.85
Other79.127.133.47
Pivot77.970.3371.39
Intent89.320.4577.58
Risk83.941.1425.90

Pivot, Intent, and Risk exhibit substantially higher normalized safety gains, supporting their value as supervision targets.

(3) Effectiveness under a matched token budget.

To test whether TF's advantage comes from semantic selection rather than supervision sparsity alone, Table G3 summarizes the comparison in Appendix F.4 (Table 8) with Random Mask and Top-KL under an approximately matched 2% token budget and a fixed rollout horizon:

Table G3. Comparison of token selection strategies under a matched 2% supervision budget.

Selection StrategySafety Avg. ↑Reasoning Avg. ↑
Random Mask80.3661.33
Top-KL85.2761.63
Ours (TF)90.0262.77

TF outperforms Top-KL by 4.75 points and Random Mask by 9.66 points in safety, while achieving the best reasoning retention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.