acceptodds
Under review as a conference paper at ICLR 2027

Seed-Level Instability in LLM Safety Alignment: Regime Shifts in Large-Margin Preference Optimization

Abstract

As large language models (LLMs) are deployed more widely, reliable safety alignment becomes increasingly important. However, we identify seed-level safety-regime instability in large-margin, label-based SafeDPO: different training seeds produce sharply different refusal profiles. Across 20 seeds on Qwen2.5-7B-Instruct, OR-Toxic prefix-refusal rates range from 1.83% to 98.47% at a fixed endpoint, while instruction-following accuracy remains comparatively stable. Cross-seed averages obscure these differences in refusal behavior. We introduce Support-Completion SafeDPO (SC-SafeDPO), which restores discarded both-unsafe comparisons using their annotated relative-safety ordering, zero additional margin, and a fixed positive weight. The method preserves the original main-comparison orderings and margins and requires no additional inference-time computation. A local affine analysis relates comparison coverage to seed sensitivity. Budget-matched ablations indicate that both mixture reweighting and restored preference gradients contribute to stability. In the paired 20-seed comparison, SC-SafeDPO reduces cross-seed refusal-rate variance by approximately 77% on XSTest unsafe prompts and 87% on OR-Toxic relative to Label SafeDPO. It also lowers mean HarmBench attack success and benign false-refusal rates and improves instruction-following accuracy. Evaluations across three model backbones examine safety–helpfulness trade-offs and cross-seed variability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Safety Alignment, Training Instability, Random Seed Sensitivity, Support Completion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jAHkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that large-margin Label SafeDPO suffers seed-level safety-regime instability: with a fixed recipe (Qwen2.5-7B-Instruct, δ=10), 20 seeds give OR-Toxic prefix-refusal rates from 1.83% to 98.47%, while instruction-following stays stable, so cross-seed means hide individual checkpoints' behavior. The proposed SC-SafeDPO restores the both-unsafe pairs discarded by Label SafeDPO, using their annotated relative-safety ordering with zero extra margin and a fixed weight α — no change to main orderings/margins, no inference cost. In a paired 20-seed study it cuts cross-seed refusal variance by ~77% (XSTest unsafe) and ~87% (OR-Toxic), with better means on all five metrics; cross-backbone results and a blinded human audit are also reported.

Critical strengths
  1. Simple intervention, consistent main result. Only reuses existing annotations; the 20-seed paired, fixed-endpoint protocol shows the same direction on all five metrics.

  2. Thorough ablations. Null / shuffled / true support under matched budgets separate mixture reweighting from restored gradients, plus δ=10 vs 5, exposure-matched control, gradient/CKA diagnostics, and a cross-backbone cohort.

  3. Complete reproducibility. Anonymous repo plus appendices covering configs, data/loss definitions, seed protocols, generation settings, scoring rules, and statistics.

Critical weaknesses
  1. Figure 1 has no error bars and conflicts with Sec. 4. The caption claims "±1 standard deviation across training seeds," but none are drawn (vector inspection confirms the short segments are just marker edges). The figure also never states its cohort/seed count, and its values match neither Table 2 nor Table 4. Fix the plot or the caption, and label the data source.

  2. WildGuard re-evaluation results are missing. §4.6 and App. E.1/C.3 promise a semantic re-score of the frozen Qwen responses, and Table 12 lists those metrics, but no such numbers appear — only Table 3's mechanism study and Table 4's Gemma rows use WildGuard. Evaluator sensitivity is key to the main claim; add the table or drop the description.

  3. "Improvement on all metrics" overstates the evidence. HarmBench ASR in Table 2 improves by just 0.68pp (SDs 3.90/1.19); Gemma IFEval (0.08pp) and StrongREJECT (0.32pp) in Table 4 are negligible. SC is worse than Label on Qwen2.5-1.5B safety and Gemma OR-Hard (8.95→11.58). App. C.4 mentions paired bootstrap, yet no CIs or p-values are reported. Add them and limit the claim to significant gains.

Questions for authors

same as weakness

3 comments

We appreciate the reviewer’s careful reading of our paper and the positive assessment of SC-SafeDPO, particularly its simple intervention, paired multi-seed evaluation, mechanism ablations, and reproducibility.

The comments also helped us identify three aspects that needed clearer presentation or additional evidence: the provenance and uncertainty visualization of Figure 1, the missing semantic evaluation of frozen Qwen responses, and the statistical support for our performance claims. We address these points below.

1. Figure 1: data provenance and uncertainty visualization

Thank you for pointing out the ambiguity in Figure 1. The figure and its original caption did not make the relationship to the cross-backbone results sufficiently clear.

Figure 1 is based on the Qwen2.5-7B-Instruct subset of the cross-backbone comparison in Table 4 (Section 4.5). Each scatter point represents an individual training seed, rather than a cross-seed mean. The horizontal axis shows safety, measured as 100 minus HarmBench attack success rate, and the vertical axis shows helpfulness, measured by IFEval accuracy.

This distinction also explains why the individual points do not coincide with the values in Table 4: the table reports cross-seed means and sample standard deviations, whereas the scatter plot displays the individual seed-level outcomes.

We have revised the figure to make the error bars more visible, with clearer line widths and end caps, and adjusted the plotting order to avoid overlap with the scatter markers. The caption now explicitly identifies the cross-backbone cohort, the number of seeds, and the meaning of the ±1 standard deviation error bars.

These changes make the figure's data source and its relationship to the aggregated results in Table 4 explicit.

2. Semantic evaluation of frozen Qwen responses

The reviewer also noted that the semantic re-evaluation described in Sections 4.6 and Appendices C.3 and E.1 was not accompanied by numerical results. We have completed this evaluation using WildGuard on the original 20-seed Qwen2.5-7B-Instruct fixed-endpoint study.

An important aspect of this evaluation is that the model checkpoints, generated responses, and evaluation subsets are unchanged. Only the evaluator is different. This allows us to examine whether the differences observed with prefix-based refusal detection are also reflected in semantic judgments, without introducing variation from additional training or response generation.

The results are summarized below. All values are percentages, reported as mean ± cross-seed sample standard deviation (percentage points).

Table R1. WildGuard semantic evaluation of frozen responses across 20 training seeds.

Semantic metricLabel SafeDPOSC-SafeDPO
OR-Toxic semantic refusal rate ↑91.29 ± 3.8297.80 ± 2.40
OR-Toxic harmful-response rate ↓0.313 ± 0.3920.153 ± 0.222
OR-Hard semantic false-refusal rate ↓48.98 ± 17.4140.10 ± 11.30

The results show that SC-SafeDPO has a higher semantic refusal rate on harmful prompts (97.80% vs. 91.29%), while producing fewer semantically identified harmful responses (0.153% vs. 0.313%). On benign OR-Hard prompts, its semantic false-refusal rate is also lower (40.10% vs. 48.98%).

The reduction in cross-seed dispersion is particularly relevant to the concern raised in this review. For OR-Toxic semantic refusal, the sample standard deviation decreases from 3.82 to 2.40 percentage points, corresponding to an approximately 60.5% reduction in sample variance. Dispersion also decreases for the other two semantic metrics.

We distinguish this result from the approximately 87% variance reduction reported in the original paper, which refers to OR-Toxic prefix-based refusal. The two numbers come from different evaluation procedures and should not be conflated.

Taken together, the WildGuard results provide an independent semantic assessment of the same model outputs. The direction of the differences is consistent with the original findings, including both improved harmful-response avoidance and fewer benign false refusals.

3. Statistical evidence and interpretation of improvement claims

The reviewer raises an important point about the scope of our original statement that SC-SafeDPO improves all reported metrics. Although the observed mean differences in Table 2 favor SC-SafeDPO, this alone does not establish that every difference is statistically distinguishable from zero.

To make this distinction clearer, we have added paired uncertainty estimates for the primary 20-seed fixed-endpoint comparison.

For each metric, we calculate the mean difference between SC-SafeDPO and Label SafeDPO, as well as the ratio of their cross-seed sample variances. Confidence intervals are obtained using paired bootstrap resampling, with training-seed correspondence preserved in every replicate.

Table R2. Paired bootstrap uncertainty estimates for the primary 20-seed comparison.

MetricMean difference (pp)95% CIVariance ratio95% CI
OR-Toxic prefix refusal ↑+41.45[26.19, 52.84]0.131[0.062, 0.241]
HarmBench attack success ↓−0.68[−2.35, 0.97]0.093[0.037, 0.215]
OR-Hard prefix false refusal ↓−1.24[−8.26, 6.34]0.447[0.215, 0.913]
IFEval strict prompt accuracy ↑+1.92[1.32, 2.66]0.743[0.352, 1.712]

The paired intervals provide a more nuanced picture of the results.

For mean performance, the improvements in OR-Toxic prefix refusal and IFEval are supported by confidence intervals that exclude zero. The intervals for HarmBench attack success and OR-Hard false refusal include zero. We therefore interpret these two results as favorable observed mean differences, rather than statistically established improvements.

The variance estimates tell a somewhat different story. The confidence intervals for the OR-Toxic, HarmBench, and OR-Hard variance ratios all lie below one, indicating lower cross-seed variance for SC-SafeDPO across these three safety-related evaluations. For IFEval, the interval includes one, so the evidence does not establish a reduction in its cross-seed variance.

We have refined the corresponding claims to reflect these distinctions. In particular, we no longer present every favorable mean difference as an established improvement. Instead, we emphasize the more consistently supported finding: SC-SafeDPO substantially reduces variation in safety-related behavior across training seeds, while improving mean performance on several key evaluations.

This distinction is important to our central motivation. A preference optimization method should not only achieve favorable average safety scores, but also produce reasonably consistent safety behavior across independent training runs.

4. Overall implications for the revised paper

The additional evaluation and statistical analysis strengthen the empirical picture in complementary ways.

Figure 1 now makes the relationship between individual seed outcomes and cross-seed statistics explicit. The WildGuard evaluation extends the original prefix-based analysis to semantic judgments on the same frozen responses. The paired bootstrap analysis, in turn, clarifies which observed changes in mean performance and cross-seed variability have direct statistical support.

More broadly, these results reinforce the motivation for SC-SafeDPO: restoring the relative-safety information in both-unsafe comparisons is associated not only with improved average safety behavior, but also with more consistent outcomes across training seeds.

We appreciate the reviewer for identifying these issues. Addressing them has helped us present the experimental evidence more clearly and, importantly, sharpen the scope of the conclusions supported by our results.

Dear Reviewer jAHk,

Thank you again for your constructive comments. We have addressed your main concerns in the rebuttal and plan to incorporate them into the revision as follows: (i) Figure 1 will be redrawn with clearly visible error bars, and its caption will specify the cross-backbone cohort, the number of seeds, and the data source; we will also clarify that the plotted points correspond to individual seeds rather than the cross-seed means reported in Tables 2 and 4; (ii) the missing WildGuard semantic evaluation on the same frozen 20-seed Qwen responses will be added to §4.6 and Appendix E.1, including means, cross-seed standard deviations, and variance ratios; and (iii) paired bootstrap 95% confidence intervals for mean differences and variance ratios will be added to Appendix C.4 and summarized near Table 2.

For related points, we have provided explanations to Reviewer xU2P regarding the margin-sensitivity results and the expanded 20-seed cross-backbone evaluation, and to Reviewer xVgV regarding the semantic evaluator, fixed endpoints, and confidence intervals. These responses are closely related to your concerns and may help contextualize our reply; we would be grateful if you could review them when convenient. As the rebuttal period is approaching its end, we would greatly appreciate any further feedback or indication of whether our responses have addressed your concerns. We remain happy to provide any additional clarifications. Thank you very much for your time and consideration.

Reviewer_xVgVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates cross-seed variability in large-margin Label SafeDPO. It reports sharply different refusal profiles across training seeds despite comparatively stable instruction-following performance. SC-SafeDPO restores discarded both-unsafe comparisons using their annotated relative-safety ordering, zero additional margin, and a fixed weight. The paper reports reduced cross-seed variability in a 20-seed comparison and provides a local affine analysis and mechanism ablations.

Critical strengths
  1. The paper addresses a worthwhile reliability question: whether an alignment method produces consistent safety behavior across independent training runs. The 20-seed evaluation is a useful contribution beyond reporting average performance from a few runs.
  2. The method is simple and uses existing annotations without additional inference-time computation. The null, shuffled, and true-support conditions, together with the exposure-matched control, are sensible attempts to distinguish reweighting from additional preference supervision.
  3. The mathematical analysis is generally careful about its scope. The paper explicitly distinguishes local affine SGD dynamics from the empirical AdamW setting and does not claim that its theorem guarantees a cross-method reduction in behavioral variance.
Critical weaknesses
  1. The interpretation of the largest variability result depends heavily on the refusal evaluator. OR-Toxic is evaluated using a small, case-sensitive prefix inventory. This can conflate changes in refusal wording with changes in whether a response actually refuses or provides harmful assistance. Section 4.6 and Appendix E.1 describe a frozen-response WildGuard audit, but the corresponding semantic means, cross-seed standard deviations, and variance ratios for the 20-seed cohort are not presented. HarmBench provides useful independent evidence of variability reduction, but does not establish that the much larger OR-Toxic effect reflects comparable changes in harmful behavior. The human audit supports average response quality, rather than the claimed reduction in cross-seed dispersion.

  2. The fixed-endpoint comparison is not specified sufficiently to separate the method effect from training duration. Table 2 evaluates the two methods at their respective endpoints, but Appendix C.1 does not give the exact update counts or explain how those endpoints were chosen. Fixing an endpoint across seeds removes per-seed checkpoint reselection, but does not ensure comparable training exposure between methods. The matched-budget and exposure-matched experiments in Table 3 partially address this concern, although they use a separate cohort. A clear account of the endpoints and their selection is needed to interpret the primary 20-seed result.

  3. The uncertainty of the main improvements remains unclear. Appendix C.4 describes paired bootstrap estimation, but the paper does not report confidence intervals for the headline variance ratios or paired mean differences. For example, Table 2 reports HarmBench ASR changing from (9.39%) to (8.71%), and OR-Hard false refusal from (14.80%) to (13.56%), alongside substantial cross-seed variation. These observed mean improvements should be distinguished from statistically well-supported improvements. The reported variance reductions are promising, but their uncertainty is also important with 20 runs.

Questions for authors
  1. How much of SC-SafeDPO’s stability gain can be attributed to restored preference supervision, rather than reduced main-objective weighting or differences in training duration? The controls in Table 3 are helpful, but how do they establish this distinction for the main 20-seed comparison? Does SC retain an advantage over the smaller-margin baseline at comparable safety–helpfulness trade-offs?

  2. Which support weight was actually used in the main 20-seed comparison? α=1 or α=0.5? Please reconcile Section 4.1, Appendix B.2, and Table 8, and specify the exact training endpoints and how they were selected for each method. The caption of Figure 1 mentions error bars, but they are difficult to discern in the plot. Could you clarify how they are displayed or revise the figure to make them more clearly visible?

  3. What does one row in Table 14 represent, and how does the 378-row annotation summary relate to the 678 scored SafeDPO/SC responses in Table 15? Please clarify whether the agreement analysis covers all responses or a subset, and report the original response counts and ambiguity exclusions by method and endpoint.

7 comments

We thank the reviewer for identifying several issues that affect how the empirical results should be interpreted. We conducted additional experiments to examine whether the observed stability improvement reflects actual refusal behavior rather than changes in response wording, and whether the results extend beyond the original Qwen2.5-7B setting.

R1. Semantic evaluation of refusal behavior

We agree that prefix-based refusal detection has limitations: a response may express a refusal without using one of the predefined prefixes, or use refusal-like wording while still providing harmful assistance. To examine whether the observed effect extends beyond lexical matching, we applied WildGuard to the frozen responses from the original 20-seed Qwen2.5-7B-Instruct Label–SC comparison. Importantly, the checkpoints, generated responses, and evaluation subsets are unchanged. We report semantic refusal and harmful-response rates separately, along with false refusals on benign prompts.

Table R1. Semantic evaluation on the original 20-seed cohort (mean ± sample SD, %).

MetricLabel SafeDPOSC-SafeDPO
OR-Toxic semantic refusal ↑91.29 ± 3.8297.80 ± 2.40
OR-Toxic harmful responses ↓0.313 ± 0.3920.153 ± 0.222
OR-Hard semantic false refusal ↓48.98 ± 17.4140.10 ± 11.30

The semantic evaluation is consistent with the prefix-based results. SC-SafeDPO achieves a higher mean refusal rate on OR-Toxic, while producing fewer harmful responses. On benign OR-Hard prompts, it also has a lower false-refusal rate. More importantly for the concern raised here, the cross-seed standard deviations decrease for all three metrics. The variance of semantic OR-Toxic refusal is reduced by 60.5%.

These results suggest that the stability difference is not merely an artifact of the prefix detector. We have added the semantic measurements to Section 4.6 and the corresponding evaluation details to Appendix E.1.

R2. Cross-backbone generalization

We extended the Qwen2.5-1.5B-Instruct and Gemma-3-12B-IT experiments from three to 20 training seeds per configuration, retaining the original runs. This provides a more informative estimate of cross-seed variability and allows us to examine whether the observed changes are consistent across different model sizes and families.

Table R2. Expanded cross-backbone evaluation (mean ± sample SD, %).

BackboneMethodSeedsHarmBench ASR ↓OR-Hard false refusal ↓†IFEval ↑
Qwen2.5-1.5BLabel SafeDPO208.5 ± 0.7087.5 ± 1.835.6 ± 1.1
Qwen2.5-1.5BSC-SafeDPO208.9 ± 0.5585.1 ± 1.837.3 ± 1.0
Gemma-3-12BLabel SafeDPO2026.7 ± 2.111.053 ± 1.87962.845 ± 0.770
Gemma-3-12BSC-SafeDPO2026.0 ± 1.610.825 ± 1.66163.769 ± 0.747

† OR-Hard uses prefix-based false-refusal detection for Qwen2.5-1.5B and semantic evaluation for Gemma-3-12B. Comparisons are made within each backbone.

The results are not identical across the two backbones. On Qwen2.5-1.5B, SC-SafeDPO reduces the variability of HarmBench ASR and improves OR-Hard false refusal and IFEval, although its mean HarmBench ASR is slightly higher. On Gemma-3-12B, SC-SafeDPO improves all three reported mean metrics, with lower standard deviations as well. The support weight used for Gemma is α = 0.5.

We think this distinction is important. The additional experiments support the relevance of support completion beyond the original 7B setting, but also show that its effect on mean safety and helpfulness depends on the backbone. We have included the expanded results and individual-seed measurements in the revised cross-backbone evaluation and appendix.

The additional experiments also address two methodological questions: whether the observed gains can be explained by adjusting the safety margin alone, and whether the primary comparison uses comparable training budgets.

R3. Margin sensitivity

One important question is whether the stability improvement can be obtained simply by reducing the safety margin. We therefore expanded the margin-sensitivity experiment to 20 seeds per configuration. Label SafeDPO was evaluated with δ ∈ {0, 2.5, 5, 7.5, 10}, while SC-SafeDPO used δ = 10. All configurations were trained for 1,000 optimizer updates under the same mechanism-study protocol.

Table R3. Margin sensitivity on Qwen2.5-7B-Instruct (mean ± sample SD, %).

MethodδHarmBench ASR ↓XSTest unsafe refusal ↑Semantic OR-Hard false refusal ↓IFEval ↑
Label SafeDPO016.4 ± 2.667.5 ± 7.332.0 ± 5.867.1 ± 1.0
Label SafeDPO2.512.8 ± 1.975.2 ± 6.439.0 ± 7.067.7 ± 0.9
Label SafeDPO510.2 ± 1.878.4 ± 9.546.0 ± 9.768.0 ± 1.1
Label SafeDPO7.58.8 ± 1.776.7 ± 12.051.7 ± 10.068.1 ± 1.0
Label SafeDPO107.9 ± 1.972.3 ± 20.558.0 ± 11.267.7 ± 1.2
SC-SafeDPO107.3 ± 1.185.4 ± 3.743.0 ± 7.568.5 ± 0.9

As expected, the margin has a substantial effect on both mean performance and variability. In particular, reducing δ from 10 to 5 lowers the XSTest unsafe-refusal standard deviation from 20.5 to 9.5 percentage points. Thus, a smaller-margin Label SafeDPO is a meaningful baseline for assessing the contribution of support completion.

Compared with Label SafeDPO at δ = 5, SC-SafeDPO achieves lower mean HarmBench ASR (7.3% vs. 10.2%), higher XSTest unsafe refusal (85.4% vs. 78.4%), lower semantic OR-Hard false refusal (43.0% vs. 46.0%), and higher IFEval accuracy (68.5% vs. 68.0%). Its standard deviations are also lower on all four metrics.

We further performed paired comparisons across the 20 seeds.

Table R4. Paired comparison between SC-SafeDPO and Label SafeDPO (δ = 5). Δ denotes SC minus Label; R denotes the sample variance ratio.

MetricΔ (pp)95% CI of ΔR95% CI of R
HarmBench ASR ↓−2.82[−3.71, −1.94]0.364[0.148, 0.944]
XSTest unsafe refusal ↑+7.00[2.30, 11.70]0.152[0.060, 0.383]
Semantic OR-Hard false refusal ↓−3.02[−8.60, 2.56]0.598[0.237, 1.510]
IFEval ↑+0.43[−0.29, 1.14]0.658[0.265, 1.691]

The paired confidence intervals distinguish the metrics more clearly: the mean differences for HarmBench ASR and XSTest unsafe refusal exclude zero, whereas those for semantic OR-Hard false refusal and IFEval do not. We therefore interpret the latter two mean differences more cautiously. The variance ratios nevertheless favor SC-SafeDPO, although their confidence intervals also reflect uncertainty with 20 seeds.

The comparison is useful because reducing the margin and restoring support comparisons change different aspects of the training objective. Margin adjustment changes the strength of the existing safety constraint. SC-SafeDPO additionally retains the relative-safety ordering of both-unsafe comparisons that Label SafeDPO discards. The results indicate that this additional supervision contributes beyond margin adjustment alone.

The expanded results and paired estimates are included in the revised experimental section and appendix.

R4. Training endpoints and training budgets

The two methods use different training-set sizes: 11,144 pairs for Label SafeDPO and 21,486 pairs for SC-SafeDPO. Their stopping fractions are defined relative to their respective nominal three-epoch schedules, rather than using the same fraction for both methods.

ProtocolLabel SafeDPOSC-SafeDPO
Training pairs11,14421,486
Nominal total steps (3 epochs)4,1798,058
Stopping fraction0.250.125
Actual optimizer updates1,0451,007
Non-padding training tokens, mean per seed1.888M1.911M

The resulting training budgets are closely matched: 1,045 versus 1,007 optimizer updates, and 1.888M versus 1.911M non-padding tokens per seed. Using the same stopping fraction would instead give SC-SafeDPO substantially more updates; for example, a fraction of 0.25 would result in approximately 2,014 updates.

The stopping fractions are fixed across seeds within each method. Each checkpoint is evaluated at its designated endpoint, without selecting a different checkpoint for individual seeds. The primary Qwen2.5-7B comparison uses δ = 10 and α = 1. The corresponding cosine-schedule horizons are 4,179 updates for Label SafeDPO and 8,058 for SC-SafeDPO, with 126 and 242 warmup updates, respectively.

The mechanism ablations are a separate ten-paired-seed study. Their primary conditions use 1,000 optimizer updates with matched token budgets. The additional exposure-matched control extends true-support training to 1,928 updates, matching the cumulative main-comparison exposure of Label SafeDPO at 1,000 updates.

We have clarified these distinctions in Section 4.1 and Appendices B.2, B.3, C.1, and D.1. In particular, the revised description now gives the actual update counts and token exposure rather than only the nominal stopping fractions.

Reviewer_xU2POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies seed-level instability in large-margin Label SafeDPO, showing that identical training recipes can yield sharply different refusal profiles even when instruction-following accuracy remains comparatively stable, and argues that safety alignment should therefore be evaluated across training-seed distributions.

Critical strengths

The paper exposes an important and interesting question. It is a deployment-relevant failure mode that would be hidden by standard few-seed or mean-only reporting.

Critical weaknesses

Limited generalizability of stability improvements. The central 20-seed experiments and mechanism ablations are conducted exclusively on Qwen2.5-7B. The cross-backbone evaluation uses only three seeds per configuration, which is particularly insufficient for a paper whose central claim concerns seed-level variability. Moreover, the results are inconsistent across models. On Qwen2.5-1.5B, SC-SafeDPO increases both the mean and standard deviation of HarmBench attack success. These findings cast doubt on whether the proposed method consistently improves safety alignment stability across models and scales.

Unclear necessity of SC-SafeDPO. The reported instability is primarily demonstrated with a large safety margin (δ = 10). However, Table 3 shows that simply reducing δ from 10 to 5 decreases the standard deviation of XSTest unsafe refusal from 21.21 to 9.92, without introducing additional preference data. This raises the question of whether the observed instability mainly results from an overly aggressive margin setting rather than an inherent limitation of Label SafeDPO.

Questions for authors

Given that seed-level variability is the central focus of this paper, why are the cross-backbone experiments limited to only three seeds per configuration? Can the authors provide more extensive multi-seed evaluations, particularly for Qwen2.5-1.5B, where SC-SafeDPO increases both the mean and variance of HarmBench attack success?

Table 3 shows that reducing the safety margin from δ = 10 to δ = 5 substantially reduces seed-level variability. Have the authors systematically evaluated different margin values? How does SC-SafeDPO compare against a properly tuned Label SafeDPO baseline?

4 comments

We thank the reviewer for highlighting two important questions: whether the stability improvements extend beyond Qwen2.5-7B, and whether they can be explained by choosing a smaller safety margin. These questions go to the scope and necessity of SC-SafeDPO, so we expanded the evaluation in both directions.

The additional experiments cover 20-seed evaluations on Qwen2.5-1.5B-Instruct and Gemma-3-12B-IT, a 20-seed margin-sensitivity study on Qwen2.5-7B-Instruct, and paired uncertainty estimates. The results show that the effect of SC-SafeDPO depends on the backbone and metric. They also show that reducing the margin helps Label SafeDPO, but does not reproduce all the improvements observed with support completion.

R1. Cross-backbone generalization and multi-seed evaluation

The reviewer is right that three seeds per configuration are not sufficient to characterize seed-level variability, particularly when variability is central to the paper's claim. We therefore extended the Qwen2.5-1.5B-Instruct and Gemma-3-12B-IT evaluations from three to 20 training seeds per configuration, retaining the original runs. The results below report the mean and sample standard deviation across the 20 independently trained checkpoints.

Table R1. Expanded cross-backbone evaluation. Results are mean ± sample standard deviation across 20 training seeds. Means and standard deviations are reported in percentage points.

BackboneMethodSeedsHarmBench ASR ↓OR-Hard false refusal ↓†IFEval ↑
Qwen2.5-1.5BLabel SafeDPO208.5 ± 0.7087.5 ± 1.835.6 ± 1.1
Qwen2.5-1.5BSC-SafeDPO208.9 ± 0.5585.1 ± 1.837.3 ± 1.0
Gemma-3-12BLabel SafeDPO2026.7 ± 2.111.053 ± 1.87962.845 ± 0.770
Gemma-3-12BSC-SafeDPO (α = 0.3)2026.0 ± 1.610.825 ± 1.66163.769 ± 0.747

† OR-Hard uses prefix-based false-refusal detection for Qwen2.5-1.5B and semantic false-refusal evaluation for Gemma-3-12B. Comparisons are made within each backbone.

On Qwen2.5-1.5B-Instruct, SC-SafeDPO does not improve every metric. Mean HarmBench ASR increases slightly, from 8.5% to 8.9%, although its sample standard deviation decreases from 0.70 to 0.55 percentage points. The method also lowers mean OR-Hard false refusal from 87.5% to 85.1% and raises IFEval accuracy from 35.6% to 37.3%. Thus, the additional seeds do not support a claim of uniform safety improvement on this backbone. They instead show lower observed dispersion alongside a mixed change in mean performance.

The results on Gemma-3-12B-IT are more consistent across the three reported metrics. With α = 0.3, mean HarmBench ASR decreases from 26.7% to 26.0%, semantic OR-Hard false refusal decreases from 11.053% to 10.825%, and IFEval accuracy increases from 62.845% to 63.769%. The corresponding sample standard deviations are also lower for SC-SafeDPO on all three metrics.

Taken together, the expanded evaluation provides multi-seed evidence on two additional backbones. It suggests that reduced seed-level dispersion is not unique to Qwen2.5-7B, while also making clear that changes in mean safety and helpfulness are backbone- and metric-dependent. We have therefore kept the claim narrower: SC-SafeDPO can improve cross-seed reliability, but it does not guarantee improvement on every metric or architecture.

The revised Appendix will include the seed-wise results, checkpoint details, and experimental configurations for both backbones, so that the variability and mean-performance changes can be examined separately.

The first part addressed the concern about cross-backbone generalization. This second part focuses on whether the observed instability can be addressed simply by tuning the safety margin, or whether SC-SafeDPO contributes additional supervision.

R2. Margin sensitivity and comparison with a tuned Label SafeDPO baseline

The reviewer notes that reducing the safety margin from δ = 10 to δ = 5 already lowers the XSTest unsafe-refusal standard deviation in the original mechanism study. This is an important alternative explanation. To test it directly, we expanded the margin-sensitivity study to 20 paired training seeds. Label SafeDPO was evaluated at δ ∈ {0, 2.5, 5, 7.5, 10}, alongside SC-SafeDPO at δ = 10 and α = 1. All configurations use the same 1,000-update protocol.

Table R2. Margin sensitivity on Qwen2.5-7B-Instruct. Results are mean ± sample standard deviation across 20 training seeds. Values are percentages; standard deviations are in percentage points.

MethodδHarmBench ASR ↓XSTest unsafe refusal ↑Semantic OR-Hard false refusal ↓IFEval ↑
Label SafeDPO016.4 ± 2.667.5 ± 7.332.0 ± 5.867.1 ± 1.0
Label SafeDPO2.512.8 ± 1.975.2 ± 6.439.0 ± 7.067.7 ± 0.9
Label SafeDPO510.2 ± 1.878.4 ± 9.546.0 ± 9.768.0 ± 1.1
Label SafeDPO7.58.8 ± 1.776.7 ± 12.051.7 ± 10.068.1 ± 1.0
Label SafeDPO107.9 ± 1.972.3 ± 20.558.0 ± 11.267.7 ± 1.2
SC-SafeDPO107.3 ± 1.185.4 ± 3.743.0 ± 7.568.5 ± 0.9

The margin sweep confirms that δ affects both mean performance and seed variability. In particular, reducing δ from 10 to 5 substantially lowers the variability of XSTest unsafe refusal. Margin tuning is therefore a meaningful way to mitigate part of the instability; it should be treated as a strong baseline rather than dismissed.

The comparison with δ = 5 nevertheless favors SC-SafeDPO on the four reported mean metrics, and its sample standard deviations are lower across all four. To account for the paired seeds, we also computed paired bootstrap confidence intervals.

Table R3. Paired comparison of SC-SafeDPO with Label SafeDPO at δ = 5. Each configuration contains 20 training seeds. Δ is the paired mean difference (SC minus Label), in percentage points; R is Var(SC) / Var(Label).

MetricΔ (pp)95% CI of ΔVariance ratio R95% CI of R
HarmBench ASR ↓−2.82[−3.71, −1.94]0.364[0.148, 0.944]
XSTest unsafe refusal ↑+7.00[2.30, 11.70]0.152[0.060, 0.383]
Semantic OR-Hard false refusal ↓−3.02[−8.60, 2.56]0.598[0.237, 1.510]
IFEval ↑+0.43[−0.29, 1.14]0.658[0.265, 1.691]

The paired intervals support improvements in mean HarmBench ASR and XSTest unsafe refusal, together with lower XSTest cross-seed variance. The intervals for semantic OR-Hard false refusal and IFEval include zero, so these should be interpreted as favorable observed differences rather than established mean improvements.

The distinction between margin tuning and support completion lies in the supervision used for training. Reducing δ changes the strength of the safety margin on comparisons already retained by Label SafeDPO; it does not restore the discarded both-unsafe comparisons or their relative-safety annotations. SC-SafeDPO preserves the original main-comparison orderings and margins while adding these comparisons with zero additional margin and a fixed positive weight.

The mechanism ablations provide evidence about the contributions of this additional supervision. Under matched optimizer updates and token budgets, null support isolates mixture reweighting without adding support preference gradients. Shuffled and true support use the same support pairs and weights but differ in preference orientation. The results show that mixture reweighting contributes to lower XSTest dispersion, while adding support preference gradients provides a further reduction. The annotated ordering also matters for mean safety outcomes: true support performs better than shuffled support on HarmBench ASR and semantic OR-Hard false refusal. The exposure-matched control shows that the lower variability persists when cumulative main-comparison exposure is matched.

These ablations do not establish a complete causal account of every seed-level outcome. They do, however, indicate that the observed effects cannot be attributed solely to margin adjustment or reduced exposure to the original main comparisons.

The revised Appendix will include the full margin-sweep results, paired bootstrap procedure, and details of the matched-budget and exposure-matched controls.

Scope of the revised claims

The expanded results support a more specific claim. Margin tuning can reduce instability in Label SafeDPO, but the paired comparison and mechanism ablations indicate that restoring comparisons with annotated relative-safety orderings provides additional supervision and can further improve cross-seed reliability. The effect on mean safety and helpfulness remains dependent on the backbone and evaluation metric, as shown in the first part.

We therefore do not claim that SC-SafeDPO dominates every baseline on every metric. Its contribution is to complement margin selection with previously discarded preference information, improving reliability under the evaluated settings. We hope this additional evidence and the more carefully scoped conclusion address the reviewer’s concern about the necessity of the method.

Dear Reviewer xU2P,

Thank you again for your helpful comments on cross-backbone generalizability and on the necessity of SC-SafeDPO relative to margin tuning. In the rebuttal we have addressed these points by: (i) expanding Qwen2.5-1.5B and Gemma-3-12B to 20 training seeds per configuration, reporting means and cross-seed SDs, and explicitly acknowledging the mixed Qwen1.5B HarmBench result while narrowing our claim to improved cross-seed reliability rather than uniform dominance; (ii) adding a 20-seed margin sweep over δ ∈ {0, 2.5, 5, 7.5, 10} and paired bootstrap comparisons against a tuned Label SafeDPO baseline at δ = 5, showing that SC-SafeDPO retains favorable variance and mean differences on key safety metrics, with CIs used to distinguish supported from merely favorable differences; and (iii) clarifying how matched-budget and exposure-matched controls separate margin adjustment from restored support supervision.

For related points, we have provided explanations to Reviewer xVgV regarding the semantic evaluator, fixed endpoints/training duration, confidence intervals, and human-audit counts, and to Reviewer jAHk regarding paired confidence intervals, Figure 1/error bars, and the WildGuard evaluation. We would be grateful if you could review those responses when convenient, as they contextualize the margin and backbone concerns. As the rebuttal period is approaching its end, any indication of whether our response addresses your concerns would be very helpful; we remain happy to provide further clarifications. Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.