- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper shows that large-margin Label SafeDPO suffers seed-level safety-regime instability: with a fixed recipe (Qwen2.5-7B-Instruct, δ=10), 20 seeds give OR-Toxic prefix-refusal rates from 1.83% to 98.47%, while instruction-following stays stable, so cross-seed means hide individual checkpoints' behavior. The proposed SC-SafeDPO restores the both-unsafe pairs discarded by Label SafeDPO, using their annotated relative-safety ordering with zero extra margin and a fixed weight α — no change to main orderings/margins, no inference cost. In a paired 20-seed study it cuts cross-seed refusal variance by ~77% (XSTest unsafe) and ~87% (OR-Toxic), with better means on all five metrics; cross-backbone results and a blinded human audit are also reported.
Critical strengths
-
Simple intervention, consistent main result. Only reuses existing annotations; the 20-seed paired, fixed-endpoint protocol shows the same direction on all five metrics.
-
Thorough ablations. Null / shuffled / true support under matched budgets separate mixture reweighting from restored gradients, plus δ=10 vs 5, exposure-matched control, gradient/CKA diagnostics, and a cross-backbone cohort.
-
Complete reproducibility. Anonymous repo plus appendices covering configs, data/loss definitions, seed protocols, generation settings, scoring rules, and statistics.
Critical weaknesses
-
Figure 1 has no error bars and conflicts with Sec. 4. The caption claims "±1 standard deviation across training seeds," but none are drawn (vector inspection confirms the short segments are just marker edges). The figure also never states its cohort/seed count, and its values match neither Table 2 nor Table 4. Fix the plot or the caption, and label the data source.
-
WildGuard re-evaluation results are missing. §4.6 and App. E.1/C.3 promise a semantic re-score of the frozen Qwen responses, and Table 12 lists those metrics, but no such numbers appear — only Table 3's mechanism study and Table 4's Gemma rows use WildGuard. Evaluator sensitivity is key to the main claim; add the table or drop the description.
-
"Improvement on all metrics" overstates the evidence. HarmBench ASR in Table 2 improves by just 0.68pp (SDs 3.90/1.19); Gemma IFEval (0.08pp) and StrongREJECT (0.32pp) in Table 4 are negligible. SC is worse than Label on Qwen2.5-1.5B safety and Gemma OR-Hard (8.95→11.58). App. C.4 mentions paired bootstrap, yet no CIs or p-values are reported. Add them and limit the claim to significant gains.
Questions for authors
same as weakness
3 comments
3. Statistical evidence and interpretation of improvement claims
The reviewer raises an important point about the scope of our original statement that SC-SafeDPO improves all reported metrics. Although the observed mean differences in Table 2 favor SC-SafeDPO, this alone does not establish that every difference is statistically distinguishable from zero.
To make this distinction clearer, we have added paired uncertainty estimates for the primary 20-seed fixed-endpoint comparison.
For each metric, we calculate the mean difference between SC-SafeDPO and Label SafeDPO, as well as the ratio of their cross-seed sample variances. Confidence intervals are obtained using paired bootstrap resampling, with training-seed correspondence preserved in every replicate.
Table R2. Paired bootstrap uncertainty estimates for the primary 20-seed comparison.
| Metric | Mean difference (pp) | 95% CI | Variance ratio | 95% CI |
|---|---|---|---|---|
| OR-Toxic prefix refusal ↑ | +41.45 | [26.19, 52.84] | 0.131 | [0.062, 0.241] |
| HarmBench attack success ↓ | −0.68 | [−2.35, 0.97] | 0.093 | [0.037, 0.215] |
| OR-Hard prefix false refusal ↓ | −1.24 | [−8.26, 6.34] | 0.447 | [0.215, 0.913] |
| IFEval strict prompt accuracy ↑ | +1.92 | [1.32, 2.66] | 0.743 | [0.352, 1.712] |
The paired intervals provide a more nuanced picture of the results.
For mean performance, the improvements in OR-Toxic prefix refusal and IFEval are supported by confidence intervals that exclude zero. The intervals for HarmBench attack success and OR-Hard false refusal include zero. We therefore interpret these two results as favorable observed mean differences, rather than statistically established improvements.
The variance estimates tell a somewhat different story. The confidence intervals for the OR-Toxic, HarmBench, and OR-Hard variance ratios all lie below one, indicating lower cross-seed variance for SC-SafeDPO across these three safety-related evaluations. For IFEval, the interval includes one, so the evidence does not establish a reduction in its cross-seed variance.
We have refined the corresponding claims to reflect these distinctions. In particular, we no longer present every favorable mean difference as an established improvement. Instead, we emphasize the more consistently supported finding: SC-SafeDPO substantially reduces variation in safety-related behavior across training seeds, while improving mean performance on several key evaluations.
This distinction is important to our central motivation. A preference optimization method should not only achieve favorable average safety scores, but also produce reasonably consistent safety behavior across independent training runs.
4. Overall implications for the revised paper
The additional evaluation and statistical analysis strengthen the empirical picture in complementary ways.
Figure 1 now makes the relationship between individual seed outcomes and cross-seed statistics explicit. The WildGuard evaluation extends the original prefix-based analysis to semantic judgments on the same frozen responses. The paired bootstrap analysis, in turn, clarifies which observed changes in mean performance and cross-seed variability have direct statistical support.
More broadly, these results reinforce the motivation for SC-SafeDPO: restoring the relative-safety information in both-unsafe comparisons is associated not only with improved average safety behavior, but also with more consistent outcomes across training seeds.
We appreciate the reviewer for identifying these issues. Addressing them has helped us present the experimental evidence more clearly and, importantly, sharpen the scope of the conclusions supported by our results.
Dear Reviewer jAHk,
Thank you again for your constructive comments. We have addressed your main concerns in the rebuttal and plan to incorporate them into the revision as follows: (i) Figure 1 will be redrawn with clearly visible error bars, and its caption will specify the cross-backbone cohort, the number of seeds, and the data source; we will also clarify that the plotted points correspond to individual seeds rather than the cross-seed means reported in Tables 2 and 4; (ii) the missing WildGuard semantic evaluation on the same frozen 20-seed Qwen responses will be added to §4.6 and Appendix E.1, including means, cross-seed standard deviations, and variance ratios; and (iii) paired bootstrap 95% confidence intervals for mean differences and variance ratios will be added to Appendix C.4 and summarized near Table 2.
For related points, we have provided explanations to Reviewer xU2P regarding the margin-sensitivity results and the expanded 20-seed cross-backbone evaluation, and to Reviewer xVgV regarding the semantic evaluator, fixed endpoints, and confidence intervals. These responses are closely related to your concerns and may help contextualize our reply; we would be grateful if you could review them when convenient. As the rebuttal period is approaching its end, we would greatly appreciate any further feedback or indication of whether our responses have addressed your concerns. We remain happy to provide any additional clarifications. Thank you very much for your time and consideration.
We appreciate the reviewer’s careful reading of our paper and the positive assessment of SC-SafeDPO, particularly its simple intervention, paired multi-seed evaluation, mechanism ablations, and reproducibility.
The comments also helped us identify three aspects that needed clearer presentation or additional evidence: the provenance and uncertainty visualization of Figure 1, the missing semantic evaluation of frozen Qwen responses, and the statistical support for our performance claims. We address these points below.
1. Figure 1: data provenance and uncertainty visualization
Thank you for pointing out the ambiguity in Figure 1. The figure and its original caption did not make the relationship to the cross-backbone results sufficiently clear.
Figure 1 is based on the Qwen2.5-7B-Instruct subset of the cross-backbone comparison in Table 4 (Section 4.5). Each scatter point represents an individual training seed, rather than a cross-seed mean. The horizontal axis shows safety, measured as 100 minus HarmBench attack success rate, and the vertical axis shows helpfulness, measured by IFEval accuracy.
This distinction also explains why the individual points do not coincide with the values in Table 4: the table reports cross-seed means and sample standard deviations, whereas the scatter plot displays the individual seed-level outcomes.
We have revised the figure to make the error bars more visible, with clearer line widths and end caps, and adjusted the plotting order to avoid overlap with the scatter markers. The caption now explicitly identifies the cross-backbone cohort, the number of seeds, and the meaning of the ±1 standard deviation error bars.
These changes make the figure's data source and its relationship to the aggregated results in Table 4 explicit.
2. Semantic evaluation of frozen Qwen responses
The reviewer also noted that the semantic re-evaluation described in Sections 4.6 and Appendices C.3 and E.1 was not accompanied by numerical results. We have completed this evaluation using WildGuard on the original 20-seed Qwen2.5-7B-Instruct fixed-endpoint study.
An important aspect of this evaluation is that the model checkpoints, generated responses, and evaluation subsets are unchanged. Only the evaluator is different. This allows us to examine whether the differences observed with prefix-based refusal detection are also reflected in semantic judgments, without introducing variation from additional training or response generation.
The results are summarized below. All values are percentages, reported as mean ± cross-seed sample standard deviation (percentage points).
Table R1. WildGuard semantic evaluation of frozen responses across 20 training seeds.
The results show that SC-SafeDPO has a higher semantic refusal rate on harmful prompts (97.80% vs. 91.29%), while producing fewer semantically identified harmful responses (0.153% vs. 0.313%). On benign OR-Hard prompts, its semantic false-refusal rate is also lower (40.10% vs. 48.98%).
The reduction in cross-seed dispersion is particularly relevant to the concern raised in this review. For OR-Toxic semantic refusal, the sample standard deviation decreases from 3.82 to 2.40 percentage points, corresponding to an approximately 60.5% reduction in sample variance. Dispersion also decreases for the other two semantic metrics.
We distinguish this result from the approximately 87% variance reduction reported in the original paper, which refers to OR-Toxic prefix-based refusal. The two numbers come from different evaluation procedures and should not be conflated.
Taken together, the WildGuard results provide an independent semantic assessment of the same model outputs. The direction of the differences is consistent with the original findings, including both improved harmful-response avoidance and fewer benign false refusals.