- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes CovDPO, a representation-space coverage-based framework for improving the safety alignment of LLMs against out-of-distribution (OOD) harmful prompts. The authors introduce the Hidden-State Coverage Score (HCS) to estimate the coverage of safety-preference training data in the model's hidden representation space. They use HCS to guide the selection of low-coverage prompt variations for training data augmentation and to scale the DPO loss according to the coverage deficit. Experiments on Llama-3-8B and Qwen3-8B across four OOD harmful benchmarks and three jailbreak attacks show that CovDPO-Full reduces the average attack success rate from 15.90% to 2.41% on Llama-3-8B and from 11.21% to 4.60% on Qwen3-8B, outperforming the evaluated baselines while maintaining general capabilities and low over-refusal. Ablation studies are also provided to assess the individual contributions of the proposed components.
Critical strengths
The paper establishes an empirical association between insufficient representation-space coverage of safety-preference training data and safety vulnerabilities on OOD harmful prompts. Building on this observation, it introduces the Hidden-State Coverage Score (HCS) to identify under-covered regions and guide targeted preference-data augmentation. This is a meaningful contribution because it provides a principled way to connect the diagnosis of safety-generalization failures with the construction of training data to address them.
Critical weaknesses
-
Although I agree that the coverage of safety-training data can strongly influence the safety generalization of LLMs [1], I believe the paper does not sufficiently establish the effectiveness of its proposed hidden-state coverage-based approach beyond the benefits of data augmentation. CovDPO reduces the average ASR from 25.71% to 2.41% on Llama-3-8B and from 37.18% to 4.60% on Qwen3-8B compared with standard DPO. However, the ablation results suggest that a substantial portion of these gains can be attributed to data augmentation rather than the proposed coverage-aware selection and loss weighting. Removing both components yields average ASRs of 6.86% for Llama-3-8B and 8.63% for Qwen3-8B, compared with 25.71% and 37.18%, respectively, for standard DPO. These results indicate that more than 80% of the ASR reduction can be achieved without the proposed coverage-aware mechanisms. This raises the question of whether the primary benefit comes from adding more diverse safety-preference data or specifically from the proposed coverage-aware mechanism. The authors should identify appropriate scenarios when CovDPO performs significantly better than simple data augmentation.
-
CovDPO introduces additional hyperparameters, including the coverage threshold and the time-dependent weighting coefficient , which is linearly decayed during training. However, the paper provides limited guidance on how to select these hyperparameters in practice or how sensitive the results are to their choices.
[1] Addepalli et. al., "Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?", ICLR 2025.
Questions for authors
-
Contribution of coverage-aware mechanisms: The ablation results show that removing both HCS-guided sample weighting and HCS-guided data augmentation still yields substantial improvements over standard DPO. Can the authors provide a comparison with an equally sized, coverage-agnostic data augmentation baseline to isolate the benefits of HCS-guided selection and weighting from those of simply adding more diverse preference data?
-
Sensitivity to hyperparameter choices: CovDPO introduces additional hyperparameters, including the coverage threshold and the time-dependent weighting schedule (\gamma(t)). How sensitive are the reported results to these choices, and do the authors observe consistent benefits across different settings or models?
5 comments
Author Response to Reviewer FNW1 (Part 2/3)
(Continued from Part 1/3.)
Third, we examine whether HCS itself identifies more beneficial training samples. Using the same pool of 12,019 randomly augmented candidates, we separately select the 4,000 lowest-HCS and 4,000 highest-HCS samples. Each subset is combined with the same 4,744 original preference pairs, resulting in 8,744 training pairs per configuration under matched training settings.
Table R3: Low-HCS versus high-HCS selection from the same random augmentation pool on Llama-3-8B. All evaluation metrics are percentages (%).
| Selection | OOD ASR ↓ | Jailbreak ASR ↓ | Avg. ASR ↓ | XSTest ↓ | MMLU ↑ | HellaSwag ↑ |
|---|---|---|---|---|---|---|
| Low-HCS | 11.96 | 26.75 | 18.30 | 1.60 | 59.69 | 49.11 |
| High-HCS | 20.41 | 36.67 | 27.37 | 0.80 | 60.06 | 48.43 |
Selecting low-HCS samples reduces average ASR from 27.37% to 18.30% and OOD ASR from 20.41% to 11.96%. Since both groups originate from the same random augmentation pool and use the same training data volume, this experiment provides direct evidence that HCS identifies samples more beneficial for safety generalization, rather than merely increasing data diversity or quantity.
Finally, we examine whether the improvements persist without increasing the total training dataset size. On Llama-3-8B, we replace 4,000 original preference pairs with 4,000 HCS-guided augmented pairs, keeping the total at 8,744 training pairs.
Table R4: Fixed-size training data comparison on Llama-3-8B. All evaluation metrics are percentages (%).
| Method | ID Pairs | Aug. Pairs | OOD ASR ↓ | Jailbreak ASR ↓ | Avg. ASR ↓ | XSTest ↓ | MMLU ↑ | HellaSwag ↑ |
|---|---|---|---|---|---|---|---|---|
| Standard DPO | 8,744 | 0 | 20.74 | 32.33 | 25.71 | 0.00 | 57.10 | 42.00 |
| CovDPO (Fixed-Size) | 4,744 | 4,000 | 1.06 | 4.42 | 2.50 | 2.00 | 58.10 | 51.19 |
| CovDPO-Full | 8,744 | 8,000 | 0.54 | 4.92 | 2.41 | 0.40 | 57.45 | 54.10 |
With the same total number of training pairs as Standard DPO, CovDPO reduces average ASR from 25.71% to 2.50%. The fixed-size configuration also achieves an average ASR comparable to CovDPO-Full, at 2.50% versus 2.41%, but exhibits higher XSTest over-refusal of 2.00% versus 0.40%. These results show that the improvements cannot be explained solely by increased training data volume.
Overall, these complementary experiments demonstrate that HCS-guided selection and coverage weighting provide substantial additional benefits beyond our structured augmentation framework. In particular, the random augmentation controls directly support the effectiveness of HCS-guided sample selection, with the strongest additional gains observed on OOD harmful requests. We will include these additional comparisons in Appendix F.1 of the revised manuscript.
Author Response to Reviewer FNW1 (Part 3/3)
The sensitivity of CovDPO to the coverage threshold and time-dependent weighting coefficient remains unclear.
We thank the reviewer for this valuable suggestion. We conduct additional sensitivity experiments on Qwen3-8B, varying the coverage threshold and weighting schedule. The default configuration uses a coverage threshold of 0.21 and linearly decays the weighting coefficient from 80 to 45 during the first half of training, keeping it fixed thereafter. We compare alternative thresholds of 0.18 and 0.24, as well as constant weighting coefficients of 45 and 80. The results are presented in Table R2.
Table R2: Hyperparameter sensitivity analysis on Qwen3-8B. All values are percentages (%).
| Method | Avg. ASR ↓ | AdvBench ↓ | SORRY-Bench ↓ | HEx-PHI ↓ | HarmEval ↓ | Prefill ↓ | GCG ↓ | PAIR ↓ | OOD ASR ↓ | Jailbreak ASR ↓ | MMLU ↑ | HellaSwag ↑ | XSTest ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CovDPO-Full | 4.60 | 0.00 | 4.22 | 0.33 | 1.64 | 20.50 | 0.50 | 5.00 | 1.55 | 8.67 | 60.00 | 66.85 | 0.80 |
| θ = 0.18 | 2.12 | 0.00 | 3.33 | 0.00 | 0.00 | 11.50 | 0.00 | 0.00 | 0.83 | 3.83 | 58.43 | 57.20 | 4.40 |
| θ = 0.24 | 3.11 | 0.00 | 4.67 | 0.33 | 0.00 | 15.25 | 0.00 | 1.50 | 1.25 | 5.58 | 63.12 | 68.39 | 8.40 |
| Fixed γ = 45 | 3.08 | 0.00 | 4.22 | 0.00 | 0.36 | 13.50 | 0.50 | 3.00 | 1.15 | 5.67 | 58.62 | 63.48 | 10.80 |
| Fixed γ = 80 | 2.11 | 0.00 | 3.78 | 0.33 | 0.18 | 9.25 | 0.25 | 1.00 | 1.07 | 3.50 | 61.47 | 68.70 | 4.40 |
Across different coverage thresholds and weighting schedules, CovDPO consistently maintains low average ASR, ranging from 2.11% to 3.11% across the alternative configurations. However, over-refusal is more sensitive to these choices, with XSTest increasing to 4.40%–10.80%, compared with only 0.80% under the default configuration. These results indicate that while alternative settings can achieve lower ASR, the default configuration provides a more favorable safety–over-refusal trade-off. We will include these hyperparameter sensitivity results in Appendix F.3 of the revised manuscript.
We appreciate your thoughtful comments. We hope our response addresses your concerns. Please let us know if there are any additional questions, and we will be happy to discuss further.
Author Response to Reviewer FNW1 (Part 1/3)
We thank the reviewer for the thoughtful and constructive feedback and recognition of our contributions. We have conducted additional experiments and provide point-by-point responses addressing the benefits of coverage-aware mechanisms beyond data augmentation, comparisons under matched data budgets, and hyperparameter sensitivity below.
R1: Contribution of Coverage-Aware Mechanisms.
Response W1.1: Additional Benefits Beyond Coverage-Agnostic Augmentation.
We thank the reviewer for this valuable comment. We address this concern through four complementary comparisons, examining the additional contribution of coverage-aware mechanisms, their effectiveness against random augmentation, the role of HCS in sample selection, and the influence of training data volume.
First, we quantify the additional contribution of coverage-aware mechanisms. Our existing w/o Both ablation in Table 2 provides an equal-sized, coverage-agnostic augmentation baseline. Both CovDPO-Full and w/o Both use our proposed structured preference-data mutation and verification framework, generating 8,000 validated preference pairs. The difference is that w/o Both removes HCS-guided selection and coverage weighting while retaining the same mutation and verification procedures.
Table R1: Relative ASR reductions from the proposed augmentation framework and coverage-aware mechanisms.
We acknowledge the reviewer's observation that augmentation without HCS accounts for a substantial portion of the total absolute ASR reduction. However, this proportion does not fully characterize the additional contribution of coverage-aware mechanisms. As shown in Table R1, under the same augmentation budget, HCS-guided selection and weighting further reduce the remaining average ASR by 64.9% on Llama and 46.7% on Qwen, with particularly pronounced improvements on OOD benchmarks.
Second, we directly compare CovDPO with ordinary random augmentation under the same data budget. We randomly select 8,000 augmented preference pairs from a pool of 12,019 randomly augmented candidates and combine them with the same 8,744 original preference pairs.
Table R2: Equal-sized random augmentation versus CovDPO-Full on Llama-3-8B. All evaluation metrics are percentages (%).
Under the same augmentation budget, CovDPO-Full reduces average ASR from 7.38% to 2.41%, a 67.3% relative reduction. The improvement is especially pronounced on OOD benchmarks, where ASR decreases from 3.91% to 0.54%, while XSTest remains at 0.40%. This comparison directly demonstrates substantial robustness gains beyond ordinary random augmentation, particularly for OOD safety generalization.
(Continued in Part 2/3.)