acceptodds
Under review as a conference paper at ICLR 2027

CovDPO: Bridging Representation-Space Coverage Gaps in LLM Safety Alignment

10 views

Abstract

Despite their effectiveness on in-distribution requests, safety-aligned models often generalize poorly to out-of-distribution (OOD) harmful inputs. Across two model families, we find that harmful inputs associated with safety failures tend to lie in regions weakly covered by the training safety-preference data, indicating that safety generalization failures are systematically associated with gaps in representation-space coverage. We therefore propose CovDPO, a coverage-aware framework that connects coverage diagnosis with preference data augmentation and optimization. CovDPO introduces the Hidden-State Coverage Score (HCS) to measure the local coverage of safety-preference data in representation space. HCS serves two complementary roles: it guides preference-data augmentation toward under-covered regions associated with safety failures, and it provides coverage-dependent loss scaling for the DPO objective. Across four OOD harmful benchmarks and three jailbreak attacks, CovDPO outperforms the strongest baseline on both model families, reducing the average attack success rate from 15.90% to 2.41% on Llama-3-8B and from 11.21% to 4.60% on Qwen3-8B, while maintaining low over-refusal.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety Alignment, Direct Preference Optimization, Out-of-Distribution Generalization, Representation-Space Coverage, Preference Data Augmentation, Jailbreak Robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FNW1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes CovDPO, a representation-space coverage-based framework for improving the safety alignment of LLMs against out-of-distribution (OOD) harmful prompts. The authors introduce the Hidden-State Coverage Score (HCS) to estimate the coverage of safety-preference training data in the model's hidden representation space. They use HCS to guide the selection of low-coverage prompt variations for training data augmentation and to scale the DPO loss according to the coverage deficit. Experiments on Llama-3-8B and Qwen3-8B across four OOD harmful benchmarks and three jailbreak attacks show that CovDPO-Full reduces the average attack success rate from 15.90% to 2.41% on Llama-3-8B and from 11.21% to 4.60% on Qwen3-8B, outperforming the evaluated baselines while maintaining general capabilities and low over-refusal. Ablation studies are also provided to assess the individual contributions of the proposed components.

Critical strengths

The paper establishes an empirical association between insufficient representation-space coverage of safety-preference training data and safety vulnerabilities on OOD harmful prompts. Building on this observation, it introduces the Hidden-State Coverage Score (HCS) to identify under-covered regions and guide targeted preference-data augmentation. This is a meaningful contribution because it provides a principled way to connect the diagnosis of safety-generalization failures with the construction of training data to address them.

Critical weaknesses
  1. Although I agree that the coverage of safety-training data can strongly influence the safety generalization of LLMs [1], I believe the paper does not sufficiently establish the effectiveness of its proposed hidden-state coverage-based approach beyond the benefits of data augmentation. CovDPO reduces the average ASR from 25.71% to 2.41% on Llama-3-8B and from 37.18% to 4.60% on Qwen3-8B compared with standard DPO. However, the ablation results suggest that a substantial portion of these gains can be attributed to data augmentation rather than the proposed coverage-aware selection and loss weighting. Removing both components yields average ASRs of 6.86% for Llama-3-8B and 8.63% for Qwen3-8B, compared with 25.71% and 37.18%, respectively, for standard DPO. These results indicate that more than 80% of the ASR reduction can be achieved without the proposed coverage-aware mechanisms. This raises the question of whether the primary benefit comes from adding more diverse safety-preference data or specifically from the proposed coverage-aware mechanism. The authors should identify appropriate scenarios when CovDPO performs significantly better than simple data augmentation.

  2. CovDPO introduces additional hyperparameters, including the coverage threshold and the time-dependent weighting coefficient γ(t)\gamma(t), which is linearly decayed during training. However, the paper provides limited guidance on how to select these hyperparameters in practice or how sensitive the results are to their choices.

[1] Addepalli et. al., "Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?", ICLR 2025.

Questions for authors
  1. Contribution of coverage-aware mechanisms: The ablation results show that removing both HCS-guided sample weighting and HCS-guided data augmentation still yields substantial improvements over standard DPO. Can the authors provide a comparison with an equally sized, coverage-agnostic data augmentation baseline to isolate the benefits of HCS-guided selection and weighting from those of simply adding more diverse preference data?

  2. Sensitivity to hyperparameter choices: CovDPO introduces additional hyperparameters, including the coverage threshold and the time-dependent weighting schedule (\gamma(t)). How sensitive are the reported results to these choices, and do the authors observe consistent benefits across different settings or models?

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer FNW1 (Part 1/3)

We thank the reviewer for the thoughtful and constructive feedback and recognition of our contributions. We have conducted additional experiments and provide point-by-point responses addressing the benefits of coverage-aware mechanisms beyond data augmentation, comparisons under matched data budgets, and hyperparameter sensitivity below.

R1: Contribution of Coverage-Aware Mechanisms.{\color{#00aa00}\text{R1: Contribution of Coverage-Aware Mechanisms.}}

Weakness-1:{\color{#00aa00}\text{Weakness-1:}} The contribution of HCS-guided augmentation and coverage weighting beyond simply adding more diverse preference data is not sufficiently established.

Response W1.1: Additional Benefits Beyond Coverage-Agnostic Augmentation.{\color{#00aa00}\text{Response W1.1: Additional Benefits Beyond Coverage-Agnostic Augmentation.}}

We thank the reviewer for this valuable comment. We address this concern through four complementary comparisons, examining the additional contribution of coverage-aware mechanisms, their effectiveness against random augmentation, the role of HCS in sample selection, and the influence of training data volume.

First, we quantify the additional contribution of coverage-aware mechanisms. Our existing w/o Both ablation in Table 2 provides an equal-sized, coverage-agnostic augmentation baseline. Both CovDPO-Full and w/o Both use our proposed structured preference-data mutation and verification framework, generating 8,000 validated preference pairs. The difference is that w/o Both removes HCS-guided selection and coverage weighting while retaining the same mutation and verification procedures.

Table R1: Relative ASR reductions from the proposed augmentation framework and coverage-aware mechanisms.

BackboneStandard DPO → w/o Both (Avg. ASR)w/o Both → CovDPO-Full (Avg. ASR)w/o Both → CovDPO-Full (OOD ASR)w/o Both → CovDPO-Full (Jailbreak ASR)
Llama-3-8B25.71% → 6.86% (−73.3%)6.86% → 2.41% (−64.9%)2.95% → 0.54% (−81.7%)12.08% → 4.92% (−59.3%)
Qwen3-8B37.18% → 8.63% (−76.8%)8.63% → 4.60% (−46.7%)3.54% → 1.55% (−56.2%)15.42% → 8.67% (−43.8%)

We acknowledge the reviewer's observation that augmentation without HCS accounts for a substantial portion of the total absolute ASR reduction. However, this proportion does not fully characterize the additional contribution of coverage-aware mechanisms. As shown in Table R1, under the same augmentation budget, HCS-guided selection and weighting further reduce the remaining average ASR by 64.9% on Llama and 46.7% on Qwen, with particularly pronounced improvements on OOD benchmarks.

Second, we directly compare CovDPO with ordinary random augmentation under the same data budget. We randomly select 8,000 augmented preference pairs from a pool of 12,019 randomly augmented candidates and combine them with the same 8,744 original preference pairs.

Table R2: Equal-sized random augmentation versus CovDPO-Full on Llama-3-8B. All evaluation metrics are percentages (%).

MethodID PairsAug. PairsOOD ASR ↓Jailbreak ASR ↓Avg. ASR ↓XSTest ↓MMLU ↑HellaSwag ↑
Random Augmentation8,7448,0003.9112.007.380.4059.4754.24
CovDPO-Full8,7448,0000.544.922.410.4057.4554.10

Under the same augmentation budget, CovDPO-Full reduces average ASR from 7.38% to 2.41%, a 67.3% relative reduction. The improvement is especially pronounced on OOD benchmarks, where ASR decreases from 3.91% to 0.54%, while XSTest remains at 0.40%. This comparison directly demonstrates substantial robustness gains beyond ordinary random augmentation, particularly for OOD safety generalization.

(Continued in Part 2/3.)

AuthorsOpenReview ↗

Author Response to Reviewer FNW1 (Part 2/3)

(Continued from Part 1/3.)

Response W1.2: Effectiveness of HCS-Guided Sample Selection.{\color{#00aa00}\text{Response W1.2: Effectiveness of HCS-Guided Sample Selection.}}

Third, we examine whether HCS itself identifies more beneficial training samples. Using the same pool of 12,019 randomly augmented candidates, we separately select the 4,000 lowest-HCS and 4,000 highest-HCS samples. Each subset is combined with the same 4,744 original preference pairs, resulting in 8,744 training pairs per configuration under matched training settings.

Table R3: Low-HCS versus high-HCS selection from the same random augmentation pool on Llama-3-8B. All evaluation metrics are percentages (%).

SelectionOOD ASR ↓Jailbreak ASR ↓Avg. ASR ↓XSTest ↓MMLU ↑HellaSwag ↑
Low-HCS11.9626.7518.301.6059.6949.11
High-HCS20.4136.6727.370.8060.0648.43

Selecting low-HCS samples reduces average ASR from 27.37% to 18.30% and OOD ASR from 20.41% to 11.96%. Since both groups originate from the same random augmentation pool and use the same training data volume, this experiment provides direct evidence that HCS identifies samples more beneficial for safety generalization, rather than merely increasing data diversity or quantity.

Response W1.3: Fixed-Size Training Data Comparison.{\color{#00aa00}\text{Response W1.3: Fixed-Size Training Data Comparison.}}

Finally, we examine whether the improvements persist without increasing the total training dataset size. On Llama-3-8B, we replace 4,000 original preference pairs with 4,000 HCS-guided augmented pairs, keeping the total at 8,744 training pairs.

Table R4: Fixed-size training data comparison on Llama-3-8B. All evaluation metrics are percentages (%).

MethodID PairsAug. PairsOOD ASR ↓Jailbreak ASR ↓Avg. ASR ↓XSTest ↓MMLU ↑HellaSwag ↑
Standard DPO8,744020.7432.3325.710.0057.1042.00
CovDPO (Fixed-Size)4,7444,0001.064.422.502.0058.1051.19
CovDPO-Full8,7448,0000.544.922.410.4057.4554.10

With the same total number of training pairs as Standard DPO, CovDPO reduces average ASR from 25.71% to 2.50%. The fixed-size configuration also achieves an average ASR comparable to CovDPO-Full, at 2.50% versus 2.41%, but exhibits higher XSTest over-refusal of 2.00% versus 0.40%. These results show that the improvements cannot be explained solely by increased training data volume.

Overall, these complementary experiments demonstrate that HCS-guided selection and coverage weighting provide substantial additional benefits beyond our structured augmentation framework. In particular, the random augmentation controls directly support the effectiveness of HCS-guided sample selection, with the strongest additional gains observed on OOD harmful requests. We will include these additional comparisons in Appendix F.1 of the revised manuscript.

AuthorsOpenReview ↗

Author Response to Reviewer FNW1 (Part 3/3)

R2: Sensitivity to Hyperparameter Choices.{\color{#00aa00}\text{R2: Sensitivity to Hyperparameter Choices.}}

Weakness-2:{\color{#00aa00}\text{Weakness-2:}} The sensitivity of CovDPO to the coverage threshold and time-dependent weighting coefficient remains unclear.

Response W2: Hyperparameter Sensitivity Analysis.{\color{#00aa00}\text{Response W2: Hyperparameter Sensitivity Analysis.}}

We thank the reviewer for this valuable suggestion. We conduct additional sensitivity experiments on Qwen3-8B, varying the coverage threshold and weighting schedule. The default configuration uses a coverage threshold of 0.21 and linearly decays the weighting coefficient from 80 to 45 during the first half of training, keeping it fixed thereafter. We compare alternative thresholds of 0.18 and 0.24, as well as constant weighting coefficients of 45 and 80. The results are presented in Table R2.

Table R2: Hyperparameter sensitivity analysis on Qwen3-8B. All values are percentages (%).

MethodAvg. ASR ↓AdvBench ↓SORRY-Bench ↓HEx-PHI ↓HarmEval ↓Prefill ↓GCG ↓PAIR ↓OOD ASR ↓Jailbreak ASR ↓MMLU ↑HellaSwag ↑XSTest ↓
CovDPO-Full4.600.004.220.331.6420.500.505.001.558.6760.0066.850.80
θ = 0.182.120.003.330.000.0011.500.000.000.833.8358.4357.204.40
θ = 0.243.110.004.670.330.0015.250.001.501.255.5863.1268.398.40
Fixed γ = 453.080.004.220.000.3613.500.503.001.155.6758.6263.4810.80
Fixed γ = 802.110.003.780.330.189.250.251.001.073.5061.4768.704.40

Across different coverage thresholds and weighting schedules, CovDPO consistently maintains low average ASR, ranging from 2.11% to 3.11% across the alternative configurations. However, over-refusal is more sensitive to these choices, with XSTest increasing to 4.40%–10.80%, compared with only 0.80% under the default configuration. These results indicate that while alternative settings can achieve lower ASR, the default configuration provides a more favorable safety–over-refusal trade-off. We will include these hyperparameter sensitivity results in Appendix F.3 of the revised manuscript.

We appreciate your thoughtful comments. We hope our response addresses your concerns. Please let us know if there are any additional questions, and we will be happy to discuss further.

Reviewer_WoNbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper proposes CovDPO, a coverage-aware safety alignment framework that uses hidden-state representations to discover under-covered regions in safety preference data. It combines coverage guided data augmentation with coverage weighted DPO and improves safety generalization and jailbreak robustness.

Critical strengths

The paper motivates an OOD safety generalization improvement strategy by linking representation-space coverage analysis to preference optimization. The Hidden-State Coverage Score proposed is a unified signal for both data augmentation and loss weighting. Experimental results are also very strong (SR reduction in ASR on multiple harmful benchmarks and two model families), component ablations in Table 2 show the contribution of augmentation and weighting, while coverage analyses provide insights into how representation space coverage relates to safety failures.

Critical weaknesses
  1. The necessity of the proposed HCS formulation is not clear. While the paper shows that low HCS predicts safety failures, it is not clear whether the safety-direction projection has any advantages over simpler representation space density measures or difficulty sampling; comparisons with PCA-only kNN coverage or other hard example selection strategies would isolate contribution from HCS.

  2. The jailbreak robustness evaluation is only for fixed attack suites. As shown in Appendix E.2 GCG and PAIR attacks are generated against separate anchor models and used across all checkpoints to be compared; this ensures a consistent comparison but does not establish robustness against adaptive attacks on the aligned models (i.e., robustness against stronger model-specific adversaries). The improvements thus may overestimate robustness against stronger model-specific adversaries.

Questions for authors
  1. How does HCS compare to simpler coverage measures (PCA based kNN density without the safety-direction projection, difficulty based example selection under the same augmentation budget)?

  2. Can the authors evaluate CovDPO against adaptive GCG or PAIR attacks generated specifically for the final aligned checkpoints?

  3. What is the end-to-end computational cost of generating the augmented preference data (candidate generation + verification) in comparison with training costs and competing methods?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer WoNb (Part 1/2)

We thank the reviewer for the thoughtful and constructive feedback and recognition of our contributions. We have conducted additional experiments and provide point-by-point responses addressing the HCS formulation, comparisons with simpler coverage and sampling strategies, adaptive jailbreak robustness, and computational costs below.

R1: Ablation Studies on the HCS Design.{\color{#0077ff}\text{R1: Ablation Studies on the HCS Design.}}

Weakness-1:{\color{#0077ff}\text{Weakness-1:}} The advantages of HCS over PCA-based kNN density without the safety-direction projection and difficulty-based example selection remain unclear under the same augmentation budget.

Response W1: Comparison with Alternative Coverage Measures and Selection Strategies.{\color{#0077ff}\text{Response W1: Comparison with Alternative Coverage Measures and Selection Strategies.}}

We thank the reviewer for this valuable suggestion. To isolate the contributions of the safety axis and representation-layer choice, we conduct additional downstream training experiments on Llama-3-8B and Qwen3-8B. We compare CovDPO-Full against three alternatives: (1) w/o Safety Axis, which removes the response-derived safety-axis coordinate and uses prompt-distance nearest-neighbour coverage for augmentation selection; (2) w/ Worst Layer, which uses the layer with the lowest safety-discrimination score; and (3) w/ Last-Layer PCA, a conventional alternative using PCA-based nearest-neighbour coverage in the final-layer representation. The results are presented in Table R1.

Table R1: Ablation studies on the safety axis and layer selection in HCS. All values are percentages (%).

MethodAvg. ASR ↓AdvBench ↓SORRY-Bench ↓HEx-PHI ↓HarmBench ↓Prefill ↓GCG ↓PAIR ↓MMLU ↑HellaSwag ↑XSTest ↓
Llama-3-8B
CovDPO-Full2.410.001.780.000.3614.750.000.0057.4554.100.40
w/o Safety Axis2.850.191.560.000.1817.750.000.2558.6244.089.60
w/ Worst Layer1.780.001.330.000.3610.750.000.0057.5844.6512.40
w/ Last-Layer PCA2.330.001.110.000.1814.750.000.2556.9544.5310.80
Qwen3-8B
CovDPO-Full4.600.004.220.331.6420.500.505.0060.0066.850.80
w/o Safety Axis3.520.005.560.670.9116.500.001.0064.0073.875.60
w/ Worst Layer2.600.003.110.000.3612.250.252.2561.9669.3716.80
w/ Last-Layer PCA3.740.005.330.330.5519.000.250.7561.8570.1910.40

HCS achieves lower over-refusal than the simpler coverage alternatives on both backbones. Removing the safety axis increases XSTest over-refusal from 0.40% to 9.60% on Llama and from 0.80% to 5.60% on Qwen. Using the worst layer or last-layer PCA likewise raises over-refusal to 10.40–16.80%, despite sometimes achieving lower ASR. CovDPO-Full thus maintains low ASR while achieving the lowest over-refusal across both backbones. We also compare with Staged-Competence, a difficulty-aware curriculum baseline, in Table 1. CovDPO-Full reduces average ASR from 15.90% to 2.41% on Llama and from 11.21% to 4.60% on Qwen.

Overall, these results demonstrate that the safety axis and representation-layer selection contribute to a better safety–over-refusal trade-off. We will incorporate these additional ablation results into Section 4.5 of the revised manuscript.

AuthorsOpenReview ↗

Author Response to Reviewer WoNb (Part 2/2)

R2: Adaptive Jailbreak Evaluation.{\color{#0077ff}\text{R2: Adaptive Jailbreak Evaluation.}}

Question-2:{\color{#0077ff}\text{Question-2:}} The original jailbreak evaluations use fixed attack suites, leaving robustness against attacks specifically optimized for the final aligned checkpoints unclear.

Response Q2: Evaluation Against Adaptive GCG Attacks.{\color{#0077ff}\text{Response Q2: Evaluation Against Adaptive GCG Attacks.}}

We thank the reviewer for this valuable suggestion. Due to computational constraints, our original GCG and PAIR evaluations use attacks generated against fixed anchor models rather than attacks optimized for each evaluated checkpoint. To address this concern, we conduct additional Adaptive GCG attacks on the Llama-3-8B model family. Specifically, we independently optimize GCG attacks against each final aligned checkpoint using 100 optimization steps and the same 400 test cases, rather than reusing attacks generated against separate anchor models.

Table R2: Adaptive GCG attack results on Llama-3-8B models.

ModelSuccessful AttacksASR ↓
Unaligned257 / 40064.25%
ShaPO161 / 40040.25%
Standard DPO54 / 40013.50%
DOOR39 / 4009.75%
Staged-Competence12 / 4003.00%
CovDPO-Full0 / 4000.00%

As shown in Table R2, CovDPO-Full achieves 0.00% ASR under Adaptive GCG attacks, compared with 3.00% for the strongest competing baselines. These results provide additional evidence that CovDPO's robustness improvements extend beyond fixed transfer attacks to adversarial optimization directly targeting the final aligned checkpoints. We will include these additional results in Appendix F.4 of the revised manuscript.

R3: Computational Cost of Data Augmentation.{\color{#0077ff}\text{R3: Computational Cost of Data Augmentation.}}

Weakness-3:{\color{#0077ff}\text{Weakness-3:}} The end-to-end computational cost of augmented preference data generation, including candidate generation and verification, is not clearly reported or compared with training costs.

Response W3: End-to-End Computational Cost Analysis.{\color{#0077ff}\text{Response W3: End-to-End Computational Cost Analysis.}}

We thank the reviewer for this valuable suggestion. We provide a breakdown of the computational cost of CovDPO, including augmented preference data generation, verification, and subsequent training.

Table R3: Computational cost of CovDPO.

StageData SizeGPU-hours
Augmentation and Verification8,000 retained pairs47.0
Full-parameter DPO Training16,744 pairs, 2 epochs3.5
Total—50.5

Generating and verifying 8,000 valid augmented preference pairs requires approximately 47 GPU-hours, corresponding to about 21 GPU-seconds per retained pair. We then combine these with 8,744 original preference pairs and perform two epochs of full-parameter DPO training, requiring approximately 3.5 GPU-hours. The total computational cost is therefore approximately 50.5 GPU-hours.

The main computational cost of CovDPO comes from data augmentation and verification rather than training. Although this introduces additional upfront overhead, the generated preference pairs can be reused across training runs. We will include this computational breakdown in Appendix E.2 of the revised manuscript.

We appreciate your thoughtful comments. We hope our response addresses your concerns. Please let us know if there are any additional questions, and we will be happy to discuss further.

Reviewer_A4SHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CovDPO, which uses representation-space coverage to guide safety-preference data augmentation and DPO loss weighting. Its Hidden-State Coverage Score combines PCA-reduced prompt representations with a safety-axis projection and measures local coverage through nearest-neighbour distances. Experiments on two abliterated 8B model families report lower attack success rates across harmful-prompt benchmarks and fixed jailbreak suites. The paper also includes coverage analysis, component ablations, and a generated safety-preference dataset.

Critical strengths

For the strengths,

  • The paper connects a useful diagnosis to a concrete training method. Low coverage is associated with safety failures on both model families. Appendix F also checks this within benchmarks and controls for prompt length and harmful category, which makes the motivation more convincing.

  • The safety gains are substantial: average ASR drops from 15.90% to 2.41% on Llama and from 11.21% to 4.60% on Qwen compared with the strongest baseline. The equal-sized random-mutation ablations are useful. General capability and over-refusal are also evaluated.

Critical weaknesses

For the weaknesses,

  • The contribution of the specific HCS design is not fully clear. The random-mutation control supports coverage-guided selection, but does not isolate the response-derived safety axis or layer choice from simpler prompt-distance selection. Appendix F checks the coverage-ASR association without comparing downstream training under these alternatives. The weighting results also need explanation: on Qwen, removing weighting changes average ASR only from 4.60% to 4.71%, but increases XSTest over-refusal from 0.80% to 15.20%. Appendix E.1 mentions softmin aggregation without defining it, so the actual objective is difficult to check.

  • The jailbreak results have some limit. GCG and PAIR are generated against fixed anchor models and reused across checkpoints, which evaluates transfer robustness. Resistance to attacks adapted to each trained model remains unclear. Training uses seed 42 without uncertainty estimates. The fallback judge's calibration uses the same author-labelled AdvBench responses used to refine its rubric, and over-refusal is measured by refusal-phrase matching. I think an independent scoring check is especially useful given the near-zero ASRs.

Questions for authors

For the questions,

  • What is the exact loss, including softmin aggregation? Why does weighting have little effect on Qwen ASR but a large effect on over-refusal, and is this result consistent across seeds?

  • Under matched budgets, how does HCS compare with nearest-neighbour selection without the safety-axis coordinate?

  • Are there results from attacks regenerated against the trained checkpoints or an independently labelled scoring subset? These would help assess the robustness of the reported gains.

Please also refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer A4SH (Part 1/3)

We thank the reviewer for the thoughtful and constructive feedback and recognition of our findings. We have conducted additional experiments and provide point-by-point responses addressing the HCS design, loss formulation, component contributions, adaptive jailbreak robustness, and evaluation reliability below.

R1: Ablation Studies on the HCS Design.{\color{#f26921}\text{R1: Ablation Studies on the HCS Design.}}

Weakness-1:{\color{#f26921}\text{Weakness-1:}} The contributions of the response-derived safety axis and layer selection in HCS remain unclear without matched-budget downstream comparisons against simpler prompt-distance nearest-neighbour selection.

Response W1:{\color{#f26921}\text{Response W1:}} Component-level Ablations of the HCS Design.

We thank the reviewer for this valuable suggestion. To isolate the contributions of the safety axis and representation-layer choice, we conduct additional downstream training experiments on Llama-3-8B and Qwen3-8B. We compare CovDPO-Full against three alternatives: (1) w/o Safety Axis, which removes the response-derived safety-axis coordinate and uses prompt-distance nearest-neighbour coverage for augmentation selection; (2) w/ Worst Layer, which uses the layer with the lowest safety-discrimination score; and (3) w/ Last-Layer PCA, a conventional alternative using PCA-based nearest-neighbour coverage in the final-layer representation. The results are presented in Table R1.

Table R1: Ablation studies on the safety axis and layer selection in HCS. All values are percentages (%).

MethodAvg. ASR ↓AdvBench ↓SORRY-Bench ↓HEx-PHI ↓HarmBench ↓Prefill ↓GCG ↓PAIR ↓MMLU ↑HellaSwag ↑XSTest ↓
Llama-3-8B
CovDPO-Full2.410.001.780.000.3614.750.000.0057.4554.100.40
w/o Safety Axis2.850.191.560.000.1817.750.000.2558.6244.089.60
w/ Worst Layer1.780.001.330.000.3610.750.000.0057.5844.6512.40
w/ Last-Layer PCA2.330.001.110.000.1814.750.000.2556.9544.5310.80
Qwen3-8B
CovDPO-Full4.600.004.220.331.6420.500.505.0060.0066.850.80
w/o Safety Axis3.520.005.560.670.9116.500.001.0064.0073.875.60
w/ Worst Layer2.600.003.110.000.3612.250.252.2561.9669.3716.80
w/ Last-Layer PCA3.740.005.330.330.5519.000.250.7561.8570.1910.40

The main benefit of the HCS design is consistently lower over-refusal across both backbones. Removing the safety axis increases XSTest over-refusal from 0.40% to 9.60% on Llama and from 0.80% to 5.60% on Qwen, while average ASR changes from 2.41% to 2.85% and from 4.60% to 3.52%, respectively. Using the worst layer lowers average ASR to 1.78% and 2.60%, but raises over-refusal to 12.40% and 16.80%. Last-layer PCA likewise yields higher over-refusal of 10.80% and 10.40%. Thus, although some alternatives achieve lower ASR, CovDPO-Full maintains low ASR while achieving the lowest over-refusal on both backbones.

Overall, these results demonstrate that the safety axis and representation-layer selection contribute to a better safety–over-refusal trade-off, rather than simply minimizing ASR. We will incorporate these additional ablation results into Section 4.5 of the revised manuscript.

AuthorsOpenReview ↗

Author Response to Reviewer A4SH (Part 2/3)

R2: Coverage Weighting and Multi-Seed Robustness.{\color{#f26921}\text{R2: Coverage Weighting and Multi-Seed Robustness.}}

Weakness-2:{\color{#f26921}\text{Weakness-2:}} The coverage-weighting objective and its effects on Qwen's ASR and over-refusal require further explanation, along with uncertainty estimates across multiple random seeds.

Response W2.1: Exact Objective and Adaptive Softmin Aggregation.{\color{#f26921}\text{Response W2.1: Exact Objective and Adaptive Softmin Aggregation.}}

We thank the reviewer for these valuable comments. We first clarify the exact training objective and explain the motivation behind combining coverage weighting with softmin aggregation.

Let hih_i denote the HCS of sample ii and ℓi\ell_i its standard DPO loss. For each microbatch BB, our training objective is

\frac{1}{\lvert B\rvert}\sum_{i\in B} \exp\left(-\ell_i\left[1+\gamma(t)\max(0,\tau-h_i)^2\right]\right) \right).$$ Here, $\gamma(t)$ controls the weighting strength and $\tau$ is the coverage threshold. Writing $w_i$ for the coverage weight in brackets, the effective gradient coefficient is $$a_i=\frac{\partial\mathcal{L}_B}{\partial\ell_i} =w_i\underbrace{\frac{\exp(-w_i\ell_i)} {\sum_{j\in B}\exp(-w_j\ell_j)}}_{p_i}.$$ Coverage weighting and softmin aggregation jointly provide a **loss-adaptive weighting mechanism**. HCS identifies low-coverage samples, but directly assigning them large training contributions may overemphasize samples with high initial losses. Softmin adjusts their effective contributions according to the current weighted loss, moderating the influence of high-loss samples while allowing coverage weighting to increase their effective gradient coefficients as their losses decrease. Specifically, holding the current losses and other weights fixed, the effective gradient coefficient increases with the coverage weight when $w_i\ell_i(1-p_i)<1$, as derived from $$\frac{\partial a_i}{\partial w_i}=p_i[1-w_i\ell_i(1-p_i)]$$ **Table R2: Proportion of weighted training records satisfying the positive gradient-sensitivity condition (%).** | Backbone | Early | Middle | Late | Overall | |:---|:---:|:---:|:---:|:---:| | Llama-3-8B | 93.83 | 98.25 | 98.97 | **97.01** | | Qwen3-8B | 88.56 | 96.09 | 97.66 | **94.10** | Table R2 shows that **increasing the coverage weight raises the effective gradient coefficient for more weighted samples as training progresses**. ${\color{#f26921}\text{Response W2.2: Qwen's Weighting Effects and Multi-Seed Robustness.}}$ To examine whether the original Qwen result is consistent across seeds, we evaluate **CovDPO-Full** and **w/o Coverage Weighting** using seeds **42, 43, and 44**. Table R3 reports the per-seed results and their mean and standard deviation. **Table R3. Coverage-weighting ablation on Qwen3-8B across three seeds. Each cell reports Full / w/o Weight in %.** | Seed | Avg. ASR ↓ | OOD ASR ↓ | XSTest ↓ | |:---|:---:|:---:|:---:| | 42 | 4.60 / 4.71 | 1.55 / 2.38 | 0.80 / 15.20 | | 43 | 1.97 / 5.10 | 0.77 / 2.18 | 0.40 / 2.80 | | 44 | 2.39 / 5.53 | 1.62 / 2.61 | 3.60 / 7.60 | | **Mean ± SD** | **2.99 ± 1.41** / 5.11 ± 0.41 | **1.31 ± 0.47** / 2.39 ± 0.22 | **1.60 ± 1.74** / 8.53 ± 6.25 | **Across all three seeds, CovDPO-Full consistently achieves lower average ASR, OOD ASR, and XSTest over-refusal than the weighting ablation.** Across seeds, the mean of the average ASR drops from **5.11% to 2.99%**, and mean XSTest over-refusal drops from **8.53% to 1.60%**. These results demonstrate that **coverage weighting improves both overall safety robustness and benign-request responsiveness across different random seeds**. Overall, the objective and training diagnostics explain how coverage weighting interacts with softmin to adapt sample contributions, while the multi-seed results demonstrate the benefits of coverage weighting for safety robustness and over-refusal. We will incorporate the exact objective, training diagnostics, and multi-seed results into the appendix of the revised manuscript.
AuthorsOpenReview ↗

Author Response to Reviewer A4SH (Part 3/3)

R3: Adaptive Jailbreak Robustness.{\color{#f26921}\text{R3: Adaptive Jailbreak Robustness.}}

Weakness-3:{\color{#f26921}\text{Weakness-3:}} The original GCG and PAIR evaluations rely on attacks generated against fixed anchor models, leaving robustness to attacks adapted to each trained checkpoint unclear.

Response W3: Evaluation Against Adaptive GCG Attacks.{\color{#f26921}\text{Response W3: Evaluation Against Adaptive GCG Attacks.}}

We thank the reviewer for this valuable suggestion. Due to computational constraints, our original GCG and PAIR evaluations use attacks generated against fixed anchor models rather than attacks optimized for each evaluated checkpoint. To address this concern, we conduct additional Adaptive GCG attacks on the Llama-3-8B model family. Specifically, we independently optimize GCG attacks against each final aligned checkpoint using 100 optimization steps and the same 400 test cases, rather than reusing attacks generated against separate anchor models.

Table R3: Adaptive GCG attack results on Llama-3-8B models.

ModelSuccessful AttacksASR ↓
Unaligned257 / 40064.25%
ShaPO161 / 40040.25%
Standard DPO54 / 40013.50%
DOOR39 / 4009.75%
Staged-Competence12 / 4003.00%
CovDPO-Full0 / 4000.00%

As shown in Table R3, CovDPO-Full achieves 0.00% ASR under Adaptive GCG attacks, compared with 3.00% for the strongest competing baselines. These results provide additional evidence that CovDPO's robustness improvements extend beyond fixed transfer attacks to adversarial optimization directly targeting the final aligned checkpoints. We will include these additional results in Appendix F.4 of the revised manuscript.

R4: Judge Calibration and Over-refusal Evaluation.{\color{#f26921}\text{R4: Judge Calibration and Over-refusal Evaluation.}}

Weakness-4:{\color{#f26921}\text{Weakness-4:}} The judge calibration relies on author-labelled responses used for rubric refinement, while refusal-phrase matching may miss over-refusals. Independent scoring verification is needed to assess the reliability of the reported results.

Response W4: Manual Verification of Safety and Over-refusal Evaluations.{\color{#f26921}\text{Response W4: Manual Verification of Safety and Over-refusal Evaluations.}}

We thank the reviewer for this valuable suggestion. To verify the reliability of our automated evaluations, we manually reviewed model responses on HEx-PHI and XSTest for CovDPO-Full on Llama-3-8B. The results are presented in Table R4.

Table R4: Manual verification of automated safety and over-refusal evaluations. All values are percentages (%).

BenchmarkSamplesAutomaticManual
HEx-PHI ↓3000.000.00
XSTest ↓2500.404.00

For HEx-PHI, manual inspection of all 300 responses confirmed the reported 0.00% ASR, providing an additional check using data separate from the AdvBench rubric-refinement set. For XSTest, manual evaluation identified 10 refusals among 250 responses, yielding an over-refusal rate of 4.00%, compared with 0.40% under the original phrase-matching evaluation.

These results support the reliability of our near-zero ASR findings while showing that automatic XSTest scoring may underestimate absolute over-refusal. Given the limited annotation resources, we retain the standardized benchmark scores for comparisons.

We appreciate your thoughtful comments. We hope our response addresses your concerns. Please let us know if there are any additional questions, and we will be happy to discuss further.

Comments on the paper

2 comments

Dear Reviewer,

As the discussion period is nearing its end, we would like to kindly follow up and ensure that we have adequately addressed your concerns.

If you have any remaining questions or feedback, we would greatly appreciate hearing from you and would be happy to provide further clarification.

Thank you again for your time and thoughtful review.

AuthorsOpenReview ↗

Author Summary of Rebuttal and Additional Experiments

I. Acknowledgments

We thank all reviewers for their constructive feedback, which helped strengthen our empirical validation and methodological clarity.

Before the discussion, we appreciate the recognition from Reviewers A4SH, WoNb (Rating: 3), and FNW1 (Rating: 2):

  • Reviewer A4SH highlighted that our paper "connects a useful diagnosis to a concrete training method", and recognized the substantial safety improvements and comprehensive coverage analyses.
  • Reviewer WoNb acknowledged that HCS provides a unified signal for both data augmentation and loss weighting, and described our experimental results as "very strong."
  • Reviewer FNW1 recognized the connection between representation-space coverage and OOD safety failures as "a meaningful contribution", highlighting the principled motivation behind our approach.

During the discussion, we conducted additional experiments addressing HCS design, benefits beyond data augmentation, training stability, adaptive jailbreak robustness, evaluation reliability, hyperparameter sensitivity, and computational costs.


II. Key Strengths

Reviewers highlighted strengths across three dimensions:

  • Motivation and Unified Coverage-Aware Framework

    • Establishing an empirical connection between insufficient representation-space coverage and OOD safety vulnerabilities, and linking this diagnosis to targeted safety-alignment interventions (A4SH, WoNb, FNW1).
    • Introducing HCS as a unified signal for coverage-guided preference-data augmentation and coverage-weighted DPO (A4SH, WoNb, FNW1).
  • Safety Generalization and Robustness

    • Achieving substantial reductions in average ASR, from 15.90% to 2.41% on Llama-3-8B and from 11.21% to 4.60% on Qwen3-8B, compared with the strongest evaluated baselines (A4SH, WoNb, FNW1).
    • Evaluating safety across multiple OOD harmful benchmarks and jailbreak attacks, alongside general capabilities and over-refusal (A4SH, FNW1).
  • Empirical Analysis and Methodological Clarity

    • Providing component ablations and coverage analyses that help explain the relationship between coverage and safety failures (A4SH, WoNb, FNW1).
    • Presenting a clear research motivation and a structured framework supported by empirical validation (A4SH, FNW1).

III. Key Concerns and Our Responses

We conducted additional experiments and provided clarifications to address the reviewers' major concerns, summarized below.

Key ConcernsReviewersOur Response
HCS Design and Alternative Coverage Measures. Whether the safety axis and layer selection outperform simpler coverage measures.A4SH, WoNbWe conducted matched-budget ablations on Llama and Qwen, comparing CovDPO with w/o Safety Axis, w/ Worst Layer, and w/ Last-Layer PCA, demonstrating a better safety–over-refusal trade-off.
Benefits Beyond Data Augmentation.FNW1We conducted four controlled comparisons: coverage-agnostic augmentation, random augmentation, low-HCS versus high-HCS selection, and fixed-size training, demonstrating the additional benefits of coverage-aware mechanisms beyond data diversity or volume.
Training Objective, Stability, and Hyperparameters. Clarification of softmin aggregation, coverage weighting, seed stability, and hyperparameter sensitivity.A4SH, FNW1We clarified the exact training objective and effective gradient coefficients, conducted three-seed evaluations of coverage weighting on Qwen, and provided hyperparameter sensitivity analyses.
Jailbreak Robustness and Evaluation Reliability. Concerns about fixed attack suites, near-zero ASRs, and automatic refusal detection.A4SH, WoNbWe conducted Adaptive GCG attacks optimized against the final aligned checkpoints and performed manual verification on HEx-PHI and XSTest, providing additional robustness evidence.
Computational Cost. End-to-end overhead of data generation, verification, and training.WoNbWe provided a detailed GPU-hour breakdown of augmentation, verification, and full-parameter DPO training, clarifying upfront costs and data reusability.

IV. Commitment to Revision

We will incorporate the above experiments and methodological clarifications into the revised manuscript (Section 4.5 and Appendix), strengthening the empirical evidence for coverage-aware mechanisms, training stability, robustness, and evaluation reliability while clarifying computational costs.

We sincerely thank the Area Chair and all reviewers for their time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.