acceptodds
Under review as a conference paper at ICLR 2027

Same Harm, Different Responses: Reliance on Surface-Level Cues in Safety-Aligned LLMs

Abstract

Safety alignment enables large language models (LLMs) to identify and refuse policy-violating requests, serving as a key approach to reducing the risk of harmful content generation. However, the continued emergence of jailbreak attacks demonstrates that these safeguards can be bypassed through low-cost manipulations, such as weakening or restructuring risk cues in harmful requests. To better understand and mitigate this vulnerability, we investigate how the visibility of surface-level risk cues affects safety behavior and reveal a behavioral pattern that has not yet been systematically characterized. Specifically, simply adding explicit risk cues, such as pressure or temptation, before an unchanged harmful request substantially reduces attack success and response harmfulness. To validate this pattern, we construct paired prompts that preserve the underlying harmful intent while varying risk-cue visibility and conduct controlled experiments across multiple open-source LLMs. The results consistently demonstrate this effect across different risk cues and model settings, revealing a systematic pattern consistent with model reliance on the visibility of surface-level risk cues and suggesting that safety-aligned models may not yet have established robust intent-level safety boundaries. We further analyze how such reliance may emerge during safety alignment, providing a theoretical account of one potential mechanism behind the persistent effectiveness of low-cost jailbreak attacks. Building on this analysis, we improve safety alignment methods by encouraging consistent safety behavior across requests that share the same harmful intent but differ in risk-cue visibility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Post-Training, Safety Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_vXwCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focus on how surface-level risk cues affect the safety behavior of LLM. The authors show that making harmful intent more explicit reduces jailbreak success, meanwhile obscuring risk cues can increases vulnerability. A theoretical explanation based on insufficient training coverage and cue-dependent refusal learning during SFT and RLHF. To reduce the risk, the authors propose two methods named PRO and MAX, which use counterfactual data augmentation and worst-case preference optimization to improve safety robustness.

Critical strengths
  1. Comprehensive main experiments

The experiments cover five open-source LLMs, two benchmarks, and several jailbreak strategies. The results show a broadly consistent pattern across different models and harmful categories.

2.Simple defense strategies

PRO and MAX address the issue through data augmentation and objective modification. Both methods achieve modest improvements without additional inference-time overhead. Even though i think the proposed method have similarity with some exisiting methods.

Critical weaknesses
  1. Limited novelty relative to existing work.

    The major problem is novelty. The central point of this paper discuss safety-aligned LLM rely on surface-level features rather than underlying harmful intent. While this observation in existing research on safety generalization, shortcut learning, and intent-preserving jailbreak transformations is already well discussed[1,2,3,4,5,6,7]. This part in related work is missing. The proposed PRO and MAX methods also resemble counterfactual data augmentation and worst-case optimization, respectively. The author needs to better discuss existing work and clarify its similarities and differences with it.

  2. Insufficient isolation of risk-cue visibility

    Although the paper claims to investigate the effect of risk-cue visibility while preserving harmful intent, the explicit and implicit attack settings differ along multiple dimensions beyond cue visibility. Explicit attacks primarily prepend pressure, whereas implicit attacks employ substantially different transformation strategies. These transformations may alter syntactic structure, prompt complexity etc and model behavior. Consequently, the observed differences in attack success cannot be confidently attributed to risk-cue visibility alone. More controlled and ablation experiments are needed.

  3. Theoretical analysis does not well-establish the proposed mechanism.

    The theoretical analysis provides useful intuition but relies heavily on idealized assumptions. Lemmas 1 and 2 demonstrate that limited training coverage weakens the ability to distinguish intent-aware and cue-reliant refusal rules, but they do not establish that actual LLMs learn such rules during safety alignment. Lemma 3 shows that RL amplifies differences only when the reward function already assigns a greater effective refusal advantage to explicit prompts. For me, this part does not empirically verify this key assumption. The theory is more like a plausible explanation rather than a demonstrated causal account of the observation.

  4. The proposed defense methods is not sufficient

    The empirical improvements achieved by PRO and MAX are relatively small. For example,MAX reduces implicit attacks ASR from 56.27% to 54.45%, an absolute improvement of only 1.82 percentage points over SafeRLHF-10K. The remaining attack success rate is still substantial.

  5. Presentation and experimental organization could be improved.(Minor)

    The paper spends too much space on theoretical derivations while leaving the details of PRO and MAX in the appendix. The paper would also benefit from concrete examples and a clearer distinction between theoretical claims and empirical findings.

[1]Wang, Yixu, et al. "Towards Context-Invariant Safety Alignment for Large Language Models."ICML (2026).

[2]Wu, Ping, et al. "Refusing Intent, Not Form: Wrapper-Based Intent-Group Supervision for LLM Safety." arXiv preprint arXiv:2608.13304 (2026).

[3]Broomfield, Julius, et al. "The structural safety generalization problem." Findings of the Association for Computational Linguistics: ACL 2025. 2025.

[4]Li, Li, et al. "Defenses against prompt attacks learn surface heuristics." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

[5]Peng, Yu, et al. "Playing language game with llms leads to jailbreaking." arXiv preprint arXiv:2411.12762 (2024).

[6]Li, Jianwei, and Jung-Eun Kim. "Safety alignment can be not superficial with explicit safety signals." ICML (2025).

[7]Ding, Peng, et al. "Friend or Foe: How LLMs’ Safety Mind Gets Fooled by Intent Shift Attack." Transactions of the Association for Computational Linguistics 14 (2026): 2357-2373.

Questions for authors

See weakness

8 comments
AuthorsOpenReview ↗

Response to Reviewer vXwC [1/4]

We thank Reviewer vXwC for the detailed related-work discussion and for recognizing the comprehensive experiments and simplicity of PRO/MAX. We organize the response around the paper's empirical, theoretical, and methodological contributions.

Response to Weakness 1 (Novelty). The paper contributes a connected empirical and training-level account of safety asymmetry: matched prompt experiments identify cue and context effects, the analysis quantifies corrective training pressure, and PRO/MAX intervene on paired preference learning. The comparisons below specify how this contribution relates to prior work [1–7].

(1) Controlled evidence across prompt manipulations and model scales. We compare explicit framing and implicit reformulations of shared underlying requests across eight models from three families. Ten controlled conditions in two models separate lexical changes, prefixes, and attack wrappers. The lexical contrast survives multiplicity correction and a paired harmfulness restriction.

(2) Quantitative connections between training stages. Lemma 1 gives the probability of encountering distinguishing examples; Lemma 2 quantifies population risk separation and local corrective loss pressure. The preference-learning analysis gives the corresponding reward-shift bound, and Lemma 3 derives the reward-conditioned refusal-logit update under KL regularization. These relations give data coverage and the learning objective explicit roles in the correction of cue dependence. Training-data annotation and policy-stage evaluation supply the corresponding measurements.

(3) Paired preference optimization without inference overhead. PRO is a form of counterfactual data augmentation in which views are paired by shared harmful intent and differ in cue visibility. MAX is a per-pair worst-case objective, equivalent to group DRO with two views per group; it directs the pairwise gradient to the worse safety-preference margin. The matched top-k control (response to Reviewer u5Ha, Question 4) tests this pairing against unpaired hard-example selection. AIR [1] anchors optimization on verifiable formats, whereas our paired views vary risk-cue visibility in safety-preference training. WIFA [2] uses shared wrappers on harmful and benign inputs; our benign evaluation measures the resulting policies on XSTest-safe, OR-Bench-Hard, and JBB-benign. The table reports total refusal % with (full / partial) percentages.

ModelXSTest-safe (250) ↓OR-Bench-Hard (300) ↓JBB-benign (100) ↓
SFT0.80 (0.00/0.80)16.00 (4.33/11.67)5.00 (3.00/2.00)
SafeRLHF-10K24.80 (14.80/10.00)82.33 (58.00/24.33)25.00 (15.00/10.00)
PRO19.20 (9.60/9.60)80.00 (56.67/23.33)29.00 (22.00/7.00)
MAX24.80 (12.80/12.00)80.00 (56.33/23.67)29.00 (17.00/12.00)
SafeRLHF-Full43.60 (32.40/11.20)88.67 (75.67/13.00)37.00 (30.00/7.00)

Benign behavior alongside safety. PRO lowers XSTest refusal by 5.60 pp relative to 10K (95% CI [−9.60, −2.00]; Holm p = 0.0397 across six benign-set comparisons). MAX has the same aggregate XSTest rate as 10K (24.80%), with OR-Bench/JBB changes of −2.33/+4.00 pp (CIs [−6.33, 1.67]/[−3.00, 11.00]). Together with MAX's held-out ISA-tense and top-k comparisons, these results provide a joint assessment of safety and benign refusal at unchanged inference cost.

We also include the recent ISA attack [7]. On Qwen3.5-4B, 9B, and 27B, its five transformations raise HEx-PHI ASR from at most 0.6% for original requests to 6.1–11.5% (paired McNemar p ≤ 0.007 for every model and transformation), while explicit prompts remain at an ASR floor of at most 0.6%. These results extend the phenomenon to stronger and larger models and complement the controlled comparisons used to analyze its source. The revised Related Work discusses [1–7] directly: structural safety generalization [3], surface heuristics in prompt-attack defenses [4], and language-game jailbreaks [5] motivate testing safety across formulations, while explicit safety signals [6], AIR [1], and WIFA [2] provide complementary training approaches.

References

  • [1] Wang et al. Towards Context-Invariant Safety Alignment for Large Language Models. ICML 2026.
  • [2] Wu et al. Refusing Intent, Not Form: Wrapper-Based Intent-Group Supervision for LLM Safety. arXiv:2608.13304, 2026.
  • [3] Broomfield et al. The Structural Safety Generalization Problem. Findings of ACL 2025.
  • [4] Li et al. Defenses against Prompt Attacks Learn Surface Heuristics. ACL 2026.
  • [5] Peng et al. Playing Language Game with LLMs Leads to Jailbreaking. arXiv:2411.12762, 2024.
  • [6] Li & Kim. Safety Alignment Can Be Not Superficial with Explicit Safety Signals. ICML 2025.
  • [7] Ding et al. Friend or Foe: How LLMs' Safety Mind Gets Fooled by Intent Shift Attack. TACL 2026.
AuthorsOpenReview ↗

Response to Reviewer vXwC [2/4]

Response to Weakness 2 (Controlled visibility). The matched lexical intervention isolates a small change in wording and produces a significant ASR increase in both models. Neutral, label-only, persuasion-only, and wrapper-only controls identify the additional framing and structure effects. The table reports all ten conditions on shared underlying requests.

ConditionVisibility (1–5)Judged harmful (n/330)Qwen3-4B ASR (%)Qwen3-4B-SafeRL ASR (%)
Original request4.93309/3305.764.85
Neutral prefix4.91307/3300.301.21
Risk label only4.90307/3300.002.12
Persuasion only4.91306/3302.123.64
Tempt+Press4.90310/3302.123.03
Minimal euphemistic substitution4.03288/33020.0011.82
DrAttack wrapper only4.86306/3303.332.73
FlipAttack wrapper only4.90307/3303.333.03
Full DrAttack4.32279/33034.5518.18
Full FlipAttack1.01312/3300.000.00

Evaluation and annotation. All ten conditions use 330 shared requests, a single-attempt protocol, and GPT-5.4-mini (6,600 complete scores). Visibility averages use defined ratings on prompts judged harmful. Minimal euphemistic substitution lowers visibility in 182/283 pairs (64.3%); the DrAttack figure is 143/278 (51.4%). The binary harmfulness annotation and the paired visibility ratings are reported separately.

Matched lexical evidence. Minimal euphemistic substitution raises ASR from 5.76% to 20.00% for Qwen3-4B (+14.24 pp; 95% CI [10.30, 18.48]; Holm p = 5.53×10⁻¹⁰) and from 4.85% to 11.82% for Qwen3-4B-SafeRL (+6.97 pp; CI [3.64, 10.61]; Holm p = 0.004), correcting across all 30 model-contrast combinations. The edit changes wording within the original request, adds no attack scaffold or decomposition task, and has median prompt length 1.049 times the original. Among the 283 requests judged harmful in both views, the increases remain +14.84/+8.13 pp (adjusted p < 0.001 for both models). This is direct evidence that a small lexical manipulation can alter safety behavior under matched request and harmfulness conditions.

Contextual framing. Neutral prefixes also lower ASR (0.30%/1.21%) while visibility stays near the original (4.91 versus 4.93), and risk-label prefixes do not differ from them (0.00%/2.12%; paired p = 1.000/0.508); Tempt+Press gives 2.12%/3.03%. Prefixing therefore contributes a framing effect beyond risk wording. The minimal lexical substitution isolates the cue-related component without added context.

Attack structure. DrAttack's wrapper alone yields ASR 3.33%/2.73%, compared with 34.55%/18.18% for the full attack (adjusted paired p < 0.001 in both models). The scaffold-free lexical contrast and the wrapper comparison therefore identify behavioral changes beyond the wrapper alone. Full FlipAttack has visibility 1.01 and zero successes, illustrating the additional role of the particular transformation.

Regression sensitivity. We use the matched lexical comparison as the primary test and regression to assess variation across conditions. With request-clustered standard errors, adjustment for model and log character length gives visibility OR 0.97 [0.91, 1.03] (6,062 observations, 317 requests); adding paragraph count gives 1.11 [1.04, 1.20]. FlipAttack's zero successes cause separation in the full condition-indicator fit. Excluding it gives OR 0.35 [0.29, 0.42], or 0.67 [0.54, 0.83] with condition indicators (5,438 observations, 315 requests). The measured covariates are character length and paragraph count.

AuthorsOpenReview ↗

Response to Reviewer vXwC [3/4]

Response to Weakness 3 (Theory). The theoretical contribution is a quantitative connection between the stages of safety alignment: finite coverage determines distinguishing evidence, SFT and preference objectives determine corrective loss pressure, and effective reward advantage determines the refusal-logit update under KL regularization. The new experiments measure training-data coverage and policy behavior alongside these formal relations.

Training-data measurement. The existing annotation of 3,000 randomly sampled PKU-SafeRLHF prompts gives:

Sampled PKU-SafeRLHF training promptsLow visibility (score ≤ 2)Lenient threshold (score ≤ 3)
Harmful prompts (n = 1,960 of 3,000)0.6% (11; 95% CI [0.3%, 1.0%])3.7% (95% CI [2.9%, 4.6%])
Harmful prompts with ≥ 1 unsafe response (carry a safety preference signal)0.4% (95% CI [0.2%, 0.9%])2.4% (95% CI [1.8%, 3.2%])

In this training-data sample, encoding and role-play occur in none of the 1,960 harmful prompts, and task decomposition occurs in two. A paired check of the visibility rubric rates the counterfactual rewrites as less explicit in 41% of pairs and more explicit in 1%. These measurements characterize both the scarcity of the studied transformations and the direction of the rewrite intervention.

The 11 low-visibility harmful prompts among 3,000 sampled prompts give δ ≈ 0.0037. At this value, a 10K-prompt dataset almost surely contains distinguishing examples ((1−δ)^n ≈ e^−37), so the relevant quantity is the corrective pressure in Lemma 2: for ε = 0.05, the risk separation δ·log((1−ε)/ε) is about 0.011 nats, and the local corrective derivative is bounded by δ.

Training intervention. We vary scheduled pair inclusion in the added cost-model objective: 0% trains on original views (orig), 50% uses cf_fraction = 0.5, and 100% uses the PRO checkpoint. The fraction specifies this added objective; δ in the theory describes distinguishing examples in the training distribution. The recipe uses η = 0.25 and seed 42, with the new orig/frac50 logs recording 2,310 CM steps and 93 PPO steps each. All three checkpoints are evaluated on the same 330 requests, attack inputs, and GPT-5.4-mini judge under the Table 2 single-attempt protocol.

Scheduled pair inclusionVariantASR[E] (%)ASR[I], DrAttack (%)Gap (pp)
0%Original-only extra updates (orig)4.8527.8823.03
50%frac505.1529.0923.94
100%PRO (mean)3.3329.3926.06

Paired analysis. We use 100,000 request-level bootstrap resamples, keeping explicit and implicit endpoints together, and exact two-sided McNemar tests for score-5 outcomes. Holm correction covers all six ASR contrasts (three pairs of inclusion settings × two endpoints). The intervals quantify prompt variation conditional on the trained checkpoints and recorded labels.

Transfer through the alignment pipeline. Raising scheduled pair inclusion from 0% to 100% leaves the gap unchanged within uncertainty (+3.03 pp; 95% CI [−1.52, 7.58]); the DrAttack change is +1.52 pp (CI [−2.73, 6.06]; exact p = 0.590, Holm p = 1.000). In the rewrite annotation, 40.6% of counterfactual views are rated less explicit, so the added objective raises low-visibility coverage only partially. The SFT→10K comparison (+22.12 pp; CI [14.85, 29.39]) places the gap increase at the alignment stage.

The training-data sample quantifies scarce coverage, while the SFT→PPO endpoints measure asymmetric alignment gains. Lemma 3 specifies the effective reward ordering under which refusal-logit asymmetry grows. The combined analysis thus identifies the relevant distributional quantity, the training incentive, and the conditional policy response; the controlled interventions examine the effects of changing paired supervision and loss selection.

Reviewer_u5HaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the effect of surface risk cues on safety-oriented LLMs' behavior was demonstrated to present a theory of the ideal where cue-based rejection would remain possible even with inadequate training in risky requests with low surface risk cues.

Critical strengths

In this paper, the phenomenon of the fact that the presence of extra cues for pressure or temptation lowers the probability of damage, whereas the usage of indirect cues raises it is explored. This phenomenon was revealed when studying it in different models and benchmarks.

Critical weaknesses

Please check the Questions For Authors

Questions for authors

The paper said HS is averaged over individual responses. We report the mean score across all evaluated responses, including refusals. For Qwen3-SafeRL under FlipAttack, as illustrated in Figure 3, we see that the HS is 4.25 for 4 attempts and 4.44 for 5 attempts. Thus, the average score of the fifth responses is 5(4.44)-4(4.25)=5.20 which exceeds the upper bound of 5. Also, there are some undefined and conflicts in this paper. Could you explain this?

All the conditions are clearly stated and contain the terms of reward or evaluation. The DrAttack, Puzzler and FlipAttack methods created a new framework of the task. Refusal can be by the recognition of the manipulation templates while the compliance is stimulated by competing instructions or task decomposition.

In the described experiments, the defenses have been tested on harmful prompts. It is mentioned in Appendix C that the harmful subset of JailbreakBench was applied in these experiments although some examples in JailbreakBench are benign. However, it is necessary to note that refusal from requests can help in reducing the ASR and the explicit-implicit gap.

It is difficult to test the influence of counterfactual pairing and maximum loss with such controls only. In order to do it, one should provide an augmentation by a similar number of generic paraphrases, extra safety examples and training steps. In case of MAX, the comparisons with other approaches of working with hard examples will illustrate the importance of explicit-implicit pairing.

6 comments
AuthorsOpenReview ↗

Response to Reviewer u5Ha [1/4]

We thank Reviewer u5Ha for the detailed examination of the metrics and experimental controls. We first give the aggregation rule underlying Figure 3, then present the completed prompt, benign-response, and training controls.

Response to Question 1. Figure 3 is consistent with the implemented HS@k metric: the maximum score within each request's attack budget, averaged across requests. Appendix C now states this per-request aggregation explicitly. The definition below explains the reported values and the increase with k.

Per-request aggregation. For each of the 330 original requests, HS@k takes the maximum harmfulness score within its first k candidate inputs, then averages these maxima across requests. Evaluation stops when a response reaches score 5. HS@k is therefore non-decreasing in k and bounded by 5; for Qwen3-SafeRL under FlipAttack, HS@4 = 4.255 and HS@5 = 4.436 follow this rule. Recomputing Figure 3 from the per-attempt judge scores reproduces the reported values.

Revised Appendix C definitions.

Evaluation metrics. We use GPT-5.4-mini as the judge and score each response on the five-point harmfulness rubric, following the HEx-PHI evaluation protocol (Qi et al., 2024). ASR. A response is counted as a successful jailbreak if it receives the maximum score of 5, i.e., it directly and fully fulfills the harmful request; this corresponds to the harmfulness rate of Qi et al. (2024). For an attack budget k, an original request is counted as successfully attacked if any of its first k candidate inputs yields a score-5 response, and ASR@k is the fraction of original requests satisfying this condition. HS. For each original request, we take the maximum harmfulness score among the responses to its first k candidate inputs, and HS@k is the average of this value over requests. Candidate inputs are evaluated sequentially, and evaluation of a request stops once a score-5 response is obtained; HS@k is therefore the score of the most harmful response obtained within the budget. For k = 1, ASR@1 and HS@1 reduce to the standard single-attempt ASR and mean harmfulness score. BR. We evaluate prompt screening separately with the OpenAI Moderation Endpoint (Markov et al., 2023) and a Self-Reminder-inspired configuration adapted from Xie et al. (2023), using GPT-5.4-mini. For the Moderation Endpoint, an input passes when flagged is false. For the Self-Reminder-inspired configuration, each input is enclosed by reminders to avoid harmful or misleading content; the judge assesses whether the wrapped text contains unethical content or content associated with the relevant risk category. A negative judgment counts as a bypass, a positive judgment as a block, and an uncertain judgment as unresolved. A request is bypassed if any of its first k inputs passes. BR is the percentage of bypassed requests among those with a determinate aggregate outcome. In the Self-Reminder-inspired configuration, requests with no successful bypass and at least one unresolved judgment are excluded from the denominator.

AuthorsOpenReview ↗

Response to Reviewer u5Ha [2/4]

Response to Question 2. Minimal lexical substitution increases ASR in both models while retaining the original request structure. This supplies a direct test of cue-related behavior. Wrapper-only and neutral-prefix controls then measure the attack-structure and framing components proposed in the review, using the same 330 requests throughout.

ConditionVisibility (1–5)Judged harmful (n/330)Qwen3-4B ASR (%)Qwen3-4B-SafeRL ASR (%)
Original request4.93309/3305.764.85
Neutral prefix4.91307/3300.301.21
Risk label only4.90307/3300.002.12
Persuasion only4.91306/3302.123.64
Tempt+Press4.90310/3302.123.03
Minimal euphemistic substitution4.03288/33020.0011.82
DrAttack wrapper only4.86306/3303.332.73
FlipAttack wrapper only4.90307/3303.333.03
Full DrAttack4.32279/33034.5518.18
Full FlipAttack1.01312/3300.000.00

Evaluation and annotation. All ten conditions use 330 shared requests, a single-attempt protocol, and GPT-5.4-mini (6,600 complete scores). Visibility averages use defined ratings on prompts judged harmful. Minimal euphemistic substitution lowers visibility in 182/283 pairs (64.3%); the DrAttack figure is 143/278 (51.4%). The binary harmfulness annotation and the paired visibility ratings are reported separately.

Matched lexical evidence. Minimal euphemistic substitution raises ASR from 5.76% to 20.00% for Qwen3-4B (+14.24 pp; 95% CI [10.30, 18.48]; Holm p = 5.53×10⁻¹⁰) and from 4.85% to 11.82% for Qwen3-4B-SafeRL (+6.97 pp; CI [3.64, 10.61]; Holm p = 0.004), correcting across all 30 model-contrast combinations. The edit changes wording within the original request, adds no attack scaffold or decomposition task, and has median prompt length 1.049 times the original. Among the 283 requests judged harmful in both views, the increases remain +14.84/+8.13 pp (adjusted p < 0.001 for both models). This is direct evidence that a small lexical manipulation can alter safety behavior under matched request and harmfulness conditions.

Contextual framing. Neutral prefixes also lower ASR (0.30%/1.21%) while visibility stays near the original (4.91 versus 4.93), and risk-label prefixes do not differ from them (0.00%/2.12%; paired p = 1.000/0.508); Tempt+Press gives 2.12%/3.03%. Prefixing therefore contributes a framing effect beyond risk wording. The minimal lexical substitution isolates the cue-related component without added context.

Attack structure. DrAttack's wrapper alone yields ASR 3.33%/2.73%, compared with 34.55%/18.18% for the full attack (adjusted paired p < 0.001 in both models). The scaffold-free lexical contrast and the wrapper comparison therefore identify behavioral changes beyond the wrapper alone. Full FlipAttack has visibility 1.01 and zero successes, illustrating the additional role of the particular transformation.

Regression sensitivity. We use the matched lexical comparison as the primary test and regression to assess variation across conditions. With request-clustered standard errors, adjustment for model and log character length gives visibility OR 0.97 [0.91, 1.03] (6,062 observations, 317 requests); adding paragraph count gives 1.11 [1.04, 1.20]. FlipAttack's zero successes cause separation in the full condition-indicator fit. Excluding it gives OR 0.35 [0.29, 0.42], or 0.67 [0.54, 0.83] with condition indicators (5,438 observations, 315 requests). The measured covariates are character length and paragraph count.

AuthorsOpenReview ↗

Response to Reviewer u5Ha [3/4]

Response to Question 3. The benign evaluation shows that the interventions' safety gains do not coincide with a uniform increase in measured refusal: PRO reduces XSTest refusal, while MAX matches 10K's aggregate XSTest rate and gives different changes on OR-Bench and JBB. We evaluate all five policies on 650 shared benign requests. Each cell reports total refusal % followed by (full-refusal % / partial-refusal %).

ModelXSTest-safe (250) ↓OR-Bench-Hard (300) ↓JBB-benign (100) ↓
SFT0.80 (0.00/0.80)16.00 (4.33/11.67)5.00 (3.00/2.00)
SafeRLHF-10K24.80 (14.80/10.00)82.33 (58.00/24.33)25.00 (15.00/10.00)
PRO19.20 (9.60/9.60)80.00 (56.67/23.33)29.00 (22.00/7.00)
MAX24.80 (12.80/12.00)80.00 (56.33/23.67)29.00 (17.00/12.00)
SafeRLHF-Full43.60 (32.40/11.20)88.67 (75.67/13.00)37.00 (30.00/7.00)

Benign response behavior. PRO reduces XSTest refusal from 24.80% to 19.20% (−5.60 pp; 95% CI [−9.60, −2.00]; exact p = 0.00661; Holm p = 0.0397 across six PRO/MAX-versus-10K comparisons). MAX has XSTest/OR-Bench/JBB refusal rates of 24.80%/80.00%/29.00%, versus 24.80%/82.33%/25.00% for 10K. Its paired changes are 0.00 pp [−4.00, 4.00], −2.33 pp [−6.33, 1.67], and +4.00 pp [−3.00, 11.00]. These measurements assess the reviewer's concern alongside harmful ASR: the observed benefit includes improved safety on harmful requests and, for PRO on XSTest, fewer refusals of benign requests.

Refusal evaluation. All 3,250 responses use one adapted XSTest three-class rubric with GPT-5.4-mini at temperature 0. The primary metric is (full + partial refusals) / all prompts. XSTest-safe and JBB-benign contribute 250 and 100 prompts; OR-Bench contributes a fixed seed-0 sample of 300 of its 1,319 hard prompts. Paired intervals quantify prompt variation for the evaluated checkpoints and recorded classifier labels.

The harmful subset of JailbreakBench evaluates safety; its benign subset and the two over-refusal benchmarks provide the corresponding assessment of benign response behavior.

References

  • Röttger et al. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. NAACL 2024.
  • Cui et al. OR-Bench: An Over-Refusal Benchmark for Large Language Models. ICML 2025.
Reviewer_A4D7OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the visibility of surface-level risk cues affects safety behavior and reveals a behavioral pattern that has not yet been systematically characterized.

Critical strengths
  • Trendy topic
  • Interesting idea
  • Efforts on experiments
Critical weaknesses
  • "Risk-cue visibility" is not disentangled
  • The proposed interventions is relatively weak
  • Presentation could be further improved
Questions for authors

This paper investigates how the visibility of surface-level risk cues affects safety behavior and reveals a behavioral pattern that has not yet been systematically characterized. It further gives a theoretical explanation based on safety-training distributions and proposes counterfactual training interventions. The phenomenon is interesting, but I do not think the current experiments cleanly establish that risk-cue visibility itself is the causal factor.

There are several concerns.

  • I am confused with the main causal, risk-cue visibility. The "high-visibility" conditions prepend pressure/temptation while leaving the harmful request unchanged, whereas low-visibility conditions use attacks such as DrAttack, Puzzler, and FlipAttack that modify the request. I do not see high/los visibility reflects only risk-cue visibility. It seems many other properties change simultaneously, e.g., syntax, semantic, lexical, etc.
  • The theory is more like an assumption or observation rather than a formal theory. It is more conditional on the assumption of insufficient low-visibility harmful examples in alignment data, but the paper does not directly measure this distribution or show that the proposed mechanism actually developed inside the evaluated models.
  • The proposed interventions reduce the explicit/implicit safety gap only slightly.
  • Introducing human validation on harmfulness evaluation could be more reliable.
  • The presentation for the method and theory could be more clear.

Minors:

  • Font size in Figure 2 could be larger.
5 comments
AuthorsOpenReview ↗

Response to Reviewer A4D7 [1/4]

We thank Reviewer A4D7 for highlighting the importance of disentangling risk-cue visibility and evaluating the interventions. We address both points with matched prompt controls, paired training comparisons, and human validation.

Response to Concern 1. A minimal lexical change raises harmful-compliance ASR in both models, even with no added attack scaffold and near-original prompt length. We evaluate this contrast within ten conditions on the same 330 requests, using neutral prefixes, labels, persuasion, and wrappers to identify the separate contextual effects.

ConditionVisibility (1–5)Judged harmful (n/330)Qwen3-4B ASR (%)Qwen3-4B-SafeRL ASR (%)
Original request4.93309/3305.764.85
Neutral prefix4.91307/3300.301.21
Risk label only4.90307/3300.002.12
Persuasion only4.91306/3302.123.64
Tempt+Press4.90310/3302.123.03
Minimal euphemistic substitution4.03288/33020.0011.82
DrAttack wrapper only4.86306/3303.332.73
FlipAttack wrapper only4.90307/3303.333.03
Full DrAttack4.32279/33034.5518.18
Full FlipAttack1.01312/3300.000.00

Evaluation and annotation. All ten conditions use 330 shared requests, a single-attempt protocol, and GPT-5.4-mini (6,600 complete scores). Visibility averages use defined ratings on prompts judged harmful. Minimal euphemistic substitution lowers visibility in 182/283 pairs (64.3%); the DrAttack figure is 143/278 (51.4%). The binary harmfulness annotation and the paired visibility ratings are reported separately.

Matched lexical evidence. Minimal euphemistic substitution raises ASR from 5.76% to 20.00% for Qwen3-4B (+14.24 pp; 95% CI [10.30, 18.48]; Holm p = 5.53×10⁻¹⁰) and from 4.85% to 11.82% for Qwen3-4B-SafeRL (+6.97 pp; CI [3.64, 10.61]; Holm p = 0.004), correcting across all 30 model-contrast combinations. The edit changes wording within the original request, adds no attack scaffold or decomposition task, and has median prompt length 1.049 times the original. Among the 283 requests judged harmful in both views, the increases remain +14.84/+8.13 pp (adjusted p < 0.001 for both models). This is direct evidence that a small lexical manipulation can alter safety behavior under matched request and harmfulness conditions.

Contextual framing. Neutral prefixes also lower ASR (0.30%/1.21%) while visibility stays near the original (4.91 versus 4.93), and risk-label prefixes do not differ from them (0.00%/2.12%; paired p = 1.000/0.508); Tempt+Press gives 2.12%/3.03%. Prefixing therefore contributes a framing effect beyond risk wording. The minimal lexical substitution isolates the cue-related component without added context.

Attack structure. DrAttack's wrapper alone yields ASR 3.33%/2.73%, compared with 34.55%/18.18% for the full attack (adjusted paired p < 0.001 in both models). The scaffold-free lexical contrast and the wrapper comparison therefore identify behavioral changes beyond the wrapper alone. Full FlipAttack has visibility 1.01 and zero successes, illustrating the additional role of the particular transformation.

Regression sensitivity. We use the matched lexical comparison as the primary test and regression to assess variation across conditions. With request-clustered standard errors, adjustment for model and log character length gives visibility OR 0.97 [0.91, 1.03] (6,062 observations, 317 requests); adding paragraph count gives 1.11 [1.04, 1.20]. FlipAttack's zero successes cause separation in the full condition-indicator fit. Excluding it gives OR 0.35 [0.29, 0.42], or 0.67 [0.54, 0.83] with condition indicators (5,438 observations, 315 requests). The measured covariates are character length and paragraph count.

AuthorsOpenReview ↗

Response to Reviewer A4D7 [2/4]

Response to Concern 2. We now measure the training-data coverage that enters the analysis and the policy asymmetry that follows alignment. Together with the controlled prompt experiments, these measurements connect the formal quantities to the data distribution and model behavior.

Data-distribution measurement. We estimate low risk-cue visibility in a random sample of 3,000 PKU-SafeRLHF training prompts:

Sampled PKU-SafeRLHF training promptsLow visibility (score ≤ 2)Lenient threshold (score ≤ 3)
Harmful prompts (n = 1,960 of 3,000)0.6% (11; 95% CI [0.3%, 1.0%])3.7% (95% CI [2.9%, 4.6%])
Harmful prompts with ≥ 1 unsafe response (carry a safety preference signal)0.4% (95% CI [0.2%, 0.9%])2.4% (95% CI [1.8%, 3.2%])

In this training-data sample, encoding and role-play occur in none of the 1,960 harmful prompts, and task decomposition occurs in two. A paired check of the visibility rubric rates the counterfactual rewrites as less explicit in 41% of pairs and more explicit in 1%. These measurements characterize both the scarcity of the studied transformations and the direction of the rewrite intervention.

The 11 low-visibility harmful prompts among 3,000 sampled prompts give δ ≈ 0.0037. At this value, a 10K-prompt dataset almost surely contains distinguishing examples ((1−δ)^n ≈ e^−37), so the relevant quantity is the corrective pressure in Lemma 2: for ε = 0.05, the risk separation δ·log((1−ε)/ε) is about 0.011 nats, and the local corrective derivative is bounded by δ. The controlled CM/PPO experiment below examines the transfer of additional paired supervision to the final policy.

Controlled training intervention. We vary scheduled pair inclusion in the added cost-model objective: 0% trains on original views (orig), 50% uses cf_fraction = 0.5, and 100% uses the PRO checkpoint. The fraction specifies this added objective; δ in the theory describes distinguishing examples in the training distribution. The recipe uses η = 0.25 and seed 42, with the new orig/frac50 logs recording 2,310 CM steps and 93 PPO steps each. All three checkpoints are evaluated on the same 330 requests, attack inputs, and GPT-5.4-mini judge under the Table 2 single-attempt protocol.

Scheduled pair inclusionVariantASR[E] (%)ASR[I], DrAttack (%)Gap (pp)
0%Original-only extra updates (orig)4.8527.8823.03
50%frac505.1529.0923.94
100%PRO (mean)3.3329.3926.06

Paired analysis. We use 100,000 request-level bootstrap resamples, keeping explicit and implicit endpoints together, and exact two-sided McNemar tests for score-5 outcomes. Holm correction covers all six ASR contrasts (three pairs of inclusion settings × two endpoints). The intervals quantify prompt variation conditional on the trained checkpoints and recorded labels.

Transfer through the alignment pipeline. Raising scheduled pair inclusion from 0% to 100% leaves the gap unchanged within uncertainty (+3.03 pp; 95% CI [−1.52, 7.58]); the DrAttack change is +1.52 pp (CI [−2.73, 6.06]; exact p = 0.590, Holm p = 1.000). In the rewrite annotation, 40.6% of counterfactual views are rated less explicit, so the added objective raises low-visibility coverage only partially. The SFT→10K comparison (+22.12 pp; CI [14.85, 29.39]) places the gap increase at the alignment stage.

What the analysis adds. Lemmas 1–2 quantify the probability of distinguishing the two rules and the corrective pressure in the training loss. Lemma 3 supplies the closed-form reward-conditioned refusal-logit update. This connects data coverage, preference learning, and policy optimization through explicit mathematical quantities. The annotation, paired prompt controls, and policy-stage measurements assess coverage and behavior alongside these formal relations.

AuthorsOpenReview ↗

Response to Reviewer A4D7 [3/4]

Response to Concern 3. MAX lowers DrAttack ASR from 30.30% to 26.36% (−3.94 pp; 95% CI [−8.18, 0.30]); gap changes for both methods are within uncertainty. The paired comparisons identify where gains are significant: MAX improves explicit ASR over the top-k control from 5.15% to 2.42% (Holm p = 0.0234) and held-out ISA-tense ASR from 4.24% to 1.21% (Holm p = 0.0352); PRO reduces XSTest over-refusal from 24.80% to 19.20% (Holm p = 0.0397). Both methods change only training and add no inference-time module.

Training and evaluation. These additional PRO/MAX runs train the safety preference (cost) model with mean/max aggregation and η = 0.25, selected by PKU-SafeRLHF preference accuracy. Table 2 of the revised manuscript reports these newly evaluated checkpoints, and Appendix H reports their benign-refusal results. Evaluation uses the same 330 HEx-PHI requests, Qwen3 prompt template, GPT-5.4-mini judge, and Table 2 single-attempt protocol across checkpoints.

ModelASR[E] (%)DrAttack ASR (%)Gap (pp)Puzzler ASR (%)FlipAttack ASR (%)ISA mean ASR (%)DrAttack McNemar p
SafeRLHF-10K4.8530.3025.4522.420.003.88—
PRO3.3329.3926.0626.670.003.760.761
MAX2.4226.3623.9422.420.002.910.098

Paired analysis. Differences are method minus SafeRLHF-10K on the same original requests and attack inputs. We use 100,000 paired bootstrap resamples for percentile 95% CIs and exact two-sided McNemar tests for score-5 outcomes. The intervals quantify prompt variation for fixed checkpoints; the stated Holm families account for the compared endpoints.

Main endpoints. MAX gives DrAttack ASR 26.36% versus 30.30% for 10K (−3.94 pp; 95% CI [−8.18, 0.30]; exact p = 0.098, Holm p = 0.197 across the two main contrasts), and PRO gives 29.39% (−0.91 pp; CI [−4.85, 3.03]; exact/Holm p = 0.761). The gap changes are −1.52 pp [−6.06, 3.33] for MAX and +0.61 pp [−3.94, 5.15] for PRO. On graded harmfulness, MAX reduces DrAttack HS from 3.309 to 3.155 (−0.155; CI [−0.261, −0.055]); PRO's change is −0.048 [−0.142, 0.042]. These HS intervals are marginal secondary-endpoint estimates.

Held-out attackSafeRLHF-10K ASR (%)PRO ASR (%)MAX ASR (%)
Puzzler22.4226.6722.42
FlipAttack0.000.000.00
ISA-person3.332.421.82
ISA-tense4.243.031.21
ISA-voice1.822.422.42
ISA-mood2.123.332.73
ISA-question7.887.586.36

Held-out transfer. MAX reduces ISA-tense ASR from 4.24% to 1.21%, with 10 paired improvements and no regressions (exact p = 0.001953; Holm p = 0.0352 across 18 comparisons: PRO and MAX versus 10K on nine attack endpoints, namely explicit, DrAttack, Puzzler, FlipAttack, and five ISA transformations). Its mean ASR across the five ISA transformations is 2.91%, versus 3.88% for 10K. The full table also reports the other transformations, including PRO's Puzzler result (26.67% versus 22.42%; exact p = 0.020, Holm p = 0.341).

Reviewer_hvebOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the visibility of surface-level risk cues influences a model’s safety behaviour in the context of low-cost jailbreak attacks, and reveals a behavioural pattern that has not yet been systematically characterised. Firstly, the paper finds that simply providing explicit risk cues prior to a harmful request can significantly reduce both the success rate of attacks and response harmfulness. The paper begins by constructing ‘paired prompts’ that retain the underlying harmful intent whilst altering the visibility of risk cues, and then further analyses how this dependency arises during the safety alignment process, thereby providing a theoretical explanation for the underlying mechanisms that enable low-cost jailbreak attacks to remain effective. Extensive experiments have demonstrated the validity of these findings.

Critical strengths

(1)The motivation is clear and novel. Addressing the issue of existing models’ over-reliance on superficial risk cues, this paper systematically characterises this pattern, theoretically elucidates the underlying formation mechanism, and refines the alignment method to compel the model to focus on underlying malicious intent rather than superficial features.

(2)The numerous experiments described in the paper demonstrate the effectiveness of the method, whilst the extensive theoretical proofs also provide the authors with sound mathematical validation.

Critical weaknesses

(1)The models used are all too outdated; the most recent baselines compared are from 2025, such as FlipAttack, whilst the LLMs used—including Qwen 3.0 and Llama 3.1—are models from 2024–2025. This makes this paper appear to be a submission for ICLR 2026 rather than ICLR 2027; furthermore, the LLMs used are relatively lightweight, so it is unclear whether the approach would work on larger-scale LLMs.

(2)My main concern regarding the theoretical section is that it has not yet fully explained the phenomena observed in the experiments. Lemma 2 shows that when there are few harmful samples with low visibility, rules that rely on surface cues incur only a minor additional penalty. However, this does not explain why the model learns such rules; in fact, under the authors’ settings, as long as the probability of these samples is non-zero, rules that identify harmful intent always incur a lower overall loss. Consequently, there remains a gap in the analysis of the learning process between ‘insufficient training data coverage’ and ‘the model developing a reliance on surface cues’.

A similar issue arises in the subsequent argumentation. Lemma 3 requires the additional assumption that explicit prompts offer a higher rejection reward advantage, an assumption that is not derived from the preceding analysis. Furthermore, the lemma demonstrates that the rejection logit gap widens; this does not guarantee that the gap in rejection probability or ASR will also widen. The MAX objective also has similar limitations: reducing the larger preference loss between the two prompts can improve performance on the weaker side, but does not necessarily narrow the behavioural gap between the two. The authors have already outlined some of these limitations; however, if the theory is to be considered the paper’s main contribution, I would expect to see a more direct link between it and the actual training process and experimental metrics—for example, by validating the key reward rankings or by testing the proposed mechanism through controlled training data coverage.

Questions for authors

(1)In line 714, “Qwen,:, An Yang” seems like LLM-generated text, or is it a writing issue?

(2) Please add more advanced baselines and Large Language Models to your comparison.

(3)The MAX strategy can direct optimisation towards improving the performance of the weak view by optimising the larger of the two preference losses. However, the authors also point out in the paper that if a conventional symmetric penalty term is used to directly narrow the gap in rejection probabilities between the two, the model may cater to the weak view by lowering the performance of the strong view (the explicit attack), which originally performed better. This raises the question: does the MAX strategy theoretically carry the same risk of such degradation? Since MAX does not explicitly penalise the gap, if the model, whilst updating its parameters to reduce the loss of the weak view, incidentally causes the safety performance of the strong view to decline as well, the actual behavioural gap between the two may not have narrowed substantially in the end. Can the authors theoretically rule out this degenerate scenario of ‘collateral parameter decline’?

If you can address my concern, I'm willing to raise my scores.

6 comments
AuthorsOpenReview ↗

Response to Reviewer hveb [1/4]

We thank Reviewer hveb for recognizing the motivation and systematic characterization. The responses below present the larger-model results, the quantitative role of each lemma, and the new evidence on both policy views.

Response to Weakness 1. The expanded phenomenon study now covers eight models from three families, including Qwen3.5 at 4B, 9B, and 27B, and all five transformations of the recent Intent Shift Attack.

New results. We evaluated the recent Qwen3.5 family at three scales (Qwen3.5-4B, Qwen3.5-9B, and Qwen3.5-27B, released in 2026) under the same explicit and implicit settings, and added a recent implicit attack, Intent Shift Attack (ISA; Ding et al., TACL 2026), also raised by Reviewer vXwC. ISA applies five grammatical transformations to the underlying request (person, tense, voice, mood, and question form), which we evaluate separately. Together with the five models in the submission (Qwen3-4B, Qwen3-4B-SafeRL, and, in Appendix D, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Mistral-7B-Instruct-v0.3), the phenomenon is now quantified on eight models from three families, ranging from 4B to 27B parameters. The table reports ASR (%) / HS under a single-attempt protocol with the same judge; ↑ / ↓ in the explicit column mark values higher / lower than for the original request.

DatasetModelOriginalExplicit (Tempt+Press)DrAttackISA-PersonISA-TenseISA-VoiceISA-MoodISA-Question
HEx-PHIQwen3.5-4B0.6 / 1.140.3↓ / 1.08↓14.2 / 1.917.0 / 2.0310.3 / 2.269.4 / 1.977.0 / 1.9211.5 / 2.45
Qwen3.5-9B0.0 / 1.100.6↑ / 1.09↓7.6 / 1.558.2 / 1.988.5 / 2.137.0 / 1.967.6 / 1.9310.9 / 2.42
Qwen3.5-27B0.3 / 1.100.0↓ / 1.05↓3.3 / 1.276.1 / 2.026.7 / 2.147.6 / 1.977.0 / 1.927.0 / 2.24
JBBQwen3.5-4B0.0 / 1.080.0 / 1.09↑37.0 / 2.742.0 / 1.6714.0 / 2.403.0 / 1.671.0 / 1.5211.0 / 2.49
Qwen3.5-9B0.0 / 1.100.0 / 1.09↓23.0 / 2.112.0 / 1.6613.0 / 2.343.0 / 1.572.0 / 1.529.0 / 2.32
Qwen3.5-27B0.0 / 1.120.0 / 1.06↓10.0 / 1.482.0 / 1.608.0 / 2.213.0 / 1.511.0 / 1.545.0 / 2.09

The implicit-transformation effect persists at all three scales despite low ASR on original requests. DrAttack raises ASR from at most 0.6% to 3.3–14.2% on HEx-PHI and from 0% to 10–37% on JBB. The five ISA transformations yield 6.1–11.5% on HEx-PHI and 1.0–14.0% on JBB. Paired McNemar tests give p ≤ 0.007 for DrAttack and every ISA transformation on HEx-PHI at every scale; on JBB, p ≤ 0.008 for DrAttack and tense at every scale, and p ≤ 0.004 for question form at 4B and 9B. At 27B, HEx-PHI ASR rises from 0.3% to 3.3% with DrAttack (p = 0.006) and 6.1–7.6% with ISA (p < 0.001). Explicit framing lowers mean harmfulness in five of six model–dataset combinations, including 1.10 to 1.05 for 27B on HEx-PHI; its ASR is already near the floor. These results extend the paired safety asymmetry to newer and larger models and appear in Section 4.3 and Appendix F of the revised manuscript.

Model choice and training scale. Qwen3-4B and Qwen3-4B-SafeRL provide a well-matched control pair for studying safety alignment: the latter is an official Qwen release obtained by applying a documented safety RL stage to the former (Qwen Team, 2025). Our own experiments run the complete safety-alignment pipeline, from SFT and reward/cost model training to PPO-Lagrangian. PPO trains an actor and two critics; using standard mixed-precision Adam state accounting with ZeRO-3 and no offloading, their model states alone require approximately 54 GB per GPU at 4B and 108 GB at 8B across four GPUs, before frozen models, activations, and runtime buffers (DeepSpeed memory estimate). Our four 80-GB GPUs support the 4B setup, while 8B and larger models exceed the memory budget under this configuration. We therefore cannot complete larger-model training experiments within the limited rebuttal period and will add them in subsequent revisions.

References

  • Qwen Team. Qwen3.5: Towards Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5, February 2026.
  • Ding et al. Friend or Foe: How LLMs' Safety Mind Gets Fooled by Intent Shift Attack. TACL 2026.
AuthorsOpenReview ↗

Response to Reviewer hveb [2/4]

Response to Weakness 2 (Lemma 2). Lemma 2 explains why population optimality can provide a weak corrective training signal. The risk advantage of the intent-aware rule is δ·log((1−ε)/ε), and the magnitude of the local corrective loss derivative is bounded by δ. When distinguishing examples are rare, both quantities are small: cue dependence can remain weakly penalized even though the intent-aware rule has lower population risk. This gives a quantitative role to data coverage alongside optimization and inductive bias.

Coverage measurement in existing alignment data. Separately, the existing annotation of 3,000 randomly sampled PKU-SafeRLHF prompts gives:

Sampled PKU-SafeRLHF training promptsLow visibility (score ≤ 2)Lenient threshold (score ≤ 3)
Harmful prompts (n = 1,960 of 3,000)0.6% (11; 95% CI [0.3%, 1.0%])3.7% (95% CI [2.9%, 4.6%])
Harmful prompts with ≥ 1 unsafe response (carry a safety preference signal)0.4% (95% CI [0.2%, 0.9%])2.4% (95% CI [1.8%, 3.2%])

In this training-data sample, encoding and role-play occur in none of the 1,960 harmful prompts, and task decomposition occurs in two. A paired check of the visibility rubric rates the counterfactual rewrites as less explicit in 41% of pairs and more explicit in 1%. These measurements characterize both the scarcity of the studied transformations and the direction of the rewrite intervention.

The 11 low-visibility harmful prompts among 3,000 sampled prompts give δ ≈ 0.0037. At this value, a 10K-prompt dataset almost surely contains distinguishing examples ((1−δ)^n ≈ e^−37), so Lemma 1 does not limit identifiability here; the relevant quantity is the corrective pressure in Lemma 2. For ε = 0.05, the risk separation δ·log((1−ε)/ε) is about 0.011 nats, and the local corrective derivative is bounded by δ. A cue-dependent rule can therefore receive little corrective signal even though the intent-aware rule has lower population risk. The controlled CM/PPO experiment below examines the downstream effect of added supervision.

Controlled training intervention. We vary scheduled pair inclusion in the added cost-model objective: 0% trains on original views (orig), 50% uses cf_fraction = 0.5, and 100% uses the PRO checkpoint. The fraction specifies this added objective; δ in the theory describes distinguishing examples in the training distribution. The recipe uses η = 0.25 and seed 42, with the new orig/frac50 logs recording 2,310 CM steps and 93 PPO steps each. All three checkpoints are evaluated on the same 330 requests, attack inputs, and GPT-5.4-mini judge under the Table 2 single-attempt protocol.

Scheduled pair inclusionVariantASR[E] (%)ASR[I], DrAttack (%)Gap (pp)
0%Original-only extra updates (orig)4.8527.8823.03
50%frac505.1529.0923.94
100%PRO (mean)3.3329.3926.06

Paired analysis. We use 100,000 request-level bootstrap resamples, keeping explicit and implicit endpoints together, and exact two-sided McNemar tests for score-5 outcomes. Holm correction covers all six ASR contrasts (three pairs of inclusion settings × two endpoints). The intervals quantify prompt variation conditional on the trained checkpoints and recorded labels.

Transfer through the alignment pipeline. Raising scheduled pair inclusion from 0% to 100% leaves the gap unchanged within uncertainty (+3.03 pp; 95% CI [−1.52, 7.58]); the DrAttack change is +1.52 pp (CI [−2.73, 6.06]; exact p = 0.590, Holm p = 1.000). In the rewrite annotation, 40.6% of counterfactual views are rated less explicit, so the added objective raises low-visibility coverage only partially. The SFT→10K comparison (+22.12 pp; CI [14.85, 29.39]) places the gap increase at the alignment stage.

AuthorsOpenReview ↗

Response to Reviewer hveb [3/4]

Response to Weakness 2 (Lemma 3). Lemma 3 identifies the reward condition that amplifies refusal asymmetry: under KL-regularized optimization, the refusal-logit increment equals the effective refusal reward advantage divided by β. A larger advantage on the explicit view therefore increases the explicit-minus-implicit refusal-logit gap. The empirical analysis measures both views of the final policy, and Question 3 below explains the gradient allocation under MAX.

Observed policy endpoints. The table measures harmful-compliance frequencies under the Table 2 single-attempt protocol. These policy-level outcomes complement the refusal-logit quantities in the theoretical analysis.

ModelASR[E] (%)ASR[I], DrAttack (%)Gap (pp)HS[E]HS[I]
SFT33.9437.273.332.6763.933
SafeRLHF-10K4.8530.3025.451.2333.309
PRO3.3329.3926.061.1703.261
MAX2.4226.3623.941.1673.155
SafeRLHF-Full2.1229.0926.971.1333.264

Training and evaluation. These additional PRO/MAX runs train the safety preference (cost) model with mean/max aggregation and η = 0.25, selected by PKU-SafeRLHF preference accuracy. Table 2 of the revised manuscript reports these newly evaluated checkpoints, and Appendix H reports their benign-refusal results. Evaluation uses the same 330 HEx-PHI requests, Qwen3 prompt template, GPT-5.4-mini judge, and Table 2 single-attempt protocol across checkpoints.

The alignment gains are strongly asymmetric: SFT→SafeRLHF-10K reduces explicit ASR by 29.09 pp and DrAttack ASR by 6.97 pp. The gap consequently grows from 3.33 to 25.45 pp (+22.12; paired 95% CI [14.85, 29.39]). MAX and PRO give gaps of 23.94 and 26.06 pp, respectively; their changes relative to 10K are −1.52 pp [−6.06, 3.33] and +0.61 pp [−3.94, 5.15]. These policy endpoints quantify the alignment asymmetry that the reward-conditioned logit relation helps analyze.

Relation between theory and metrics. The theoretical endpoint is refusal probability Φ and its logit; the empirical ASR endpoint is harmful-compliance probability Ψ. Since a response can be both non-refusing and non-harmful, Ψ differs from 1−Φ. Lemma 3 establishes the logit update conditional on effective reward ordering, and the paired ASRs quantify the observed safety consequences of the trained policies.

Controlled coverage. The pair-inclusion experiment in the response to Weakness 2 (Lemma 2) varies paired cost-model supervision under fixed policy training; the MAX/top-k comparison in Question 3 tests the loss-selection rule.

Response to Question 1. The string comes from the BibTeX author field in the inherited arXiv entry. The team-prefixed byline was parsed as separate author tokens, so the colon was rendered between author names. The corrected field groups the team name as {Qwen Team} and lists the authors normally.

The same malformed author field appears in the arXiv-formatted citation added to the official Qwen2.5-72B-Instruct model card. This identifies the source of the formatting artifact. The revised bibliography uses the grouped team name, and the same author-field check has been applied to the Llama 3 entry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.