- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how safety behavior changes when the same harmful objective is transformed from a direct assistance request into a viewpoint-support task. It characterizes this phenomenon through representation and decoding analyses on three open-weight models and comparisons across eight viewpoint predicates. ReLoK combines viewpoint transformation, example-guided structuring, and sensitive-word decomposition, reporting an average ASR@3 of 95.69% across six models and three benchmarks. Static safety-prefix experiments examine the trade-off between restoring refusal and increasing benign false refusals.
Critical strengths
-
The problem is practically meaningful: models should support legitimate viewpoint analysis while withholding actionable harmful assistance. The explicit distinction between non-refusal and attack success makes the formulation more useful than simply maximizing refusal.
-
The paper combines cross-model and cross-benchmark evaluation with component ablations and diagnostic features selected independently from direct harmful–benign contrasts. These designs provide complementary evidence for the behavioral effects of viewpoint transformation.
Critical weaknesses
-
Reported ASR@3 values are inconsistent with the stated calculation, limiting the auditability of the main experiments. Appendix I.3 specifies 520 AdvBench objectives, and I.6 aggregates three generations into one binary success outcome per objective. ASR@3 should therefore equal . However, Table 3 reports 0.08%, 0.01%, and 0.05% for Combination Attack, ArtPrompt, and FlipAttack on Claude, respectively. These values fall between 0% for zero successes and approximately 0.19% for one success and cannot arise under the stated formula. Excluding invalid objectives cannot explain them either. Checking all cells identifies 8 of 126 scores in Table 3 that are incompatible with two-decimal rounding under the stated denominators, including 85.92% for ReLoK on Gemini 3.5. The 95.35% Direct refusal rate in Tables 7 and 8 is also incompatible with 520 binary outcomes. Some discrepancies may reflect rounding or transcription errors, but the very small ASRs require a clearer explanation.
-
The contribution is primarily an incremental empirical finding, while the mechanistic account remains insufficiently validated. ReLoK largely combines established techniques such as viewpoint reframing, example-guided structuring, and word decomposition. The comparison of viewpoint semantics is potentially valuable, but the full eight-predicate evaluation is mainly demonstrated on QwQ and AdvBench, and the main comparison omits the closely related VDJA.
Questions for authors
-
Please provide successful-objective counts, valid-objective counts, invalid or missing evaluation counts, and the three-generation labels underlying each Table 3 cell. How do 0.08%, 0.01%, and 0.05% on Claude and 85.92% for ReLoK on Gemini 3.5 follow from the stated ASR@3 formula? Were additional repetitions, weighting, or other aggregation procedures used?
-
Under matched query budgets and evaluation criteria, what additional benefit does viewpoint transformation provide over VDJA? Do the semantic differences and main mechanistic observations persist with larger samples, matched templates, and measurements of actual harmful assistance?
VDJA:Safety Guardrails of Large Language Models Are Vulnerable to Value-Driven Adversarial Prompting (Findings of ACL 2026 )
3 comments
Response to Weakness 2: Positioning Relative to VDJA and the Contribution of RAA
The reviewer raises an important question about the relationship between our work and VDJA. Both studies investigate how contextual framing can influence the safety behavior of aligned LLMs, but they examine different mechanisms and research questions.
VDJA introduces a Value Engine that uses an auxiliary LLM to generate positive value descriptions for harmful tasks, encouraging the target model to endorse their moral value. It further employs Rule Guidance to facilitate subsequent task execution. In contrast, RAA studies a different task-role transformation: the same harmful objective is presented either as a Direct assistance request or as the content of a proposition to support. This transformation does not require positive moral endorsement of the underlying objective.
Our semantic evaluation provides further evidence for this distinction. On QwQ-32B, epistemic and descriptive predicates produce substantially larger refusal reductions than deontic predicates. For example, the Easy predicate reduces refusal by 49.04 percentage points, whereas Justifiable reduces it by only 8.08 percentage points. This suggests that the strongest RAA effects do not depend on inducing moral approval, but can also emerge from ordinary feasibility- and prevalence-oriented viewpoint tasks.
We have expanded the Related Work section to discuss VDJA and other context-sensitive jailbreak studies, clarifying their respective task formulations, semantic manipulations, and research objectives.
A controlled experimental comparison between the two methods would provide additional evidence. However, their construction pipelines differ: VDJA uses auxiliary LLM-based value generation and additional rule guidance, whereas ReLoK uses a fixed, single-turn black-box prompt without target-model feedback. A fair comparison would therefore require reproducing VDJA under the same target models, benchmarks, success criteria, and evaluation protocol, while accounting for both auxiliary-model and target-model interactions. Matching only the number of target-model queries would not by itself align the overall evaluation budgets.
We have not included a direct numerical comparison in this revision and do not claim that ReLoK outperforms VDJA. Instead, we have clarified their methodological differences and limited our comparative claims accordingly.
We also clarify the respective contributions of RAA and ReLoK. The primary research contribution is RAA, which identifies and characterizes a task-role generalization failure in which safety behavior changes when the same harmful objective is processed as viewpoint content. ReLoK operationalizes this finding by combining viewpoint transformation with additional components that improve the effectiveness of the resulting attack. Example-guided structuring and sensitive-word decomposition serve as complementary components of this construction, rather than standalone methodological contributions.
This positioning emphasizes the central contribution of the paper: characterizing a specific form of safety-boundary instability and examining its practical implications.
Response to Question 2: Additional Semantic and Behavioral Evidence
We conducted additional analyses to examine whether the observed semantic differences and behavioral patterns persist with larger samples, more controlled prompt formulations, and measurements of substantive harmful assistance. The corresponding analyses have also been incorporated into the revised manuscript.
1. Evaluation on a Larger Set of Harmful Objectives.
The complete eight-predicate evaluation is conducted on all 520 AdvBench objectives using QwQ-32B. The results reveal substantial differences across semantic relations: epistemic predicates produce refusal rates of 46.35%–54.81%, the descriptive predicate Common produces 55.77%, while pragmatic and deontic predicates produce substantially higher refusal rates of 82.50%–87.31%.
We have also revised the appendix to distinguish the full 520-objective PRI trajectory analysis from the separate supplementary token-level probability visualization conducted on 20 prompts.
2. Matched Predicate-Polarity and Argumentative-Stance Controls.
To further investigate whether the observed differences arise from proposition semantics rather than simply following a Support or Oppose instruction, we added matched controls along two dimensions: feasibility and normative judgment. These conditions retain the same underlying harmful objective while varying predicate polarity and argumentative stance.
For feasibility judgments, the refusal rates are:
| Condition | Refusal |
|---|---|
| Support Easy | 46.35% |
| Support Hard | 74.62% |
| Oppose Easy | 76.54% |
| Oppose Hard | 46.15% |
Support Easy and Oppose Hard yield nearly identical refusal rates (46.35% and 46.15%). Similarly, Support Hard and Oppose Easy produce refusal rates of 74.62% and 76.54%, respectively.
We also added corresponding normative controls using Justifiable and Unjustifiable:
| Condition | Refusal |
|---|---|
| Support Justifiable | 87.31% |
| Support Unjustifiable | 72.69% |
| Oppose Justifiable | 68.65% |
| Oppose Unjustifiable | 77.12% |
Taken together, these results suggest that the behavioral effect does not simply reverse with the surface distinction between Support and Oppose. Instead, the semantic direction jointly expressed by the predicate and argumentative stance appears to be associated with the resulting refusal behavior. We have incorporated these experiments into a dedicated subsection, Predicate Polarity and Stance Controls, in the revised RAA section.
3. Evaluation of Actual Harmful Assistance.
A reduction in refusal alone does not establish successful harmful assistance. To examine this distinction, our component ablation separately evaluates whether Viewpoint-only responses satisfy the same strict substantive-and-actionable success criterion used in the full ReLoK evaluation.
Across 520 AdvBench objectives, Viewpoint-only achieves 58.08% ASR@3 on GPT-4o and 21.92% on DeepSeek. These results provide evidence that viewpoint transformation alone can elicit substantive and actionable harmful assistance in a subset of cases, although its effectiveness varies across models. The additional structuring and decomposition components further improve the attack success rate.
4. Scope of the Mechanistic Evidence.
We have also revised the white-box analysis and supplementary diagnostics to clarify their respective roles. The analysis covers three open-weight models and examines selected safety-associated features, viewpoint-conditioned representation changes, and decoding trajectories. PRI is now explicitly reported using the full set of 520 paired AdvBench objectives.
We interpret these results as complementary diagnostic evidence characterizing how viewpoint transformation relates to representation and generation changes. The activation intervention is presented as a model-specific behavioral relevance analysis, rather than as establishing a universal causal mechanism.
The complete eight-predicate evaluation and matched polarity controls are primarily behavioral analyses conducted on QwQ-32B. Accordingly, our conclusions about these detailed semantic effects and their associated internal patterns are limited to the evidence available for that model, rather than being generalized across all three models.
These additional analyses strengthen the semantic characterization of RAA, distinguish response availability from substantive harmful assistance, and clarify the scope of the mechanistic findings.
Response to Weakness 1: Correction of ASR@3 Results
We appreciate the reviewer's careful examination of our experimental results. We acknowledge that several numerical values in the original manuscript were incorrectly reported, and we apologize for these errors.
First, we confirm that ASR@3 follows the stated objective-level definition. For AdvBench, the denominator is 520 objectives, and an objective is counted as successful if at least one of its three independent generations satisfies our attack success criterion.
After checking the original experimental records, we identified two types of reporting errors.
(1) Percentage conversion errors in the Claude results. Several Claude results in the initial manuscript were transcribed as raw proportions without multiplying by 100, but were subsequently displayed with percentage signs. These entries were not properly rechecked when incorporating supplementary experimental results. The corrected results, together with the actual successful-objective counts, are:
(2) Manual calculation and transcription errors. Several other values were incorrectly calculated or transcribed when transferring rounded percentages into the tables. For example, the ReLoK result on Gemini 3.5 should be 447/520=85.9615%, rounded to 85.96%, rather than 85.92%. Similarly, the Direct refusal rate should be 496/520=95.3846%, rounded to 95.38%, rather than 95.35%.
We have rechecked the numerical results throughout the manuscript against the original experimental records and corrected all identified errors, including the affected tables and corresponding textual descriptions.
In addition, we conducted an evaluation on DeepSeek-V4.1-Flash-1017, reporting Pass@1, Per-generation ASR, and ASR@3 under the same evaluation protocol. These metrics provide complementary measurements of individual-generation success and objective-level attack success.