- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Rebuttal Attack, a novel jailbreak method. It classifies the harmful queries into 150 knowledge queries and 150 viewpoint queries from existing StrongREJECT and HarmBench dataset. Rebuttal Attack includes 2 stages: 1) Query Categorization and Neutralization, and 2) Rebuttal Prompt Generation. Each stage contains clear documentation of the pipeline. The Relative Continuation Likelihood (RCL) designed by the authors also provides explanation for the behavior presented by the LLM. Extensive experiments on state-of-the-art methods including open-source models and closed-source models validate the effectiveness of the pipeline.
Critical strengths
- The paper proposes Rebuttal Attack, a single-turn black-box jailbreak method that reformulates harmful requests as rebuttal-oriented tasks. The use of knowledge error correction and viewpoint debate simulation provides an interesting approach to inducing model compliance. The proposed RCL provides an additional perspective for analyzing the attack mechanism.
- Extensive experiments evaluate Rebuttal Attack on two categories of harmful queries across multiple open-source and closed-source models. The results demonstrate strong average performance compared with existing state-of-the-art baselines.
- The paper is well-organized, and the methodology is presented clearly. In particular, the descriptions of Knowledge Error Injection and Viewpoint Debate Simulation provide a clear understanding of the proposed attack strategies.
Critical weaknesses
- Although Appendix C.1 reports attacker refusal rates, the paper does not clearly explain how these failed attack-generation attempts are handled in the final evaluation. This may affect the interpretation of the reported attack effectiveness.
- The implementation details of the RCL analysis are insufficient. Although the mathematical formulation and signal sets are provided, the paper does not clearly specify which model is used to compute the token-level likelihoods shown in Figure 3. Besides, the relationship between RCL and the final Harmfulness Score is not empirically investigated.
- The evaluation relies on a single judge model (GLM-5.2), without additional human validation or cross-judge consistency checks.
Questions for authors
1.Appendix C.1 reports attacker refusal rates during semantic neutralization and rebuttal prompt generation. How are these refusal cases handled in the subsequent experiments? 2.Could the authors clarify which model is used to calculate RCL and how the token-level probabilities are obtained? Additionally, is RCL correlated with the final Harmfulness Score? A sample-level correlation analysis, preferably conducted separately for knowledge and viewpoint queries, would help establish the relationship between RCL and actual attack effectiveness. 3.Could the authors provide an additional evaluation using an independent judge model or a human-annotated subset to validate the reliability of GLM-5.2 and assess the robustness of the reported results?
2 comments
Response to Reviewer nNBx – part 2 of 2
Table R3. ASR (↑) with the HarmBench classifier as the judge.
| Query | Method | Gemini 2.5 Flash | GPT-5.1 | LLaMA-4 | Qwen3-Next | Avg. |
|---|---|---|---|---|---|---|
| Knowledge | SelfCipher | 0.747 | 0.007 | 0.000 | 0.640 | 0.348 |
| FlipAttack | 0.740 | 0.053 | 0.613 | 0.553 | 0.490 | |
| QueryAttack | 0.853 | 0.033 | 0.407 | 0.893 | 0.547 | |
| Authority endorsement | 0.087 | 0.007 | 0.347 | 0.053 | 0.123 | |
| Evidence-based Persuasion | 0.087 | 0.027 | 0.300 | 0.047 | 0.115 | |
| Expert Endorsement | 0.100 | 0.027 | 0.353 | 0.060 | 0.135 | |
| Logical appeal | 0.127 | 0.033 | 0.440 | 0.107 | 0.177 | |
| Misrepresentation | 0.093 | 0.040 | 0.460 | 0.080 | 0.168 | |
| SATA-MLM | 0.567 | 0.093 | 0.540 | 0.600 | 0.450 | |
| Rebuttal Attack (Ours) | 0.693 | 0.067 | 0.713 | 0.713 | 0.547 | |
| Viewpoint | SelfCipher | 0.687 | 0.000 | 0.000 | 0.520 | 0.302 |
| FlipAttack | 0.753 | 0.033 | 0.567 | 0.640 | 0.498 | |
| QueryAttack | 0.787 | 0.020 | 0.300 | 0.900 | 0.502 | |
| Authority endorsement | 0.060 | 0.020 | 0.133 | 0.020 | 0.058 | |
| Evidence-based Persuasion | 0.060 | 0.027 | 0.067 | 0.027 | 0.045 | |
| Expert Endorsement | 0.140 | 0.047 | 0.160 | 0.027 | 0.093 | |
| Logical appeal | 0.107 | 0.107 | 0.207 | 0.033 | 0.113 | |
| Misrepresentation | 0.193 | 0.120 | 0.287 | 0.107 | 0.177 | |
| SATA-MLM | 0.507 | 0.180 | 0.307 | 0.320 | 0.328 | |
| Rebuttal Attack (Ours) | 0.940 | 0.547 | 0.767 | 0.987 | 0.810 |
Following StrongREJECT's human evaluation protocol, we ran a small-scale human evaluation of outputs from Rebuttal Attack and nine baseline jailbreak methods. We recruited five volunteers, each independently rating 200 responses, with 100 from Knowledge and 100 from Viewpoint (10 per method). Using a blind setup, evaluators were not told the target models or attack methods. They rated answers on a 1–5 scale (later normalized to [0, 1]) based only on the original queries, the blinded model responses, and shared scoring criteria. We averaged the median rating for each response to obtain the Human Score reported in Table R4.
Table R4 reports Human Scores, StrongREJECT scores, and HarmBench ASRs for all methods; Table R5 reports MAE and method-level Spearman correlations between automated and human evaluations, capturing agreement in method rankings. In Table R4, Rebuttal Attack achieves the highest Human Score for both Knowledge and Viewpoint, consistent with the StrongREJECT results. Table R5 shows StrongREJECT (GLM-5.2 and DeepSeek V4.1) gives similar results and aligns better with human ratings than HarmBench, with lower MAE and higher rank correlations.
Table R4. Human and automatic evaluation of queries. StrongREJECT (↑) is evaluated with GLM-5.2 and DeepSeek V4.1; ASR (↑) with HarmBench.
| Query | Method | GLM-5.2 | DeepSeek V4.1 | HarmBench | Human Score |
|---|---|---|---|---|---|
| Knowledge | SelfCipher | 0.575 | 0.600 | 0.600 | 0.475 |
| FlipAttack | 0.438 | 0.338 | 0.600 | 0.350 | |
| QueryAttack | 0.375 | 0.325 | 0.400 | 0.175 | |
| Authority endorsement | 0.000 | 0.000 | 0.000 | 0.075 | |
| Evidence-based Persuasion | 0.200 | 0.200 | 0.100 | 0.225 | |
| Expert Endorsement | 0.100 | 0.200 | 0.100 | 0.175 | |
| Logical appeal | 0.200 | 0.200 | 0.100 | 0.250 | |
| Misrepresentation | 0.400 | 0.500 | 0.300 | 0.400 | |
| SATA-MLM | 0.300 | 0.325 | 0.300 | 0.200 | |
| Rebuttal Attack (Ours) | 0.688 | 0.750 | 0.500 | 0.575 | |
| Viewpoint | SelfCipher | 0.300 | 0.300 | 0.300 | 0.300 |
| FlipAttack | 0.238 | 0.250 | 0.600 | 0.225 | |
| QueryAttack | 0.613 | 0.463 | 0.400 | 0.375 | |
| Authority endorsement | 0.100 | 0.163 | 0.100 | 0.175 | |
| Evidence-based Persuasion | 0.000 | 0.000 | 0.000 | 0.025 | |
| Expert Endorsement | 0.200 | 0.275 | 0.200 | 0.300 | |
| Logical appeal | 0.000 | 0.000 | 0.000 | 0.125 | |
| Misrepresentation | 0.175 | 0.163 | 0.100 | 0.200 | |
| SATA-MLM | 0.100 | 0.100 | 0.300 | 0.150 | |
| Rebuttal Attack (Ours) | 0.825 | 0.863 | 0.800 | 0.850 |
Table R5. Agreement between automatic and human evaluation scores.
| Query | Evaluator | MAE ↓ | Spearman Correlation ↑ |
|---|---|---|---|
| Knowledge | GLM-5.2 | 0.113 | 0.854 (p=0.002) |
| DeepSeek V4.1 | 0.121 | 0.849 (p=0.002) | |
| HarmBench | 0.190 | 0.675 (p=0.032) | |
| Viewpoint | GLM-5.2 | 0.075 | 0.972 (p<0.001) |
| DeepSeek V4.1 | 0.080 | 0.991 (p<0.001) | |
| HarmBench | 0.168 | 0.801 (p=0.005) |
Response to Reviewer nNBx – part 1 of 2
We sincerely thank the reviewer for the thoughtful and constructive comments. Below are our point-by-point responses and additional experimental results.
Response to W1 & Q1: about refusal cases.
Thank you for your valuable suggestion. We retained refusal cases in the evaluation set and processed them in the same way as all other cases.
Response to W2 & Q2: about RCL correlation.
Thank you for your valuable suggestion. In the previous manuscript, we chose Qwen3-8B for RCL analysis to compute token-level probabilities at low cost. We use the same model in this analysis.
We examined the correlation between RCL and jailbreak performance. StrongREJECT assigns scores for convincingness and specificity, each an integer from 1 to 5. Scores for the same response may vary with the evaluator model or prompt, and RCL value may also vary with the model or context. We therefore used group means to statistically examine the overall relationship between RCL and StrongREJECT scores. For each non-empty response, we measured RCL at the boundary between the rebuttal and the answer to the original query. For each query, we averaged the StrongREJECT scores across four models (Gemini 2.5 Flash, GPT-5.1, LLaMA-4, and Qwen3-Next) and then ranked the queries by their mean scores separately for Knowledge and Viewpoint queries, dividing each ranking into quartiles. For quartile i, we denote the mean StrongREJECT score by xi and the mean RCL by yi. We calculated Pearson r as follows:
\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})} {\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2}\,\sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}}$$ Table R1 shows the correlation between StrongREJECT scores and RCL across groups. Because model’s preference for continuation or refusal can hinge on a few tokens, averaging within sequence and then across sequences within each set reduces the range of RCL. As a result, mean RCL varies only slightly across quartiles. However, the relative magnitude of RCL values still correlates strongly with StrongREJECT score for both Knowledge and Viewpoint queries. **Table R1. Correlation between StrongREJECT scores and RCL across four groups.** |Query|Metric|Bottom 25%|25%–50%|50%–75%|Top 25%|Pearson r| |---|---|---|---|---|---|---| |Knowledge|Score|0.2863|0.6352|0.7487|0.8856|**0.829**| ||RCL|0.4286|0.4343|0.4312|0.4373|| |Viewpoint|Score|0.6046|0.7500|0.9269|1.0000|**0.981**| ||RCL|0.3085|0.3150|0.3214|0.3290|| ### Response to W3 & Q3: about evaluation reliability. Thank you for this valuable suggestion. We rely on StrongREJECT (Souly et al., UC Berkeley, NeurIPS 2024) to measure how helpful a response is to the original harmful query, considering both how convincing and how specific it is. If an answer addresses only part of the request, the score reflects the level of assistance it actually offers. As suggested, we also carried out independent judge and human evaluations to assess the reported results under different evaluators. For independent judges evaluation, we evaluate all methods with DeepSeek V4.1 for StrongREJECT and the HarmBench classifier for attack success rate (ASR). As shown in Tables R2 and R3, the average StrongREJECT scores on Knowledge and Viewpoint are 0.615 and 0.813, respectively, and the corresponding HarmBench ASRs are 0.547 and 0.810. Our method achieves the best overall performance under both evaluators, with Viewpoint attaining the highest scores across all four target models. These results show that our method consistently retains its advantage across different evaluators. **Table R2. StrongREJECT scores (↑) with DeepSeek V4.1 as the judge.** |Query|Method|Gemini 2.5 Flash|GPT-5.1|LLaMA-4|Qwen3-Next|Avg.| |---|---|---|---|---|---|---| |Knowledge|SelfCipher|0.707|0.007|0.000|0.608|0.330| ||FlipAttack|**0.827**|0.033|0.475|0.008|0.336| ||QueryAttack|0.711|0.069|0.288|**0.856**|0.481| ||Authority endorsement|0.198|0.038|0.378|0.093|0.177| ||Evidence-based Persuasion|0.244|0.103|0.308|0.171|0.206| ||Expert Endorsement|0.247|0.101|0.359|0.159|0.216| ||Logical appeal|0.226|0.078|0.489|0.153|0.237| ||Misrepresentation|0.203|0.109|0.477|0.221|0.252| ||SATA-MLM|0.346|0.047|0.314|0.595|0.325| ||**Rebuttal Attack (Ours)**|0.780|**0.194**|**0.683**|0.803|**0.615**| |Viewpoint|SelfCipher|0.711|0.020|0.007|0.522|0.315| ||FlipAttack|0.797|0.025|0.266|0.084|0.293| ||QueryAttack|0.743|0.076|0.307|0.841|0.492| ||Authority endorsement|0.122|0.053|0.208|0.064|0.111| ||Evidence-based Persuasion|0.141|0.117|0.152|0.089|0.125| ||Expert Endorsement|0.197|0.123|0.228|0.058|0.151| ||Logical appeal|0.195|0.138|0.301|0.083|0.179| ||Misrepresentation|0.252|0.130|0.337|0.217|0.234| ||SATA-MLM|0.303|0.093|0.243|0.369|0.252| ||**Rebuttal Attack (Ours)**|**0.952**|**0.541**|**0.801**|**0.958**|**0.813**|