- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes DIER, a black-box jailbreak method that uses task-residualized response embeddings to guide attack strategy retrieval. Specifically, the method computes the embedding difference between the target model's response and the original task, estimates a task-specific common direction from previously observed difference vectors for the same task, and removes this direction through orthogonal projection. The resulting residual vector is used as a retrieval representation for attack strategies within an LLM-based evolutionary search framework. The authors evaluate the method on 400 HarmBench behaviors across 21 target models and conduct representation analyses, ablation studies, and cross-model transfer experiments.
Critical strengths
This paper proposes a representation for attack strategy retrieval based on the target model's responses. By removing task-relevant directions to guide evolutionary search, it offers a novel approach for optimizing the retrieval and selection of attack strategies.
Critical weaknesses
- Limited innovation: DIER essentially follows the jailbreak search paradigm established by AutoDAN-Turbo, which relies on a strategy library and LLM-based generation. The primary distinction lies in the use of "task-residualized response representation" as the new embedding for strategy retrieval. Consequently, the substantive methodological innovation of this paper relative to existing work is rather limited.
- Insufficient experimental validation: First, the baseline results are largely cited from prior literature rather than being reproduced under a unified experimental setup; thus, the fairness of comparisons across different methods warrants further clarification. Second, the paper’s coverage of more robust closed-source models is limited, and it lacks direct comparative experiments against representative existing jailbreak methods on the closed-source models that were tested. Furthermore, given that the core innovation is the "task-residualized response representation," the absence of an ablation study directly comparing performance with and without this "task-residualization" makes it difficult to fully substantiate the independent contribution of this design. Finally, since the authors posit that this representation improves strategy retrieval, could it be extended to other similar automated jailbreak optimization algorithms to further validate the method's generalizability, practical effectiveness, and impact on average query consumption?
Questions for authors
-
The clarity of the paper’s writing could be improved. Certain parts are currently difficult to follow; I suggest refining the text to make the description of the proposed method clearer and more accessible. For instance, even after reading the abstract, it remains challenging to intuitively grasp the specific method being proposed and how it fundamentally differs from existing research.
-
Regarding the use of "Judges" in the experiments: The experimental setup involves two distinct types of Judges. Please clarify the specific roles each of these Judges plays within the experiments.
-
Regarding performance gains and search efficiency: In the main experiments, DIER achieves an average ASR of 86.4%, compared to 86.1% for AutoDAN-Turbo—an improvement of only 0.3 percentage points. However, the paper highlights a performance gain of approximately 31 percentage points; I suggest explicitly stating the baseline against which this figure is calculated. Given that the overall performance improvement of DIER over AutoDAN-Turbo is quite marginal, does the proposed method offer any advantages in terms of search efficiency or computational cost? While the paper reports the query index at which the first successful attack occurs, it appears to lack a direct comparison of efficiency against existing methods like AutoDAN-Turbo.
-
Regarding the experimental categories in Table 3: Table 3 presents results across four categories—"All," "Standard," "Contextual," and "Copyright"—yet the paper does not clearly define what these categories entail. Please provide a detailed explanation of the definitions for each category.