- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a jailbreak method for vision-lanuage models, called UniCal. The authors claimed that most universal jailbreak attacks rely on surrogate-side optimization and query-based attacks often require fresh target queries for every request. Therefore these methods face a trade-off between reuse and target-specific adaptation. To address the limitation, UniCal moves rarget adaptation from the high-dimensional input space to a reusable low-dimensional behavioral coordinate space. First, a held-out-validated cross-model refusal and compliance coordinate space is built, which aligns heterogeneous surrogate states into a shared low-dimensional representation. Second, a request-independent generator maps shared coordinates to image-text carriers, allowing the attack to be calibrated once to a new target. Experimental results on MM-SafetyBench, USB, and VLJailbreakBench show that UniCal supports jailbreak reuse across unseen requests and models with a single calibration.
Critical strengths
- The three contributions in this paper are well-structured and the motivation is clear, which is to build a black-box jailbreak attack that is adaptable across many unseen requests.
- The authors observed refusal and compliance may admit a compact behavioral representation that can be aligned across models. Then they applied this observation to UniCal and built a reusable low-dimensional behavioral coordinate space. The idea is reasonal and will inspire subsequent jailbreak method development.
- This paper covers a wide range of comparison methods, such as FigStep, MML, JailBound, ZO-SPSA, and GAMBIT. And these methods are very new. Under this setup, the experimental results in this paper are convincing.
- The method section in this paper is detailed. From compliance coordinates building, universal carrier designing, and decision-label calibration, the authors give key formulations and implementation details. It benefits the reimplementation of this method and the reproduction of the experimental results.
Critical weaknesses
- In the experiment section, there are two main tables to show the quantitative results but the corresponding analysis is insufficient. The experimental analysis about Table 1-2 and Figure 3-5 only have one paragraph with four sentences. The reader can get very little information from this. And these analyses are very simple, doesn't have any deep insights and findings. This weakness causes this paper not readable.
- The loss function for generator training contains five terms with four trade-off coefficients lambda(s). But the paper does not give the value of each lambda. This is a critical missing for the experiment.
- The authors claimed that transfer-based attacks remain fixed when tranferred to a new target, but did not discuss why they are not good. In other words, what are the limitations of transfer-based methods? Have the proposed method (UniCal) addressed these limitations?
- The proposed method, UniCal, does not realize state-of-the-art results compared other methods mentioned in the paper. So I think this method does not bring any performance imporvement in this field, which weakens the paper's contribution.
- In the method comparison tables, such as Table 1 and Table 2, there are many numbers. Why don't the authors mark the best resutls with highlighted font or color? It is a good way to improve the readability of research papers.
- How many categories does the jailbreak have in authors' mind? Two (transfer-based and query-based) or three (transfer-based, query-based, and universal attack)? How should the comparison methods be classified?
- Can the learned shared coordinates be visualizaed? I want to know if it is readable and meaningful for human.
Questions for authors
- The insights and findings from the experimental analyzes.
- What limitations has the proposed method addressed?
- The details about the four trade-off coefficients.
7 comments
We Are Truly Grateful to Reviewer RjGE for the Thoughtful Comments and Generous Scholarly Guidance
Dear Reviewer RjGE,
Thank you again for your encouraging and constructive feedback. As the discussion period is approaching its deadline, we would be grateful if you could revisit our responses and revisions and share whether your concerns have been adequately addressed. We warmly welcome any further questions or discussion.
We sincerely appreciate the time and care you have devoted to reviewing our work and hope you have a wonderful life filled with happiness!
Best regards,
The Authors
Thanks for the rebuttal. The response has answered most of questions and concerns I rised. I still have one question: what limitations does this paper address? Please clarify them clearly.
Response to Reviewer RjGE
We sincerely thank the reviewer for the encouraging assessment of UniCal's motivation, experimental breadth, and methodological detail. We especially appreciate the suggestions to draw clearer findings from the experiments and improve reproducibility. We have refined the presentation accordingly.
1. Experimental insights and performance (W1, W4).
The results reveal three complementary findings. First, one-time calibration improves strict success from 76.6% to 81.7% while retaining fixed-carrier reuse (Table 3). Independently, a carrier prepared on surrogates reaches 80.1% on Molmo and 77.9% on Phi-Vision without target feedback (Table 1), providing evidence of zero-feedback transfer to unseen architectures.
Second, shared behavioral coordinates are useful beyond representation-level prediction. They achieve 0.859 held-out refusal–compliance AUC versus 0.737 for PCA (Figure 5), and 81.7% versus 77.4% strict success in the calibrated attack pipeline (Table 3). Third, benign-utility preservation produces a meaningful trade-off: removing its loss raises strict success slightly from 81.7% to 82.4%, but lowers benign utility from 93.5% to 85.6% (Table 4). Together, these results illuminate the roles of calibration, behavioral parameterization, and utility preservation.
Table 2 also reports higher raw success for certain native-access baselines, including PolyJailbreak (93.2% versus 81.7%). We present these effectiveness results alongside a separate, matched label-only efficiency comparison. In that comparison, UniCal's 192-query calibration reaches its break-even point after eight reused requests relative to label-only Random Search (Figure 4). Section 5 now draws these findings and their evaluation conditions together more explicitly.
2. Generator coefficients and reproducibility (W2).
We appreciate the request for explicit coefficient values. Appendix A.2 now reports λ_b = 1.0, λ_v = 0.5, λ_t = 0.05, and λ_u = 1.0. In Equation (7), the coordinate-realization term has unit weight; these four coefficients weight the behavioral, visual, textual, and benign-utility losses, respectively. This specifies the weighting of the five-term generator objective.
3. Motivation, taxonomy, visualization, and tables (W3, W5–W7).
Zero-feedback transfer enables immediate reuse without incorporating feedback from a new target. Target-interactive methods can adapt to that target, sometimes with repeated request-level optimization. UniCal studies one label-only calibration followed by unchanged carrier reuse, combining target-specific adaptation with reuse across unseen requests. These are not three mutually exclusive attack categories: transfer versus target interaction concerns feedback access, while universal versus request-specific concerns what is reused.
We have clarified the access and adaptation regimes in the text and improved the presentation of Tables 1–2, including the identification of leading results. For interpretability, Figure 5(a) shows the first two dimensions of the eight-dimensional shared space, with model identities and refusal/compliance labels. Figure 5(b–c) further shows cross-model alignment and held-out predictability. These views make the organization of learned behavior visible at the distributional level, without assigning a separate semantic name to every coordinate axis.
4. Ethical considerations.
We appreciate the reviewer's attention to dual-use risks. We have added Section 7, Ethical Considerations, describing the potential for misuse and our reporting of aggregate evaluation results rather than harmful model completions. This positions the work as an analysis of multimodal safety failure modes intended to inform stronger safeguards.
We sincerely appreciate the reviewer's constructive assessment. We hope the clearer experimental findings, explicit training coefficients, and revised presentation help address the concerns, and would be grateful if the reviewer could reconsider the overall assessment in light of these revisions.