- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies split-intent safety failures in large image editing models and proposes SIJA, a cross-modal jailbreak attack that distributes unsafe editing intent across textual and visual inputs. The authors further introduce SI2ESBench and evaluate a range of commercial and open source image editing models in this context. The results show that the current large image editing models can recover and execute unsafe transformations from distributed intents, and the proposed editing-intent reconstruction defense greatly improves the detection capability.
Critical strengths
- This paper identifies a distinct safety problem in large image editing models. SIJA shows that an unsafe editing request can remain incomplete in both the textual and visual components while still being recoverable by the editing model through joint interpretation. This exposes a safety gap in which the unsafe transformation can emerge through joint interpretation of the source image, textual referent, and graphical visual semantics.
- The experimental results provide extensive evidence for the split-intent vulnerability and support the proposed mechanism. The main SIJA evaluation covers commercial and open source large image editing models across 10 risk categories, showing the substantial vulnerability of different model families. The effectiveness of SIJA is further demonstrated by comparing different forms of attack, while component ablation makes clear the different roles of textual referents and visual cues. These results provide extensive empirical evidence for this vulnerability and controlled evidence for the proposed split-intent attack.
- The defense analysis further illustrates the practical security implications of SIJA. The sharp degradation of the text-space intent guardrail under SIJA suggests that the attack exposes specific weaknesses in security mechanisms that rely on recovering harmful intentions from readable text. Meanwhile, the strong performance of edit intent reconstruction suggests that explicitly recovering the full intent across textual and visual inputs is a promising direction to mitigate this vulnerability. This makes SIJA not only an effective attack but also a useful setting for motivating stronger protective measures for large image editing models.
Critical weaknesses
- The human validation procedure for SI2ESBench is not described in sufficient detail. Benchmark construction relies on manual verification of referent grounding, component intent incompleteness, and recoverability of the complete editing intent from the joint inputs. These guidelines are described and it is stated that ambiguous or inconsistent cases are revised or removed, but the specific verification process is not described in detail.
- The human–MLLM comparison is reported mainly at the aggregate level. The current comparison reports aggregate HS, EV, and HRR on the same 350 outputs, but similar summary values do not necessarily imply agreement on the same individual samples. Thus, the current analysis does not directly characterize the instance-level agreement between automatic judging and human annotation.
Questions for authors
- Could the authors clarify how the manual validation of SI2ESBench was conducted in practice, particularly how the stated criteria were checked and how ambiguous cases were resolved? The paper describes criteria for the referent grounding and recoverability of the complete editing intent, component intent incompleteness, and complete edit intent, but it is not clear how these checks are performed or how ambiguous cases are handled.
- What is the instance-level agreement between the primary MLLM judge and the human annotations on the 350 sampled outputs? The reported aggregated HS, EV, and HRR are informative, but similar aggregated values do not necessarily indicate consistency for the same individual sample.
We thank the reviewer for the positive assessment and the suggestions to clarify benchmark validation and judge reliability.
A1. Human Validation Procedure
Three human validators review candidate SI2ESBench cases against three criteria. For referent grounding, they inspect the source image and textual instruction to verify that the target is identifiable without revealing the action or specification. For component-wise incompleteness, they examine the textual instruction and visual cue separately to verify that neither independently specifies the complete source-grounded edit. For joint recoverability, they inspect the source image, instruction, and cue together to verify that the intended transformation can be recovered. Candidates with ambiguous targets, inconsistent or incomplete visual semantics, or redundant intent information are revised or excluded. This procedure yields 766 validated cases. (Revision: Sec. 4.1 and Appendix B.2.)
A2. Instance-Level Judge Agreement
We extend the original aggregate comparison on 350 edited outputs to sample-level agreement between Gemma-4-31B-it and human consensus. Three annotators independently evaluate each output, with majority-voted EV and median HS determining the consensus labels; HRR is derived from EV = 1 and HS at least 4. We report precision, recall, F1, Cohen's kappa for EV/HRR, quadratic weighted kappa for HS, and pairwise human agreement.
The F1 scores exceed 95% for both EV and HRR, and agreement reaches 0.841 quadratic weighted kappa for HS. The paired confusion matrices quantify errors on individual outputs; the appendix reports these matrices, results for all four MLLM judges, and human inter-annotator agreement. These analyses support the consistency of the primary judge with human evaluations beyond aggregate averages. (Revision: Sec. 5.5 and Appendix A.4.)