- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The manuscript proposes Mixture of Multimodal Auto-Red-Teamers (MoMART), an automated black-box red-teaming framework for evaluating modality subsets under matched conditions. MoMART combines modality-specific jailbreak strategies across text, image, audio, and video, and iteratively refines attacks using feedback from an LLM-based judge. Experiments on HarmBench demonstrate higher attack success rates than existing baselines, with additional analyses showing that multimodal composition can expose vulnerabilities not observed in individual modalities.
Critical strengths
The manuscript provides systematic evaluations of modality combinations and behavior-level analyses that offer useful insights into vulnerabilities arising from jointly delivered multimodal attacks. However, concerns remain regarding the methodological novelty and the fairness of experimental comparisons.
Critical weaknesses
1. Limited Methodological Novelty and Missing Baselines
MoMART primarily combines existing modality-specific jailbreak techniques with feedback-guided refinement, without introducing a substantially new attack mechanism or optimization algorithm. Although the manuscript discusses related compositional methods such as HACA and JMLLM, these methods are not included in the experimental comparisons. Comparisons with such approaches under compatible modality settings are important for establishing MoMART’s contributions beyond existing multimodal jailbreak frameworks.
2. Incomplete Attack-Budget Control and Efficiency Evaluation
The shared retry limit applies to each behavior-strategy trial rather than the overall attack procedure, and auxiliary LLM calls are not budget-matched across methods. Consequently, the substantial ASR improvements in Table 2 are difficult to assess without comparisons under matched aggregate attack budgets. Furthermore, Table 9 reports target queries to first success only for successful behaviors, without measuring end-to-end runtime or the computational costs of multimodal artifact generation. These limitations make it difficult to determine whether MoMART’s performance and efficiency advantages hold under comparable overall computational budgets.
Questions for authors
Please see the weaknesses above.
Thank you for the detailed response and additional experiments. After reconsidering the paper, I have decided to lower my rating from 3 (Weak Accept) to 2 (Weak Reject) for two reasons.
1. Incomplete comparison under a matched query budget The authors report query costs for other attack methods under a 300-query budget but do not provide their corresponding attack success rates (ASRs). In addition, Table 6 reports different strategy counts across modality combinations, while the efficiency evaluation in Table 9 uses 15 strategies. Without the ASRs from the new runs and clearer information about the evaluation settings, it remains unclear whether MoMART's original performance advantage holds under the same total query budget.
2. Limitations of the joint-only analysis Section 4.3.1 states that the comparison uses the "exact unmodified constituent artifacts." However, the authors clarify in their response that singleton attacks were generated and optimized independently. Therefore, the joint-only results show that multimodal attacks found successful cases missed by the evaluated singleton attacks. They do not directly establish that the same attack materials fail individually but succeed when combined.
Overall Assessment I acknowledge the value of the systematic multimodal evaluation and the additional experiments. However, these two limitations weaken the evidence supporting the paper's main conclusions. I therefore believe that 2 (Weak Reject) is a more appropriate rating.