- Rating
- 2(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces FusionBreak to focus on attacking multimodal models while preserving correct unimodal predictions. They also introduce a detector-aware loss to further reduce the detection rate. Experiments on two audio-visual datasets demonstrate the effectiveness of the proposed method.
Critical strengths
-
This paper is well-written, I can quickly understand its research objective. The research question is also interesting: the authors try to attack multimodal fusion while preserving correct unimodal predictions, thus hiding the attack at the unimodal level.
-
Proposed method can reduce the detection ratio under different detectors.
Critical weaknesses
-
From my perspective, the main contribution of this paper is introducing a new attack setting and modifying the loss function of PGD rather than proposing a novel algorithm. Meanwhile, I'm not sure about the practical importance of the proposed setting.
-
The paper only compares the proposed method with standard PGD, without including other adversarial attack methods. Beyond the FOSR metric, Appendix D shows that FusionBreak can also exhibit lower detection rates than PGD without the detector-aware loss. Thus, I think this potential to evade detectors could be an important contribution of the method. However, this also needs more comparisons with other baselines considering evading detection, e.g., PGD with detector-aware loss.
-
Datasets and targeted model are limited. (2 datasets and CNN)
Questions for authors
-
Can the authors provide some practical scenarios or applications where the proposed fusion-only attack would be particularly relevant? This is important for me to estimate the value of proposed method.
-
Can the proposed method generalize to more advanced pretrained multimodal models and other datasets?
8 comments
Many thanks for your rebuttal and clarification
I agree that attack fusion embedding is important. But my major concern is: why do we need to attack the fusion embedding while not changing the prediction in a single modality? So I say understanding this setting is really important for me. Based on this, I say that "Detector evasion" may be a potential reason?
For the limited technical contribution: I think my review is not suitable. I check some classical papers, I think one of their core contribution is also to propose an important setting and design proper training methods (e.g., VLAttack or Adversarial Prompt Tuning). But here, "important setting" is critical, for example, VLAttack is useful because they revealed the blind spot that more modalities naturally introduce more vulnerabilities.
If possible, can u give me a brief introduction to your motivation of your proposed settings? I will increase my rating to 2 (based on the current rebuttal). Indeed, I think your setting is interesting, but I still maintain the negative assessment before understanding the applications or motivation. Thanks again for your rebuttal.
Regarding the 3. Additional baselines and 4. Stronger models and datasets
Can u include these results in your current submission?
Clarification of Contributions, Practical Importance, and Expanded Evaluation
We respectfully disagree that a clear rejection adequately reflects the scientific contributions and evidence presented in our work, particularly given the positive aspects acknowledged in the review.
1. Practical importance
The failure pattern studied in our paper has been observed in previous research.
For example, Lingenfelser et al. (ICMI 2011, DOI: 10.1145/2070481.2070487) investigated multimodal emotion recognition and reported cases where both audio and video classifiers made correct predictions, but the fused classifier produced an incorrect result.
Our work investigates whether an adversary can deliberately induce this type of failure.
The broader security importance of multimodal fusion has also been demonstrated in autonomous driving. Hallyburton et al. (USENIX Security 2022) studied adversarial attacks against camera-LiDAR fusion systems. Although their setting differs from ours, it illustrates the practical consequences of vulnerabilities in multimodal decision-making.
Our original experiments achieve FOSR values of 77.09% on RAVDESS and 97.67% on CREMA-D, compared with 0% for standard PGD. This demonstrates why conventional attack success rates alone do not capture the failure condition we study.
2. Methodological contribution
We respectfully disagree that our contribution is simply a modification of the PGD loss.
Our original work introduces a constrained adversarial objective, decision-aware candidate selection, and an evaluation metric that explicitly requires correct unimodal predictions.
We further investigated this problem through an additional paired perturbation rescaling procedure and strict interaction checks.
The procedure searches combinations of audio and video perturbation strengths. A strict interaction success occurs only when the individual perturbations and their additive reference preserve the fused decision, while their joint application causes an incorrect prediction.
This allows us to distinguish joint interaction failures from ordinary disagreement between classifiers.
3. Additional baselines
We expanded the experimental evaluation to an 18-method portfolio, including conventional PGD, feasible PGD, preservation-guided search, strict PGD with different optimization budgets, single-modality attacks, and component ablations.
For example, on AVE cross-attention, the 20-step strict PGD baseline achieves a strict interaction success rate of 65.7%, compared with 91.9% for the combined rescaling and direct-search procedure.
We also evaluated an 80-step PGD baseline and measured computational costs. These comparisons distinguish the contribution of the additional search procedure from simply increasing gradient optimization.
Detector evasion is a complementary analysis, not the central contribution of our paper.
4. Stronger models and datasets
We expanded the experiments from two to three datasets and evaluated four fusion architectures using pretrained wav2vec 2.0, DeiT, AST, and CLIP encoders.
On AVE, the evaluated cross-attention models achieve clean fused balanced accuracies of 93.1% and 94.8%. The combined attack achieves approximately 91.9% strict interaction success on both seeds.
These experiments demonstrate the studied failure condition in stronger pretrained audio-visual systems.
5. Overall assessment
The review acknowledges the importance of our research question and the effectiveness of our original method against several generic detectors. Together with the methodological contributions and additional experimental evidence, we believe these findings deserve a more balanced assessment.
Regardless of whether the reviewer changes their rating, we maintain that a score of 1 does not fairly reflect the scientific contribution and supporting evidence of our work.