- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Multi-Modal Typography, a systematic study of how misleading speech and on-screen text affect audio-visual reasoning in MLLMs. It examines the cross-modal effects of unimodal and multimodal attacks, as well as the influence of attack properties. The experiments cover audio-visual reasoning, content moderation, and offline robot task selection, and evaluate prompting and fine-tuning as mitigation methods.
Critical strengths
The experiments are comprehensive, covering different attack modalities and their combinations, as well as several mitigation methods.
Critical weaknesses
-
The motivation is not sufficiently clear given the existing work on typographic attacks and audio injection attacks. The paper should explain more clearly what questions remain unanswered and how its study addresses them.
-
The attack setting of the main experiments assumes knowledge of the question and answer space and directly suggests a wrong answer, while random speech and noise barely affect accuracy in Table 4a. The practical relevance of this setting is unclear when the attacker does not know the user’s question or answer choices. Moreover, while the paper analyzes answer scores and attention, it remains unclear whether explicit reasoning can reduce the attacks’ effectiveness. Comparing direct answering with chain-of-thought reasoning before giving the final answer would help clarify this.
Questions for authors
Please see the weaknesses