- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AV-RIFT, a task-agnostic transferable adversarial attack against audio-visual (AV) models. Instead of relying on access to downstream target models, AV-RIFT constructs adversarial examples using pretrained audio and visual encoders, semantic reference features, and lightweight cross-modal projection networks trained on auxiliary paired data. The method combines two objectives: (1) an intra-modal semantics-aware contrastive (ISC) attack that moves adversarial representations away from semantically similar references and toward dissimilar ones, and (2) an inter-modal misalignment (IM) attack that reduces the correspondence between synchronized audio and visual representations in a learned shared embedding space. These objectives are jointly optimized through alternating projected gradient updates under modality-specific perturbation budgets. The authors evaluate AV-RIFT on five audio-visual tasks and 20 models, including task-specific architectures and general audio-visual models. The results demonstrate improved attack transferability over the selected baselines across multiple tasks and settings. Additional ablation studies and alternative surrogate experiments provide evidence for the effectiveness of the proposed objectives.
Critical strengths
-
A meaningful threat model and an audio-visual-specific attack formulation. The paper investigates an important and relatively underexplored problem: transferable attacks across heterogeneous audio-visual tasks without accessing the downstream target models. This setting is valuable because many deployed systems expose neither model parameters nor inference gradients. The proposed formulation also goes beyond independently perturbing the two modalities. In particular, the distinction between instance-level feature displacement and event-level semantic displacement is well motivated. Learning lightweight projections to measure audio-visual correspondence between independently pretrained encoders is a reasonable approach to constructing a cross-modal attack objective. The two objectives are clearly formulated and tailored to the characteristics of audio-visual data.
-
Broad experimental evaluation and encouraging transferability results. The evaluation covers five heterogeneous tasks and 20 target models, using task-appropriate performance metrics and attack-success criteria. AV-RIFT achieves the highest average ASR among the compared methods across all five tasks, with an average improvement of 12.87 percentage points over the strongest baseline selected separately for each task. The component ablation provides evidence that the ISC and IM objectives are complementary. The alternative surrogate experiments further show that the proposed attack remains stronger than the combined unimodal baseline when different pretrained encoders are used, although absolute effectiveness varies considerably with the surrogate choice.
Critical weaknesses
-
Strong dependence on labeled evaluation data. The proposed attack assumes access to the complete labeled test set to construct semantically similar and dissimilar reference banks. Although this does not violate the stated zero-query setting, it limits the practical applicability of the attack to real-world black-box services, where such annotations are generally unavailable. The paper mentions a feature-neighborhood fallback but does not evaluate its effectiveness. Clarifying this assumption and examining attack performance without test-set labels would strengthen the practical threat model.
-
Insufficient validation of the proposed attack mechanisms. Although the ablation study demonstrates that the intra-modal semantic contrastive (ISC) and inter-modal misalignment (IM) objectives are complementary, it does not directly establish that they cause the intended semantic shifts and cross-modal alignment failures. Furthermore, the main baselines primarily involve independently optimized unimodal attacks, making it difficult to isolate the benefits of the proposed objectives from those of joint optimization. More direct mechanism analysis and controlled comparisons would strengthen the methodological contribution.
-
Limited evidence for broader transferability and perceptual quality. Attack effectiveness varies considerably across tasks and surrogate encoders, with relatively modest improvements on audio-visual segmentation and substantially lower absolute ASR when alternative surrogates are used. These variations deserve further analysis to clarify the method's generalization boundaries. Additionally, the paper lacks perceptual-quality evaluations of the perturbed audio and video, making it difficult to assess the practical significance of the chosen perturbation budgets.
Questions for authors
Q1. Dependence on labeled reference banks. How does AV-RIFT perform when the adversary does not have access to the labeled evaluation set? In particular, can the authors report results using the proposed feature-neighborhood fallback or an auxiliary-only semantic reference bank? This would clarify the extent to which the reported transferability depends on evaluation-set annotations.
Q2. Validation of the proposed attack mechanisms. Can the authors provide direct evidence that ISC changes event-level semantic neighborhoods and that IM disrupts audio-visual correspondence in a way that contributes to downstream prediction failures? For example, analyses of surrogate-space event-neighborhood changes and the relationship between cross-modal similarity reduction and downstream attack success would help substantiate the mechanism.
Q3. Isolation of methodological contributions. How does AV-RIFT compare with a matched surrogate-space transfer baseline using comparable optimization steps and perturbation budgets? In particular, can the authors clarify how much of the improvement arises from semantic reference guidance, cross-modal misalignment, and joint alternating optimization, respectively?
Follow-up on Our Rebuttal
Thank you for your thoughtful comments and constructive feedback. We have carefully addressed your concerns through additional experiments and detailed responses in our rebuttal. We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns or if there are any remaining questions. We would be glad to discuss them further.