acceptodds
Under review as a conference paper at ICLR 2027

AV-RIFT: Task-Agnostic Transferable Adversarial Attacks on Audio-Visual Models

Abstract

Audio-visual models integrate auditory and visual cues to provide a comprehensive understanding of real-world scenes and have been applied to diverse tasks (e.g., event localization and question answering), largely benefiting from pretrained unimodal encoders that provide well-generalized representations. However, the open accessibility of these encoders also exposes a shared adversarial attack surface across downstream models, which remains unexplored by existing audio-visual attacks. This paper, for the first time, explores this threat by proposing AV-RIFT, a transferable audio-visual attack that solely utilizes publicly available encoders to mislead diverse downstream models. It jointly perturbs audio and video by disrupting semantic relationships both within and across modalities. Specifically, since heterogeneous audio-visual tasks commonly revolve around the semantics of sounding events shared by different instances, merely disrupting instance-level features on pretrained encoders does not necessarily alter the perceived event. AV-RIFT therefore introduces an intra-modal semantics-aware contrastive attack that steers each modality's representations away from semantically similar references and toward dissimilar ones. Moreover, to exploit the semantic consistency between audio and visual streams overlooked by independent unimodal perturbations, it further introduces an inter-modal misalignment attack that reduces the similarity between synchronized audio and visual representations in a projection space learned to bridge independently pretrained encoders. By exploiting the event-centric semantics and temporal synchronization inherent in audio-visual data, AV-RIFT is tailored to audio-visual models rather than being a direct adaptation of attacks designed for other modalities. Experiments on audio-visual tasks and models show that AV-RIFT consistently achieves state-of-the-art attack transferability and remains effective under existing defenses.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
audio-visual learning, adversarial attack

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o1fvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AV-RIFT, a task-agnostic transferable adversarial attack against audio-visual (AV) models. Instead of relying on access to downstream target models, AV-RIFT constructs adversarial examples using pretrained audio and visual encoders, semantic reference features, and lightweight cross-modal projection networks trained on auxiliary paired data. The method combines two objectives: (1) an intra-modal semantics-aware contrastive (ISC) attack that moves adversarial representations away from semantically similar references and toward dissimilar ones, and (2) an inter-modal misalignment (IM) attack that reduces the correspondence between synchronized audio and visual representations in a learned shared embedding space. These objectives are jointly optimized through alternating projected gradient updates under modality-specific perturbation budgets. The authors evaluate AV-RIFT on five audio-visual tasks and 20 models, including task-specific architectures and general audio-visual models. The results demonstrate improved attack transferability over the selected baselines across multiple tasks and settings. Additional ablation studies and alternative surrogate experiments provide evidence for the effectiveness of the proposed objectives.

Critical strengths
  1. A meaningful threat model and an audio-visual-specific attack formulation. The paper investigates an important and relatively underexplored problem: transferable attacks across heterogeneous audio-visual tasks without accessing the downstream target models. This setting is valuable because many deployed systems expose neither model parameters nor inference gradients. The proposed formulation also goes beyond independently perturbing the two modalities. In particular, the distinction between instance-level feature displacement and event-level semantic displacement is well motivated. Learning lightweight projections to measure audio-visual correspondence between independently pretrained encoders is a reasonable approach to constructing a cross-modal attack objective. The two objectives are clearly formulated and tailored to the characteristics of audio-visual data.

  2. Broad experimental evaluation and encouraging transferability results. The evaluation covers five heterogeneous tasks and 20 target models, using task-appropriate performance metrics and attack-success criteria. AV-RIFT achieves the highest average ASR among the compared methods across all five tasks, with an average improvement of 12.87 percentage points over the strongest baseline selected separately for each task. The component ablation provides evidence that the ISC and IM objectives are complementary. The alternative surrogate experiments further show that the proposed attack remains stronger than the combined unimodal baseline when different pretrained encoders are used, although absolute effectiveness varies considerably with the surrogate choice.

Critical weaknesses
  1. Strong dependence on labeled evaluation data. The proposed attack assumes access to the complete labeled test set to construct semantically similar and dissimilar reference banks. Although this does not violate the stated zero-query setting, it limits the practical applicability of the attack to real-world black-box services, where such annotations are generally unavailable. The paper mentions a feature-neighborhood fallback but does not evaluate its effectiveness. Clarifying this assumption and examining attack performance without test-set labels would strengthen the practical threat model.

  2. Insufficient validation of the proposed attack mechanisms. Although the ablation study demonstrates that the intra-modal semantic contrastive (ISC) and inter-modal misalignment (IM) objectives are complementary, it does not directly establish that they cause the intended semantic shifts and cross-modal alignment failures. Furthermore, the main baselines primarily involve independently optimized unimodal attacks, making it difficult to isolate the benefits of the proposed objectives from those of joint optimization. More direct mechanism analysis and controlled comparisons would strengthen the methodological contribution.

  3. Limited evidence for broader transferability and perceptual quality. Attack effectiveness varies considerably across tasks and surrogate encoders, with relatively modest improvements on audio-visual segmentation and substantially lower absolute ASR when alternative surrogates are used. These variations deserve further analysis to clarify the method's generalization boundaries. Additionally, the paper lacks perceptual-quality evaluations of the perturbed audio and video, making it difficult to assess the practical significance of the chosen perturbation budgets.

Questions for authors

Q1. Dependence on labeled reference banks. How does AV-RIFT perform when the adversary does not have access to the labeled evaluation set? In particular, can the authors report results using the proposed feature-neighborhood fallback or an auxiliary-only semantic reference bank? This would clarify the extent to which the reported transferability depends on evaluation-set annotations.

Q2. Validation of the proposed attack mechanisms. Can the authors provide direct evidence that ISC changes event-level semantic neighborhoods and that IM disrupts audio-visual correspondence in a way that contributes to downstream prediction failures? For example, analyses of surrogate-space event-neighborhood changes and the relationship between cross-modal similarity reduction and downstream attack success would help substantiate the mechanism.

Q3. Isolation of methodological contributions. How does AV-RIFT compare with a matched surrogate-space transfer baseline using comparable optimization steps and perturbation budgets? In particular, can the authors clarify how much of the improvement arises from semantic reference guidance, cross-modal misalignment, and joint alternating optimization, respectively?

1 comment
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you for your thoughtful comments and constructive feedback. We have carefully addressed your concerns through additional experiments and detailed responses in our rebuttal. We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns or if there are any remaining questions. We would be glad to discuss them further.

Reviewer_69piOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AV-RIFT, a task-agnostic black-box transferable attack on audio-visual models. In the main black-box setting, it generates adversarial examples using pretrained surrogate encoders, semantic reference banks, and auxiliary-trained cross-modal projection heads without accessing downstream target models. The method combines an intra-modal semantics-aware contrastive (ISC) attack and an inter-modal misalignment (IM) attack to disrupt event-level semantic relationships and cross-modal consistency between synchronized audio and visual representations. Experiments across five audio-visual tasks and 20 models show that AV-RIFT outperforms existing transfer attacks in most settings and remains effective against several defenses.

Critical strengths
  1. The paper investigates a practically relevant and relatively underexplored attack scenario, where publicly available pretrained encoders are used to attack downstream audio-visual models across different tasks without accessing the target models, providing valuable insights into shared adversarial vulnerabilities in multimodal systems.
  2. The experiments cover five tasks (AVEL, AVS, AVER, AVR, and AVQA), including both task-specific and general-purpose audio-visual models. The evaluation also considers different surrogate encoders, perturbation budgets, defenses, and component ablations, demonstrating broad experimental coverage and substantial attack performance improvements in the main experiments.
Critical weaknesses
  1. Section 3.1 and Appendix B.2 indicate that ISC constructs semantic reference banks using the complete labeled test set, including answer and question-type annotations for AVQA. Although Appendix B.2 describes an alternative reference construction strategy based on unlabeled segment relationships, no corresponding experiments are reported, leaving its effectiveness in genuinely label-free black-box settings unclear.
  2. The current experiments do not sufficiently disentangle the benefits of semantic supervision from those of the proposed method. The main baselines in Table 1 use instance-level feature attacks without semantic labels, whereas AV-RIFT constructs positive and negative semantic reference sets using ground-truth labels.
  3. Although the paper compares several existing attacks, direct experimental comparisons with semantic-guided or contrastive-learning-based transfer attacks are missing.
  4. Although Figure 4(a) provides component ablations, the full method improves average ASR by only 2.64 percentage points over ISC alone, and the ablation covers only AVEL, AVER, and AVR. These results are insufficient to fully explain the contribution of IM to attack transferability.
Questions for authors
  1. How much does AV-RIFT's attack performance decrease across the five tasks when ISC uses feature-space neighborhoods or an independent unlabeled reference bank without access to ground-truth test labels?
  2. Could the authors introduce baselines using the same semantic reference information to verify whether ISC's advantage comes from its semantic contrastive objective rather than additional semantic supervision?
  3. Could the authors provide comparisons with existing semantic-guided or contrastive-learning-based transfer attacks under the same surrogate encoders, perturbation budgets, and optimization settings?
  4. Could the authors report the audio-visual matching quality in the learned projection space and the changes in cross-modal similarity before and after the attack?
  5. What are the main reasons for the substantial performance variations across different surrogate encoders?
Reviewer_3KVROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AV-RIFT is a task-agnostic transferable adversarial attack for audio-visual models that exploits shared pretrained encoders across downstream tasks. It combines an intra-modal semantics-aware contrastive attack with an inter-modal misalignment attack to generate transferable perturbations without accessing target models. Evaluations across five audio-visual tasks, 16 task-specific models, and four general audio-visual models demonstrate improved transferability over baselines, with additional analyses of defenses and component ablations.

Critical strengths
  • AV-RIFT is evaluated across five diverse audio-visual tasks and a wide range of task-specific and general audio-visual models. The consistent improvements across these settings provide meaningful evidence of cross-task attack transferability.
  • The proposed method combines an intra-modal semantics-aware contrastive attack with an inter-modal misalignment attack, targeting both semantic relationships within each modality and correspondence across modalities. The ablation results further support the complementary contributions of these two objectives.
Critical weaknesses
  • Although AV-RIFT does not access downstream target models, its main experiments rely on task-specific semantic annotations to construct reference sets, including question- and answer-related tags in AVQA. The paper mentions a label-free alternative based on feature-space neighborhoods but provides no corresponding attack results. This limits the evidence supporting its practical task-agnostic setting.
  • The baseline comparisons do not fully isolate the contributions of the proposed objectives. The main multimodal baseline, I2V + SoundBreak, combines independently optimized audio and visual attacks without a cross-modal loss. Moreover, although Appendix C.2 describes a proxy-TMA variant, the reported comparison uses task-aware TMA. Evaluating proxy-TMA under comparable settings would help clarify the benefit of AV-RIFT's semantics-aware objective.
Questions for authors

The following questions are related to the critical weaknesses identified above.

  • Could the authors evaluate the feature-space-neighborhood variant mentioned in Sections 3.1/3.2 across the five tasks without ground-truth labels or task-specific semantic annotations? How does its ASR compare with the labeled-reference setting?
  • How does the proxy-TMA variant described in Appendix C.2 perform under the same surrogate encoders, perturbation budgets, and evaluation protocol as AV-RIFT? This comparison would help clarify whether the improvement comes from the semantics-aware objective or joint audio-visual optimization with a cross-modal loss.
2 comments

We thank the reviewer for the constructive feedback. Our responses are as follows.

Q1: Although AV-RIFT does not access the target models, its main experiments rely on task-specific semantic annotations, including question- and answer-related tags in AVQA, to construct the reference sets. The label-free alternative based on feature-space neighborhoods is mentioned but not evaluated.

A1. We evaluate the feature-space-neighborhood variant (Sec. 3.2) on all five tasks without any labels or annotations. For each query segment ss and modality mm, we rank the clean features of all other segments in the evaluation split by cosine similarity to gs,mg_{s,m}, and take the KK most and KK least similar ones as Rs,msim/dis\mathcal{R}_{s,m}^{\mathrm{sim/dis}} (K=3K=3). All other settings follow Table 1.

Table R1: Label-free vs. labeled reference construction (%).

MethodAVEL (Acc./ASR)AVS (mIoU/ASR)AVER (Acc./ASR)AVR (mAP/ASR)AVQA (Acc./ASR)Avg. ASR
Clean72.87 / –76.86 / –72.45 / –77.68 / –72.61 / ––
I2V + SoundBreak31.30 / 65.4652.85 / 18.1047.92 / 39.2452.81 / 42.9758.76 / 26.8138.52
AV-RIFT (Label-free)19.13 / 85.5250.14 / 19.6442.14 / 44.7740.68 / 56.2055.72 / 38.1448.85
AV-RIFT (Labeled)7.43 / 93.1650.02 / 20.4740.38 / 48.2336.35 / 60.6348.44 / 44.0853.31

Labels are not a prerequisite for AV-RIFT. The label-free variant stays within 4.46 points of the labeled one and still outperforms I2V + SoundBreak by 10.34 points on every task. It uses only the unlabeled clean inputs to be perturbed, with no labels or target-model queries, so the task-agnostic threat model holds without annotations.

Ablation on K. We vary K∈{1,3,5,10}K \in \{1, 3, 5, 10\} with all other settings fixed.

Table R2: ASR (%) of label-free AV-RIFT with different K.

TaskK=1K=3K=5K=10
AVEL83.9685.5282.1786.04
AVS19.3619.6419.9919.08
AVER41.3244.7739.0640.30
AVR49.9056.2056.8553.32
AVQA38.0838.1437.1636.79
Avg.46.5248.8547.0547.11

The average ASR stays within 46.52–48.85% and above I2V + SoundBreak (38.52%) for every KK, so the label-free variant is insensitive to KK.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you for your thoughtful comments and constructive feedback. We have carefully addressed your concerns through additional experiments and detailed responses in our rebuttal. We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns or if there are any remaining questions. We would be glad to discuss them further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revisions

We sincerely thank all three reviewers for their time, thoughtful feedback, and constructive suggestions. Their comments have helped us strengthen the experimental evidence and improve the manuscript.

A central question was whether AV-RIFT relies on ground-truth semantic annotations or prior knowledge of the target test distribution to construct its reference banks. To address this, we evaluated a fully label-free variant that estimates semantic relatedness directly from feature-space neighborhoods without any annotations, demonstrating that it remains within 4.46 points of the labeled version and consistently outperforms baselines across all five tasks (Table 5). To further eliminate reliance on the target data distribution, we evaluated an even more stringent setting where references are harvested from completely independent external datasets (Table 7). AV-RIFT retains its strong transferability in this setting. To isolate whether the gains stem from semantic references or our core algorithmic design, we conducted controlled comparisons against strengthened baselines equipped with the same semantic references and cross-modal losses (Table 8). AV-RIFT continues to outperform the strongest enhanced baseline across all tasks, confirming that its advantage arises from the joint intra- and inter-modal formulation rather than semantic supervision alone.

Reviewers also asked for direct mechanistic evidence demonstrating that the proposed objectives produce their intended representation-level effects, and whether the components contribute across all tasks. We therefore expanded the component ablation to all five tasks (Table 10) and measured same-event neighborhood precision in the surrogate feature spaces (Table 11). The results show that our intra-modal contrastive attack successfully pushes adversarial features out of the original event's semantic neighborhood. Concurrently, our inter-modal misalignment attack drives synchronized audio-visual cosine similarity below that of randomly mismatched clean pairs across all evaluated tasks (Table 12). We also added modality-wise comparisons against representative contrastive video and audio attacks, where AV-RIFT achieves superior transferability under single-modality perturbations (Table 9).

Finally, we further expanded the appendix with Linear CKA similarity analyses explaining surrogate-target compatibility (Table 16), segment-level cross-modal retrieval evaluations validating the learned projection space (Table 12), an ablation confirming the benefit of alternating optimization (Table 14), and quantitative perceptual quality evaluations across budgets (Table 17). We also revised the manuscript to improve the clarity and consistency of its presentation, terminology, and notation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.