acceptodds
Withdrawn from ICLR 2027

Selective Listening: Mechanism-Guided Control of Audio Influence in Large Audio-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Large audio-language models (LALMs) exploit multimodal evidence, yet task-irrelevant audio can alter text-reasoning decisions when listening is unnecessary. Aggregate Accuracy can hide this paired drift because audio-induced repairs and damages may cancel. Paired drift analysis and targeted interventions identify architecture-specific, intervention-sensitive late audio pathways as actionable control points. We introduce ICAP-Gate, which applies mechanism-guided, task-conditioned control to each model's pathway. Across four LALMs, two reasoning benchmarks, and environmental-sound and natural-speech interference, ICAP-Gate has lower point estimates for Influence Rate and Answer Flip than ungated inference in all 16 full-split model–condition evaluations. Fixed suppression degrades automatic speech recognition (ASR) across all four models, whereas ICAP-Gate matches ungated ASR performance by preserving the pathway for explicit audio-demand instructions. ICAP-Gate has lower paired-drift point estimates than mitigation prompting in all four evaluated settings and provides competitive stabilization relative to eight-sample Self-Consistency while using one generation per query; in controlled ARC measurements, Self-Consistency incurs – ungated latency. These results establish selective modality influence control as a design principle for robust multimodal reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Audio-language models, Multimodal reasoning robustness, Irrelevant-audio interference, Mechanistic pathway intervention, Task-conditioned modality control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nsebOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Irrelevant audio can degrade text reasoning and increase output instability in large audio-language models (LALMs). This work introduces ICAP-Gate, a mechanism-guided, task-conditioned approach to controlling audio pathways to mitigate such instability.

Critical strengths

The paper focuses on an important robustness challenge in LALMs: prediction instability induced by task-irrelevant audio. By jointly evaluating Influence Rate and Answer Flip, the study provides a comprehensive evaluation of audio interference, supported by experiments across multiple models and diverse audio interference conditions.

Critical weaknesses

My main concern is that the proposed routing mechanism relies on a keyword-based rule rather than a learned or semantic relevance-aware approach. This raises serious concerns about whether the method can reliably determine when audio information is needed.

My second critical concern is that the paper evaluates audio capability preservation only through ASR, without considering comprehensive audio understanding and reasoning benchmarks such as MMAU or MMAR. Furthermore, ICAP-Gate sets the gating factor to 1 for explicit ASR instructions, making its behavior identical to ungated inference by construction. Therefore, the identical WER results in Table 6(a) provide limited evidence of effective selective modality control.

The paper does not clearly explain why the selected late audio layers are more effective than other layers. Comparisons with simpler methods, such as audio masking, or with alternative layer selections are needed to justify the proposed mechanism-guided approach.

Questions for authors

How does the instruction-conditioned router handle implicit audio requests?

Could you include a learnable instruction-conditioned router as a comparison?

Could you evaluate audio understanding on benchmarks such as MMAU or MMAR?

Could you explain why the selected late audio layers are more effective than simpler approaches, such as audio masking or selecting other layers?

Reviewer_GQE7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how irrelevant audio changes text-reasoning decisions and proposes ICAP-Gate, which attenuates an audio pathway unless the instruction contains an explicit audio-demand cue. Across four models, two reasoning benchmarks, and environmental and speech interference, the authors report lower Influence Rate and Answer Flip point estimates in all 16 full-split conditions, though not every improvement is statistically significant. Speech-recognition WER matches ungated inference for explicit audio-demand requests. Comparisons support a single-generation cost advantage over eight-sample Self-Consistency, which remains more stable. The results demonstrate reduced drift in the tested settings, not general semantic identification of useful versus distracting audio.

Critical strengths
  • The problem in Section 1 is important: a model should not change its answer merely because irrelevant audio is available. Section 2 acknowledges that this interference was already observed in prior work. The added interest is connecting paired decision drift to actionable internal pathways and task-conditioned control, rather than claiming a new failure mode.

  • Table 1's four-model, two-benchmark evaluation with environmental and speech interference is substantial for the behavioral drift claim. Section 3.1 separates repairs from damages that aggregate accuracy can hide. The scope supports reduced drift in the tested conditions, while the broader ability to distinguish useful audio from interference remains unresolved.

  • Section 4.3 compares with prompting and Self-Consistency from the directly related study, so the baselines are relevant to the problem. Tables 2–3 support a useful single-generation cost trade-off, though Self-Consistency remains more stable. This is evidence of efficiency at the tested operating points, not general superiority over alternative mitigation strategies.

Critical weaknesses
  • The stated objective does not establish why internal soft attenuation is needed. With the same router, conditional audio removal reproduces Equation (1)'s clean branch under deterministic inference and preserves explicit audio-demand requests. Section 3.3 gives no separate utility requirement for retaining weakened audio, so the internal intervention's added value remains unclear.

  • Equation (8) implements instruction-template switching rather than evidence-based relevance selection. The rule does not determine whether audio contributes to the answer and, as Table 17 shows, mishandles implicit dependence and negation. The authors acknowledge this scope; unchanged-pathway ASR preservation does not establish separation of useful audio from interference.

Questions for authors
  • What task constraint or measured utility makes attenuated audio preferable to conditional removal under the same router? For sampled decoding, please separate audio effects from sampling variation.

  • How should the method handle implicit audio dependence, negation, or requests containing both useful and distracting audio? Is the intended contribution limited to template-conditioned modality switching?

Reviewer_1q24OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how irrelevant audio changes answers to text-based questions. It uses paired metrics to measure changes that overall accuracy may hide. The proposed ICAP-Gate scales the output of a selected late audio-encoder layer when the instruction does not explicitly require audio. Experiments cover four models, two reasoning benchmarks, environmental sounds, and unrelated speech.

Critical strengths
  1. The paired analysis shows how overall accuracy can hide audio-induced answer changes.
  2. The method requires no training or additional generations, and is evaluated across several models and interference conditions.
Critical weaknesses
  1. The practical benefit is unclear. Once the router decides that audio is unnecessary, the same rule could simply remove the audio input. This would avoid audio interference and save computation. The paper does not establish an advantage over this simpler alternative.

  2. The ASR preservation result is largely expected. For explicit ASR requests, the gate is set to one and the model is unchanged. Matching the original WER therefore provides limited evidence that the method reliably preserves useful audio processing.

  3. The mechanism analysis identifies useful intervention sites, but provides limited explanation of why interference occurs. More systematic comparisons with other layers and scaling locations would help establish what is special about the selected late layers.

Questions for authors

Using the same routing rule, how does internal scaling compare with directly removing audio in terms of accuracy, drift, and latency?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.