acceptodds
Under review as a conference paper at ICLR 2027

How Sound Meets Language: Causal Mechanisms of Audio-Language Integration in Large Audio Language Models

Abstract

How do large audio language models (LALMs) integrate task-relevant auditory evidence into language computation? We introduce AudioEvidence, a controlled testbed with temporal evidence annotations, matched non-evidence controls, and counterfactual sound replacements, to trace this process across nine LALMs. At the backbone input, the causal effects of auditory evidence remain temporally localized. Within the backbone, eight models share a depth-wise organization: auditory evidence states exert stronger effects early, direct evidence access is most consequential at intermediate layers, and text-state effects strengthen with depth. Joint interventions establish that blocking intermediate evidence access weakens the influence of subsequent text states on the answer. The critical access stage also supports additional tasks and longer audio inputs. Head contributions vary in concentration across models, with MiMo-Audio exhibiting sparse organization that parallels retrieval-head findings in text LLMs. High evidence-attention scores alone do not establish causal importance. Finally, amplifying selected attention heads in MiMo-Audio improves audio hallucination detection and reduces task deviations in speech recognition. These findings provide mechanistic priors for more efficient audio scaling and modality integration.

Primary area:
interpretability and explainable AI
Keywords:
Large Audio Language Models, Interpretability, Modality Integration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1WuFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discusses AudioEvidence, a synthetic testing protocal with anotated evidence intervals, matched non-evidence controls, and counterfactual pairs. Using state patching, attention blocking, and head ablation across nine LALMs, the work reports 3 findings:

  • evidence effects stay localized at the backbone input;
  • in 8 models, test -> evidence access at intermediate layers drives later text-state effects;
  • head level concentration appears clearly in MiMO-Audio

As a test results, scaling selected MiMo-Audio heads gives small gaims on two benchmarks (HalluAudio and Librispeech).

Critical strengths
  • the counterfactual design is good initiative, with careful setup supporting it (non evidence controls ; Full/Null question contrast etc)
  • 9 models do cover enough variance in modeling
  • paper honestly reports that the retrieval-head hypothesis mostly fails and that attention scores do not predict causal importance.
  • The main findings transfer to other tasks, -30s inputs, external benchmarks, and speech.
Critical weaknesses
  • The main finding is partly forced by the architecture: The answer is read at a text position, so information must move from audio to text at some depth, and a crossover in state effects is expected. So the evidence flow findings are not suprising
  • Later audio positions can attend to E and pass its information on to text. The claims should say “direct access,” or the experiments should also block audio -> E attention.
  • The head-ablation results are not evidence-specific. Notably, MiMo with ablated heads only got 47.7% MMAU while the same model with silent audio got 58.6, meaning that the ablation demages more than audio retrieval. Also heads are zeroed at all positions, not only on paths to audio.
  • Only two results are significant, both in one model, which is not plausible. Notably, LibriSpeech test-clean is close to saturated, and strong models commonly report WER below 2%. MiMo-Audio’s 3.44% baseline is weak, and basically 89% of the gain comes from fixing 17 utterances where it output Chinese. This shows better task adherence, not better recognition. Probably a harder sets would be a good choice here (noisy / accented, or long form speech).
  • MiMo was chosen because it was the only model with concentrated heads, so these results are a best case. G.5 reports only selected positive cases, so please report all configurations.
  • The novelty over prior work is unclear. Similar depth-wise transfer is reported by Zhang et al. (2025), Suharitdamrong et al. (2026), Chen et al. (2026b), and Fan et al. (2026). Please state the difference clearly in the introduction.
  • Figure 1 motivates the work with MMAU accuracy under silence. The intervention is never tested on MMAU or MMAR, even though the same heads are validated there by ablation, and it is never tested under silence. Either run these evaluations or reframe the introduction.

Minor issues:

  • There are too many “evidence” terms. The paper uses evidence states, evidence access, evidence attention, evidence-attending heads, access intervals, and access envelopes. These are spread across sections and mix “positions E” with “information the model uses.” Given many terms discussed, please consider introduce glossary in the paper.
  • Figure 2 is hard to read: Panel (a) shows Full/Null without saying what it is for. Panel (b) mixes the counterfactual pair with an unrelated task list. Panel (c) does not show which positions are affected, the patching direction, or what is measured. Caption is not useful here also.
  • The phrase “surprisingly, these intervals overlap” is not surprising given Weakness 1.
  • The experiments locate where evidence has influence, not what computation happens, so “causal mechanisms” overstates the result. “Priors for efficient audio scaling” is never tested. The RVQ and training-scale discussion is speculation.
Questions for authors

Already explain my questions in the weakness section.

11 comments
AuthorsOpenReview ↗

Response to Reviewer 1WuF — Part 1/6

Response to Reviewer 1WuF — Part 1/6

Thank you for reviewing our work.


1. (Excerpt) The main finding is partly forced by the architecture: The answer is read at a text position, so information must move from audio to text at some depth, and a crossover in state effects is expected.

The paper asks: when a model uses audio to answer, at what depths and through which components does it access the answer-determining sound, and how does this access shape subsequent text computation? Reading an answer from a text position specifies where information must ultimately act, but not this process. Section 3 therefore constructs counterfactuals that replace only the target sound while preserving the question and other waveform content; Section 4 tests whether answer-related representation effects move with that sound; Section 5.2 then localizes and validates critical access using target evidence E versus matched non-evidence N and candidate intervals versus equal-width early/late controls. Eight models have evidence- and depth-selective access intervals, with model-dependent locations and widths.

Table 5 further tests the causal effect of critical evidence access on subsequent text computation. Changing answer preference by blocking a pathway establishes pathway importance; changing it by patching a state establishes state importance. Observing these separately does not establish whether the measured state effect depends on that pathway. The original joint intervention therefore uses inputs A/B differing only in the target sound: the recipient B runs normally, while E or N access is blocked only in A, within the identified interval and before the prespecified measurement layer. A's subsequent text states are then patched into the same B. In all eight models, E blocking weakens the states' ability to transfer A's answer preference more than N blocking, passing the prespecified primary comparison.

The cross-model comparison thus addresses two linked questions: which evidence-integration patterns are shared, and which component properties vary with the model and task? The original nine models span different audio interfaces, backbones, and training settings. Eight share causally validated critical access stages and access–state links, with model-dependent interval locations, widths, and head contributions. Added selection and transfer experiments further establish task dependence. Shared access organization and differing component implementations together constitute our answer to How Sound Meets Language. We will make this central aim and its quantitative evidence explicit in the introduction and remove “surprisingly.”


AuthorsOpenReview ↗

Response to Reviewer 1WuF — Part 2/6

Response to Reviewer 1WuF — Part 2/6

2. Later audio positions can attend to E and pass its information on to text. The claims should say “direct access,” or the experiments should also block audio -> E attention.

The original operator targets direct access from text positions to evidence E to locate the depth at which the model depends on this connection. The paper tests this stage using within-interval E/N and equal-width early/late controls.

Thank you for identifying the possibility of relay through other audio positions. This lets us test whether the critical stage persists when that pathway is included. Keeping the original interval fixed, we compare Direct (blocking post-audio text positions from attending to E), Relay (blocking non-E audio positions from attending to E), and Joint (both). All three interventions act on every head within the original critical interval:

ModelBaselineDirectRelayJointAdditional Relay drop (pp)
MiMo56.25%29.58%56.88%27.29%2.29
Qwen2.5-Omni78.13%38.96%77.29%38.33%0.63
Qwen2-Audio57.29%25.21%57.08%25.21%0.00
Step66.88%40.21%67.71%38.33%1.88
AF368.33%33.13%67.71%33.96%-0.83
Kimi65.21%33.96%64.79%31.25%2.71
MOSS71.25%31.67%70.83%31.25%0.42
Qwen3-Omni80.21%49.17%78.75%30.63%18.54

Each model uses 480 questions. The last column is Direct−Joint; negative values indicate improvement; bold values are significant after Holm8 correction across eight models.

Direct-access blocking causes substantial damage in all eight models, while additional relay contributions vary by model. The additional accuracy losses in Kimi and Qwen3 pass the correction; Qwen3 falls further from 49.17% to 30.63%, showing a substantial relay contribution after direct access is blocked. Additional margin damage is significant in MiMo, Qwen2.5, Step, Kimi, and Qwen3; Qwen2-Audio shows a significant reverse effect.

Crucially, all 16 accuracy comparisons between the critical interval and equal-width early/late intervals under Joint pass Holm16. Critical-stage depth selectivity therefore persists when audio relay is included. We will consistently call the original operator “direct evidence access” and report model-specific additional relay contributions.


AuthorsOpenReview ↗

Response to Reviewer 1WuF — Part 3/6

Response to Reviewer 1WuF — Part 3/6

3. The head-ablation results are not evidence-specific. Notably, MiMo with ablated heads only got 47.7% MMAU while the same model with silent audio got 58.6, meaning that the ablation demages more than audio retrieval. Also heads are zeroed at all positions, not only on paths to audio.

Whole-head ablation in the original paper tests whether heads selected on controlled evidence are causally important for independent, broader tasks. Zeroing outputs at all positions affects multiple computations supported by those heads, whereas silence changes the audio input. Performance below the silent-audio condition can therefore include damage beyond evidence access, motivating targeted pathway controls.

Thank you for identifying why this distinction matters for the retrieval-head hypothesis. We add two direct tests: comparison with random ablation on P0 tests retrieval-task contributions; E/N pathway blocking with the same heads fixed tests direct evidence-access contributions.

ModelP0 baselineWhole-head drop − mean random drop (pp)E-pathway drop (pp)N-pathway drop (pp)
MiMo-Audio56.25%22.6420.63−3.75
Qwen2.5-Omni78.13%11.3918.33−2.50
Qwen2-Audio57.29%1.537.71−0.42
Step-Audio-2-mini66.88%6.393.96−2.08
Audio Flamingo 368.33%1.045.421.67
Kimi-Audio65.21%4.511.67−0.83
MOSS-Audio71.25%10.497.71−1.04
Qwen3-Omni80.21%5.006.04−0.63

Each model uses 480 questions and global top-5% TQ heads; random controls use the same budget. E/N block the same heads' access to target/matched non-target audio; negatives indicate improvements.

First, retrieval-task losses exceed random controls in six models (Holm16), extending the effect originally most apparent in MiMo. Second, E/N margin differences are significant in all eight models and primary-TQ accuracy differences in seven, supporting direct evidence-access contributions. Including TAll, 16 margin and 14 accuracy comparisons pass Holm32; margin is the correct-minus-strongest-incorrect option score.

We therefore extend the original MiMo-centered finding to heads participating in direct evidence access across multiple models, with task contributions and transfer scope depending on model and selection task.

The introduction will use the primary TQ result 77.4%→59.4%; the 47.7% cited in the comment will be explicitly labeled as the TAll ablation result.


4. (Excerpt) Only two results are significant, both in one model, which is not plausible. […] MiMo-Audio’s 3.44% baseline is weak, and basically 89% of the gain comes from fixing 17 utterances where it output Chinese. This shows better task adherence, not better recognition. […] Probably a harder sets would be a good choice here (noisy / accented, or long form speech).

Task adherence is already the paper's interpretation of the ASR gain. Section 6 explicitly states that the WER improvement primarily comes from fewer hallucination-related errors and task deviations, especially Chinese translations in place of requested English transcriptions. The paper reports that such outputs account for 89.1% of the net word-error reduction, with breakdowns and examples in Appendix G (Table 9).

The original steering motivation is therefore to test whether evidence-related heads provide inference-time control points for hallucination-related behavior and task adherence. To test whether this effect extends to more difficult speech, we keep MiMo's Top80, α=1.3 configuration fixed and evaluate LibriSpeech test-other:

ScopeUtterancesBaseline WERAmplified WERWER change (pp), 95% CI
All test-other29396.6122%6.4268%−0.1853 [−0.3797, −0.0254]

The configuration is unchanged; confidence intervals use paired bootstrap over 33 speakers. Overall WER also improves on harder test-other, consistent with the original behavioral-control framing. In the additional breakdown, seven baseline outputs containing Chinese characters account for 56 of 97 net errors removed, again supporting task adherence as a source of benefit.

Across models, the original Table 10 already reports positive HalluAudio configurations for Qwen2-Audio, Qwen2.5-Omni, and Qwen3-Omni, based on pointwise 95% confidence intervals. The complete configurations show how amplification effects vary across models and settings.


Reviewer_rnJwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AudioEvidence, a controlled benchmark to trace how audio-LLMs process auditory evidence. Across nine models, the authors use mechanistic interventions (state swaps and attention blocking) to show that eight models share a clear trajectory: early evidence encoding, a critical mid-stage integration window, and text-dominated later layers. Building on this, they propose a head-amplification technique that effectively cuts hallucination and speech errors.

Critical strengths

The experimental study is thorough and clearly presented within its stated scope (focusing primarily on multiple-choice tasks with 10-second audio).

Critical weaknesses

Generalization to realistic long-form audio (where events are always correlated rather than isolated) and open-ended QA is not convincingly demonstrated.

Direct attention-head intervention relies too heavily on strong human priors, making it impractical for real-world model training and inference.

Questions for authors

Do the critical access intervals and head contributions persist with complex event dependencies or open-ended QA? What scope does the evidence support?

Could you elaborate on the key differences between your work and prior studies, such as [From Senses to Decisions], [Causal Tracing of Audio–Text Fusion], and others?

9 comments
AuthorsOpenReview ↗

Response to Reviewer rnJw — Part 1/3

Response to Reviewer rnJw — Part 1/3

Thank you for reviewing our work.


1. Generalization to realistic long-form audio (where events are always correlated rather than isolated) and open-ended QA is not convincingly demonstrated.

The original paper already tests transfer from single-event retrieval to multi-event tasks and from 10s to 30s; the new experiments further test evidence access in open-ended QA. Controlled construction lets us change only the answer-determining event while preserving the question and other sounds, identifying the evidence responsible for changes in answers and internal states.

Appendix F applies intervals identified on 10s P0 to 30s retrieval, event verification, and temporal ordering. Across eight models, blocking evidence within these intervals has stronger effects than matched non-evidence blocking and equal-width early/late controls; counting shows greater model variation. The existing evidence thus covers duration changes and multi-event relations beyond the initial retrieval setting.

For open-ended QA, the new experiments remove all candidate options and measure answer preference through reference-answer log-probabilities: all eight models retain evidence selectivity, four support persistence of the path–state relationship, and MOSS also retains critical-depth selectivity, as detailed below. We distinguish two generalization dimensions: controlled task and response-format extensions have empirical support; dense event dependencies in realistic long-form audio require separate validation. We will organize the revised results around these dimensions to make the demonstrated scope clear.


AuthorsOpenReview ↗

Response to Reviewer rnJw — Part 2/3

Response to Reviewer rnJw — Part 2/3

2. Do the critical access intervals and head contributions persist with complex event dependencies or open-ended QA? What scope does the evidence support?

The original P1–P3 tasks test event verification, ordering, and counting, and Appendix F transfers stages identified on 10s P0 to other tasks and 30s inputs. This design tests reuse of a fixed stage across tasks. To test whether these mechanisms extend to open-ended QA, we retain the original intervals and state-measurement layers and separately measure evidence selectivity, the path–state link, and depth changes across eight models.

1. Does evidence selectivity depend on candidate options? We remove candidate options and compare target-evidence E blocking with matched non-evidence N blocking within the original critical interval:

ModelAnswer preference after E blocking (nats/token)Answer preference after N blocking (nats/token)
MiMo0.77400.9539
Qwen2.5-Omni1.72822.2156
Qwen2-Audio1.66862.0268
Step2.08012.1991
AF31.75562.2430
Kimi2.07232.1508
MOSS0.85022.0067
Qwen3-Omni1.42972.0280

Each model uses 240 pairs. The metric is the mean token log-probability difference between correct answer A and counterfactual answer B (nats/token); higher values favor A. References are used only for scoring, not supplied as options. All eight E/N differences pass Holm24, supporting evidence selectivity at the original stage in open-ended QA. The correction family includes E/N, early-layer, and late-layer comparisons for each of eight models.

2. Does upstream evidence access still affect subsequent text computation? Showing that evidence blocking affects answers alone does not establish this relationship. We hold recipient input B fixed and compare the answer preferences produced by patching A states after E or N blocking. N−E below is the difference in A−B margin between these two patching conditions: a positive value means that blocking target-evidence access weakens downstream states’ support for answer A more than blocking non-evidence does.

ModelPath–state effect N−E (nats/token)
MiMo-0.0073
Qwen2.5-Omni-0.0043
Qwen2-Audio+0.0898
Step+0.1036
AF3-0.0143
Kimi+0.0138
MOSS+1.5308
Qwen3-Omni+0.3239

Each model uses 240 pairs; bold values are positive effects passing eight-model Holm8 correction. Qwen2-Audio, Step, MOSS, and Qwen3 support this path–state relationship; their normal-state-patching positive controls also pass the separate Holm8 family. Step, MOSS, and Qwen3 also pass under summed sequence log-probability; Qwen2-Audio is supported by the prespecified mean-token metric.

Persistence of the path–state relationship and invariance of critical depth are distinct conclusions. MOSS also retains selectivity of the original interval over early/late controls; some other models show stronger late-layer effects. These experiments support a causal access–state relationship across response formats in multiple models while characterizing changes in depth. The conclusions use reference-answer probabilities; complete depth controls and free-generation results are provided separately.

3. What is the scope of head contributions and transfer? Added experiments test source-task and cross-task effects separately: on controlled sound retrieval, selected-head ablations exceed random controls in six models (Holm16), and E/N comparisons with the same heads support direct evidence-access contributions in eight models (margin, Holm32). In the nine-model speech comparisons, ContextQA heads cause source-task losses of 35.21/42.50 points in Qwen2-Audio/Gemma; Qwen3's letter-string heads cause a 14.17-point source loss but only 1.30/2.70-point losses on MMAU/MMAR. Selected components can therefore matter for their source task while their cross-task effects depend on the model and selection conditions.


AuthorsOpenReview ↗

Response to Reviewer rnJw — Part 3/3

Response to Reviewer rnJw — Part 3/3

3. Direct attention-head intervention relies too heavily on strong human priors, making it impractical for real-world model training and inference.

Evidence annotations serve to construct attributable causal controls: they specify which answer-relevant sound is changed or blocked while other sounds and the question remain fixed. Positions come from source-event annotations and construction records, independently of attention and predictions; critical layers and heads are identified through model measurements and independent validation.

Annotations specify the sound to test; model interventions determine the mechanistic conclusion. Section 4 first verifies that representation positions supporting answer recovery move when the same sound moves. Access intervals and head rankings are then identified on discovery data and evaluated on independent examples using E/N, critical versus equal-width early/late intervals, and evidence/non-evidence access through selected heads. Important layers or heads are not manually prescribed.

Practical use separates offline identification from online intervention. Section 6 and Appendix G.1 already fix the head set and scaling coefficient after offline ranking and calibration. New inputs require no correct answers, evidence timestamps, or per-example head reselection; inference applies fixed head-output scaling while leaving model weights unchanged. Offline evidence supervision therefore does not require per-example human judgment at deployment. For training, Section 7 positions the cross-model observations as motivation for controlled future studies; the intervention validated here uses fixed components at inference time.


4. Could you elaborate on the key differences between your work and prior studies, such as [From Senses to Decisions], [Causal Tracing of Audio–Text Fusion], and others?

The paper examines both the causal link between evidence and language decisions and whether that organization is shared across models. Appendix B.1 covers nine models spanning discrete, continuous, hybrid, and direct-waveform interfaces, with different backbones, dense/MoE architectures, and training scales. The original experiments identify a critical direct-access stage shared by eight models and quantify model differences in its location, width, and head contributions; added speech selection and transfer experiments further characterize task dependence.

The value of cross-architecture coverage is to distinguish shared evidence-integration organization from model- and task-dependent component implementations. Our contribution relative to prior work combines causal identification of specific evidence with comparison of shared patterns and differences across audio interfaces and models:

Prior workEstablished findingFurther question and contribution of this paper
Zhang et al., Cross-modal Information FlowStages through which visual information affects question representations and predictionsFrom overall stages to specific answer-relevant sounds: is the critical access stage evidence- and depth-selective? We provide quantitative causal evidence across eight models.
Suharitdamrong et al., From Senses to DecisionsPathway importance among audio/video, question, and prediction positionsFrom category-level pathway importance to target-evidence access: is damage attributable to answer-relevant evidence rather than other content? We distinguish E/N causal contributions.
Chen et al., Causal Tracing of Audio–Text FusionAudio presence versus absence affects fusion layers and text statesFrom whole-audio effects to target-event identity: how does a specific sound change affect the answer? We connect that change to answer-related state effects.
Fan et al., Can We Read the Mind of an Audio LLM?Middle-layer audio-driven representations have causal effectsFrom state effects to their evidence source: do downstream answer-related state effects depend on upstream evidence access? Table 5 causally tests this relationship.

We will state both advances explicitly in the introduction and Related Work. “Mechanism” refers specifically to how evidence access affects subsequent text computation, measured through pathway blocking and text-state patching.

Reviewer_QcWfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how large audio language models (LALMs) integrate task-relevant auditory evidence into language computation. The authors build AudioEvidence, a testbed of synthetic 10-s and 30-s clips composed from AudioSet Strong and AudioEdit events, with four multiple-choice tasks (P0 retrieval, P1 verification, P2 temporal ordering, P3 counting), temporal evidence annotations, size-matched non-evidence positions, and counterfactual pairs that change the answer by replacing only the evidence interval. Using state patching, attention-path blocking, and head ablation/scaling on 9 LALMs, they report that evidence effects remain temporally localized at the backbone input, that in eight models evidence-state effects dominate early and text-state effects dominate later, and that text-to-evidence access at intermediate layers is critical and shapes the later text-state effects. A sparse retrieval-head organization is observed clearly only in MiMo-Audio.

Critical strengths
  1. The counterfactual design is careful and is the most reusable part of the paper. Waveforms outside the evidence interval are identical at the sample level, every intervention is compared against matched non-evidence positions and equal-width early and late intervals, and the critical intervals are selected on a discovery split and validated on a separate evaluation split with paired bootstrap intervals and Holm correction. This isolates the contribution of answer-determining evidence much more precisely than whole-audio or modality-level perturbations used in prior LALM interpretability work.

  2. The comparison across nine models with discrete, continuous, hybrid, and encoder-free audio interfaces separates what is shared from what is model-dependent. The observation that intermediate-layer access matters in all eight qualifying models while head-level concentration varies, together with the finding in Appendix E.5 that strong evidence attention in Step-Audio-2-mini, Qwen2-Audio, and MOSS-Audio extends beyond the causally critical interval, is a useful audio-specific data point on the gap between attention weights and causal importance.

Critical weaknesses
  1. The main mechanistic finding is largely expected, and several claims are framed more strongly than the results support.

In a decoder-only model, the answer is read out at text positions, so once evidence has been copied into text states, patching the audio positions in later layers should have little effect. The crossing of evidence-state and text-state curves in Fig. 5 is therefore close to what the architecture predicts, and similar information flow has been reported for VLMs. The joint intervention in Table 5, where blocking access leaves less evidence in subsequent text states, also points in a direction that is nearly expected by construction, although the matched N control makes the result clean.

The retrieval-head hypothesis is supported in one of nine models, yet the abstract and introduction present it as paralleling text-LLM findings. The introduction also reports the headline drop (77.8% to 47.7%) under the TAllT_{All} ranking, whereas Sec. 5.1 states that TQT_Q is the primary ranking, under which the drop is to 59.4%. The proposed reasons for MiMo-Audio being different (6.25 positions per second, RVQ discretization, over 100M hours of training) are not tested. Finally, the claim that the findings provide priors for more efficient audio scaling is not supported by any training or scaling experiment.

  1. Baselines needed to interpret the effect sizes are missing, and the inference-time control results are weak and partly inconsistent.

(a) Per-model baseline accuracies on AudioEvidence are not reported for any task or duration. Without them, the 26.7 to 39.6 point drops from critical-interval blocking cannot be put in context. The scattered numbers that are reported suggest this matters: Gemma 4 is at 29.4% and 20.8% on the P1 relocation set against a 25% chance level (Sec. C.3), and Qwen2-Audio is at 12.08% on P3 (Sec. F.2). It is unclear how analyses on model-task combinations at or below chance should be interpreted.

(b) The control gains in Sec. 6 are small and not clearly tied to auditory grounding. The AHa-Bench interval includes zero, and full-interval scaling gives no gain on either hallucination benchmark (Table 6). On LibriSpeech, 269 of the 302 removed word errors come from 17 utterances where the baseline output Chinese translations, and the WER change on the remaining 2,603 utterances is −0.063 points with an interval including zero (Table 9), so the gain mainly reflects task adherence. On both hallucination benchmarks, FPR decreases while FNR increases (Table 7), which is consistent with a shift in response bias, but there is no comparison with random head amplification at the same K and α, with simple logit bias calibration, or with prior steering methods such as MATA (Wang et al., 2025) and audio-specialist head steering (Glazer et al., 2026). The selected α = 1.3 is also the largest value in the calibration grid (Fig. 21).

Questions for authors

Clarifications:

  1. Could you report the baseline accuracy of all nine models on AudioEvidence P0 to P3, for both the 10-s and 30-s versions? How are model-task combinations at or below chance handled in the state and pathway analyses?

  2. Why does the introduction report the TAllT_{All} ablation result while Sec. 5.1 names TQT_Q as the primary ranking?

  3. How large is the discovery sample used for the inference-time ranking in Sec. 6, and how much does the ranking overlap with the one from the 480-example P0 discovery set?

Suggestions that would affect my assessment:

  1. A layer-matched random ablation baseline for the head ablations in Sec. 5.1, and a comparison with a ranking that does not use question conditioning.

  2. For the Top80, α = 1.3 configuration, comparisons with random head amplification at the same K and α, with a logit bias calibration that produces a similar FPR/FNR shift, and with MATA or the steering method of Glazer et al. (2026).

7 comments
AuthorsOpenReview ↗

Response to Reviewer QcWf — Part 1/6

Response to Reviewer QcWf — Part 1/6

Thank you for reviewing our work.


1. (Excerpt) The main mechanistic finding is largely expected, and several claims are framed more strongly than the results support. […] The joint intervention in Table 5, where blocking access leaves less evidence in subsequent text states, also points in a direction that is nearly expected by construction, although the matched N control makes the result clean.

The paper asks: when a model uses audio to answer, at what depths and through which components does it access the answer-determining sound, and how does this access shape subsequent text computation? Reading an answer from a text position specifies where information must ultimately act, but not this process. Section 3 therefore constructs counterfactuals that replace only the target sound while preserving the question and other waveform content; Section 4 tests whether answer-related representation effects move with that sound; Section 5.2 then localizes and validates critical access using target evidence E versus matched non-evidence N and candidate intervals versus equal-width early/late controls. Eight models have evidence- and depth-selective access intervals, with model-dependent locations and widths.

Table 5 further tests the causal effect of critical evidence access on subsequent text computation. Changing answer preference by blocking a pathway establishes pathway importance; changing it by patching a state establishes state importance. Observing these separately does not establish whether the measured state effect depends on that pathway. The original joint intervention therefore uses inputs A/B differing only in the target sound: the recipient B runs normally, while E or N access is blocked only in A, within the identified interval and before the prespecified measurement layer. A's subsequent text states are then patched into the same B. In all eight models, E blocking weakens the states' ability to transfer A's answer preference more than N blocking, passing the prespecified primary comparison.

The cross-model comparison thus addresses two linked questions: which evidence-integration patterns are shared, and which component properties vary with the model and task? The original nine models span different audio interfaces, backbones, and training settings. Eight share causally validated critical access stages and access–state links, with model-dependent interval locations, widths, and head contributions. Added selection and transfer experiments further establish task dependence. Shared access organization and differing component implementations together constitute our answer to How Sound Meets Language. We will make this central aim and its quantitative evidence explicit in the introduction and remove “surprisingly.”


AuthorsOpenReview ↗

Response to Reviewer QcWf — Part 2/6

Response to Reviewer QcWf — Part 2/6

2. The retrieval-head hypothesis is supported in one of nine models, yet the abstract and introduction present it as paralleling text-LLM findings.

The proposed reasons for MiMo-Audio being different (6.25 positions per second, RVQ discretization, over 100M hours of training) are not tested.

Section 5.1 starts from the text-LLM retrieval-head hypothesis and tests whether audio evidence use is concentrated in a small set of heads. The paper ranks heads by question-conditioned evidence attention, then ablates fixed sets on MMAU/MMAR to test their causal importance for broader audio tasks. MiMo shows the most pronounced concentration in the original results, while effects in other models vary with ranking and budget.

Whole-head ablation measures a component's overall role across positions, including other computations it supports. To identify its evidence-access contribution, we add two complementary tests: selected-head versus matched-budget random ablation on controlled sound retrieval P0, and blocking only E or N attention access with the selected head set held fixed. These results extend the original MiMo-centered observation to direct evidence-access contributions across models, and further test how contributions vary with the selection task.

ModelP0 baselineWhole-head drop − mean random drop (pp)E-pathway drop (pp)N-pathway drop (pp)
MiMo-Audio56.25%22.6420.63−3.75
Qwen2.5-Omni78.13%11.3918.33−2.50
Qwen2-Audio57.29%1.537.71−0.42
Step-Audio-2-mini66.88%6.393.96−2.08
Audio Flamingo 368.33%1.045.421.67
Kimi-Audio65.21%4.511.67−0.83
MOSS-Audio71.25%10.497.71−1.04
Qwen3-Omni80.21%5.006.04−0.63

P0 is controlled sound-event retrieval: 480 questions per model, global top-5% TQ heads, and matched-budget random controls. E/N block the same heads' access to target/matched non-target audio; negative drops indicate improvements.

Whole-head losses exceed random controls in six models, so retrieval-task contributions are not unique to MiMo. Significant E/N margin differences in all eight models show that these components directly participate in evidence access. Under primary TQ, accuracy differences are significant in seven models. Whole-head comparisons use Holm16; pathway comparisons use Holm32 across TQ/TAll and accuracy/margin, yielding 16 significant margin and 14 significant accuracy comparisons. Margin is the correct-minus-strongest-incorrect option score.

To test how the selection task affects the components identified, we extend the speech branch in Appendix H. The original spoken causal QA task (ContextQA) asks for the cause of an event stated in a sentence, using four answer options; the added tasks require recovery of six-letter or six-character alphanumeric strings spoken character by character. All three tasks cover nine models, using 10s audio, 480 selection examples and 480 independent evaluation examples. The original GE score selects the top 5% of heads across the model using TQ.

ContextQA uses whole-sentence evidence. Both string tasks ask “What is the code?”, with no spoken prefix, format instruction, or options; they use whole-string evidence and exact matching after six-character extraction. Ablation on the head-selection task (the source task) measures contribution to that task; ablation of the same fixed heads on MMAU/MMAR measures cross-task effects. Each benchmark has 1000 questions, scored by valid-option scores at the first answer position. Content and response format jointly define the three selection conditions.

Source-task contribution: baseline (%) / ablation drop (pp).

ModelContextQA baseline / dropLetters baseline / dropMixed baseline / drop
MiMo100.00 / 4.7915.83 / 6.2531.04 / 9.79
Step100.00 / 0.002.71 / 2.717.08 / 6.67
AF3100.00 / 0.214.17 / 3.9626.04 / 3.75
Qwen2.5-Omni100.00 / 1.6722.08 / 22.0811.67 / 9.17
Kimi100.00 / 0.002.29 / 0.6317.71 / 17.08
MOSS100.00 / 3.5410.00 / 8.7526.67 / 17.71
Qwen2-Audio97.50 / 35.210.83 / 0.622.92 / 1.25
Gemma4100.00 / 42.505.83 / 5.8313.96 / 13.33
Qwen3-Omni100.00 / 0.0030.63 / 14.1737.71 / 2.92

Among the eight models with no-audio/silence controls, ContextQA baselines are 97.50%–100%, versus 23.54%–26.88% without audio and 24.58%–27.29% with silence, confirming audio dependence. Despite the shared task, selected heads cause 35.21/42.50-point losses in Qwen2-Audio/Gemma, whereas Step/Kimi show no accuracy loss. Audio dependence and the task contribution of a small selected head set are therefore distinct properties to measure.

AuthorsOpenReview ↗

Response to Reviewer QcWf — Part 3/6

Response to Reviewer QcWf — Part 3/6

Transfer performance: targeted drop − existing random drop (pp). Each cell gives MMAU / MMAR:

ModelContextQA → MMAU / MMARLetters → MMAU / MMARMixed → MMAU / MMAR
MiMo5.10 / 8.8013.20 / 18.2019.40 / 21.70
Step3.07 / 5.030.07 / 1.73-0.63 / 2.43
AF32.63 / 4.373.93 / 6.672.83 / 5.47
Qwen2.5-Omni4.40 / 3.9340.00 / 33.838.40 / 9.23
Kimi3.30 / 0.832.70 / 2.834.50 / 4.63
MOSS0.07 / 1.570.27 / 0.87-0.13 / 0.57
Qwen2-Audio9.60 / 3.20-0.50 / 0.70-1.70 / -0.50
Gemma421.00 / 17.572.80 / 2.175.30 / 1.77
Qwen3-Omni2.50 / 2.301.50 / 1.801.20 / 2.10

Historical Global 5% controls use their corresponding baselines; cross-protocol differences are descriptive.

Selection-task effects differ across models. Qwen2.5's letter-selected heads cause 31.60/24.60 points more MMAU/MMAR damage than its mixed-string heads; in MiMo, mixed-string heads instead cause 6.20/3.50 points more damage than letter heads. Newly evaluated ContextQA heads also produce larger transfer losses in Gemma than either string head set. Qwen3's letter-selected heads cause a 14.17-point source-task loss, versus raw MMAU/MMAR losses of 1.30/2.70 points, further distinguishing source-task contribution from transfer effects. The same random reference is used within each model and target task, leaving differences between selected head sets unchanged.

Together, the experiments identify heads with direct evidence-access contributions across multiple models, whose source- and transfer-task effects vary with model and selection task. P0 random controls support selective localization, fixed-head E/N controls identify direct-access contributions, and speech extensions characterize task and transfer scope. This refines the original cross-model differences into direct-access contributions and task-transfer scope.

The abstract and introduction will emphasize “direct evidence-access contributions across models and their task dependence.” RVQ, audio-position density, and training scale retain their hypothesis status from Section 7.


3. The introduction also reports the headline drop (77.8% to 47.7%) under the [TAll] ranking, whereas Sec. 5.1 states that [TQ] is the primary ranking, under which the drop is to 59.4%.

Why does the introduction report the [TAll] ablation result while Sec. 5.1 names [TQ] as the primary ranking?

The paper compares TQ at question positions with TAll across post-audio input positions, using TQ as primary in Section 5.1. To align the introduction with that analysis, we will report TQ: 77.4%→59.4%, an 18.0-percentage-point drop, and explicitly label 77.4%→47.7% as a supplementary TAll result. The original 77.8% came from an earlier discovery experiment; 77.4% is the baseline corresponding to the reported ablation. The revised introduction and Section 5.1 will be consistent.


4. Finally, the claim that the findings provide priors for more efficient audio scaling is not supported by any training or scaling experiment.

The motivation for scaling priors is to use cross-model observations to inform future studies of computation allocation and modality integration. Appendix B.1 and Section 7 compare nine models spanning discrete audio codes, continuous features, hybrid representations, and direct waveform inputs, with different backbones, dense/MoE architectures, and training scales. The aim is to distinguish shared evidence-use patterns from model-dependent component organization.

The empirical findings identify a critical direct-access stage in eight models, while head-level concentration and transfer vary across models. These observations provide concrete questions for subsequent training studies, including where to allocate audio–language computation and whether component organization changes with the audio interface.

Section 7 already identifies RVQ, temporal compression, and training scale as potentially interacting hypotheses and states that they were not independently manipulated. We will preserve the cross-model observations and motivation, while clarifying “priors” as “mechanistic observations to inform subsequent controlled training studies,” rather than demonstrated training-efficiency gains.


Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response — A — Part 1/5

We thank the three reviewers for their comments.

Our central aim is to distinguish causal organization shared across audio–language models from component contributions that vary with the model and task. Nine models span different audio interfaces, backbones, and training settings: causal tests in eight identify critical evidence-access stages and their effects on subsequent text computation, while head-level concentration and transfer scope vary across models and tasks. Additional experiments test these components' direct evidence-access contributions and whether critical-stage selectivity persists when audio relay is included.

A. Core contribution: from overall cross-modal interaction to the causal link between specific evidence and language decisions

The paper asks: when a model uses audio to answer, at what depths and through which components does it access the answer-determining sound, and how does this access shape subsequent text computation? Reading an answer from a text position specifies where information must ultimately act, but not this process. Section 3 therefore constructs counterfactuals that replace only the target sound while preserving the question and other waveform content; Section 4 tests whether answer-related representation effects move with that sound; Section 5.2 then localizes and validates critical access using target evidence E versus matched non-evidence N and candidate intervals versus equal-width early/late controls. Eight models have evidence- and depth-selective access intervals, with model-dependent locations and widths.

Table 5 further tests the causal effect of critical evidence access on subsequent text computation. Changing answer preference by blocking a pathway establishes pathway importance; changing it by patching a state establishes state importance. Observing these separately does not establish whether the measured state effect depends on that pathway. The original joint intervention therefore uses inputs A/B differing only in the target sound: the recipient B runs normally, while E or N access is blocked only in A, within the identified interval and before the prespecified measurement layer. A's subsequent text states are then patched into the same B. In all eight models, E blocking weakens the states' ability to transfer A's answer preference more than N blocking, passing the prespecified primary comparison.

The cross-model comparison thus addresses two linked questions: which evidence-integration patterns are shared, and which component properties vary with the model and task? The original nine models span different audio interfaces, backbones, and training settings. Eight share causally validated critical access stages and access–state links, with model-dependent interval locations, widths, and head contributions. Added selection and transfer experiments further establish task dependence. Shared access organization and differing component implementations together constitute our answer to How Sound Meets Language. We will make this central aim and its quantitative evidence explicit in the introduction and remove “surprisingly.”

This addresses QcWf (Q1), 1WuF (Q1, Q6), and rnJw (Q4); the comparison with prior work appears at the end.

AuthorsOpenReview ↗

General Response — B.1–B.2 — Part 2/5

General Response — B.1–B.2 — Part 2/5

B. Retrieval-head hypothesis: direct evidence-access contributions across models, with model- and task-dependent transfer

Section 5.1 starts from the text-LLM retrieval-head hypothesis and tests whether audio evidence use is concentrated in a small set of heads. The paper ranks heads by question-conditioned evidence attention, then ablates fixed sets on MMAU/MMAR to test their causal importance for broader audio tasks. MiMo shows the most pronounced concentration in the original results, while effects in other models vary with ranking and budget.

Whole-head ablation measures a component's overall role across positions, including other computations it supports. To identify its evidence-access contribution, we add two complementary tests: selected-head versus matched-budget random ablation on controlled sound retrieval P0, and blocking only E or N attention access with the selected head set held fixed. These results extend the original MiMo-centered observation to direct evidence-access contributions across models, and further test how contributions vary with the selection task.

We thank 1WuF (Q3) for the evidence-specificity question and QcWf (Q2) for the question on generality. They motivated separate tests of component importance, direct-access contributions, and transfer scope.

B.1 Do selected heads matter for retrieval? Six models show losses exceeding random controls

To test contributions on a retrieval task directly, we evaluate controlled sound-event retrieval P0 and compare against matched-budget random ablations, testing whether selected-head contributions exceed general component-removal damage.

ModelP0 baselineTQ drop − mean random drop (pp)
MiMo56.25%22.64
Qwen2.5-Omni78.13%11.39
Qwen2-Audio57.29%1.53
Step66.88%6.39
AF368.33%1.04
Kimi65.21%4.51
MOSS71.25%10.49
Qwen3-Omni80.21%5.00

Controlled sound-event retrieval P0 uses 480 questions per model and global top-5% TQ heads; random controls use the same budget. Bold values indicate differences significant after Holm16 correction across eight models × TQ/TAll; pp denotes percentage points. Retrieval-task contributions are not unique to MiMo, and transfer differences on broad benchmarks do not establish absence of retrieval components elsewhere.

TQ/TAll use the original GE=Full−Null ranking with question positions/all post-audio input positions, respectively. MiMo, Step, AF3, Kimi, and MOSS are short for MiMo-Audio, Step-Audio-2-mini, Audio Flamingo 3, Kimi-Audio, and MOSS-Audio, respectively.

B.2 Do these components directly access evidence? Fixed-head E/N comparisons support this role

Whole-head losses on P0 can still include damage to other computations. To test direct evidence-access contributions, we hold each head set fixed and block only its text-position access to target evidence E or matched non-evidence N:

ModelBaselineE-pathway drop (pp)N-pathway drop (pp)Additional E versus N drop (pp)
MiMo56.25%20.63−3.7524.38
Qwen2.5-Omni78.13%18.33−2.5020.83
Qwen2-Audio57.29%7.71−0.428.13
Step66.88%3.96−2.086.04
AF368.33%5.421.673.75
Kimi65.21%1.67−0.832.50
MOSS71.25%7.71−1.048.75
Qwen3-Omni80.21%6.04−0.636.67

Each model uses 480 questions and the same budget; negative drops indicate improvements and differences use unrounded values. Under primary TQ, E/N differences are significant in eight models for margin and seven for accuracy. Including TAll, 16 margin and 14 accuracy comparisons pass Holm32 (eight models × two rankings × two metrics). Margin is the correct-minus-strongest-incorrect option score; bold denotes significant accuracy differences.

B.1 establishes retrieval-task importance; B.2 identifies direct evidence-access contributions. Together they establish task contributions and direct-access roles across models.

AuthorsOpenReview ↗

General Response — B.3 — Part 3/5

General Response — B.3 — Part 3/5

B.3 Do heads selected by the same score have consistent task contributions and transfer scope?

To test how the selection task affects the components identified, we extend the speech branch in Appendix H. The original spoken causal QA task (ContextQA) asks for the cause of an event stated in a sentence, using four answer options; the added tasks require recovery of six-letter or six-character alphanumeric strings spoken character by character. All three tasks cover nine models, using 10s audio, 480 selection examples and 480 independent evaluation examples. The original GE score selects the top 5% of heads across the model using TQ.

ContextQA uses whole-sentence evidence. Both string tasks ask “What is the code?”, with no spoken prefix, format instruction, or options; they use whole-string evidence and exact matching after six-character extraction. Ablation on the head-selection task (the source task) measures contribution to that task; ablation of the same fixed heads on MMAU/MMAR measures cross-task effects. Each benchmark has 1000 questions, scored by valid-option scores at the first answer position. Content and response format jointly define the three selection conditions.

Source-task contribution: baseline (%) / ablation drop (pp).

ModelContextQA baseline / dropLetters baseline / dropMixed baseline / drop
MiMo100.00 / 4.7915.83 / 6.2531.04 / 9.79
Step100.00 / 0.002.71 / 2.717.08 / 6.67
AF3100.00 / 0.214.17 / 3.9626.04 / 3.75
Qwen2.5-Omni100.00 / 1.6722.08 / 22.0811.67 / 9.17
Kimi100.00 / 0.002.29 / 0.6317.71 / 17.08
MOSS100.00 / 3.5410.00 / 8.7526.67 / 17.71
Qwen2-Audio97.50 / 35.210.83 / 0.622.92 / 1.25
Gemma4100.00 / 42.505.83 / 5.8313.96 / 13.33
Qwen3-Omni100.00 / 0.0030.63 / 14.1737.71 / 2.92

Among the eight models with no-audio/silence controls, ContextQA baselines are 97.50%–100%, versus 23.54%–26.88% without audio and 24.58%–27.29% with silence, confirming audio dependence. Despite the shared task, selected heads cause 35.21/42.50-point losses in Qwen2-Audio/Gemma, whereas Step/Kimi show no accuracy loss. Audio dependence and the task contribution of a small selected head set are therefore distinct properties to measure.

Transfer performance: targeted drop − existing random drop (pp). Each cell gives MMAU / MMAR:

ModelContextQA → MMAU / MMARLetters → MMAU / MMARMixed → MMAU / MMAR
MiMo5.10 / 8.8013.20 / 18.2019.40 / 21.70
Step3.07 / 5.030.07 / 1.73-0.63 / 2.43
AF32.63 / 4.373.93 / 6.672.83 / 5.47
Qwen2.5-Omni4.40 / 3.9340.00 / 33.838.40 / 9.23
Kimi3.30 / 0.832.70 / 2.834.50 / 4.63
MOSS0.07 / 1.570.27 / 0.87-0.13 / 0.57
Qwen2-Audio9.60 / 3.20-0.50 / 0.70-1.70 / -0.50
Gemma421.00 / 17.572.80 / 2.175.30 / 1.77
Qwen3-Omni2.50 / 2.301.50 / 1.801.20 / 2.10

Historical Global 5% controls use their corresponding baselines; cross-protocol differences are descriptive.

Selection-task effects differ across models. Qwen2.5's letter-selected heads cause 31.60/24.60 points more MMAU/MMAR damage than its mixed-string heads; in MiMo, mixed-string heads instead cause 6.20/3.50 points more damage than letter heads. Newly evaluated ContextQA heads also produce larger transfer losses in Gemma than either string head set. Qwen3's letter-selected heads cause a 14.17-point source-task loss, versus raw MMAU/MMAR losses of 1.30/2.70 points, further distinguishing source-task contribution from transfer effects. The same random reference is used within each model and target task, leaving differences between selected head sets unchanged.

Together, the experiments identify heads with direct evidence-access contributions across multiple models, whose source- and transfer-task effects vary with model and selection task. P0 random controls support selective localization, fixed-head E/N controls identify direct-access contributions, and speech extensions characterize task and transfer scope. This refines the original cross-model differences into direct-access contributions and task-transfer scope.

The introduction will report primary TQ 77.4%→59.4%, with 77.4%→47.7% labeled as supplementary TAll.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.