- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper discusses AudioEvidence, a synthetic testing protocal with anotated evidence intervals, matched non-evidence controls, and counterfactual pairs. Using state patching, attention blocking, and head ablation across nine LALMs, the work reports 3 findings:
- evidence effects stay localized at the backbone input;
- in 8 models, test -> evidence access at intermediate layers drives later text-state effects;
- head level concentration appears clearly in MiMO-Audio
As a test results, scaling selected MiMo-Audio heads gives small gaims on two benchmarks (HalluAudio and Librispeech).
Critical strengths
- the counterfactual design is good initiative, with careful setup supporting it (non evidence controls ; Full/Null question contrast etc)
- 9 models do cover enough variance in modeling
- paper honestly reports that the retrieval-head hypothesis mostly fails and that attention scores do not predict causal importance.
- The main findings transfer to other tasks, -30s inputs, external benchmarks, and speech.
Critical weaknesses
- The main finding is partly forced by the architecture: The answer is read at a text position, so information must move from audio to text at some depth, and a crossover in state effects is expected. So the evidence flow findings are not suprising
- Later audio positions can attend to E and pass its information on to text. The claims should say “direct access,” or the experiments should also block audio -> E attention.
- The head-ablation results are not evidence-specific. Notably, MiMo with ablated heads only got 47.7% MMAU while the same model with silent audio got 58.6, meaning that the ablation demages more than audio retrieval. Also heads are zeroed at all positions, not only on paths to audio.
- Only two results are significant, both in one model, which is not plausible. Notably, LibriSpeech test-clean is close to saturated, and strong models commonly report WER below 2%. MiMo-Audio’s 3.44% baseline is weak, and basically 89% of the gain comes from fixing 17 utterances where it output Chinese. This shows better task adherence, not better recognition. Probably a harder sets would be a good choice here (noisy / accented, or long form speech).
- MiMo was chosen because it was the only model with concentrated heads, so these results are a best case. G.5 reports only selected positive cases, so please report all configurations.
- The novelty over prior work is unclear. Similar depth-wise transfer is reported by Zhang et al. (2025), Suharitdamrong et al. (2026), Chen et al. (2026b), and Fan et al. (2026). Please state the difference clearly in the introduction.
- Figure 1 motivates the work with MMAU accuracy under silence. The intervention is never tested on MMAU or MMAR, even though the same heads are validated there by ablation, and it is never tested under silence. Either run these evaluations or reframe the introduction.
Minor issues:
- There are too many “evidence” terms. The paper uses evidence states, evidence access, evidence attention, evidence-attending heads, access intervals, and access envelopes. These are spread across sections and mix “positions E” with “information the model uses.” Given many terms discussed, please consider introduce glossary in the paper.
- Figure 2 is hard to read: Panel (a) shows Full/Null without saying what it is for. Panel (b) mixes the counterfactual pair with an unrelated task list. Panel (c) does not show which positions are affected, the patching direction, or what is measured. Caption is not useful here also.
- The phrase “surprisingly, these intervals overlap” is not surprising given Weakness 1.
- The experiments locate where evidence has influence, not what computation happens, so “causal mechanisms” overstates the result. “Priors for efficient audio scaling” is never tested. The RVQ and training-scale discussion is speculation.
Questions for authors
Already explain my questions in the weakness section.
11 comments
Response to Reviewer 1WuF — Part 2/6
Response to Reviewer 1WuF — Part 2/6
2. Later audio positions can attend to E and pass its information on to text. The claims should say “direct access,” or the experiments should also block audio -> E attention.
The original operator targets direct access from text positions to evidence E to locate the depth at which the model depends on this connection. The paper tests this stage using within-interval E/N and equal-width early/late controls.
Thank you for identifying the possibility of relay through other audio positions. This lets us test whether the critical stage persists when that pathway is included. Keeping the original interval fixed, we compare Direct (blocking post-audio text positions from attending to E), Relay (blocking non-E audio positions from attending to E), and Joint (both). All three interventions act on every head within the original critical interval:
| Model | Baseline | Direct | Relay | Joint | Additional Relay drop (pp) |
|---|---|---|---|---|---|
| MiMo | 56.25% | 29.58% | 56.88% | 27.29% | 2.29 |
| Qwen2.5-Omni | 78.13% | 38.96% | 77.29% | 38.33% | 0.63 |
| Qwen2-Audio | 57.29% | 25.21% | 57.08% | 25.21% | 0.00 |
| Step | 66.88% | 40.21% | 67.71% | 38.33% | 1.88 |
| AF3 | 68.33% | 33.13% | 67.71% | 33.96% | -0.83 |
| Kimi | 65.21% | 33.96% | 64.79% | 31.25% | 2.71 |
| MOSS | 71.25% | 31.67% | 70.83% | 31.25% | 0.42 |
| Qwen3-Omni | 80.21% | 49.17% | 78.75% | 30.63% | 18.54 |
Each model uses 480 questions. The last column is Direct−Joint; negative values indicate improvement; bold values are significant after Holm8 correction across eight models.
Direct-access blocking causes substantial damage in all eight models, while additional relay contributions vary by model. The additional accuracy losses in Kimi and Qwen3 pass the correction; Qwen3 falls further from 49.17% to 30.63%, showing a substantial relay contribution after direct access is blocked. Additional margin damage is significant in MiMo, Qwen2.5, Step, Kimi, and Qwen3; Qwen2-Audio shows a significant reverse effect.
Crucially, all 16 accuracy comparisons between the critical interval and equal-width early/late intervals under Joint pass Holm16. Critical-stage depth selectivity therefore persists when audio relay is included. We will consistently call the original operator “direct evidence access” and report model-specific additional relay contributions.
Response to Reviewer 1WuF — Part 3/6
Response to Reviewer 1WuF — Part 3/6
3. The head-ablation results are not evidence-specific. Notably, MiMo with ablated heads only got 47.7% MMAU while the same model with silent audio got 58.6, meaning that the ablation demages more than audio retrieval. Also heads are zeroed at all positions, not only on paths to audio.
Whole-head ablation in the original paper tests whether heads selected on controlled evidence are causally important for independent, broader tasks. Zeroing outputs at all positions affects multiple computations supported by those heads, whereas silence changes the audio input. Performance below the silent-audio condition can therefore include damage beyond evidence access, motivating targeted pathway controls.
Thank you for identifying why this distinction matters for the retrieval-head hypothesis. We add two direct tests: comparison with random ablation on P0 tests retrieval-task contributions; E/N pathway blocking with the same heads fixed tests direct evidence-access contributions.
| Model | P0 baseline | Whole-head drop − mean random drop (pp) | E-pathway drop (pp) | N-pathway drop (pp) |
|---|---|---|---|---|
| MiMo-Audio | 56.25% | 22.64 | 20.63 | −3.75 |
| Qwen2.5-Omni | 78.13% | 11.39 | 18.33 | −2.50 |
| Qwen2-Audio | 57.29% | 1.53 | 7.71 | −0.42 |
| Step-Audio-2-mini | 66.88% | 6.39 | 3.96 | −2.08 |
| Audio Flamingo 3 | 68.33% | 1.04 | 5.42 | 1.67 |
| Kimi-Audio | 65.21% | 4.51 | 1.67 | −0.83 |
| MOSS-Audio | 71.25% | 10.49 | 7.71 | −1.04 |
| Qwen3-Omni | 80.21% | 5.00 | 6.04 | −0.63 |
Each model uses 480 questions and global top-5% TQ heads; random controls use the same budget. E/N block the same heads' access to target/matched non-target audio; negatives indicate improvements.
First, retrieval-task losses exceed random controls in six models (Holm16), extending the effect originally most apparent in MiMo. Second, E/N margin differences are significant in all eight models and primary-TQ accuracy differences in seven, supporting direct evidence-access contributions. Including TAll, 16 margin and 14 accuracy comparisons pass Holm32; margin is the correct-minus-strongest-incorrect option score.
We therefore extend the original MiMo-centered finding to heads participating in direct evidence access across multiple models, with task contributions and transfer scope depending on model and selection task.
The introduction will use the primary TQ result 77.4%→59.4%; the 47.7% cited in the comment will be explicitly labeled as the TAll ablation result.
4. (Excerpt) Only two results are significant, both in one model, which is not plausible. […] MiMo-Audio’s 3.44% baseline is weak, and basically 89% of the gain comes from fixing 17 utterances where it output Chinese. This shows better task adherence, not better recognition. […] Probably a harder sets would be a good choice here (noisy / accented, or long form speech).
Task adherence is already the paper's interpretation of the ASR gain. Section 6 explicitly states that the WER improvement primarily comes from fewer hallucination-related errors and task deviations, especially Chinese translations in place of requested English transcriptions. The paper reports that such outputs account for 89.1% of the net word-error reduction, with breakdowns and examples in Appendix G (Table 9).
The original steering motivation is therefore to test whether evidence-related heads provide inference-time control points for hallucination-related behavior and task adherence. To test whether this effect extends to more difficult speech, we keep MiMo's Top80, α=1.3 configuration fixed and evaluate LibriSpeech test-other:
| Scope | Utterances | Baseline WER | Amplified WER | WER change (pp), 95% CI |
|---|---|---|---|---|
| All test-other | 2939 | 6.6122% | 6.4268% | −0.1853 [−0.3797, −0.0254] |
The configuration is unchanged; confidence intervals use paired bootstrap over 33 speakers. Overall WER also improves on harder test-other, consistent with the original behavioral-control framing. In the additional breakdown, seven baseline outputs containing Chinese characters account for 56 of 97 net errors removed, again supporting task adherence as a source of benefit.
Across models, the original Table 10 already reports positive HalluAudio configurations for Qwen2-Audio, Qwen2.5-Omni, and Qwen3-Omni, based on pointwise 95% confidence intervals. The complete configurations show how amplification effects vary across models and settings.
Response to Reviewer 1WuF — Part 1/6
Response to Reviewer 1WuF — Part 1/6
Thank you for reviewing our work.
The paper asks: when a model uses audio to answer, at what depths and through which components does it access the answer-determining sound, and how does this access shape subsequent text computation? Reading an answer from a text position specifies where information must ultimately act, but not this process. Section 3 therefore constructs counterfactuals that replace only the target sound while preserving the question and other waveform content; Section 4 tests whether answer-related representation effects move with that sound; Section 5.2 then localizes and validates critical access using target evidence E versus matched non-evidence N and candidate intervals versus equal-width early/late controls. Eight models have evidence- and depth-selective access intervals, with model-dependent locations and widths.
Table 5 further tests the causal effect of critical evidence access on subsequent text computation. Changing answer preference by blocking a pathway establishes pathway importance; changing it by patching a state establishes state importance. Observing these separately does not establish whether the measured state effect depends on that pathway. The original joint intervention therefore uses inputs A/B differing only in the target sound: the recipient B runs normally, while E or N access is blocked only in A, within the identified interval and before the prespecified measurement layer. A's subsequent text states are then patched into the same B. In all eight models, E blocking weakens the states' ability to transfer A's answer preference more than N blocking, passing the prespecified primary comparison.
The cross-model comparison thus addresses two linked questions: which evidence-integration patterns are shared, and which component properties vary with the model and task? The original nine models span different audio interfaces, backbones, and training settings. Eight share causally validated critical access stages and access–state links, with model-dependent interval locations, widths, and head contributions. Added selection and transfer experiments further establish task dependence. Shared access organization and differing component implementations together constitute our answer to How Sound Meets Language. We will make this central aim and its quantitative evidence explicit in the introduction and remove “surprisingly.”