- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the underutilization of audio information in LALMs and relates it to limitations in audio representation and audio-text alignment during pretraining. To address this issue, the authors propose FAME, a hierarchical training framework that strengthens the audio pathway through a MoE design. It first trains five aspect-specific expert heads on separate datasets, then learns a question-driven soft router, and subsequently adapts the router, projector, and LLM through expert-level and notion-level focus augmentation together. Experiments across four benchmarks and multiple backbones demonstrate the effectiveness of FAME and provide support for the overall training strategy.
Critical strengths
-
The three-level framework is intuitively well motivated by the observed audio utilization problem. Moving from aspect-specific representation learning to question-guided routing and then to higher-level focus augmentation forms a reasonably coherent progression, and the roles of the individual components are discussed through step-by-step ablations.
-
FAME produces improvements across a diverse set of backbones rather than being tied to a single base model. In addition, the expert-level evaluation provides useful evidence that four of the five expert heads acquire their intended types of information.
Critical weaknesses
-
Although the overall pipeline is introduced, several important architectural choices are not examined in sufficient depth. The MoE encoder seems to consist of a shared frozen audio encoder and five linear expert heads. It remains unclear why these particular choices are appropriate or how sensitive the results are to them. Comparisons among different expert architectures, routing mechanisms, or feature fusion strategies would make the technical design more convincing. The presentation also relies heavily on one dense overview figure, and a clearer module-wise description would help.
-
The motivation in the Introduction moves among general audio reasoning limitations (although I do not find all of these arguments particularly convincing), audio encoding, and audio-text alignment. The paper then largely attributes audio under-utilization to the encoder structure and training objectives, but this causal explanation is not sufficiently established. Existing LALMs already perform explicit audio-text alignment during pretraining, and the current results do not isolate the main limitation clearly.
-
More direct evidence is needed to show that FAME genuinely improves audio encoding and utilization. Since FAME introduces substantial additional supervision from multiple audio datasets, some end-to-end gains may come from exposure to this data rather than from the proposed architectural design itself. A data-matched conventional fine-tuning baseline would help separate these effects.
Questions for authors
- Could the authors provide a more detailed analysis of the MoE-based audio encoder architecture? Were alternative expert structures, routing strategies, or fusion mechanisms considered, and are there ablations targeting these architectural choices?
- Can the authors provide more direct experimental evidence that existing LALMs underutilize audio and that FAME specifically mitigates this problem, beyond downstream accuracy and aggregate attention weights (which prior studies have shown that they may correlate poorly with feature importance and can change substantially without corresponding changes in model predictions) ?
[1] NAACL 2019, Attention is not Explanation. [2] ACL 2020, Learning to Deceive with Attention-Based Explanations.
4 comments
Response to Reviewer UkQj (2)
Weakness 3
LoRA baseline: We would like to further clarify that our training and evaluation datasets are entirely distinct in terms of both their content and task formulation. Specifically, the training data consist exclusively of basic audio classification tasks, as illustrated by the examples in Table 7 (e.g., identifying musical instruments). In contrast, the evaluation datasets consist of reasoning-oriented tasks. Given this substantial difference in task distributions, we would not necessarily expect standard LoRA fine-tuning on the training data to yield substantial improvements on the reasoning benchmarks. Nevertheless, in response to the reviewer's request, we are reporting this additional experiment to provide a clearer and more comprehensive comparison:
| Backbone | Arm | MMAU | MuCho | Clotho (bal.) | ADQA |
|---|---|---|---|---|---|
| Audio-Flamingo-3 | Native | 59.14 | 76.66 | 68.50 | 41.71 |
| +FAME | 59.94 | 74.55 | 76.45 | 44.95 | |
| Pure LoRA | 59.25 | 77.00 | 70.84 | 42.71 | |
| Gemma-3n-E2B | Native | 46.43 | 63.02 | 58.65 | 35.93 |
| +FAME | 50.12 | 60.07 | 60.06 | 38.37 | |
| Pure LoRA | 47.69 | 61.68 | 58.90 | 36.92 | |
| Voxtral-Mini-3B | Native | 45.17 | 57.71 | 57.98 | 40.49 |
| +FAME | 50.24 | 53.33 | 61.47 | 45.02 | |
| Pure LoRA | 46.68 | 55.60 | 52.01 | 41.56 | |
| Qwen2-Audio-7B | Native | 43.29 | 52.40 | 54.96 | 36.39 |
| +FAME | 49.43 | 62.43 | 60.78 | 41.92 | |
| Pure LoRA | 44.53 | 54.03 | 56.32 | 40.42 | |
| Qwen2.5-Omni-3B | Native | 57.04 | 70.94 | 81.02 | 43.11 |
| +FAME | 57.78 | 72.28 | 82.86 | 44.36 | |
| Pure LoRA | 55.37 | 71.34 | 81.06 | 42.63 | |
| Ultravox v0.5 | Native | 41.49 | 54.68 | 51.57 | 33.57 |
| +FAME | 45.24 | 54.71 | 53.00 | 35.41 | |
| Pure LoRA | 42.35 | 53.33 | 50.95 | 33.61 | |
| GLM-4-Voice-9B | Native | 41.07 | 41.03 | 50.20 | 28.83 |
| +FAME | 43.86 | 44.82 | 50.95 | 37.19 | |
| Pure LoRA | 40.36 | 43.08 | 50.24 | 31.13 |
It shows that the Pure LoRA results are quite similar to the Native version, and even worse, which is expected as this LoRA is finetuning the model to undertake simple classification tasks such as emotion recognition instead of audio reasoning task. It is also a proof that the gain is from our methodology instead of more exposure to more audios.
Furthermore, most of the MoE-based methods included in Table 3 do not report comparisons against standard LoRA fine-tuning. Our competitive, and in some cases superior, performance compared with these existing approaches provides further evidence of the effectiveness of our framework. The additional pure-LoRA experiment will allow us to establish this comparison more directly.
Questions to authors
Q1: We believe this question has been addressed in weakness 1. Please let us know if more concerns remain.
Q2:
More investigation on exploitation on audio modality:
Previous studies have reported modality bias and the underutilisation of non-textual modalities in multimodal LLMs, including visual and audio inputs (e.g., (https://arxiv.org/abs/2505.18657); (https://arxiv.org/pdf/2603.27187)). These findings provide broader evidence supporting our motivation that LALMs may not fully exploit their audio inputs.
We also appreciate the reviewer's concern that attention visualisation alone may not provide sufficiently convincing evidence of improved audio utilisation. We agree that attention values are only one possible indication of how a model processes different modalities. To provide complementary behavioural evidence, we conducted an additional intervention experiment: we replaced the original audio with white noise (scrambled) while keeping the question unchanged, and evaluated the model's performance under this condition. Interestingly, the FAME-enhanced models exhibit a greater performance degradation when the original audio is replaced with white noise, suggesting that their predictions have become more sensitive to the audio input. This provides additional behavioural evidence that FAME encourages the model to make greater use of audio information:
| Backbone | Native real | Native scrambled | Native drop | FAME real | FAME scrambled | FAME drop | FAME − native, scrambled |
|---|---|---|---|---|---|---|---|
| Audio-Flamingo-3 | 59.14 | 48.92 | 10.22 | 59.94 | 46.46 | 13.48 | −2.46 |
| Qwen2.5-Omni-3B | 57.04 | 43.68 | 13.36 | 57.78 | 41.01 | 16.77 | −2.67 |
| Gemma-3n-E2B | 46.43 | 35.79 | 10.64 | 50.12 | 36.49 | 13.63 | +0.70 |
| Voxtral-Mini-3B | 45.17 | 36.09 | 9.08 | 50.24 | 35.16 | 15.08 | −0.93 |
| Qwen2-Audio-7B | 43.29 | 33.93 | 9.36 | 49.43 | 34.51 | 14.92 | +0.58 |
| Ultravox v0.5 | 41.49 | 36.48 | 5.01 | 45.24 | 36.27 | 8.97 | −0.21 |
| GLM-4-Voice-9B | 41.07 | 38.79 | 2.28 | 43.86 | 36.39 | 7.47 | −2.40 |
Response to Reviewer UkQj (3)
We acknowledge that establishing a direct causal relationship between internal attention patterns and modality utilisation is a challenging problem in multimodal explainability research. Although we find this direction particularly interesting and worth further investigation, a comprehensive analysis of the underlying mechanisms would go beyond the scope of the current paper and could constitute a valuable direction for future work. However if the reviewer has any suggestions, we are happy to perform it.
In conclusion, we would like to thank the reviewer again for raising these concerns and giving us the opportunity to clarify our work. We believe that most of the concerns stem from potential misunderstandings, rather than fundamental issues with the research. We kindly encourage the reviewer to reconsider these points in light of our responses, and kindly ask the reviewer to reconsider the overall score in light of these clarifications.
Thank you for the detailed response, particularly the pure-LoRA baseline and white-noise intervention. These results address most of my concerns regarding the source of the gains and actual audio utilization. I have therefore raised my score to 3.
Response to Reviewer UkQj
Thanks a lot for the reviewer to raise concerns.
We respond to the weakness point and questions one by one.
Weakness 1:Selection validation for MoE architecture: We have already conducted extensive investigations into different expert architectures and routing mechanisms (e.g., MoWE, which uses audio representations for routing) in Table 3, different skill configurations in Table 4, and the factors that enable successful audio expert training in Table 5.Regarding feature fusion strategies, we have not conducted a separate comparison because existing LALMs generally adopt a similar approach of concatenating audio and text representations. We would therefore appreciate it if the reviewer could clarify which alternative feature fusion strategies they would like us to investigate, targeting LALMs adoptation.
We have dedicated substantial discussion to these design choices in Lines 380–455 (Pages 7–9), where we examine the architectural decisions and their implications in detail. Nevertheless, we remain open to further investigation and would be happy to conduct additional experiments if the reviewer has specific suggestions regarding the expected comparisons.
We also appreciate the suggestion of including an additional figure to illustrate the module-wise design. We have already provided a module-wise description in Section 3.2, using bold subsection paragraphs to distinguish the individual components. However, given the current page limitations, we will consider adding a more comprehensive architectural illustration in the camera-ready version.
Weakness 2:Motivation issue: We would like to clarify that the motivation and the corresponding methodological design are explicitly developed throughout the Introduction. We now elaborate it step by step:First, in Line 41, we point out that existing audio LLMs are predominantly trained for tasks such as ASR, rather than tasks requiring deeper audio reasoning. In Line 44, we further highlight that these models do not sufficiently attend to audio perceptual information, while Line 48 discusses the substantial training data requirements of existing LALM training paradigms. These observations motivate us to develop a framework that encourages LALMs to better attend to and exploit fundamental audio perceptual information, which is the motivation behind our low-level MoE design.
Second, in Line 55, we highlight that audio evidence is not always fully exploited and that the alignment between audio evidence and the model's reasoning process remains insufficient. This motivates our focus-aware architecture and the corresponding mid-level training strategy, which aim to encourage the model to identify and utilise relevant audio evidence more effectively.
Finally, in Line 66, we introduce MoE as a potential architectural solution to address the aforementioned limitations. Rather than adopting a conventional MoE architecture directly, we further refine it with our focus-aware mechanism to better align expert specialisation with relevant audio skills and perceptual information.
Therefore, the proposed components are not arbitrary architectural choices; each is motivated by a specific limitation identified in the Introduction and addresses a different aspect of the overall problem. We would appreciate it if the reviewer could point out which specific part of this motivation they consider insufficiently justified or inappropriate. We would be more than happy to provide further clarification.