acceptodds
Under review as a conference paper at ICLR 2027

FAME: Multi-Level Focus-Aware Mixture-of-Experts with Auxiliary Correctness Supervision for Audio Reasoning

Abstract

Audio reasoning has been a challenging task for Large audio-language models (LALMs) due to modality imbalance where models disproportionately rely on textual cues and under-utilise audio information. This stems from typical LALMs' architectures which process audio through a single and entangling heterogeneous aspect of audio understanding into one shared representation via an audio encoder and then align it with a text language model to perform downstream tasks. This limits both audio feature encoding and the model's ability to selectively attend to the aspect of audio representation most relevant to the task. We introduce FAME, a hierarchical focus-level framework trained through a progressive three-stage curriculum. At the low level, five aspect expert heads followed by on audio encoder specialised on different aspect-labelled corpora, independently capture distinct facets of audio evidence, and a question-driven soft Mixture-of-Experts router is trained to route and blend their features. At the mid level, focus-augmentation names the relevant experts within the input question to sharpen this routing, reinforced by an auxiliary correctness objective that teaches the model to reconsider uncertain answers. At the high level, focus-augmentation further expands the model's attention to abstract, notion-level concepts beyond individual experts. Crucially, none of FAME's training data overlaps with the audio reasoning benchmarks used for evaluation, making our evaluation fully out-of-domain. Across MMAU-Pro, MuCho Music, Clotho-AQA, and ADQA, FAME improves accuracy by 2.81% over the frozen backbone, which relies on implicit, emergent expert specialisation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio reasoning, Audio large language models, Multimodality, Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UkQjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the underutilization of audio information in LALMs and relates it to limitations in audio representation and audio-text alignment during pretraining. To address this issue, the authors propose FAME, a hierarchical training framework that strengthens the audio pathway through a MoE design. It first trains five aspect-specific expert heads on separate datasets, then learns a question-driven soft router, and subsequently adapts the router, projector, and LLM through expert-level and notion-level focus augmentation together. Experiments across four benchmarks and multiple backbones demonstrate the effectiveness of FAME and provide support for the overall training strategy.

Critical strengths
  1. The three-level framework is intuitively well motivated by the observed audio utilization problem. Moving from aspect-specific representation learning to question-guided routing and then to higher-level focus augmentation forms a reasonably coherent progression, and the roles of the individual components are discussed through step-by-step ablations.

  2. FAME produces improvements across a diverse set of backbones rather than being tied to a single base model. In addition, the expert-level evaluation provides useful evidence that four of the five expert heads acquire their intended types of information.

Critical weaknesses
  1. Although the overall pipeline is introduced, several important architectural choices are not examined in sufficient depth. The MoE encoder seems to consist of a shared frozen audio encoder and five linear expert heads. It remains unclear why these particular choices are appropriate or how sensitive the results are to them. Comparisons among different expert architectures, routing mechanisms, or feature fusion strategies would make the technical design more convincing. The presentation also relies heavily on one dense overview figure, and a clearer module-wise description would help.

  2. The motivation in the Introduction moves among general audio reasoning limitations (although I do not find all of these arguments particularly convincing), audio encoding, and audio-text alignment. The paper then largely attributes audio under-utilization to the encoder structure and training objectives, but this causal explanation is not sufficiently established. Existing LALMs already perform explicit audio-text alignment during pretraining, and the current results do not isolate the main limitation clearly.

  3. More direct evidence is needed to show that FAME genuinely improves audio encoding and utilization. Since FAME introduces substantial additional supervision from multiple audio datasets, some end-to-end gains may come from exposure to this data rather than from the proposed architectural design itself. A data-matched conventional fine-tuning baseline would help separate these effects.

Questions for authors
  1. Could the authors provide a more detailed analysis of the MoE-based audio encoder architecture? Were alternative expert structures, routing strategies, or fusion mechanisms considered, and are there ablations targeting these architectural choices?
  2. Can the authors provide more direct experimental evidence that existing LALMs underutilize audio and that FAME specifically mitigates this problem, beyond downstream accuracy and aggregate attention weights (which prior studies have shown that they may correlate poorly with feature importance and can change substantially without corresponding changes in model predictions) ?

[1] NAACL 2019, Attention is not Explanation. [2] ACL 2020, Learning to Deceive with Attention-Based Explanations.

4 comments
AuthorsOpenReview ↗

Response to Reviewer UkQj

Thanks a lot for the reviewer to raise concerns.

We respond to the weakness point and questions one by one.

Weakness 1:

Selection validation for MoE architecture: We have already conducted extensive investigations into different expert architectures and routing mechanisms (e.g., MoWE, which uses audio representations for routing) in Table 3, different skill configurations in Table 4, and the factors that enable successful audio expert training in Table 5.

Regarding feature fusion strategies, we have not conducted a separate comparison because existing LALMs generally adopt a similar approach of concatenating audio and text representations. We would therefore appreciate it if the reviewer could clarify which alternative feature fusion strategies they would like us to investigate, targeting LALMs adoptation.

We have dedicated substantial discussion to these design choices in Lines 380–455 (Pages 7–9), where we examine the architectural decisions and their implications in detail. Nevertheless, we remain open to further investigation and would be happy to conduct additional experiments if the reviewer has specific suggestions regarding the expected comparisons.

We also appreciate the suggestion of including an additional figure to illustrate the module-wise design. We have already provided a module-wise description in Section 3.2, using bold subsection paragraphs to distinguish the individual components. However, given the current page limitations, we will consider adding a more comprehensive architectural illustration in the camera-ready version.

Weakness 2:

Motivation issue: We would like to clarify that the motivation and the corresponding methodological design are explicitly developed throughout the Introduction. We now elaborate it step by step:

First, in Line 41, we point out that existing audio LLMs are predominantly trained for tasks such as ASR, rather than tasks requiring deeper audio reasoning. In Line 44, we further highlight that these models do not sufficiently attend to audio perceptual information, while Line 48 discusses the substantial training data requirements of existing LALM training paradigms. These observations motivate us to develop a framework that encourages LALMs to better attend to and exploit fundamental audio perceptual information, which is the motivation behind our low-level MoE design.

Second, in Line 55, we highlight that audio evidence is not always fully exploited and that the alignment between audio evidence and the model's reasoning process remains insufficient. This motivates our focus-aware architecture and the corresponding mid-level training strategy, which aim to encourage the model to identify and utilise relevant audio evidence more effectively.

Finally, in Line 66, we introduce MoE as a potential architectural solution to address the aforementioned limitations. Rather than adopting a conventional MoE architecture directly, we further refine it with our focus-aware mechanism to better align expert specialisation with relevant audio skills and perceptual information.

Therefore, the proposed components are not arbitrary architectural choices; each is motivated by a specific limitation identified in the Introduction and addresses a different aspect of the overall problem. We would appreciate it if the reviewer could point out which specific part of this motivation they consider insufficiently justified or inappropriate. We would be more than happy to provide further clarification.

AuthorsOpenReview ↗

Response to Reviewer UkQj (2)

Weakness 3

LoRA baseline: We would like to further clarify that our training and evaluation datasets are entirely distinct in terms of both their content and task formulation. Specifically, the training data consist exclusively of basic audio classification tasks, as illustrated by the examples in Table 7 (e.g., identifying musical instruments). In contrast, the evaluation datasets consist of reasoning-oriented tasks. Given this substantial difference in task distributions, we would not necessarily expect standard LoRA fine-tuning on the training data to yield substantial improvements on the reasoning benchmarks. Nevertheless, in response to the reviewer's request, we are reporting this additional experiment to provide a clearer and more comprehensive comparison:

BackboneArmMMAUMuChoClotho (bal.)ADQA
Audio-Flamingo-3Native59.1476.6668.5041.71
+FAME59.9474.5576.4544.95
Pure LoRA59.2577.0070.8442.71
Gemma-3n-E2BNative46.4363.0258.6535.93
+FAME50.1260.0760.0638.37
Pure LoRA47.6961.6858.9036.92
Voxtral-Mini-3BNative45.1757.7157.9840.49
+FAME50.2453.3361.4745.02
Pure LoRA46.6855.6052.0141.56
Qwen2-Audio-7BNative43.2952.4054.9636.39
+FAME49.4362.4360.7841.92
Pure LoRA44.5354.0356.3240.42
Qwen2.5-Omni-3BNative57.0470.9481.0243.11
+FAME57.7872.2882.8644.36
Pure LoRA55.3771.3481.0642.63
Ultravox v0.5Native41.4954.6851.5733.57
+FAME45.2454.7153.0035.41
Pure LoRA42.3553.3350.9533.61
GLM-4-Voice-9BNative41.0741.0350.2028.83
+FAME43.8644.8250.9537.19
Pure LoRA40.3643.0850.2431.13

It shows that the Pure LoRA results are quite similar to the Native version, and even worse, which is expected as this LoRA is finetuning the model to undertake simple classification tasks such as emotion recognition instead of audio reasoning task. It is also a proof that the gain is from our methodology instead of more exposure to more audios.

Furthermore, most of the MoE-based methods included in Table 3 do not report comparisons against standard LoRA fine-tuning. Our competitive, and in some cases superior, performance compared with these existing approaches provides further evidence of the effectiveness of our framework. The additional pure-LoRA experiment will allow us to establish this comparison more directly.

Questions to authors

Q1: We believe this question has been addressed in weakness 1. Please let us know if more concerns remain.

Q2:

More investigation on exploitation on audio modality:

Previous studies have reported modality bias and the underutilisation of non-textual modalities in multimodal LLMs, including visual and audio inputs (e.g., (https://arxiv.org/abs/2505.18657); (https://arxiv.org/pdf/2603.27187)). These findings provide broader evidence supporting our motivation that LALMs may not fully exploit their audio inputs.

We also appreciate the reviewer's concern that attention visualisation alone may not provide sufficiently convincing evidence of improved audio utilisation. We agree that attention values are only one possible indication of how a model processes different modalities. To provide complementary behavioural evidence, we conducted an additional intervention experiment: we replaced the original audio with white noise (scrambled) while keeping the question unchanged, and evaluated the model's performance under this condition. Interestingly, the FAME-enhanced models exhibit a greater performance degradation when the original audio is replaced with white noise, suggesting that their predictions have become more sensitive to the audio input. This provides additional behavioural evidence that FAME encourages the model to make greater use of audio information:

BackboneNative realNative scrambledNative dropFAME realFAME scrambledFAME dropFAME − native, scrambled
Audio-Flamingo-359.1448.9210.2259.9446.4613.48−2.46
Qwen2.5-Omni-3B57.0443.6813.3657.7841.0116.77−2.67
Gemma-3n-E2B46.4335.7910.6450.1236.4913.63+0.70
Voxtral-Mini-3B45.1736.099.0850.2435.1615.08−0.93
Qwen2-Audio-7B43.2933.939.3649.4334.5114.92+0.58
Ultravox v0.541.4936.485.0145.2436.278.97−0.21
GLM-4-Voice-9B41.0738.792.2843.8636.397.47−2.40
AuthorsOpenReview ↗

Response to Reviewer UkQj (3)

We acknowledge that establishing a direct causal relationship between internal attention patterns and modality utilisation is a challenging problem in multimodal explainability research. Although we find this direction particularly interesting and worth further investigation, a comprehensive analysis of the underlying mechanisms would go beyond the scope of the current paper and could constitute a valuable direction for future work. However if the reviewer has any suggestions, we are happy to perform it.

In conclusion, we would like to thank the reviewer again for raising these concerns and giving us the opportunity to clarify our work. We believe that most of the concerns stem from potential misunderstandings, rather than fundamental issues with the research. We kindly encourage the reviewer to reconsider these points in light of our responses, and kindly ask the reviewer to reconsider the overall score in light of these clarifications.

Thank you for the detailed response, particularly the pure-LoRA baseline and white-noise intervention. These results address most of my concerns regarding the source of the gains and actual audio utilization. I have therefore raised my score to 3.

Reviewer_jM4nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FAME has 5 audio expert heads for various tasks. It includes a question-driven MoE router to route each question to the relevant expert. The authors evaluate it on 7 audio language models and 4 benchmarks.

Critical strengths
  • The authors have evaluated several models and benchmarks, which is the main strength of the paper. They include several MoE and adaptation methods.
  • The authors describe that expert roles have explicit supervision and separate training.
  • The authors explain that training and eval use one RTXA6000.
Critical weaknesses
  • Standard Lora baseline is not included (trained on the same data without experts or focus prompts). The paper has + LLM-Lora steps and no standard Lora.
  • It's not clear how the model reconsiders internally when eval uses a single forward pass to select the highest probability ans token. (Sec F page 16, L863).
  • The improvements over other methods are small. eg., Tab 3. MMAUpro results are not consistent (49.13 vs 49.43)
  • The OOD claim needs more explanation, author need to explain sample overlap and concept overlap while establishing out of domain.
Questions for authors
  • Please explain how FAME would compare with standard Lora finetuning with no experts on the same data.
  • Please elaborate on the answer reconsiderations during evaluation.
  • As the training topics in MMAR and MusicBench, and all these topics overlap with music QA, can you please provide a clear definition of OOD?
2 comments
AuthorsOpenReview ↗

Response to Reviewer jM4n

Thanks a lot for the reviewer to raise concerns.

Please allow us to reiterate that the core framework of FAME is to leverage audio understanding tasks, such as instrument classification, as pre-training tasks. These tasks do not involve reasoning questions. The resulting model is then evaluated on reasoning tasks using widely adopted reasoning benchmarks with their provided questions and audio as the prompt. We do not intentionally modify any of these prompts to optimise their performance through prompt engineering. We would therefore like to emphasise that the pre-training and evaluation stages serve distinct purposes, and that our framework does not rely on reasoning questions during pre-training, which is fully OOD test. In other words, OOD test definition is: (1) no reasoning questions are used in training; (2) question content: no training and test questions overlap; (3) audio recordings: we removed training clips that also appear in the evaluation sets (explain below for MuCho). However for concept side, so some basic acoustic concepts are shared, and this is intended and our core motivation and training method.our hypothesis is that LALMs lack explicit, well-grounded perceptual knowledge of these low-level concepts, and that strengthening it in the audio pathway helps downstream reasoning.

We respond to your weakness point and questions one by one.

Weakness 1:

Standard LoRA baseline: We would like to further clarify that our training datasets and evaluation datasets are entirely distinct in terms of both their content and task formulation. Specifically, the training data consist exclusively of basic audio classification tasks, as illustrated by the examples in Table 7 (e.g., identifying musical instruments). In contrast, the evaluation sets consist of reasoning-oriented tasks. Given this substantial difference in task distributions, we would not necessarily expect standard LoRA fine-tuning on the training data to yield strong improvements on the reasoning benchmarks. Nevertheless, in response to the reviewer's request, we are showing this additional experiment below to provide a clearer and more comprehensive comparison.

BackboneArmMMAUMuChoClotho (bal.)ADQA
Audio-Flamingo-3Native59.1476.6668.5041.71
+FAME59.9474.5576.4544.95
Pure LoRA59.2577.0070.8442.71
Gemma-3n-E2BNative46.4363.0258.6535.93
+FAME50.1260.0760.0638.37
Pure LoRA47.6961.6858.9036.92
Voxtral-Mini-3BNative45.1757.7157.9840.49
+FAME50.2453.3361.4745.02
Pure LoRA46.6855.6052.0141.56
Qwen2-Audio-7BNative43.2952.4054.9636.39
+FAME49.4362.4360.7841.92
Pure LoRA44.5354.0356.3240.42
Qwen2.5-Omni-3BNative57.0470.9481.0243.11
+FAME57.7872.2882.8644.36
Pure LoRA55.3771.3481.0642.63
Ultravox v0.5Native41.4954.6851.5733.57
+FAME45.2454.7153.0035.41
Pure LoRA42.3553.3350.9533.61
GLM-4-Voice-9BNative41.0741.0350.2028.83
+FAME43.8644.8250.9537.19
Pure LoRA40.3643.0850.2431.13

It shows that the Pure LoRA results are quite similar to the Native version, and even worse, which is expected as this LoRA is finetuning the model to undertake simple classification tasks such as emotion recognition instead of audio reasoning task. It is also a proof that the gain is from our methodology instead of more exposure to more audios.

Furthermore, most of the MoE-based methods included in Table 3 do not report comparisons against standard LoRA fine-tuning. Given that our method achieves competitive or even better performance than these existing approaches, this also provides additional evidence of the effectiveness of our framework compared with standard LoRA fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer jM4n (2)

Weakness 2

Reconsider mechanism for inference: Exactly. We would like to clarify that this is precisely one of our contributions, rather than an inconsistency between the training and testing settings. We intentionally designed this mechanism to ensure a fair evaluation across all baselines. We cannot allow our model to reconsider its answers multiple times while other baselines are only given a single attempt. Instead, we incorporate reconsideration as part of the training process, enabling the model to develop the ability to reconsider its initial predictions during reasoning. Ideally, this should become an automatic behaviour once the model has learned the underlying strategy effectively, without requiring additional attempts or interventions at inference time. The component-wise improvements reported in Table 2 provide further evidence that the model benefits from this training strategy. We should not control the token count without deliberately interfering with the inference and evaluation process. Changing Chain of Thought (such as using self-consistency method (https://arxiv.org/abs/2203.11171)) may influence or impair the performance of the model reasoning (https://arxiv.org/abs/2408.02442).

Weakness 3

Improvement extent: The improvements are consistently above 3 percentage points, achieved across relatively large evaluation datasets. Noting that we are improving based on already well-performed LALMs and even some of them achieve SOTA score. Furthermore, reasoning is inherently challenging, and these gains are obtained on top of frontier LLMs that have already been pre-trained on massive amounts of data, making the improvements particularly meaningful.

Regarding the reported score, we apologise for the typo: the correct value is 49.43, rather than the previously stated figure. This was a typographical error caused by the adjacent positions of the 1 and 4 keys on the keyboard (lol).

Weakness 4

OOD explanation and content overlap: We would like to emphasise that the training question sets, including MusicBench, consist entirely of classification tasks, which are designed to teach the model fundamental, low-level audio knowledge, as described in Line 229 and Table 7. In contrast, the testing question sets, including MuChoMusic, consist of reasoning questions provided by the respective benchmarks. The training and testing sets therefore differ fundamentally in both task formulation and evaluation objectives. This is why we argue that there is no data leakage between training and testing.

Furthermore, if the concern is that the same audio recordings might appear in both MusicBench and MuChoMusic but be associated with different questions, we have also explicitly de-duplicated the audio recordings to eliminate such overlap. As further evidence, Table 7 reports only 12,904 MusicBench samples in our training set, whereas the official MusicBench dataset contains over 52,000 samples. This reflects our selective data curation and de-duplication process. We ensure that there is no overlap in either the questions or the audio recordings between the training and testing sets. We will make these distinctions and the de-duplication procedure much clearer in the revised manuscript.

Questions to authors

We believe we have answered the questions before. In conclusion, we would like to thank the reviewer again for raising these concerns and giving us the opportunity to clarify our work. We believe that most of the concerns stem from potential misunderstandings, rather than fundamental issues with the research. We kindly encourage the reviewer to reconsider these points in light of our responses, and kindly ask the reviewer to reconsider the overall score in light of these clarifications.

Reviewer_cPnkOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FAME is a framework for improving audio reasoning by explicitly organizing the audio representations and training around different audio understanding scenarios. Multiple experts capture different levels details from the audios like events, entities, structure, modifiers, and relations. A question-conditioned soft router combines their representations with the outputs of the original audio. Subsequent training introduces expert-specific focus instructions, an auxiliary answer-correctness prediction objective, and randomly assigned instructions drawn from 13 broader audio concepts.

Critical strengths
  • The framework was tested across multiple large audio language models across 4 benchmarks, outperforming baselines in 25 out of 28 settings. Consistent gains on models like Qwen2-Audio show that the general recipe has real practical value.

  • Breaking audio down into distinct aspects provides a clear, interpretable design. Specialised training clearly transfers to unseen data in some cases, most notably on UrbanSound8K, where the event expert hits 82.5% versus the baseline's 66.1%.

Critical weaknesses
  • The ablation adds components on top of each other without matching training steps, data exposure, or parameter counts. Without comparing against plain LoRA or basic prompt baselines, it’s unclear if the complex architecture is needed, especially since the lead over the best baseline is only ~0.76%.

  • The auxiliary task trains the model to predict correctness, but inference uses a single forward pass without an actual answer-revision step. Figure 2 also doesn't control for token count or test what happens when audio is shuffled, meaning the claims about deeper acoustic grounding aren't firmly backed up.

  • What kind of averaging was used for getting scores on the 620 questions? If an average or weighting was used, it needs to be clearly explained.

  • MusicBench (training) and MuChoMusic (testing) both pull from MusicCaps. Just using different dataset names doesn't guarantee the raw audio clips don't overlap, so a source-level deduplication check is missing, and would strengthen the paper if those details are provided.

  • The paper could benefit from adding more recent baselines which already report improved scores over the presented baseline.

  • One thing that stands out is that the gains are significantly higher on older models (Qwen2-Audio) and significantly lower on newer models. This raises a concern that the models trained on newer and more data which align better with the recent benchmarks already perform good and older models could benefit just from the new datasets. In my experience, both Ultravox v0.5 and Voxtral-mini (part of Mistral’s Voxtral family) are speech-centric models, rather than general-purpose audio understanding models, and were trained mostly for speech related tasks. Can the authors report breakdown of scores on MMAU and MMAU-Pro for these models?

Questions for authors
  • The paper tests on 3,336 samples out of MMAU-Pro’s 5,305 total instances. The exact filtering rules used to select this subset need to be documented. I am assuming that the authors filtered spatial and multi-audio subset, but filtering should be explicitly mentioned.

  • How does the method compare to a standard LoRA model trained for the exact same steps and parameter budget? Also, can you provide error bars/standard deviations for the Table 3 results?

  • Does the model actually revise its answers at test time, or does it just pick the best token in a single pass? Also, please clarify whether the audio encoder was fully frozen or partially updated as suggested in Appendix F.

  • See weakness section.

10 comments
AuthorsOpenReview ↗

Response to Reviewer cPnk

Thanks a lot for the reviewer to raise concerns.

Please allow us to reiterate that the core framework of FAME is to leverage audio understanding tasks, such as instrument classification, as pre-training tasks. These tasks do not involve reasoning questions. The resulting model is then evaluated on reasoning tasks using widely adopted reasoning benchmarks with their provided questions and audio as the prompt. We do not intentionally modify any of these prompts to optimise their performance through prompt engineering. We would therefore like to emphasise that the pre-training and evaluation stages serve distinct purposes, and that our framework does not rely on reasoning questions during pre-training, which is fully OOD test. We explain further we do not have question and audio leakage later on.

We respond to your weakness point and questions one by one.

Weakness 1:

Baseline comparison: We would like to clarify that the purpose of the ablation is to examine whether each individual component contributes additional gains over the plain baseline (see Table 2). The baseline reported in the first row of Table 2 is precisely the comparison against the basic prompt baseline.

Ablations: Since our objective is to evaluate the contribution of each component relative to the baseline, rather than to compare our model against itself, we do not see a clear rationale for controlling the data size each step, particularly given that the underlying pre-training data of foundational LLMs such as Qwen2.5-Audio is not fully accessible or controllable. Table 2 as a whole provides evidence that each of the proposed architectural components are necessary to achieve the observed improvements. We would also note that we do not consider the proposed architecture particularly complex, especially when compared with the pre-trained stage of architectures of large-scale models such as DeepSeek.

trival improvement: Additionally, could you please clarify where the figure of approximately 0.76% is reported? We cannot locate this figure in either the tables or the main text. In contrast, Table 1 clearly shows that our method achieves considerable improvements on three out of the four test datasets, with gains of over 3% compared against already strong pre-trained LLM baselines. The main point of Table 3 is to compare different MoE encoders in this Table 3 while Table 1 is mainly for comparison with SOTA models. We hope this clarify this point which could be a misunderstanding of our paper.

Weakness 2:

auxiliary training and inference settings: We would like to clarify that this is precisely one of our contributions, rather than an inconsistency between the training and testing settings. Our intention is to allow the model to reconsider its initial responses when it is uncertain or does not sufficiently trust its own predictions, while ensuring that the model is evaluated under a fair and consistent prompting setting. Therefore, we believe that our augmentation strategy is methodologically justified and correctly designed.

Regarding Figure 2, as previously mentioned, we should not control the token count without deliberately interfering with the inference and evaluation process. Changing Chain of Thought may influence or impair the performance of the model reasoning (https://arxiv.org/abs/2408.02442). The attention values presented in Figure 2 are intended to demonstrate that our training strategy mitigates the modality imbalance issue during audio reasoning.

Finally, we would like to emphasise that we have never claimed to shuffle the audio.

Weakness 3:

Score calculation on 620 questions: We swap the positions of the answer choices to balance the distribution of correct answers. For example, if the original options are A. 1 and B. 2, with A (i.e., 1) being the correct answer, we swap the two options to obtain A. 2 and B. 1, making B the new correct answer. Specifically, in the original Clotho annotations, 515 questions have A as the correct answer, while only 105 have B as the correct answer. By swapping the options for 205 questions originally answered by A, we obtain an equal distribution of 310 A and 310 B correct answers. This eliminates the answer-position imbalance in our evaluation set. We will clarify this procedure and revise the relevant description in Appendix F accordingly.

AuthorsOpenReview ↗

Response to Reviewer cPnk (2)

Weakness 4:

Content leakage and overlap concern: We would like to emphasise that the training question sets, including MusicBench, consist entirely of classification tasks, which are designed to teach the model fundamental, low-level audio knowledge, as described in Line 229 and Table 7. In contrast, the testing question sets, including MuChoMusic, consist of reasoning questions provided by the respective benchmarks. The training and testing sets therefore differ fundamentally in both task formulation and evaluation objectives. This is why we argue that there is no data leakage between training and testing.

Furthermore, if the concern is that the same audio recordings might appear in both MusicBench and MuChoMusic but be associated with different questions, we have also explicitly de-duplicated the audio recordings to eliminate such overlap. As further evidence, Table 7 reports only 12,904 MusicBench samples in our training set, whereas the official MusicBench dataset contains over 52,000 samples (official MusicBench dataset). This reflects our selective data curation and de-duplication process. We ensure that there is no overlap in either the questions or the audio recordings between the training and testing sets. We will make these distinctions and the de-duplication procedure much clearer in the revised manuscript.

Weakness 5:

More baselines: We thank the reviewer for the suggestion. Noting that the baselines are all from recent (in Table 1 we have many from 2025 and 2026). If you are referring to particular baseline, we are happy to add the results.

Weakness 6:

Performance only on out-dated LALMs: We would like to point out that the benefits of our method are not limited to older models such as Qwen2-Audio. As shown in Table 1, improvements are observed across multiple models, including Audio Flamingo 3, a strong and relatively recent audio LLM, which achieves gains of 7.95 points on Clotho AQA and 3.24 points on ADQA. Therefore, the claim that our method primarily benefits older models is not supported by the experimental results.

Breakdown value of specified LALMs: Thank you for raising the question regarding the breakdown of results on the MMAU-Pro dataset. We provide the detailed breakdown below. The results show that, for speech-related questions, which are of particular interest given the reviewer's concern that the model has already been extensively trained on speech understanding tasks, FAME still delivers moderate improvements. This further demonstrates the necessity and effectiveness of FAME, even when the backbone model has already received substantial speech-related training. More importantly, for the other sub-tasks, FAME achieves even more substantial gains, highlighting its broader benefits across different audio understanding capabilities.

BackboneMusicSoundSpeech
Audio-Flamingo-3−0.40+2.27+1.02
Qwen2.5-Omni-3B+0.40+3.02−1.03
Gemma-3n-E2B+0.80+10.47+0.47
Voxtral-Mini-3B+4.06+7.90+3.42
Qwen2-Audio-7B+8.84+6.15+1.82
Ultravox v0.5+0.77+10.53+1.36
GLM-4-Voice-9B+9.47−6.83+3.26

Note that we have never tried to test MMAU dataset, which could be a factual error of the suggestion.

Question for authors:

Q1:

filtering strategy for MMAU-Pro: We would like to assure you that we did not intentionally filter out any questions from the dataset. As stated in the paper, for simplicity, we selected multiple-choice questions for our evaluation, while the remaining questions are open-ended. The distinction between these two types of questions lies solely in how the generated answers are evaluated, rather than in the underlying audio understanding task. Specifically, open-ended questions require evaluating the generated responses, whereas multiple-choice questions allow for more straightforward and consistent evaluation. The core aim for our experiments is to test whether LALMs have ability to reason over audio questions, which could be tested via either types of the questions. We also note that each multiple-choice question has at least two answer options, as questions with fewer options would be overly simplistic. We will clarify these details in the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer cPnk (3)

Q2:

As stated previously, we use LoRA for model fine-tuning, although we are not entirely clear why this is directly relevant to the concern raised. Our training and reasoning evaluation sets are entirely distinct, and the resulting model could also be used as a pre-trained foundation model for subsequent applications. Nevertheless, to further address the reviewer's concerns, we also report the results obtained using pure LoRA fine-tuning:

BackboneArmMMAUMuChoClotho (bal.)ADQA
Audio-Flamingo-3Native59.1476.6668.5041.71
+FAME59.9474.5576.4544.95
Pure LoRA59.2577.0070.8442.71
Gemma-3n-E2BNative46.4363.0258.6535.93
+FAME50.1260.0760.0638.37
Pure LoRA47.6961.6858.9036.92
Voxtral-Mini-3BNative45.1757.7157.9840.49
+FAME50.2453.3361.4745.02
Pure LoRA46.6855.6052.0141.56
Qwen2-Audio-7BNative43.2952.4054.9636.39
+FAME49.4362.4360.7841.92
Pure LoRA44.5354.0356.3240.42
Qwen2.5-Omni-3BNative57.0470.9481.0243.11
+FAME57.7872.2882.8644.36
Pure LoRA55.3771.3481.0642.63
Ultravox v0.5Native41.4954.6851.5733.57
+FAME45.2454.7153.0035.41
Pure LoRA42.3553.3350.9533.61
GLM-4-Voice-9BNative41.0741.0350.2028.83
+FAME43.8644.8250.9537.19
Pure LoRA40.3643.0850.2431.13

It shows that the Pure LoRA results are quite similar to the Native version, and even worse, which is expected as this LoRA is finetuning the model to undertake simple classification tasks such as emotion recognition instead of audio reasoning task.

Regarding Table 3, its sole purpose is to investigate which MoE configuration is most suitable for our framework. Importantly, selecting the MoE configuration is only one stage of our overall framework; we further apply our subsequent training strategies after the MoE integration. Therefore, Table 3 should be interpreted as an architectural investigation rather than the final performance evaluation of FAME. Nevertheless, to further address the reviewer's concerns, we re-run the evaluation four times, and report the significant test results:

BenchmarkFAME mean ± SDGain ± SD95% CI on gainp
MMAU-Pro49.36 ± 0.21+6.07 ± 0.21[+5.73, +6.40]<.001
MuCho61.91 ± 0.48+9.51 ± 0.48[+8.74, +10.28]<.001
Clotho (balanced)60.84 ± 1.11+5.88 ± 1.11[+4.12, +7.64].0018
ADQA41.87 ± 0.38+5.48 ± 0.38[+4.88, +6.08]<.001
Mean of 453.49 ± 0.27+6.73 ± 0.27[+6.31, +7.16]<.001

The results demonstrate significant improvements over the baseline while achieving competitive performance compared with previous MoE architectures. Beyond accuracy, unlike less structured MoE alternatives, our design provides an explicit breakdown of the skill sets being utilised, offering greater interpretability and explainability.

Q3:

Inference settings: No. During inference, it pick the best token in a single pass. The audio tower is fully frozen in Line 195, and the trained part is expert head. We will make it more clear in Appendix F again.

In conclusion, we would like to thank the reviewer again for raising these concerns and giving us the opportunity to clarify our work. We believe that most of the concerns stem from potential misunderstandings, rather than fundamental issues with the research. We have therefore provided detailed clarifications and pointed to the specific lines, tables, and sections in the manuscript that address these concerns. We kindly encourage the reviewer to reconsider these points in light of our responses. In particular, we hope the reviewer will recognise that the issues raised primarily concern clarifications of the manuscript, rather than fundamental shortcomings in the motivation, experimental design, or methodology that would necessitate major revisions. In this regard, we respectfully believe that the concerns may not fully justify the current strong reject recommendation, and kindly ask the reviewer to reconsider the overall score in light of these clarifications.

Ultimately, our shared goal is to advance reliable and rigorous audio understanding research and to contribute to the broader effort against the proliferation of low-quality, AI-generated content. We sincerely hope our work can contribute to this important effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.