acceptodds
Under review as a conference paper at ICLR 2027

Escape the Language Prior: Mitigating Late-Stage Modality Collapse in Audio Reasoning via Modality-Aware Policy Optimization

Abstract

Audio and omni-modal large language models exhibit impressive cross-modal reasoning capabilities. However, standard reinforcement learning methods such as GRPO do not explicitly weight individual tokens by their dependence on the non-text source modality. This exacerbates late-stage modality collapse during extended chain-of-thought generation, where models progressively abandon the primary source signal in favor of compressed textual priors, leading to confident but ungrounded hallucinations. To address this, we introduce Modality-Aware Policy Optimization (MAPO), a novel dual-branch reinforcement learning framework. First, MAPO dynamically concentrates the policy gradient on modality-critical tokens using a modality relevance mask, which is derived from the cross-modal differential entropy between an audio-ablated reference and the multimodal policy. Second, it integrates an auxiliary attention loss branch that applies a targeted, temporally scaled penalty to the model's internal attention distributions. This encourages sustained attention to the source modality throughout the reasoning trace. Evaluations on complex audio reasoning benchmarks demonstrate that MAPO substantially improves long-horizon reasoning fidelity and multimodal instruction following, achieving highly competitive performance on several key benchmarks among open-weights models. By relying strictly on native statistical signals rather than domain-specific inductive biases, MAPO offers a promising foundation for mitigating epistemic collapse across diverse multimodal systems.

Primary area:
foundation or frontier models, including LLMs
Keywords:
audio reasoning, multimodal large language models, reinforcement learning, chain-of-thought reasoning, multimodal grounding, policy optimization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_TM2sOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors focused on attention decrease of the input aduio in long-term aduio reaonsing. Specifically, they proposed MAPO, which assigns weights to different tokens in GRPO. Also, an additional loss was proposed to encourage models to focus on the input audio itself. I think the problem is practical, the method is reasonable, and the authors have provided more evaluations apart from accuracy performance.

Critical strengths
  • First of all, I think the problem is practical, and the proposed technique is reaonable. These strengths highlight the authors indeed have a deep understanding of this direction.

  • Second, the tehnique part is clear and easy-to-understand. The relationships between MAPO and GRPO is clear, and the authors also provide the experimental comparisons between MAPO and GRPO under similar settings.

  • Third, the evaluation part is sound as the authors provide an experiment about no/mask/wrong audios apart from accuracy performance. A bad case is also provided in the appendix, which help readers know the shortcomings of their proposed method.

Critical weaknesses

I am not an expert about audio, I mainly focus on textual language models. I raise some concerns from my own perspective, and I hope it will be benifical for the authors to improve the manuscript.

  • I am not fully sure whether the entropy difference during training only reflects the effect of audio. The text-only reference model is frozen, while the policy model keeps changing.

  • Tokens in the same response still share the same sequence-level advantage. For example, an audio-related token could be correct, but receive a negative update because the final answer is wrong.

  • There are quite a few missing results in the appendix. It would be better if the authors could explain these missing results, or fill in as many as possible.

  • Minor issues: some figures are difficult to read. For example, the labels and legends in Figures 2 and 8 are quite small, and the text above the plots in Figure 8 is shortened. Also, the authors have provided rich information in the appendix, but the main text still has some blank space, so I think maybe they can organize the manuscript better.

Questions for authors

Please refer to the Critical Weaknesses part.

5 comments

Thank you for the thoughtful review and for recognizing the importance of evaluating audio use alongside answer accuracy.

1. Effect of the frozen reference on the entropy signal and token weights

I am not fully sure whether the entropy difference during training only reflects the effect of audio. The text-only reference model is frozen, while the policy model keeps changing.

The entropy difference in training combines the effect of audio with a smaller shift introduced by training, and the token weights it produces stay close to those from a same-checkpoint reference. To separate the effect of audio from changes introduced by training, we measured next-token predictive entropy in three conditions: trained MAPO with audio, the same checkpoint without acoustic features, and the base model without acoustic features. The question, saved response prefix, and token positions were held fixed. Comparing MAPO with and without audio measures how audio changes the trained model's uncertainty. Comparing the two text-only predictions measures the entropy shift associated with training. An additional experiment gave the following mean absolute entropy differences:

ComparisonMean absolute entropy difference
MAPO w/ versus w/o audio0.445 nats
MAPO w/o audio versus base w/o audio0.244 nats
MAPO w/ audio versus base w/o audio0.622 nats

The text-only shift is therefore smaller than the within-MAPO audio/text effect, at about 55% of its magnitude. Because the reference entropy enters training only through the token weights of the two branches, we also examined how reference choice affects the resulting token weights in a separate replay of 100 MMAU questions and 44,269 saved response tokens, using the released MAPO weighting rule. At the final checkpoint, which is the furthest in training from the frozen reference, frozen-reference and same-checkpoint weights have Spearman rank correlations of 0.860 for the policy branch and 0.866 for the inverse-attention branch. Changing the reference redistributes 2.8% and 2.5% of normalized weight mass, respectively. This close agreement at the end of training indicates that the two choices of reference give similar training signals.

We use the frozen reference to reduce GPU memory overhead by reusing the model already maintained for GRPO's KL penalty, avoiding a separate entropy-reference model. A fixed reference also allows the text-only pass to be computed once per rollout batch and reused across the four optimizer updates (Appendix H.2), whereas a reference that follows the policy would need to be recomputed after every update. Given the smaller text-only shift and the strong agreement in normalized token weights, we regard this as a reasonable balance between preserving the weighting signal and limiting memory use and training time in our training setup.

2. Audio-critical tokens and correctness of reasoning steps

Tokens in the same response still share the same sequence-level advantage. For example, an audio-related token could be correct, but receive a negative update because the final answer is wrong.

The example is helpful because it distinguishes audio relevance from semantic correctness. A correct audio-related token is updated according to all the sampled completions in which it appears, not only the one with the wrong answer, because the advantage is computed relative to the eight completions sampled for the same question (Section 2). Within a completion that scores below the group average, the shared advantage does give every token a negative term, yet the same token receives a positive term wherever it appears in a completion that scores above the average, and when all eight completions receive the same reward the advantage is zero for each of them. The net effect therefore depends on whether the token appears more often in the higher-scoring or the lower-scoring completions, which over training favors tokens that contribute to higher rewards. Audio relevance enters through MAPO's weights, which are positive and change only the magnitude of each token's term, not its direction, so a larger weight on an audio-critical token does not by itself lower its probability, and it strengthens the reinforcement of a token that appears mostly in the higher-scoring completions.

Feedback on an individual token within a single response, as opposed to across the group, would require a finer-grained semantic assessment of the generated CoT, such as step-wise process rewards that evaluate individual reasoning steps. This form of supervision is complementary to MAPO's focus on acoustic grounding. Section 2 discusses MAPO's compatibility with sequence-level reward strategies, and in the same way step-wise CoT rewards could further refine which intermediate steps should be reinforced, while MAPO's weights determine their relative emphasis according to audio relevance. The two approaches can therefore guide the same policy optimization through complementary roles: reasoning supervision supplies finer-grained feedback on correctness, and MAPO directs learning toward audio-critical positions. We will add this discussion to the revision.

3. Completing the appendix baseline comparisons

There are quite a few missing results in the appendix. It would be better if the authors could explain these missing results, or fill in as many as possible.

To fill in as many of the missing results as possible at this stage, we evaluated the public Covo-Audio-Chat and Step-Audio-R1 checkpoints on MMAR and MMAU-Pro, using their released model weights and audio encoders. The additional results are:

ModelMMAR accuracy (%)MMAU-Pro score (%)
Covo-Audio-Chat50.8047.40
Step-Audio-R170.2057.27

We also located additional published results in the MiMo-Audio paper (Table 8). These will be included as cited scores from that evaluation:

ModelPublished MMAR accuracy (%)Published MMAU-Pro score (%)
Audio Flamingo 358.5051.70
MiMo-Audio-7B-Instruct63.6053.35

We will also incorporate our new evaluations and their category breakdowns, together with these additional cited scores, into the appendix, with clear attribution of each source. These additions broaden the coverage of public audio models and make the comparison easier to assess, and the dashes that remain are mostly for human baselines and proprietary systems.

4. Figure readability and manuscript organization

Minor issues: some figures are difficult to read. For example, the labels and legends in Figures 2 and 8 are quite small, and the text above the plots in Figure 8 is shortened. Also, the authors have provided rich information in the appendix, but the main text still has some blank space, so I think maybe they can organize the manuscript better.

Thank you for the helpful suggestions on presentation. We will do our best to incorporate them in the revision by improving the readability of the labels and legends in Figures 2 and 8, restoring the full text above the plots in Figure 8, and using the available main-text space to present key appendix findings more prominently.

Reviewer_kaLYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a modification of GRPO named Modality-Aware Policy Optimization which is intended to sustain acoustic grounding during long reasoning traces. MAPO has 2 branches where its first branch reweights token-level policy losses using the absolute predictive-entropy difference between the live audio-conditioned policy and an audio-ablated reference and its second branch penalizes low audio attention at selected content-bearing tokens, with inverse-relevance weights, increasing temporal weights, and an advantage/failure-dependent prefactor.

Critical strengths
  • Table 1 demonstrates consistent improvements over GRPO across both LoRA and full fine-tuning setups, showing minimal variance across three random seeds. Across the four evaluation benchmarks, full fine-tuning yields a 2.30 percentage point boost.

  • POS gating is well-justified and the ablation in Table 10 provides clear evidence that part-of-speech gating is specifically contributes to the downstream performance gains.

  • The extra computational overhead is modest and well-documented, adding only about 11.3% to training time over the measured window. Importantly, all auxiliary modules are discarded during inference, meaning the method introduces no extra runtime latency or memory footprint at test time.

  • The paper does a rigorous grounding checks beyond raw attention heatmaps and rather than relying solely on surface-level attention maps to argue for better acoustic grounding, it includes thorough controls: accounting for expanding text key length, evaluating silence, mismatched audio, and text-only baselines, testing continuations conditioned on identical prefix tokens, and selectively masking high-, low-, and randomly attended acoustic segments.

  • The authors openly acknowledge the limitations of their interventions, noting that prefix activations are recomputed once audio is removed and clarifying that these tests do not prove every intermediate reasoning step is strictly faithful. This makes the validation much more credible.

Critical weaknesses
  • While Section 3.3 runs a diagnostic comparing the same model with and without audio, the actual training phase compares the updating multimodal policy against a frozen text-only reference model. As a result, the calculated entropy difference mixes genuine audio grounding with policy drift and calibration shifts during RL.

  • Predictive entropy alone is too coarse to identify audio-dependent tokens. Because entropy is a single scalar summary, two completely different token distributions can have the exact same entropy value. A large gap in entropy does not necessarily mean a token truly depends on acoustic evidence. The metric works fine as a practical heuristic, but treating it as proof of true modality dependence requires stronger validation, such as a same-checkpoint audio ablation during training or a distribution divergence check.

  • Key ablation variants are missing from the evaluation. Table 1 tests GRPO, mask-only, and mask-plus-attention variants under LoRA, but leaves out an attention-only baseline. Furthermore, the full fine-tuning experiments do not break down these intermediate branches. This makes it impossible to tell whether the gains come from the differential entropy mechanism or simply from simpler POS-gated attention regularization.

  • Baseline comparisons confound multiple training variables. Completion lengths vary across setups, and the primary MAPO and GRPO experiments resume from their own distinct AVQA-trained checkpoints. These differences make it difficult to attribute final performance to the core training method rather than multi-stage compounding effects.

  • Improvements do not generalize beyond reasoning models. The reported performance gains are limited strictly to the CoT policy. Appendix I shows that applying the method to the Instruct policy actually leads to a slight performance drop on MMAU. While this negative result is useful to see, it contradicts the claim that the framework serves as a general-purpose optimization recipe across different model types, and I would suggest to tone down the claim.

Questions for authors
  • How do the masks and held-out results change when both entropy terms are evaluated with the same evolving checkpoint? Can you quantify how much of the fixed-reference signal is associated with policy drift rather than audio conditioning?

  • What are the results of attention-only training and of a simple uniform/POS-gated attention penalty under the same update budget? Does differential-entropy reweighting still add value under full fine-tuning?

  • If MAPO and GRPO start the 29k-example stage from the identical AVQA checkpoint and use matched generation budgets, do the gains remain? Please report response lengths alongside evaluation accuracy to clarify the accuracy/computation trade-off.

6 comments

Thank you for the careful review and for recognizing the value of the behavioral grounding controls. Several of the questions below call for additional training runs of the 30B model at the scale described in Section 4, which were beyond the compute available to us during the rebuttal period. We therefore address these questions with new analyses of the existing checkpoints, a smaller-scale training study on a second model family, and the same-setting comparisons already in the paper.

1. Effect of the frozen reference on the entropy signal and token weights

How do the masks and held-out results change when both entropy terms are evaluated with the same evolving checkpoint? Can you quantify how much of the fixed-reference signal is associated with policy drift rather than audio conditioning?

To separate the effect of audio from changes introduced by training, we measured next-token predictive entropy in three conditions: trained MAPO with audio, the same checkpoint without acoustic features, and the base model without acoustic features. The question, saved response prefix, and token positions were held fixed. Comparing MAPO with and without audio measures how audio changes the trained model's uncertainty. Comparing the two text-only predictions measures the entropy shift associated with training. An additional experiment gave the following mean absolute entropy differences:

ComparisonMean absolute entropy difference
MAPO w/ versus w/o audio0.445 nats
MAPO w/o audio versus base w/o audio0.244 nats
MAPO w/ audio versus base w/o audio0.622 nats

The text-only shift is therefore smaller than the within-MAPO audio/text effect, at about 55% of its magnitude. Because the reference entropy enters training only through the token weights of the two branches, we also examined how reference choice affects the resulting token weights in a separate replay of 100 MMAU questions and 44,269 saved response tokens, using the released MAPO weighting rule. At the final checkpoint, which is the furthest in training from the frozen reference, frozen-reference and same-checkpoint weights have Spearman rank correlations of 0.860 for the policy branch and 0.866 for the inverse-attention branch. Changing the reference redistributes 2.8% and 2.5% of normalized weight mass, respectively. This close agreement at the end of training indicates that the two choices of reference give similar training signals.

We use the frozen reference to reduce GPU memory overhead by reusing the model already maintained for GRPO's KL penalty, avoiding a separate entropy-reference model. A fixed reference also allows the text-only pass to be computed once per rollout batch and reused across the four optimizer updates (Appendix H.2), whereas a reference that follows the policy would need to be recomputed after every update. Given the smaller text-only shift and the strong agreement in normalized token weights, we regard this as a reasonable balance between preserving the weighting signal and limiting memory use and training time in our training setup.

2. Validating the entropy gap as an audio-dependence signal

Predictive entropy alone is too coarse to identify audio-dependent tokens. Because entropy is a single scalar summary, two completely different token distributions can have the exact same entropy value. A large gap in entropy does not necessarily mean a token truly depends on acoustic evidence. The metric works fine as a practical heuristic, but treating it as proof of true modality dependence requires stronger validation, such as a same-checkpoint audio ablation during training or a distribution divergence check.

Our same-checkpoint comparison with exact KL shows that the absolute entropy gap provides a useful signal for prioritizing tokens whose predictions respond to audio. At a fixed checkpoint and with an identical text prefix, a nonzero entropy gap between audio-present and audio-removed predictions necessarily implies different token distributions and therefore positive KL divergence. Audio can make predictions either more concentrated or more uncertain. The absolute entropy gap captures both effects. KL also detects changes that preserve entropy, so their empirical agreement helps establish how well entropy tracks distribution changes during reasoning.

We compared the absolute entropy gap with exact KL(audio-present || audio-removed) using the base Qwen3-Omni-Thinking model on 100 MMAU questions, covering 45,702 token positions. Both conditions used identical saved response prefixes. The agreement holds across sound, music, and speech:

Question subsetMean within-response Spearman correlation [95% CI]Top-20% token overlap
All0.865 [0.857, 0.873]63.5%
Sound0.841 [0.828, 0.854]59.6%
Music0.854 [0.839, 0.868]60.8%
Speech0.901 [0.888, 0.914]70.1%

Results are averaged over questions, with 95% question-bootstrap confidence intervals. The overall correlation remains 0.846 after removing broad position trends within each response. The 63.5% top-token overlap is well above the 20.1% expected under independent selections, showing that the two signals prioritize substantially overlapping tokens. Because MAPO assigns a soft weight to every token rather than selecting a top set, the rank correlation is the statistic most directly related to how the signal is used. The agreement in both token rankings and high-score selections supports the entropy gap as a practical reweighting signal for positions affected by audio. The behavioral controls in Appendix E complement this signal-level validation by examining how changes to the audio affect final answers. We agree with the reviewer that the entropy gap is best described as a practical signal of audio dependence, and we will describe it this way in the revision and include this comparison in the appendix.

3. Contributions of entropy reweighting and the attention branch

What are the results of attention-only training and of a simple uniform/POS-gated attention penalty under the same update budget? Does differential-entropy reweighting still add value under full fine-tuning?

The attention branch was designed on top of the mask-reweighting branch, motivated by our observation that the entropy gap correlates with the trend of audio attention decay during late-stage reasoning (Section 3.3). When the model stops using audio, the gap can become small and leave the mask with little signal, so the attention loss targets substantive tokens with small absolute entropy gaps and places greater weight on later CoT positions. Its intended effect is to restore audio use at these positions and increase their entropy gaps, allowing the mask to give them greater relative weight in subsequent policy updates. Thus, at a conceptual level, attention-only training without the mask is not a configuration the method is designed for, since the policy update would then weight all tokens uniformly and could not give greater weight to the positions where audio use is restored.

Similarly, a uniform attention penalty would spread optimization weight across all response tokens, including function words. The POS-gate ablation in Appendix G tests this directly under full fine-tuning: removing the gate lowers the four-benchmark average from 73.25% to 72.39%, with a lower mean on every benchmark. To test the attention-only case empirically, we ran a small-scale experiment on MOSS-Audio-8B-Thinking, a smaller audio reasoning model from a different model family, training each variant for 128 steps with LoRA rank 16, alpha 32 under the same rollout and update budget. The attention-only variant adds the POS-gated attention penalty from the reviewer's question to GRPO, using uniform token weights in both the policy and attention losses so that entropy-based weighting is removed from both branches. The benchmark accuracies are as follows, with MMSU evaluated on the full 5,000-example set:

BenchmarkGRPOAttention-onlyFull MAPO
MMAU74.70%75.20%76.90%
MMSU73.64%73.72%74.06%
MMAR64.00%64.20%65.40%
MMAU-Pro56.12%56.25%57.48%

Attention-only training reaches a four-benchmark average of 67.34%, slightly above GRPO at 67.12%, while full MAPO reaches 68.46% and is above attention-only on all four benchmarks, by 0.34 to 1.70 points.

Together with the LoRA ablation in Table 1, where the modality relevance mask alone raises the four-benchmark average from 69.48% for GRPO to 70.46% and the full method reaches 71.39%, the MOSS study supports the complementary roles of the two components. Entropy-based weighting improves on GRPO without the attention loss, attention regularization alone gives a smaller gain on MOSS, and the combination gives the best four-benchmark average on both model families. Under full fine-tuning, the available comparison is for the complete method, whose advantage over GRPO is maintained when all parameters are updated, with the four-benchmark average improving from 70.95% to 73.25% (Table 1).

Reviewer_zXPmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper title: Escape the language prior: Mitigating late-stage modality collapse in audio reasoning via modality-aware policy optimization.

The paper proposes an optimization over GRPO for further attention on collapsed modalities (i.e., audio sequence), in extended chain-of-thoughts generation. The authors proposed a reinforcement learning method by reweighting the policy gradient with the entropy difference of text-only reference against multimodal policy and matched it with an attention loss to detect and penalize low audio attention tokens.

The authors set up an interesting evaluation to identify and track the audio attention across token positions in the chain-of-thoughts generation at inference time, allowing clear hypothesis framing that audio modality tends to collapse in long sequence reasoning. Once the gap is evaluated, the authors introduce a cross-modal differential entropy solution to track which tokens rely on the audio perception against text-only tokens and motivate stronger focus on larger differences through a modality relevance mask. To further improve policy gradient awareness from GRPO, they add an auxiliary loss with attention to directly penalize audio modality collapse.

The problem is well defined: to strengthen long-reasoning generation, they propose a clear method, but further explained results could help in assessing whether increasing attention on audio helps assert and generalize the model's capacity to reason.

Critical strengths
  • The problem is well motivated and clear; the method is well-detailed and follows a strong chain of thoughts.

  • In Table 1, MAPO seems to improve consistently over the baselines. Table 1 supports the relevance, with overall improvements from the modality relevance mask and the attention loss across both modality-centric and capability-centric benchmarks. Further, with LoRA fine-tuning, the method still slightly outperforms GRPO with low standard deviation across modality-centric and capability-centric benchmarks.

Critical weaknesses
  • Section 3.2 seems truly important to support the claim of late-stage audio modality collapse. It would help to have further evaluation and clearer examples. In Figure 2, the question "How to make this song less funny?" should be highlighted, and maybe a less ambiguous example would help a clear understanding.

  • For Figures 4 and 5, many reported methods seem to have close accuracy scores (76-78), resulting in difficulty assessing a true rank without variance information. The proposed method improves on the Qwen3-Omni-Thinking baseline, but Audio-DeepThinker (He et al., 2026) seems to perform better, with higher accuracy on the modality-centric benchmarks MMAU (78.50/77.83) and MMAR (74.00/70.83). Although MAPO seems stronger on capability-centric benchmarks, with greater results on MMSU (77.26/79.35), clear assessment is difficult as one result for a competing method is noted as unavailable. Further explanation or comparison between methods on other tasks could be relevant. Competing scores could be further developed to see the limitations and strengths of the method.

Questions for authors
  • For Section 3.2, it would help to have a less ambiguous example of question asked (i.e., in Figure 2, the question "How to make this song less funny?") or further statistics over multiple tests to further back up the late-stage audio modality collapse.
  • What could be the reasons for the MMAR gap in results between MAPO and Audio-Deep Thinker?
  • For Figures 4 and 5, many reported methods seem to have close accuracy scores (76-78), resulting in difficulty assessing a true rank without variance information. Were the base model results re-evaluated under the same MAPO protocol?
  • In Table 1, it would be interesting to see the MAPO mask-only ablation results for full fine-tuning as well, is it possible?
  • Are the default weights of the reward formulation ablated/explained? Defending or citing the choice of default values λ_acc=2, λ_format=1, and λ_cons=1 would strengthen the method design choices.
  • Further results or development moved to the main paper would help a clear assessment of the strength of the method (e.g., showing method robustness to language-prior hallucinations; to support the third contribution statement. Reducing the space allocated to Figures 4 and 5, or placing Algorithm 1 in the appendix, would help gain space).
  • Mentioning in the main paper the computational overhead that MAPO adds to GRPO would be interesting.

Small corrections:

  • The text in Figure 2 should be the same size as the main text for readability.
  • Figure 2 would gain from some clarifications or a clearer visual distinction between mean and max. The pink entropy is not labelled: does it denote negative values? The "reasoning end" in purple in the labelled box is not shown in the graph. The (a) and (b) notation should be added within the title of each subgraph.
  • Figure 3 is neither mentioned nor explained in the text.
  • Optional: An appendix summary would help to map ablated results.
  • Optional: the ablation results may come after the main results or, as they directly support the method design choices, the section could be renamed.

Rebuttal: After a strong rebuttal made by the authors, the paper has gained substantial clarity on the results; which was my main concern. I know decide to raise my score from 2 (weak reject) to 3 (weak accept).

8 comments

Thank you for the thoughtful review and for recognizing the consistency of MAPO's improvements over GRPO. We appreciate your emphasis on connecting those improvements to the model's use of audio during reasoning.

1. Evidence for late-stage audio modality collapse and clarity of Figure 2

For Section 3.2, it would help to have a less ambiguous example of question asked (i.e., in Figure 2, the question "How to make this song less funny?") or further statistics over multiple tests to further back up the late-stage audio modality collapse.

The qualitative examples and controlled analyses in the appendix may help clarify both the attention pattern and its behavioral implications. Appendix C.4 provides a direct illustration using the question, "Identify the emotion of each speaker." Base Thinking and mask-only MAPO show declining audio attention later in their responses and both incorrectly conclude that both speakers express sadness. Full MAPO retains more audio attention and correctly distinguishes the speakers' emotions. This example makes the task and the differences between the models easier to follow. We will revise Figure 2 with a clearer example and make the full question and model predictions more prominent.

The aggregate analysis in Appendix E.1 further supports the pattern across examples. It covers 200 MMAU prompts and aligns each response at the point where the model moves from describing the audio to reasoning about the question. After accounting for the growing number of text keys, Base Thinking's per-key audio preference still declines by 0.313 across this transition (166 responses), while MAPO's is preserved (+0.033, 157 responses). On the 143 prompts aligned for both models, the paired MAPO-minus-Base contrast is +0.338, with a 95% confidence interval of [0.260, 0.420], and the max-head reduction supports the same ordering. The audio also remains necessary after this point. In Appendix E.3, on 200 audio-essential MMAR examples, removing the audio after Base Thinking has already generated a quarter, a third, or half of its response lowers accuracy by 17.0, 14.0, and 6.0 points, so the generated text does not replace the source audio. Finally, the paired interventions in Appendix E.2 show that MAPO relies on the audio more than GRPO does: on 200 audio-essential MMAU examples, removing audio reduces MAPO's accuracy from 67% to 12%, compared with 60% to 20% for GRPO. We propose bringing these key findings into Section 3.2 and explicitly pointing readers to the detailed appendix analyses. Together, these results show that the decline in audio attention holds across examples, that continued access to the audio matters for the final answer, and that MAPO strengthens both attention to the audio and reliance on it.

2. MMAR comparison with Audio-DeepThinker

What could be the reasons for the MMAR gap in results between MAPO and Audio-Deep Thinker?

Differences in the starting policy and reasoning supervision offer two plausible explanations for the MMAR gap. Audio-DeepThinker starts from Qwen3-Omni-Instruct, while MAPO starts from Qwen3-Omni-Thinking. In their own two-stage ablation, Audio-DeepThinker reaches 74.0% on MMAR from Instruct and 70.0% from Thinking, showing that the starting policy substantially affects performance under their recipe. MAPO starts from the Thinking model because it is designed to operate on an existing reasoning trace (Appendix I), and from this starting point it reaches 70.83%, compared with their 70.0%. Importantly, Audio-DeepThinker also rewards similarity to reference CoTs, providing guidance on the reasoning path and key steps. This additional supervision may benefit MMAR's challenging audio reasoning tasks. MAPO uses correctness, format, and reasoning-consistency rewards without reference-CoT supervision. These differences are documented in Audio-DeepThinker, Table 4 and Section 3.3.

Within our setup, MAPO improves MMAR accuracy from 66.90% for the base Thinking model and 68.43% for GRPO to 70.83%, supporting the benefit of modality-aware optimization for an existing reasoning policy. MAPO also achieves 79.35% on MMSU compared with 77.26% for Audio-DeepThinker, indicating complementary strengths across tasks. Together, these results support MAPO's contribution to improving audio reasoning through sustained use of acoustic evidence. In fact, as discussed in Section 2, MAPO "remains fully compatible with these sequence-level reward strategies." Audio-DeepThinker's reference-based rewards can guide the reasoning content alongside MAPO's token-level reweighting and attention objective.

3. Evaluation protocol and uncertainty in Figures 4 and 5

For Figures 4 and 5, many reported methods seem to have close accuracy scores (76-78), resulting in difficulty assessing a true rank without variance information. Were the base model results re-evaluated under the same MAPO protocol?

The Qwen3-Omni baselines were re-evaluated locally using the same benchmark splits, output normalization, and scoring pipeline as MAPO. Results for external systems, including Audio-DeepThinker and Step-Audio-R1, are taken from their published reports.

For MAPO and GRPO, we report the mean and sample standard deviation across three seeds (17, 42, and 1234). On MMAU, MAPO improves accuracy from 75.53 ± 0.55% for GRPO to 77.83 ± 0.25%, with an improvement in each seed. This consistency extends to all four benchmarks. We will make MAPO's uncertainty more visible in Figures 4 and 5 and clearly identify published point estimates in the captions.

Reviewer_EAwdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to augment GRPO in large audio LMM to keep long chain-of-thought reasoning more grounded in the audio signal rather than just based on text information. It introduces a policy that updates based on the entropy difference between audio-plus-text versus text-only. It then adds a new loss to encourage audio attention on content words or keywords.The authors tested on several speech reasoning benchmarks and claimed about a 3% absolute gain in accuracy.

Critical strengths

The idea is very simple and elegant, and it makes sense from a high level. The problem is valid: we do see that during long CoT, the audio LLM tends not to pay too much attention to the audio signal. Plus, the authors used three seeds and provided a reasonable statistical analysis.

Critical weaknesses

The main concern for this paper is whether the method and results are significant, because the average improvement is small. The major improvement reported is actually from instruction following, which is doubtful to be caused by paying more attention to the audio signal. This is especially apparent if you look at the MMAU-PRO benchmark dataset in instruction following: it is likely due to letting the model learn the pattern of the question. Also, one key reference was the VPPO paper in ICLR 2026. It seems KL divergence would also work. It seems the previous method should be modality agnostic. I’m not sure if the authors have a baseline using the old method.

Questions for authors
  1. Can you explain why VPPO not working? It should at least be a baseline right?
  2. If we remove instruction following subset in MMAU-Pro, what is the results?
6 comments

Thank you for your thoughtful comments. We were glad to see that you share our concern about models losing track of the audio during long reasoning traces, the problem that motivated MAPO.

1. Given the small average accuracy gain, how significant are the method and results?

We agree that the significance of the method should be established through its effect on both task performance and audio use. In fact, our results do show consistent gains across benchmarks and seeds and, more importantly, a shift toward sustained use of acoustic evidence during reasoning, which is the behavioral change we aim to encourage with MAPO. The consistency across seeds and both fine-tuning settings supports a repeatable benefit over GRPO, averaging 2.30 percentage points with full fine-tuning and 1.91 with LoRA (Table 1). On MMSU, the 2.87-point gain over GRPO supports improved spoken-language understanding. The category breakdown further shows gains over Base Thinking in perception (+2.17 points) and reasoning (+4.01), suggesting better recognition and use of acoustic information (Appendix D).

Although absolute accuracy gains depend on the backbone's perceptual and reasoning capabilities, as discussed in Appendix J, our controlled analyses help identify the intended change in audio use. MAPO better preserves audio attention than Base Thinking at the transition into reasoning, after accounting for the growing text context (Appendix E.1). On the 200-example audio-essential MMAU subset, MAPO has higher clean accuracy than GRPO (67% vs. 60%) and a larger accuracy drop without audio (55 vs. 40 points), indicating greater audio dependence (Appendix E.2). Masking highly attended audio regions also causes a larger accuracy drop than masking an equal duration at random, supporting the relevance of the attended evidence (Appendix E.4). Together, these results support our claim that MAPO strengthens acoustic grounding during reasoning and improves the model's performance across audio benchmarks.

2. Are the gains mainly from instruction following? What are the MMAU-Pro results without that subset?

We agree that the instruction-following subset should be examined separately. Part of the gain on this subset can be attributed to the format reward that GRPO and MAPO share, since GRPO also improves on it, while MAPO improves further (95.40 vs. 90.80) and its advantage over GRPO does not depend on this subset. The subset contains 87 of the 5,304 scored MMAU-Pro questions (1.6%), and the table below gives the category scores together with the overall score recomputed without it.

ModelInstruction following (87)Closed-ended (4,592)Open-ended (625)OverallOverall without instruction following
Base Thinking62.0759.6384.7762.6362.64
GRPO90.8060.1783.4863.4262.96
MAPO95.4062.0085.3265.2964.79

Each row is one evaluation run with full fine-tuning, and the overall means in Table 1 are 63.35 for GRPO and 64.98 for MAPO. Without the subset, MAPO remains 2.15 points above Base Thinking and 1.83 points above GRPO, compared with 2.66 and 1.87 with it, so instruction following accounts for less than 0.1 point of MAPO's gain over GRPO. MAPO's lead over GRPO is also nearly the same on closed-ended (+1.83) and open-ended (+1.84) questions, which together make up 98% of the benchmark. For GRPO, by comparison, instruction following accounts for 0.47 of its 0.79-point gain over the base model.

The other benchmarks support the same conclusion: with full fine-tuning, MAPO retains a 2.52-point average advantage over GRPO across MMAU, MMAR, and MMSU (Table 1).

3. Could KL divergence be used instead of the entropy-based signal?

We agree that KL divergence could also provide a useful reweighting signal. In fact, the two signals are strongly correlated in practice, supporting the entropy gap as a practical indicator of audio-conditioned distribution change. At a fixed checkpoint and with an identical text prefix, a nonzero entropy gap between audio-present and audio-removed predictions necessarily implies different token distributions and therefore positive KL divergence. Audio can make predictions either more concentrated or more uncertain. The absolute entropy gap captures both effects. KL also detects changes that preserve entropy, so their empirical agreement helps establish how well entropy tracks distribution changes during reasoning.

We compared the absolute entropy gap with exact KL(audio-present || audio-removed) using the base Qwen3-Omni-Thinking model on 100 MMAU questions, covering 45,702 token positions. Both conditions used identical saved response prefixes. The agreement holds across sound, music, and speech:

Question subsetMean within-response Spearman correlation [95% CI]Top-20% token overlap
All0.865 [0.857, 0.873]63.5%
Sound0.841 [0.828, 0.854]59.6%
Music0.854 [0.839, 0.868]60.8%
Speech0.901 [0.888, 0.914]70.1%

Results are averaged over questions, with 95% question-bootstrap confidence intervals. The overall correlation remains 0.846 after removing broad position trends within each response. The 63.5% top-token overlap is well above the 20.1% expected under independent selections, showing that the two signals prioritize substantially overlapping tokens. Together, these results support the entropy gap as a practical signal for identifying positions where audio changes the model's predictions, consistent with its role in MAPO's token reweighting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.