acceptodds
Under review as a conference paper at ICLR 2027

Rethinking, Revealing, and Reducing Real Modality Gap of Audio Language Models under Linguistic-Paralinguistic Trade-Off

Abstract

Large Audio Language Models (LALMs) perform worse on audio than on semantically equivalent text, which is known as the modality gap. Recent studies claim that the gap can be reduced without degrading the pre-trained paralinguistic capability. However, we argue that the current protocol, which measures the two sides on disjoint data respectively built for linguistic- and paralinguistic-centered tasks, ignores their real-world tension and thus cannot actually support the claim. In response, we propose ReMG, a new protocol that crosses two linguistic contents with two paralinguistic states into a matched 4-tuple, holding the remaining acoustic covariates fixed. Each audio clip then has a variant that changes along only one axis and is also queried on both axes. This leaves no shortcut for LALMs but to directly trade off linguistic against paralinguistic capability on the same audio, only under which we believe a real modality gap can be meaningfully claimed. The result under ReMG challenges the previous view: paralinguistic collapse is widespread, and the only released checkpoint shows no substantial gap reduction. ReMG helps address this situation by guiding LALM fine-tuning. We use simple on-policy distillation without stronger teachers or ground-truth labels, so as not to mix the protocol contribution with that from new knowledge learning, as existing methods do. Across nine LALMs, four baseline methods, and seven benchmarks, the proposed ReMG is demonstrated to effectively reveal and reduce the real modality gap under the linguistic-paralinguistic trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Modality Gap, Large Audio Language Models, Linguistic-Paralinguistic Trade-Off

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uwdQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether audio language models can improve spoken question answering while preserving emotion recognition. It proposes ReMG, which groups recordings with two different contents and two emotions, and tests both tasks on each recording. The authors build training and test sets using synthesized questions and human emotional speech, and fine-tune models through content and emotion distillation. Experiments evaluate existing models, compare fine-tuning methods, and test generalization to other benchmarks.

Critical strengths
  1. Evaluating content understanding and emotion recognition on the same recordings is a useful diagnostic design. It reveals task-interference failures that separate evaluations may overlook.

  2. The evaluation combines synthesized reasoning speech, human emotional recordings, and existing benchmarks, providing complementary evidence about model behavior.

Critical weaknesses
  1. The presentation is hard to follow. The term “ReMG protocol” covers evaluation, the construction of training and test sets, and the OPD-based method. The contributions of these components should be explained separately. Figures 1 and 2 are also too dense and difficult to understand.

  2. The role of quadruples (or the created dataset) in training is unclear. Equations (8)–(9) use per-example losses without explicitly using the relationships within each quadruple. It is unclear what quadruple pairing adds beyond training both tasks on each audio.

  3. The source of the gains is not fully established. Table 4 shows that most gains appear after adding privileged teacher instructions. A comparison with vanilla OPD using the same instructions is missing.

  4. The paired metrics need clarification. Equations (3)–(4) measure joint correctness, which should not exceed average accuracy. However, Table 2 reports 68.37% content invariability versus 52.27% content accuracy.

  5. A smaller gap partly comes from worse text performance. Table 2 implies that Phi-4’s text accuracy drops by 6.26 points, explaining more than half of its 10.80-point gap reduction. Text accuracy should be reported directly.

  6. The baseline comparisons have limits. Several important components of prior methods are removed or replaced. Results from these modified versions do not fully support the broad claims about the original methods.

Questions for authors
  1. How are the relationships within each quadruple used during training? What changes in the training procedure when adding the “ReMG protocol” to vanilla OPD?

  2. How does vanilla OPD perform with the same privileged teacher instructions? This comparison would help separate the effects of the protocol and teacher prompting.

  3. Could you clarify how the paired metrics are calculated? Under Equations (3)–(4), how can content invariability reach 68.37% when content accuracy is only 52.27% in Table 2?

  4. Could you report text accuracy before and after fine-tuning? How much of the gap reduction comes from improved audio accuracy, and how much comes from decreased text accuracy?

2 comments
AuthorsOpenReview ↗

Response to Reviewer uwdQ

Thank you for your time in reviewing our manuscript, and also for valuing our diagnostic design in revealing previously overlooked failures and providing complementary evidence. We hope the following response could address your concerns.


W2&Q1: Eq. 8&9 use per-example losses without explicitly using the relationships within each quadruple. How does it work beyond training both tasks on each audio? What changes in the training procedure when adding the “ReMG protocol” to vanilla OPD?

Thanks for the question. It works by injecting the quadruple relationships as data priors across the whole fine-tuning set (Lines 95-102), since each audio there is constructed under the ReMG protocol and uniquely comes from a single quadruple. DNNs are not so weak that only when we explicitly write something into a loss function can it learn that. Instead, the existence of such a relationship within the data distribution is enough. Similarly, adding the "ReMG protocol" does not need to change the standard training procedure of vanilla OPD. Just like its literal meaning, everything that happens here is an additional "protocol" (i.e., the quadruple relationships under ReMG requirements) in data organization.

Of course, we are not to deny that an explicit quadruple-aware loss might be useful. Then why not? Well, this is directly tied to the positioning of our work. Our main contribution is the ReMG protocol itself, instead of any specific methods to realize it (Lines 484-485), while both per-example and quadruple-aware losses belong to "specific methods". To emphasize our positioning, we deliberately keep other specific realizations simple, while more advanced realizations might be better left for follow-up works (Lines 502-505).


W3&Q2: Table 4 shows that most gains appear after adding privileged teacher instructions. A comparison with vanilla OPD using the same instructions is missing.

Thanks for the comment. Please kindly refer to [S1 in the "Response to Shared Questions"] above for our response.


W1. The term “ReMG protocol” covers evaluation, the construction of training and test sets, and the OPD-based method. The contributions of these components should be explained separately.

Thank you for the comment. Following the above discussions (W2&Q1, W3&Q2), we can now respond to it better. Specifically, the “ReMG protocol” strictly refers to the crafted data prior (i.e., the 2x2 data-query organization in Fig. 1 and Sec 2.1), upon which the three downstream components are clearly separated: 1) constructing data that satisfy the ReMG protocol (Sec 2.2), 2) evaluating models on such data with basically standard metrics and their pair/unit variants (Sec 2.3), and 3) fine-tuning models on such data via basically vanilla OPD (Sec 3). The core contribution of our work is to propose the “ReMG protocol” itself, while all other designs demonstrate one specific realization of it. So we deliberately used one term to emphasize this. Still, thanks to your kind suggestion, we have reviewed all positions where the term appears and refined our presentation to eliminate potential barriers to understanding in the revised PDF (Lines 24, 26, 29, 116, 127, 128, 184, 309, 350, 411, 454, 470, 475).


W4&Q3: Eq. 3&4 measure joint correctness, which should not exceed average accuracy. How can content invariability reach 68.37% when content accuracy is only 52.27% in Table 2?

Thank you for carefully pointing this out. Please kindly refer to [S2 in the "Response to Shared Questions"] above for our response.


W5&Q4: A smaller gap partly comes from worse text performance. Could you report text accuracy before and after fine-tuning?

Thank you for the comment. Please kindly refer to [S3 in the "Response to Shared Questions"] above for our response.


W6: Several important components of prior methods are removed or replaced. Results from these modified versions do not fully support the broad claims about the original methods.

Thanks for the comment, but we respectfully disagree. Please kindly refer to [S4 in the "Response to Shared Questions"] above for our response.

Thank you again for reviewing our work. As the end of the discussion phase is approaching, we would appreciate hearing your feedback and knowing whether there are any remaining concerns :)

Reviewer_a6oQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

ReMG evaluates linguistic and paralinguistic behavior on matched 2×2 audio units, using synthetic reasoning speech and human emotional speech. It introduces paired metrics and a gap–retention score, then applies self-distillation with task-specific teacher instructions. Experiments include commercial models, two tuned backbones, and external audio benchmarks.

Critical strengths

The matched design tests content and emotion on the same clips and includes a human-speech tier (§2). It reveals a concrete failure: Qwen2-Audio often answers the spoken question instead of the emotion query (§2.4). The study evaluates external benchmarks and explicitly ablates the paralinguistic loss and teacher instructions (Tables 3–4), providing evidence beyond its new benchmark alone.

Critical weaknesses

The reported metrics do not support their stated interpretation. Under Eqs. (3–4), paired joint correctness cannot exceed marginal accuracy, yet Table 2 gives Qwen2.5 Acnt=52.27 and Icnt=68.37. Also, its gap falls by 2.84 points while audio accuracy rises only 1.14; Eq. (2) implies a 1.70-point text decline. Gap closure therefore partly reflects text degradation. The protocol's independent benefit is unresolved. In Table 4, the protocol-only variant has a worse gap than Base (6.53 vs. 3.41); improvement appears after teacher instructions. There is no privilege-only control on unmatched data. Moreover, Table 9 removes central components from competing methods, limiting conclusions about the original algorithms.

Questions for authors

What exact denominators generate Table 2's paired scores? Please reconcile them with Eqs. (3–4) and report text accuracy explicitly alongside audio accuracy and gap. Does matched training still help when teacher instructions, losses, and budgets are identical? Please separate conclusions about modified baselines from those about the original methods.

2 comments
AuthorsOpenReview ↗

Response to Reviewer a6oQ

Thank you for your comments. We are glad that you believe our manuscript successfully reveals failure and provides wide evidence. Please kindly let us know if we do not address your concerns in the following response.


W1&Q1: Under Eq. 3&4, paired joint correctness cannot exceed marginal accuracy, yet Table 2 gives Qwen2.5 AcntA_{cnt}=52.27 and IcntI_{cnt}=68.37. What exact denominators generate Table 2's paired scores?

Thank you for carefully pointing this out. Please kindly refer to [S2 in the "Response to Shared Questions"] above for our response.


W2: The gap closure partly reflects text degradation. It falls by 2.84 while audio accuracy rises only 1.14, implying a 1.70 text decline.

Thank you for the comment. Please kindly refer to [S3 in the "Response to Shared Questions"] above for our response.


W3-1: The protocol's independent benefit is unresolved. In Table 4, the protocol-only variant has a worse gap than Base (6.53 vs. 3.41).

Thanks for the comment, but we respectfully disagree because the comparison object is unfair. The "protocol-only variant" is actually vanilla OPD + our protocol, which means there is another variable: the effectiveness of the OPD algorithm in the modality gap task. It would also be simple to control this variable to achieve a fair comparison, just by comparing "vanilla OPD" vs. "ReMG protocol" as already provided in Table 4. That is actually the "protocol's independent benefit". Then it would be clear that the ReMG protocol is itself powerful, as detailed in the second paragraph of [S1 in the "Response to Shared Questions"].


W3-2&Q2: The improvement appears after teacher instructions. There is no privilege-only control on unmatched data. Does matched training still help when teacher instructions, losses, and budgets are identical?

Thanks for the question. Please kindly refer to [S1 in the "Response to Shared Questions"] above first as the main part of our response to this question. Then, we would just like to further emphasize one point: "vanilla OPD" vs. "ReMG protocol" is exactly the evidence that "when teacher instructions, losses, and budgets are identical", the matched data themselves still help.


W4&Q3: Table 9 removes central components from competing methods. Please separate conclusions about modified baselines from those about the original methods.

Thanks for the comment, but we respectfully disagree because we actually remove confounding factors while retaining their central mechanisms. Please kindly refer to [S4 in the "Response to Shared Questions"] above for more details.

Thank you again for reviewing our work. As the end of the discussion phase is approaching, we would appreciate hearing your feedback and knowing whether there are any remaining concerns :)

Reviewer_sWpCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the standard way of measuring the modality gap in Large Audio Language Models (LALMs) is flawed. The modality gap is the accuracy drop on spoken questions compared with the same questions given as text. The authors propose ReMG, a 2×2 data protocol used for evaluation and a simple fine-tuning recipe built on it. The protocol groups audio clips into sets of four. Within each set, the speaker, synthesizer and recording conditions are identical, and the clips cover two different linguistic contents, each spoken in two different emotional states. Any change in model behavior within a set can therefore be traced to a difference in either the words or the emotion. Every clip is queried twice, once on its content and once on its emotion. The results show that existing methods fail to reduce the gap on ReMG and often lose emotion accuracy, while the proposed fine-tuning reduces the gap and improves emotion recognition.

Critical strengths
  • The paper evaluates the modality gap and emotion recognition on the same clips, which exposes that recent gap reduction methods such as CORD and X-OPD lose most of their emotion accuracy without reducing the gap.
  • The baselines are reproduced without external teachers, gold labels or judge models, and the code, data and checkpoints are released, which makes the comparison controlled and reproducible.
Critical weaknesses
  • I think the approach is not new and even the framing of the problem. Prior work already uses matched audio with the same content and different emotions to show that audio LMs rely on lexical content and ignore paralinguistic cues, including LISTEN (Chen et al., EACL 2026) and Heard but Not Heeded (Koduru et al., COLM 2026), and none of these works are cited. Heard but Not Heeded already shows that Qwen2-Audio is content driven and that paralinguistic information is encoded in the audio encoder but ignored at the output, which matches and explains the Qwen2-Audio collapse reported here.
  • Training on utterances with the same content and different styles to encourage acoustic sensitivity also appears in prior work (PolyTone training in ParaS2S https://openreview.net/pdf?id=CcmDDh070o).
  • Table 4 shows that most of the improvement comes from the teacher instructions, not the matched data. The matched data raises emotion accuracy only from 35.80 to 40.62 over Vanilla OPD and leaves the gap at 6.53, almost twice the base model’s 3.41. Adding the teacher instructions raises emotion accuracy to 52.56 and reduces the gap to 0.57. This weakens the claim that the improvement comes from the ReMG protocol itself.
  • The quality of the synthesized data is not evaluated by humans. Emotion labels come from TTS control and are kept only when emotion2vec or SpeechBrain agree, and the only human check is a small pilot on training speakers. There is no naturalness, speaker similarity or human emotion study on the evaluation set. This matters because automatic filtering may favor clear and exaggerated emotion, so the evaluation set may not reflect natural emotional speech.
  • Tier-R uses a single emotion contrast (neutral versus happy), so the conclusions are not shown to hold for other emotions or other paralinguistic attributes.
Questions for authors

1- In Eq. 3 and Eq. 4, I_cnt and S_cnt are the rates at which both clips in a pair are answered correctly, so they cannot exceed A_cnt. Yet Table 2 reports higher values in every row (for example, I_cnt 68.37 and S_cnt 62.96 versus A_cnt 52.27 for Qwen2.5-Omni). How are these two metrics actually computed?

2- Can you report results for at least one additional emotion pair on Tier-R, or a breakdown by emotion pair on Tier-P?

5 comments
AuthorsOpenReview ↗

Response to Reviewer sWpC (1/3)

Thank you for your detailed reviews and also for valuing our evaluation in exposing failures in the current SOTAs, our controlled reproduction, as well as our released code, data, and checkpoints. Below we respond to each point of your concerns.


W1&W2: The approach and problem framing are not new. Prior works LISTEN, Heard but Not Heeded, and ParaS2S already use matched audio with the same content and different emotions/styles to show that LALMs rely on lexical content and ignore paralinguistic cues. Heard but Not Heeded already reported a similar result for Qwen2-Audio as ReMG. ParaS2S also proposes to encourage acoustic sensitivity.

Thank you for recommending the prior works, but we respectfully disagree with your opinion. In fact, we agree with your summarizations of the prior works, but it is precisely such agreement that makes us more confused: Why do you think they are evidence not for but against our novelty? In our opinion, any single point mentioned here actually helps to confirm our unique contributions.

First, for problem framing, all the recommended prior works are about the paralinguistic perception task. That not only means "matched audio with the same content and different emotions / styles", but also means paralinguistic query (i.e., qparaq_{para}). Thus, just as you summarized for ParaS2S, they focus on "encouraging acoustic sensitivity" alone. While this is indeed also considered in our ReMG as para sensitivity SparaS_{para}, it is just one corner of our 4-tuple. Apart from it, we also consider three more corners: para invariability IparaI_{para} (different cnt + same para + para query), cnt sensitivity ScntS_{cnt} (different cnt + same para + cnt query), and cnt invariability IcntI_{cnt} (same cnt + different para + cnt query). Even from this simple statistical perspective, the recommended prior works can only cover at most one quarter of our ReMG. This first supports our novelty in noticing the empty corners and filling them.

Second, if we had only reached here, it might still be somewhat reasonable to say our "approach is not new" by relying on a similar paired control (though only similar in the format). However, our core contribution appears only after the pairs are constructed into 4-tuple units, which is then completely unique from the recommended prior works. More specifically, the ReMG does not encourage any corners alone. Instead, based on the 4-tuple units, it actually encourages perceiving, distinguishing, and directly trading off linguistic and paralinguistic features within a single audio (Lines 095-103). This makes it possible for the reduction of modality gap and maintenance of paralinguistic capability to be evaluated and achieved at the same time, which the recommended prior works that do not even consider modality gap could have never expected, and is also unseen in any previous works.

Finally, the Qwen2-Audio observation in Heard but Not Heeded actually serves as an external validation rather than overlap of our core contributions. Please note that the purpose of Table 1 is not to claim "Qwen2-Audio is content-driven" as our main discovery, but to check the different conclusions between the previous modality gap protocol and our ReMG. Specific to this case, under the previous protocol, Qwen2-Audio was still reported to achieve a high paralinguistic score (e.g., 60.81% in CORD). Thus, Heard but Not Heeded actually helps to double-confirm that the previous protocol gave false-positive assessments and masked the defect, whereas ReMG faithfully exposes it (e.g., 0.57%) while jointly measuring the modality gap. Moreover, Table 1 is only our starting point before revealing the trade-off collapse of SOTA gap-reduction methods and resolving it via fine-tuning later, neither of which is explored in Heard but Not Heeded (and other recommended prior works).

Of course, thanks to your kind recommendation, we are glad to cite all these prior works to further highlight our novelty in the revised PDF (Lines 266-269, 321-323).

AuthorsOpenReview ↗

Response to Reviewer sWpC (2/3)

W3-1: Table 4 shows that most of the improvement comes from the teacher instructions, not the matched data.

Thanks for the comment, but we respectfully disagree. Please kindly find [S1 in the "Response to Shared Questions"] above for details.


W3-2: The matched data raises emotion accuracy only from 35.80 to 40.62 over Vanilla OPD and leaves the gap at 6.53, almost twice the base model’s 3.41, which weakens the claim that the improvement comes from the ReMG protocol itself.

Thanks for the comment, but we respectfully disagree. First, + ReMG protocol itself achieves -0.29 (better) gap, +1.42 audio content acc, +4.82 audio paralinguistic acc (all these are standard metrics), and also about +3 consistent improvements across all our new unit metrics. We do not think this is so negligible as you commented. Second, it makes no sense in Table 4 to skip the middle "vanilla OPD" and compare "ReMG protocol" with "Base model", because "ReMG protocol" is not the base model + ReMG protocol but the base model + vanilla OPD + ReMG protocol. Though the gap is "almost twice" at this time, the logic is actually "+ vanilla OPD makes it worse, while further + ReMG protocol turns the trend", instead of "+ ReMG protocol makes it worse". In fact, this exactly shows that "the improvement comes from the ReMG protocol itself". More details can be found in the second paragraph of [S1 in the "Response to Shared Questions"].


W4: The quality of the synthesized data is not evaluated by humans. Emotion labels come from TTS control and are kept only when emotion2vec or SpeechBrain agree, and the only human check is a small pilot on training speakers. There is no human check on the evaluation set. This matters because automatic filtering may favor clear and exaggerated emotion, so the evaluation set may not reflect natural emotional speech.

Thanks for the comment, but we respectfully disagree. First, "only human check is a small pilot on training speakers" is a misunderstanding. The original sentence we wrote is actually "Before the full-scale run, a pilot of 20 texts × 2 engines × 2 states × 4 training-pool speakers is synthesized and passes a human listening check" (Lines 787-789). That is not about only "training speakers", but about all synthesis materials including texts, TTS engines, emotions, and speakers. In fact, the whole synthesis pipeline of Tier-R is already finished there (i.e., the remaining steps do not generate new data), so it is actually a final, comprehensive human check for "the quality of the synthesized data".

Second, would "small pilot on training" be a problem? We do not think so. On the one hand, this "pilot" does not consist of external samples. Instead, they are just part of the synthesized dataset itself. Thus, our implementation is actually equivalent to post-checking the first few samples among the synthesized dataset without cherry-picking. At the same time, we think the number of samples checked here, 320, is not "small" compared with previous practices. On the other hand, emphasizing "training" over "evaluation" is unnecessary here, because the whole synthesis pipeline of our training and evaluation splits has no differences at all. As our human check on the training set already confirmed the data quality from this pipeline, the same conclusion certainly holds for the evaluation set.

Third, regarding the concern about the automatic filtering after synthesis, the logic is that: If we do not even find exaggerated/unnatural cues when checking the synthesized superset, could there be such a problem in its subset after automatic filtering? We think the risk is minor. Still, two facts can further help to address this concern: 1) Apart from the synthesized Tier-R, we also supplement the natural-sourced Tier-P, which is based on human-recorded ESD with "natural emotion" (Lines 161-188). 2) If exaggerated/unnatural speech is really dominant in our dataset, when we fine-tuned our model as in Sec 3, a bad consequence of domain mismatch should have been exposed once we tested on those recognized external high-quality benchmarks. But the fact is, the model fine-tuned on our dataset successfully zero-shot transferred to six benchmarks under VoiceBench and MMAU, outperforming all prior works.

AuthorsOpenReview ↗

Response to Reviewer sWpC (3/3)

W5&Q2: Tier-R uses a single emotion contrast (neutral versus happy), so the conclusions are not shown to hold for other emotions or other paralinguistic attributes. Can you report results for at least one additional emotion pair on Tier-R, or a breakdown by emotion pair on Tier-P?

Thank you for the comment. We acknowledge that the more emotion pairs and paralinguistic attributes, the more ideal the situation could be. Our current implementation is mainly constrained by the cost, as the data construction pipeline is expensive and time-consuming (Table 6&7 provide an intuitive illustration). However, this does not mean our conclusions are not of general significance. Specifically, if the concern is regarding the evaluation part, then please note that the logic for our evaluation is not "we do not cover other emotions/attributes, thus cannot conclude something specific to them", but "if even a simple neutral versus happy could already reveal the flaw, then the problem must exist". This is enough to draw our main conclusions in this work. On the other hand, if the concern is regarding the fine-tuning part, then as discussed in W4 above, zero-shot transferring to widely recognized VoiceBench and MMAU (covers diverse "other emotions" and "other paralinguistic attributes" across speech, sound, and music) already confirmed our general significance.


Q1: In Eq. 3&4, IcntI_{cnt} and ScntS_{cnt} are the rates at which both clips in a pair are answered correctly, so they cannot exceed AcntA_{cnt}. Yet Table 2 reports higher values in every row. How are these two metrics actually computed?

Thank you for carefully pointing this out. Please kindly refer to [S2 in the "Response to Shared Questions"] above for our response.


Q3: In Table 1 Qwen2-Audio has 0.57% emotion recognition accuracy. In 318 of 352 cases, it ignores the emotion question and tries to answer the question spoken in the audio. This may be an instruction-following failure, not a perception failure. Can you separate the two, for example with constrained decoding over the label set?

Thanks for the question. First, this could not be a conventional "instruction-following failure", since in CORD, for example, Qwen2-Audio had no trouble following the instruction qparaq_{para}. Instead, this failure occurs only when our ReMG is applied, and our finding also holds across other LALMs in Table 1. Second, we have never thought the issue here is a simple input-end "perception failure", otherwise we would not have frozen the audio encoder and adapter while only fine-tuning the backbone (Lines 954-955). Actually, our view is, again, to "encourage LALMs to perceive, distinguish, and directly trade off linguistic and paralinguistic features within a single audio" (Lines 099-102), which obviously refers to how the model understands and processes the information. Failing to follow qparaq_{para} is exactly one manifestation of ignoring this in the previous view. Finally, we appreciate your suggested constrained decoding, but respectfully think it is unnecessary here. This is simply because we do not use LALMs like that in real-world practice. In that case, even the main task "modality gap" would lose its definition, making any discussions meaningless. So no matter what result we may get from constrained decoding, it would anyway not change our conclusion.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

Dear Reviewers,

Thank you all for taking the time to review our manuscript and provide detailed comments. In the following cells, we respond to each point to try to address your concerns. The submitted PDF is also updated accordingly, with all revised positions marked in blue. Before diving into specific points, we would like to first talk about the overall positioning of this work for better understanding.

The core view of this work is that: Prior works claim to reduce the modality gap while maintaining the paralinguistic capability, but that is an over-optimistic claim since the two sides are measured on disjoint data without modeling their real trade-off. Based on this, we 1) first propose a ReMG protocol that can model the tension, 2) then instantiate the ReMG protocol with new data and evaluation, and 3) finally fine-tune models on that data to show the potential of ReMG.

The most important thing we would like to emphasize here is that: The purpose of 2) and 3) is not to provide the ultimate solution, but to demonstrate the feasibility and effectiveness of 1), our new direction, and to alert the community about this blind spot. Thus, if evaluating 2) in isolation against works dedicated to building new benchmarks, or evaluating 3) in isolation against works dedicated to developing new algorithms in fixed-setting tasks, they may not match those works in every detail (and it would also be unfair to expect that). However, this does not mean they are not sufficient to provide clear and rigorous empirical evidence for our core protocol and conclusions in this work (as we would revisit multiple times in our detailed responses). We sincerely hope that you could keep this big picture in mind when we go into the following detailed discussions, and accordingly make a more comprehensive judgement of this work as a whole.

Finally, in the following detailed responses, for better efficiency and clarity, we tend to state our arguments and note potential misunderstandings (if any) straightforwardly. We would like to clarify that they are just scientific discussions. Regardless of whether we agree with the specific opinions or not, we sincerely appreciate and respect all reviewers for voluntarily reviewing our work and kindly helping us towards a better revision. Thank you!

Best regards,

Authors

AuthorsOpenReview ↗

Response to Shared Questions (1/3)

In the following cells, we would like to address some comments/questions raised by multiple reviewers first. We label at the end of each comments/questions by which reviewers it is mentioned, and would also refer to this cell in the responses under every single review. Thank you to the reviewers for kindly taking the time to check both parts :)


S1. Ablation shows that our most improvement comes from not the ReMG protocol but the privileged teacher instruction? [Reviewer sWpC, a6oQ, uwdQ]

First of all, we would like to respectfully point out that Table 4 actually does not show "most" improvement after adding the privilege. We guess this misunderstanding may come from a conventional gap-centered perspective (i.e., from the column of GG). However, one of the most important arguments at the beginning of our work is that: The gap value is meaningless without considering its real trade-off with paralinguistic capability (Lines 13-17, 21-23). Even existing works generally also claim the maintenance of paralinguistic capability (despite their questionable practices).

Thus, when it comes to the improvement in Table 4, if only one column is allowed to refer, it should be UU (as it considers the gap and paralinguistic capability at the same time), which actually demonstrates a larger improvement after adding the ReMG protocol (from -3.69 to 0.00) than privilege (from 0.00 to +2.84). Even if considering UU is a new metric that might not be completely trustworthy, at least all other metrics in Table 4 (either standard metrics themselves or derived directly from standard ones) should be taken into account, which also show substantial improvements of the ReMG protocol. The most important observation here is that: Vanilla OPD gets worse results than the base model, while only adding the ReMG protocol itself can successfully reverse the trend. That means the ReMG protocol itself already provides the correct optimization direction, and the following privilege is just a practical enhancement approach.

Next, why is vanilla OPD + privilege not an ablation item in our original manuscript? First, as above, + vanilla OPD and + ReMG protocol already show opposite trends without any privilege, so it is natural to think that vanilla OPD is out at the beginning. Second, the linguistic privilege is not for effectiveness but efficiency, and was proved not to help alone ("- Lpara\mathcal{L}_{para}" in Table 4). Third, the paralinguistic privilege is indeed for effectiveness, but is useful mostly because of the ReMG protocol. Specifically, only under the ReMG protocol are there such cases: An audio with meaningful linguistic content (like QA) is queried about its paralinguistic feature (like emotion). Only in that case, our paralinguistic privilege, which reminds the teacher to focus on the query instead of being misled by the linguistic content, is useful. In contrast, without the ReMG protocol, paralinguistic queries are only applied to paralinguistic datasets, where their linguistic contents are basically meaningless daily chats. In that case, a teacher does not need such a reminder to avoid being misled at all. That is why it is an "enhancement" specific to the ReMG protocol, instead of a general trick useful for others.

Finally, thanks to your suggestion, we now realize a plain vanilla OPD + privilege ablation would anyway be helpful, so we are glad to provide the following new result. Specifically, without our ReMG, vanilla OPD + privilege mainly boosts the linguistic side and still leaves the larger modality gap caused by vanilla OPD. Meanwhile, its paralinguistic side still fails to turn the decreasing trend from the base model, lagging 13.64% behind Ours and resulting in a negative U=−0.57U = -0.57. Again, this is not surprising as the paralinguistic privilege is mainly useful because of ReMG, just as explained above. This new result directly confirms that without our ReMG protocol, vanilla OPD + privilege alone cannot resolve the linguistic-paralinguistic tension while reducing the modality gap. We have also added this to Table 4 in the revised PDF.

GGAcntA_{cnt}AparaA_{para}UaU_aUhU_hUU
Base model3.4152.2739.4945.8844.99-
Vanilla OPD + privilege (as suggested)5.9757.3938.9248.1646.38-0.57
Vanilla OPD + ReMG protocol + privilege (Ours)0.5753.4152.5652.9852.98+2.84
AuthorsOpenReview ↗

Response to Shared Questions (2/3)

S2: Eq. 3&4 measure joint correctness, which should not exceed average accuracy. But Table 2 reports higher IcntI_{cnt} and ScntS_{cnt} than AcntA_{cnt}. Why? [Reviewer sWpC, a6oQ, uwdQ]

Just as implied by "correct changes according to sis_i" in the two lines above Eq. 3 (Line 253), the cnt invariability is calculated not on all data like the average accuracy but conditioned on the model correctness on sis_i (in text). The philosophy is intuitive: If the model even fails to answer sis_i correctly at first, then it would be meaningless to talk about whether an audio with sis_i maintains the invariability of "correct" responses to paralinguistic changes. The same philosophy holds for the cnt sensitivity, but not for the para invariability / sensitivity, since the latter do not have the correctness of an independent zjz_j as the reference. Thus, there should indeed be an additional condition in the equations of IcntI_{cnt} and ScntS_{cnt}. We sincerely apologize for omitting this detail and have corrected it in the revised PDF.


S3: A better gap partly comes from worse text performance. [Reviewer a6oQ, uwdQ]

That is true. However, that is not a flaw of ReMG but a pattern that should have been revealed in the modality gap task if prior works had properly considered the paralinguistic capability. Following the kind suggestion of Reviewer uwdQ, we have added the column "AtextA_{text}" to report text accuracy directly in all tables (if any) in the revised PDF. The score is simply "G+AcntG+A_{cnt}", both of which were originally reported in the tables, and the text accuracy before fine-tuning is the AtextA_{text} score in the base model. Taking Table 2 as an example, it indeed shows that ReMG suffers from worse text performance while the baselines do not. However, let's consider a simple question: When we are aiming at a smaller G=Atext−AcntG=A_{text}-A_{cnt}, would finally getting an increased AtextA_{text} be normal? No, that is counter-intuitive. Actually, that is exactly one of the issues we mentioned for the prior view (Lines 80-85), as everything becomes clear once we check the paralinguistic side: Compared with ReMG, the baselines suffer -15.34, -4.26, -26.14, -23.87 drops in AparaA_{para} score, respectively. So the truth here is simply a trade-off. Then, what kind of trade-off is more reasonable: Better text performance but worse paralinguistic capability like the baselines, or worse text performance but better paralinguistic capability like ours? For LALMs with acoustic capability as the core competitiveness to distinguish from LLMs, the answer should be clearly ours.

AuthorsOpenReview ↗

Response to Shared Questions (3/3)

S4: Table 9 removes/replaces central components of prior methods. Results from modified baselines do not fully support the claims about the original methods. [Reviewer a6oQ, uwdQ]

First, it is not original superficial form but real scientific attribution that is essential for fair comparison. Just as the "one methodology-level issue" we mentioned at the beginning (Lines 25-28, 111-117, 122-123), these prior methods learn from stronger teachers or ground-truth labels, which introduces external knowledge as a confounding factor when evaluating modality gap reduction. This is questionable because, even without any new method designed for the gap, the model can still benefit from such external knowledge, being more powerful in general rather than in cross-modal alignment, and providing potentially unfair and illusory "improvement" to the gap reduction task. To be honest, these prior methods obviously mix their own contributions with those from new knowledge learning, which makes it impossible to tell whether their gains come from their proposed mechanisms. Therefore, by keeping their core mechanism and only equalizing the external knowledge sources (Lines 416-419), it is precisely our controlled reproduction that isolates the true scientific contribution of their "original methods" for a fair comparison.

Second, we did provide a prior method as "original", which is shown as "TARS-Qwen2.5" in Table 2. It is an official publicly released checkpoint from one of the baselines, TARS, and also the only checkpoint that prior methods released to the public. The results clearly show that even this totally "original" checkpoint is not competitive with ours. We also hope more prior methods would have released their checkpoint as we did, but when it comes to reproduction, we would insist on our view above.

Finally, within the removed/replaced components among all baselines, the ones most probably close to their "central" mechanisms are those of TARS. However, it is precisely TARS that is least affected compared with the results of its "original" method (i.e., quite close to its original TARS-Qwen2.5 checkpoint), and performs the best among all baselines as in Table 2. On the contrary, for example, what we did for X-OPD is only to replace the more powerful external Qwen3-A3B-Instruct teacher with Qwen2.5-Omni-7B itself, which we do not think involves any "central" modification, but eventually its results are largely behind TARS and CKD baselines. Thus, again, it is precisely our controlled reproduction that makes clear the truly promising and robust scientific contributions among the baselines, so that our conclusions are applicable not only to support our claims, but also to benefit the community and follow-up works in understanding the "original methods". Still, we are glad to revise some statements in the PDF to make the conclusions more precise (Lines 422-426, 495-500). Everything we did was to push the field forward alongside prior methods, which we believe is also what their authors had hoped for.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.