acceptodds
Under review as a conference paper at ICLR 2027

Query-Oriented and Reasoning-Aware Perception for Versatile Speech Understanding

Abstract

Existing speech understanding systems rely on either implicit speech perception or query-agnostic speech perception, lacking the consideration of the relationship between perception and reasoning. What to perceive depends on the reasoning, while reasoning builds upon perceived information. To address this issue, we propose Query-oriented and Reasoning-aware Perception Optimization (QRPO), a reinforcement learning algorithm that trains a Perceiver to first determine what should be perceived for answering the query and then generate a reasoning-aware description for the perceived information. A Reasoner, which can be instantiated with arbitrary off-the-shelf large language model, performs downstream reasoning based on the description to generate the final response. The correctness of the response is further used as a reward signal to encourage the Perceiver to learn how to describe speech in a way that better supports downstream reasoning. We name the resulting PErceiver-REasoner framework PERE. Experimental results on the MMAU, MMSU, MMAR, and AIR-Bench benchmarks demonstrate that PERE achieves significant gains on speech understanding performance. Furthermore, PERE achieves state-of-the-art results under a more realistic open-ended setting where only the speech and query are provided without candidate answer options.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech understanding, speech language models, query-oriented perception, reasoning-aware perception, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_D6ahOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PERE, a framework for speech question answering. It consists of a perceiver and a reasoner. The perceiver's input is speech query and optionally answer options. Its outputs are perception aspects and a description of evidence. The reasoner is an LLM that answers the question from the description. The framework is trained with a special RL recipe that is based on GRPO. The training reward consists of format validity, F1, and reasoner answer correctness. The framework is evaluated over MMAU, MMSU, MMAR, and AIR-Bench and gains over other speech LMs are reported.

Critical strengths
  1. The gains seem convincing especially on the large benchmark MMSU (Table 1).
  2. The authors provide enough ablation to show what matters in the framework. Table 2 includes multiple ways of producing the intermediate text and shows that a lot of other ways do not help or even hurts.
Critical weaknesses
  1. The framework is in essence a cascade: a model that first turns speech into text and then another model understands the text. The novelty is limited and the paper should be positioned among prior cascade works like Rao et al. “Do as I Mean, Not as I Say: Sequence Loss Training for Spoken Language Understanding.” which also trains a pipeline ASR model with RL using reward from downstream task accuracy.
  2. It's not shown that indeed the framework decouples perception from reasoning. The answer-preserving SFT objective also encourages the perceiver to answer the query, so it's less clear whether the perceiver's description already encodes the answer.
Questions for authors
  1. Have the authors checked whether the descriptions outputted by the perceiver are an honest description of the audio?
  2. What are the end-to-end latency of the proposed framework compared with a single speech language model?
  3. The “Qwen2.5-Omni-7B (Fine-tuned)” baseline in Table 2 is described only as being fine-tuned “on the same training data of PERE”. Was it trained with SFT only, or also with RL?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the detailed feedback and constructive questions. Below, we provide detailed responses to each of your questions and suggestions.


Q1. Have the authors checked whether the descriptions outputted by the perceiver are an honest description of the audio?

A1. Thank you for raising this important question. To verify whether the descriptions generated by the Perceiver faithfully reflect the actual audio content, we conducted both human and model-based evaluations. Specifically, we randomly sampled 10 audio clips from the benchmarks and generated descriptions for each using both Qwen2.5-Omni-7B and PERE, resulting in 20 descriptions in total. Ten human annotators independently rated these descriptions on a 1–5 scale, with higher scores indicating greater factual accuracy with respect to the corresponding audio. Importantly, the evaluation focused exclusively on the accuracy of the described content, rather than its comprehensiveness. In addition, we employed Qwen3-Omni-30B-A3B-Instruct to evaluate descriptions for all samples across the benchmarks using the same criterion.

Human Evaluation

ModelHuman Evaluation
Qwen2.5-Omni-7B3.26
PERE (Ours)3.98

Model-based Evaluation

ModelMMAUMMSUMMARAIR-Bench
Qwen2.5-Omni-7B3.833.733.524.40
PERE (Ours)4.054.074.154.54

As shown in the results, PERE consistently achieves higher description accuracy than Qwen2.5-Omni-7B in both human and model-based evaluations. These findings provide empirical evidence that the descriptions generated by PERE faithfully reflect the input audio content.

We have added these results and the corresponding discussion to the revised manuscript.


Q2. What is the end-to-end latency of the proposed framework compared with a single speech language model?

A2. To evaluate the end-to-end inference efficiency of PERE compared with a single speech language model, we conducted additional measurements of both inference latency and token consumption per query. We compared PERE with an end-to-end model and a modular baseline. The modular baseline uses Qwen3-Captioner to generate a query-agnostic caption, which is then fed to the same Reasoner used in PERE. All measurements are conducted on a single NVIDIA H100 GPU with a batch size of 1 using MMAU samples.

Latency (Seconds / Query)

MethodPerceiverReasonerTotal
Qwen2.5-Omni-7B----0.17
Generic Captions + Reasoner20.280.4220.70
PERE (Ours)1.280.291.57

Tokens / Query

MethodPerceiverReasonerTotal
Qwen2.5-Omni-7B----5.53
Generic Captions + Reasoner508.477.32515.79
PERE (Ours)99.505.74105.24

The end-to-end baseline is the most efficient, requiring only 0.17 s and 5.53 tokens per query, as it generates answers in a single decoding pass. Compared with the query-agnostic baseline, PERE is substantially more efficient: Qwen3-Captioner generates verbose captions averaging 508.47 tokens, resulting in a Perceiver latency of 20.28 s/query, whereas PERE generates concise, query-oriented descriptions averaging only 99.50 tokens, reducing Perceiver latency to 1.28 s/query. The shorter descriptions also reduce Reasoner latency from 0.42 s to 0.29 s. These results demonstrate that PERE achieves a favorable efficiency trade-off compared with the query-agnostic modular pipeline, although it remains slower than the end-to-end baseline.

We have added these results and the corresponding discussion to the appendix of the revised manuscript.


Continued in the next comment...

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Q3. The “Qwen2.5-Omni-7B (Fine-tuned)” baseline in Table 2 is described only as being fine-tuned “on the same training data of PERE”. Was it trained with SFT only, or also with RL?

A3. The “Fine-tuned” baseline in Table 2 is trained with SFT only. No RL is involved in this variant. This variant is designed to rule out the potential effect of additional training data used in PERE. We have revised the main text to provide a clearer description of this variant.


Q4 (weakness 1). The framework is in essence a cascade: a model that first turns speech into text and then another model understands the text. The novelty is limited and the paper should be positioned among prior cascade works like Rao et al. “Do as I Mean, Not as I Say: Sequence Loss Training for Spoken Language Understanding.” which also trains a pipeline ASR model with RL using reward from downstream task accuracy.

A4. Thank you for this important comment. We agree that, at a high level, PERE can be viewed as a cascade framework that first converts speech into a textual representation and then uses a separate model for downstream understanding. In fact, our original manuscript already discusses existing cascade-based approaches, including explicit speech perception models, in Related Work (Section 2.2). As discussed there, a key limitation of most existing approaches is that their intermediate textual representations are query-agnostic, regardless of the downstream task. This observation motivates the central idea of PERE: query-oriented and reasoning-aware perception.

Compared with prior work such as Rao et al., which primarily focuses on speech content or intent for downstream spoken language understanding, PERE addresses a substantially broader perception problem. Understanding speech may require capturing not only linguistic content but also various paralinguistic and acoustic characteristics, such as emotion, prosody, rhythm, fluency, articulation, and speaker characteristics. More importantly, PERE investigates a fundamental question that is largely overlooked in conventional cascade pipelines: whether the speech-to-text mapping itself can be made query-oriented and reasoning-aware. Given a user query, PERE explicitly considers both what information should be perceived to answer the query (what to perceive) and how the relevant evidence should be described to facilitate downstream reasoning (how to describe).

Therefore, PERE goes beyond simply cascading a speech model with a downstream language model. Its key contribution lies in making the intermediate perception process explicitly conditioned on the user's query and the requirements of downstream reasoning. We have clarified this distinction and strengthened the discussion of related cascade-based approaches in Section 2.2 of the revised manuscript.


Continued in the next comment...

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q5 (weakness 2). It's not shown that indeed the framework decouples perception from reasoning. The answer-preserving SFT objective also encourages the perceiver to answer the query, so it's less clear whether the perceiver's description already encodes the answer.

A5. Thank you for raising this important concern. We would like to clarify that the answer-preserving SFT objective serves only as a gentle initialization strategy. In the perception-only SFT and the subsequent QRPO stages, the Perceiver is explicitly trained to output only the relevant aspects and their descriptions.

We also note that whether a description contains information sufficient to identify the correct answer depends naturally on the nature of the query. For perception-oriented questions, the perceived audio content itself may directly provide the answer, and it is therefore reasonable for the description to contain information corresponding to the correct option. In contrast, for questions requiring more complex reasoning, the Perceiver is expected to provide relevant perceptual evidence, leaving the more challenging reasoning process to the Reasoner, which is generally a more powerful LLM with good text reasoning capability but lacking audio perception capability.

To further investigate whether the Perceiver's descriptions already encode the answers, we analyzed the frequency with which descriptions generated under the MCQ setting contain text from the correct answer option. Note that the options are provided to the Perceiver as a list of their textual contents, rather than as letter labels (A/B/C/D). Therefore, we measure whether the generated descriptions contain text corresponding to the correct answer option (see the Perceiver prompt in the Appendix for details). The results are reported below.

MMAU Test-miniMMSU Per.MMSU Rea.MMSU AvgMMAR
PERE16.5%21.0%7.9%14.7%17.0%

In MMSU, the perception subset evaluates what the model can directly hear from speech, such as basic acoustic and linguistic cues, whereas the reasoning subset evaluates what it can further infer from those perceived cues. As shown in the table, the probability of containing the answer text varies substantially depending on the nature of the query. For perception-oriented questions, descriptions are more likely to contain text from the correct option, with this occurring in 21.0% of cases on the perception subset of MMSU. In contrast, for reasoning-oriented questions, this probability drops to only 7.9% on the reasoning subset of MMSU.

We also hope to draw the reviewer’s attention to the case studies in the Appendix for concrete examples of the descriptions generated by the Perceiver. As illustrated by these examples, the Perceiver's descriptions primarily provide perceptual information from selected aspects of the audio, rather than attempting to directly answer the query. When a query concerns information that can be directly obtained through perception, the description may naturally contain text corresponding to the correct answer. Otherwise, there is no need for the Perceiver to explicitly encode the answer in its description, as the subsequent, more capable Reasoner is responsible for performing the necessary reasoning based on the provided perceptual evidence.


We sincerely thank the reviewer for your time and expertise in providing comments to improve our manuscript. We have tried our best to address all concerns and suggestions to improve our manuscript. Thank you!

Reviewer_xijxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work tries to solve the speech understanding problem. Instead of the standard end-to-end approach, the paper proposes to separate speech perception from reasoning, training the perception module with SFT and RL and using a strong foundation model as the reasoner.

Critical strengths
  1. The idea is straightforward and easy to apply, and the method achieves strong results compared with prior work.

  2. The introduced open-ended evaluation allows a more realistic assessment of speech perception by withholding candidate answer options from the Perceiver.

Critical weaknesses
  1. A closely related framework for decoupling perception and reasoning was introduced in RAPID [1]. Although the paper cites this work, the discussion is brief and does not clearly explain the similarities or establish the contribution beyond adapting the framework to speech understanding.

  2. The ablations compare SFT + RL with SFT alone, without evaluating RL without an additional SFT stage, as used in [1]. Without this comparison, it is difficult to isolate the benefit of the introduced SFT stage.

[1] Gou et al. Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning. ICLR 2026.

Questions for authors
  1. During open-ended evaluation, are answer choices withheld from both the Perceiver and Reasoner? Section 4.3 and Appendix B.3 appear to describe different protocols.
  2. How does PERE compare with a direct adaptation of RAPID to speech?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the detailed feedback and constructive questions. Below, we provide detailed responses to each of your questions and suggestions.


Q1. During open-ended evaluation, are answer choices withheld from both the Perceiver and Reasoner? Section 4.3 and Appendix B.3 appear to describe different protocols.

A1. Thank you for pointing out this ambiguity. In the original manuscript, the term “open-ended evaluation” referred to a setting in which the Perceiver was not provided with answer options, while the Reasoner still had access to them. This design allowed us to evaluate the Perceiver's option-unaware perception capability while retaining accuracy as an objective and intuitive evaluation metric.

However, we acknowledge that describing this protocol as “open-ended” was imprecise and could cause confusion. To address this issue, we have revised the terminology and evaluation protocols in the updated manuscript. Specifically, we have renamed the original “open-ended setting” as the “perceiver-blind setting” as suggested by Reviewer XUxK, explicitly referring to the protocol in which the Perceiver cannot access the answer options, but the Reasoner can. In addition, we have introduced a strictly “open-ended setting”, in which answer options are withheld from both the Perceiver and the Reasoner. Under this setting, the system generates free-form responses, which are evaluated by an LLM judge.

We have revised Section 4.3 to clearly distinguish between the perceiver-blind and open-ended settings, and added the results under the strictly open-ended setting to Table 1 in the revised manuscript. We have also ensured that the descriptions in Section 4.3 and Appendix B.3 are fully consistent.


Q2. How does PERE compare with a direct adaptation of RAPID to speech?

A2. We thank the reviewer for pressing on the relation to RAPID (Gou et al., 2026). We agree that both methods belong to the broad family of perception–reasoning decoupling. We will expand Related Work to state this clearly. However, this paradigm remains largely unexplored in the speech domain. To the best of our knowledge, our proposed PERE is the first work to bridge this gap. Nevertheless, PERE is not a direct adaptation of RAPID to speech; the two differ along several design axes.

(1) Degree of decoupling. In PERE, the Perceiver is responsible only for deciding from which perspectives speech should be described and how it should be described. It is not required to produce a tentative solution, as in RAPID. Therefore, PERE realizes a more thorough perception–reasoning decoupling.

(2) Perceiver functionality. PERE requires the Perceiver to explicitly select the speech aspects relevant to the query before generating a description. These aspects are specifically designed for speech, and this explicit “what-to-perceive to how-to-describe” decision process is absent from RAPID. (Please see the Section 5.2 Effectiveness of explicit aspect selection)

(3) Training stage design. We deliberately introduce a dual-objective SFT stage. Without this stage, directly applying RL in a RAPID-like manner leads to limited improvements. (Please see the Q3 below.)

(4) Supervision design. PERE supervises aspect selection during SFT and employs an aspect-selection reward during RL, both of which are specific to PERE. In addition, RAPID optimizes descriptions solely through outcome-based RL, which may lead to unstable and reward-hackable captions without guaranteed query relevance. In contrast, PERE ensures that descriptions are both query-oriented and reasoning-aware through carefully designed data curation, training strategies, and supervision objectives.

(5) Data curation. PERE constructs a dedicated supervision stack: query-oriented aspect labels, reasoning-aware descriptions (with reasoning traces and answer derivations stripped), a reasoning-sufficiency filter, and a weak/strong audio-contribution split used respectively for SFT and RL.

(6) Evaluation. Beyond the conventional MCQ setting, we evaluate SLMs under a more realistic open-ended setting, where answer choices are withheld and only the speech signal and query are provided. In addition, we also evaluate SLMs under a perceiver-blind setting, which allows us to assess the Perceiver’s option-unaware perception capability while retaining accuracy as an objective and intuitive evaluation metric.


Continued in the next comment...

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q3 (weakness 2). The ablations compare SFT + RL with SFT alone, without evaluating RL without an additional SFT stage.

A3. To address this concern, we have conducted additional experiments using RL (QRPO) training alone, without prior SFT.

VariantMMAU Test-miniMMAU TestMMSU Per.MMSU Rea.MMSU AvgMMARAIR-Bench
Qwen2.5-Omni-7B + Reasoner65.7765.8745.5974.3659.5950.348.06
 + QRPO66.9767.2743.2775.4158.8361.908.05
PERE (Ours)74.4769.7348.0180.3663.1664.938.32

The results show that RL-only training brings very limited improvements on MMAU, and AIR-Bench, and can even lead to performance degradation on MMSU. Although RL-only training yields a more noticeable improvement on MMAR, its performance still falls substantially short of that achieved by PERE with both SFT and RL training stages. These results further demonstrate the importance of the SFT stage.

We have added these results and the corresponding discussion to the revised manuscript.


We sincerely thank the reviewer for your time and expertise in providing comments to improve our manuscript. We have tried our best to address all concerns and suggestions to improve our manuscript. Thank you!

Reviewer_XUxKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Updating ratings after reviewing new evidence supplied by the authors

This paper addresses speech perception conditioned on the query and aligned to downstream reasoning.

It proposes PERE, a two-stage framework: (1) a trainable Perceiver, by default Qwen2.5-Omni-7B, that selects query-relevant aspects from a fixed six-aspect taxonomy and writes a short description limited to those aspects; (2) a frozen text-only Reasoner that answers from the query and that description, plus candidate options when given. Training is two-stage: dual-objective SFT for description generation and direct answering, followed by QRPO, which is GRPO with format, aspect-selection F1, and answer-correctness rewards weighted 0.1, 0.2, and 0.7. SFT targets are distilled by Qwen3-30B from AudioMCQ/SpeechCraft CoTs and filtered by Qwen2-7B. The Reasoner is Qwen2-7B during RL training but Qwen3-30B-A3B at evaluation.

The paper reports gains on MMAU, MMSU, MMAR, and AIR-Bench, and proposes an open-ended setting in which the Perceiver does not see options, although the Reasoner still does for MMAU, MMSU, and MMAR.

Critical strengths

Perceiver training helps with the Reasoner held fixed. With the same Qwen3-30B Reasoner, MMAU test-mini rises from 65.77 for the untrained Perceiver plus Reasoner to 74.47 for full PERE in MCQ (Table 2). In the perceiver-blind setting, where baselines are also routed through the same Reasoner (Section 4.3), PERE gains 6.61 over the backbone on MMAU test-mini (58.86 to 65.47, Table 1) and still leads the next-best system on MMAU and MMSU (MMAU test-mini +5.00, MMAU test +2.74, MMSU average +3.28), though only by 0.34 points on MMAR.

Critical weaknesses

W1. The headline comparisons are not on equal terms The strength above holds within the paper's own controls, but the headline claims rest on different comparisons. In MCQ, Table 1 compares PERE (7B Perceiver plus 30B Reasoner) against direct 7B models using baseline numbers copied from papers and leaderboards, with no same-Reasoner rerun and no latency or cost reported. On AIR-Bench, the judge is the same Qwen3-30B checkpoint that writes PERE's final answers, while baselines appear to be judged on raw outputs: Qwen2.5-Omni-7B scores 7.79 alone, but 8.06 when used untrained as a Perceiver with the Reasoner (Table 2), about half of PERE's +0.53 gain to 8.32. The "open-ended" setting still gives options to the Reasoner for MMAU, MMSU, and MMAR (Section 4.3), so only AIR-Bench is open-ended at the system level. As reported, the abstract's state-of-the-art and realistic open-ended claims therefore do not rest on equal-terms comparisons.

W2. The claimed mechanism, query-oriented and faithful perception, is not verified

The dominant reward (weight 0.7) is final-answer correctness (Section 3.3), and no grounding check is reported for whether descriptions are faithful to the audio, so a description that steers the Reasoner could score well without being accurate perception. The Perceiver is given options throughout SFT and RL (Section 4.2) and is rewarded for Reasoner correctness, so it may learn to encode the answer. Withholding options improves MMSU from 63.16 in MCQ to 64.14 in the perceiver-blind setting (Table 1), which is consistent with that risk. Aspect labels are generated by a teacher that sees the query, options, CoT, and ground-truth answer (Section 3.1) and are never checked against human judgment. The aspect-reward ablation in Table 2 mostly favors keeping it, but MMAU test rises (+0.47) without it, and as single runs the effect sizes cannot be judged. Without a faithfulness check and a leakage probe, the gain in the strength above could reflect Reasoner-friendly writing rather than better perception, which keeps the recommendation below acceptance.

Questions for authors

On W1:

  1. In Table 1's AIR-Bench column, are baseline outputs judged raw or after the Reasoner?
  2. Please rerun the Table 1 MCQ baselines with the same Qwen3-30B Reasoner, prompts, and scoring, and add a fine-tuned Qwen2.5-Omni-7B plus Reasoner row. This would separate the contribution of the PERE training from that of the Reasoner and the training data.
  3. What are the latency and tokens per query for PERE compared with the direct models?
  4. Will the abstract and contributions describe the MMAU/MMSU/MMAR setting as perceiver-blind rather than open-ended, or can you add an evaluation in which the Reasoner also has no options?

On W2:

  1. On a sample of descriptions, how often do they contain claims not supported by the audio (pitch, emotion, speaker traits, lexical content)? A human or independent-model grounding check would directly support the faithfulness claim.
  2. How often do descriptions generated with options visible contain the correct option's text versus the distractors? Does accuracy hold if options are shuffled or replaced after the Perceiver runs? Why does withholding options improve MMSU?
  3. How well do the teacher's aspect labels agree with human annotation, and can you report variance across seeds for the reward ablations?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the detailed feedback and constructive questions. Below, we provide detailed responses to each of your questions and suggestions.


Q1. In Table 1's AIR-Bench column, are baseline outputs judged raw or after the Reasoner?

A1. The baseline outputs are first fed into the same Reasoner used by PERE, and the resulting responses are then evaluated by the LLM judge. We have clarified this evaluation procedure in Section 4.3 of the revised manuscript.


Q2. Please rerun the Table 1 MCQ baselines with the same Qwen3-30B Reasoner, prompts, and scoring, and add a fine-tuned Qwen2.5-Omni-7B plus Reasoner row. This would separate the contribution of the PERE training from that of the Reasoner and the training data.

A2. Thank you for this valuable suggestion. We have rerun all MCQ baselines using exactly the same Qwen3-30B Reasoner, prompts, and scoring protocol as PERE to ensure a fair comparison. We have also added a new baseline, fine-tuned Qwen2.5-Omni-7B + Reasoner, as suggested. The results are reported in the table below.

ModelMMAU Test-miniMMAU TestMMSU Per.MMSU Rea.MMSU AvgMMAR
Audio-Reasoner58.5658.6334.6364.9249.3739.46
Audio Flamingo 364.8661.3343.6571.3257.0853.40
Baichuan-Omni67.5763.5036.0169.1552.1249.66
DIFFA61.5660.1740.8568.2754.1343.54
GAMA42.9439.4328.7139.9034.1531.97
Kimi-Audio69.9763.3743.0175.2158.6448.98
Mellow40.2442.8226.0849.5937.5338.44
Qwen2-Audio-Instruct57.3645.8730.4657.4643.5335.71
Qwen2.5-Omni-3B61.5659.5741.0166.4753.3946.94
Qwen2.5-Omni-7B65.7765.8745.5974.3659.5850.34
SALMONN50.1550.1327.4851.9239.3136.39
Qwen2.5-Omni-7B (Fine-tuned)70.8767.7745.7473.5159.2152.72
PERE (Ours)74.4769.7348.0180.3663.1664.93

As shown in the table, PERE consistently and substantially outperforms all baselines across all benchmarks. These results demonstrate the effectiveness of PERE's training approach and suggest that its performance gains cannot be attributed solely to the choice of Reasoner or training data.


Q3. What are the latency and tokens per query for PERE compared with the direct models?

A3. Thank you for raising this important point. To assess the inference efficiency of PERE, we conducted additional comparisons with an end-to-end direct model and a modular baseline. The modular baseline uses Qwen3-Captioner to generate a query-agnostic caption, which is then fed to the same Reasoner used in PERE. All measurements are conducted on a single NVIDIA H100 GPU with a batch size of 1 using all MMAU samples.

Latency (Seconds / Query)

MethodPerceiverReasonerTotal
Qwen2.5-Omni-7B----0.17
Generic Captions + Reasoner20.280.4220.70
PERE (Ours)1.280.291.57

Tokens / Query

MethodPerceiverReasonerTotal
Qwen2.5-Omni-7B----5.53
Generic Captions + Reasoner508.477.32515.79
PERE (Ours)99.505.74105.24

The end-to-end baseline is the most efficient, requiring only 0.17 s and 5.53 tokens per query, as it generates answers in a single decoding pass. Compared with the query-agnostic baseline, PERE is substantially more efficient: Qwen3-Captioner generates verbose captions averaging 508.47 tokens, resulting in a Perceiver latency of 20.28 s/query, whereas PERE generates concise, query-oriented descriptions averaging only 99.50 tokens, reducing Perceiver latency to 1.28 s/query. The shorter descriptions also reduce Reasoner latency from 0.42 s to 0.29 s. These results demonstrate that PERE achieves a favorable efficiency trade-off compared with the query-agnostic modular pipeline, although it remains slower than the end-to-end baseline.

We have added these results and the corresponding discussion to the appendix of the revised manuscript.


Continued in the next comment...

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Q4. Will the abstract and contributions describe the MMAU/MMSU/MMAR setting as perceiver-blind rather than open-ended, or can you add an evaluation in which the Reasoner also has no options?

A4. We sincerely thank the reviewer for this valuable suggestion. We agree that the original use of the term open-ended was imprecise. Accordingly, we have renamed the original open-ended setting as the perceiver-blind setting throughout the revised manuscript.

Furthermore, following the reviewer's suggestion, we have conducted additional experiments under a strictly open-ended setting, in which answer options are withheld from both the Perceiver and the Reasoner. The system generates free-form responses, which are evaluated by an LLM judge. The experimental results have been added in the revised manuscript, and also presented in the table below.

ModelMMAU Test-miniMMSU Per.MMSU Rea.MMSU AvgMMARAIR-Bench
Audio-Reasoner5.414.626.795.675.236.54
Audio Flamingo 35.114.776.565.645.107.35
Baichuan-Omni5.064.326.085.184.987.27
DIFFA4.283.433.273.353.625.94
GAMA2.272.112.442.272.774.22
Kimi-Audio5.074.305.995.124.826.81
Mellow2.483.002.292.663.022.51
Qwen2-Audio-Instruct4.484.505.975.214.307.61
Qwen2.5-Omni-3B4.944.286.355.284.867.27
Qwen2.5-Omni-7B5.684.756.455.575.097.79
SALMONN1.561.322.101.702.433.00
PERE (Ours)6.205.006.985.965.628.32

With these revisions, all experiments now referred to as open-ended in the revised manuscript follow this strict protocol, where neither the Perceiver nor the Reasoner has access to answer options. We sincerely appreciate the reviewer’s feedback, which helped us clarify our evaluation protocols and strengthen the experimental evaluation.


Q5. On a sample of descriptions, how often do they contain claims not supported by the audio (pitch, emotion, speaker traits, lexical content)? A human or independent-model grounding check would directly support the faithfulness claim.

A5. Thank you for this helpful suggestion. To assess whether the generated descriptions contain claims that are not supported by the input audio, we conducted both human and model-based evaluations of their factual accuracy. Specifically, we randomly sampled 10 audio clips from the benchmarks and generated descriptions for each using both Qwen2.5-Omni-7B and PERE. Ten human annotators independently rated the resulting 20 descriptions on a 1–5 scale, with higher scores indicating greater factual accuracy with respect to the corresponding audio. Importantly, the evaluation focused exclusively on the accuracy of the described content, rather than its comprehensiveness. Thus, descriptions containing unsupported claims about aspects such as pitch, emotion, speaker traits, or lexical content would receive lower scores. Furthermore, we conducted a larger-scale evaluation using Qwen3-Omni-30B-A3B-Instruct to assess descriptions across all benchmarks under the same criterion.

Human Evaluation

ModelHuman Evaluation
Qwen2.5-Omni-7B3.26
PERE (Ours)3.98

Model-based Evaluation

ModelMMAUMMSUMMARAIR-Bench
Qwen2.5-Omni-7B3.833.733.524.40
PERE (Ours)4.054.074.154.54

As shown in the results, PERE consistently achieves higher description accuracy than Qwen2.5-Omni-7B in both human and model-based evaluations. These findings provide evidence that PERE generates descriptions that are more faithful to the input audio and less prone to including unsupported claims.

We have added the above results and analysis to the revised manuscript.


Q6. How often do descriptions generated with options visible contain the correct option's text versus the distractors? Does accuracy hold if options are shuffled or replaced after the Perceiver runs? Why does withholding options improve MMSU?

A6. Firstly, we examined the frequency with which descriptions generated under the MCQ setting contain text from the correct answer option. Note that the options are provided to the Perceiver as a list of their textual contents, rather than as letter labels (A/B/C/D); therefore, the Perceiver cannot directly output these labels (see the Perceiver prompt in the Appendix for details). Accordingly, we measure whether the generated descriptions contain text corresponding to the correct answer option. The results are reported below.


Continued in the next comment...

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

MMAU Test-miniMMSU Per.MMSU Rea.MMSU AvgMMAR
PERE16.5%21.0%7.9%14.7%17.0%

In MMSU, the perception subset evaluates what the model can directly hear from speech, such as basic acoustic and linguistic cues, whereas the reasoning subset evaluates what it can further infer from those perceived cues. As shown in the table, the probability of containing the answer text is strongly dependent on the nature of the query. When the query is perception-oriented, the description of the perceived content is more likely to contain text from the correct option. For example, this occurs in 21.0% of cases on the perception subset of MMSU. In contrast, when the query primarily involves reasoning, this probability decreases substantially, reaching only 7.9% on the reasoning subset of MMSU.

We also hope to draw the reviewer’s attention to the case studies in the Appendix for concrete examples of the descriptions generated by the Perceiver. As illustrated by these examples, the Perceiver's descriptions primarily provide perceptual information from selected aspects of the audio, rather than attempting to directly answer the query. When a query concerns information that can be directly obtained through perception, the description may naturally contain text corresponding to the correct answer. Otherwise, there is no need for the Perceiver to explicitly encode the answer in its description, as the subsequent, more capable Reasoner is responsible for performing the necessary reasoning based on the provided perceptual evidence.

Secondly, to investigate the sensitivity of PERE to option ordering, we conducted an additional experiment in which the options were randomly shuffled for both the Perceiver and the Reasoner. The results reported in the below table show negligible performance differences compared with the original setting, indicating that PERE's performance is robust to changes in option ordering and does not rely on a particular option arrangement.

MMAU Test-miniMMSU Per.MMSU Rea.MMSU AvgMMAR
Original74.4748.0180.3663.1664.93
Shuffled74.1747.9879.8963.1464.97

Finally, on MMSU, PERE’s average rises slightly from 63.16% (MCQ) to 64.14% (perceiver-blind). We attribute this to reduced option-induced perception bias: when options are visible to the Perceiver, it may latch onto lexical cues in the choices and produce option-conditioned descriptions rather than speech-grounded evidence. This interference is especially pronounced on MMSU, where options are often semantically close and thus easily confused. Withholding options forces query-conditioned listening and yields cleaner descriptions. We do not claim this gain is universal (MMAU/MMAR drop under the same protocol); it indicates that option cues can slightly hurt perception quality on MMSU.


Q7. Aspect labels are generated by a teacher that sees the query, options, CoT, and ground-truth answer (Section 3.1) and are never checked against human judgment. How well do the teacher's aspect labels agree with human annotation, and can you report variance across seeds for the reward ablations?

A7. Thank you for raising this important concern. We randomly sampled 10 test instances from the benchmarks and invited 10 human annotators to independently identify the aspects that need to be perceived for each instance. We then calculated the agreement between human annotations and the teacher-generated labels, obtaining an agreement rate of 82.2%.

In addition, we repeated the reward ablation experiments in the Table 2 of the manuscript using four different random seeds to assess the variability of the results. The variance across the four runs is reported below. Specifically, w/o R_fmt denotes removing the format reward during QRPO training, while w/o R_asp and w/o R_rsp denote removing the aspect selection reward and response reward, respectively.

AblationMMAU Test-miniMMSUMMARAIR-Bench
QRPO w/o R_fmt0.0820.0140.1830.068
QRPO w/o R_asp0.1370.0110.1160.123
QRPO w/o R_rsp0.2100.0140.0290.116

We sincerely thank the reviewer for your time and expertise in providing comments to improve our manuscript. We have tried our best to address all concerns and suggestions to improve our manuscript. Thank you!

Reviewer_XUxKOpenReview ↗

Authors addressed concerns

Authors have addressed my concern. I will be changing the rating.

Reviewer_coG9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PERE, a speech understanding framework that separates perception from downstream reasoning. A Perceiver selects query-relevant speech aspects and generates a textual description, which a frozen language model uses to answer the question. Training combines perception and answer-generation objectives during SFT, followed by QRPO, a GRPO-based procedure with format, aspect-selection, and downstream answer rewards. The authors evaluate the approach on MMAU, MMSU, MMAR, and AIR-Bench, with additional experiments on different Perceivers, Reasoners, and training components.

Critical strengths
  1. The modular interface has practical value. Generating evidence relevant to a question is a sensible alternative to producing a generic audio caption. The experiments with different Reasoners in Table 6 support the usefulness of keeping the reasoning component replaceable. The failure cases are also informative: an explicit description makes it possible to identify missing perceptual information that leads to an incorrect answer.

  2. The empirical study covers several meaningful settings. The paper evaluates three Perceiver backbones and includes ablations of the training stages, reward components, and data construction choices. The gains over the default Qwen2.5-Omni-7B backbone are encouraging, particularly on MMSU and MMAR, where MCQ accuracy improves from 60.57 to 63.16 and from 59.86 to 64.93, respectively. These results give the proposed training pipeline practical interest, even though the source of the improvements needs further investigation.

Critical weaknesses
  1. The open-ended evaluation does not fully support the claim made in the abstract and conclusion. Section 4.3 states that, on MMAU, MMSU, and MMAR, answer options are withheld from the Perceiver but still supplied to the Reasoner. The final task therefore remains multiple choice. This is a useful test of whether the Perceiver can produce helpful descriptions without seeing the options, and using the same Reasoner for the baselines is a reasonable comparison. However, it does not establish performance when candidate answers are unavailable to the system as a whole. AIR-Bench provides evidence for that setting, but the distinction is lost in the broader claims. Appendix B.3 further states that the options are removed from the Reasoner prompt during open-ended evaluation, leaving the actual protocol unclear.

  2. The experiments do not sufficiently isolate the contribution of the proposed aspect selection or establish the claimed improvement in perception. The closely related RAPID/VPO approach of Gou et al., which the paper cites, already uses query-relevant descriptions, an external text Reasoner, and downstream answer correctness to optimize perception. The speech-specific aspect representation and training design are therefore important to the contribution here. Yet removing the aspect reward still retains aspect generation, and the fine-tuned baseline in Table 2 is not described well enough to determine whether it receives comparable RL training.

    The description-quality analysis leaves a related question unresolved. Training descriptions are derived from answer-conditioned CoTs and retained according to whether they allow a model to recover the correct answer. Appendix G then evaluates description quality through downstream answer accuracy. This establishes usefulness for answering questions, but does not independently establish faithfulness to the audio. I would find the method considerably more convincing if the authors could show that its gains reflect better extraction of acoustic evidence, rather than descriptions that are simply easier for the Reasoner to map to an answer.

Questions for authors
  1. Evaluation protocol. For each of the three MCQ benchmarks, does the Reasoner receive answer options in the reported open-ended experiments? Please reconcile Section 4.3 with Appendix B.3. If the options are retained, the claims about fully open-ended understanding should be narrowed accordingly.

  2. Contribution of the training design. What training does the “Fine-tuned” baseline in Table 2 receive, including its SFT and RL data and budgets? Which comparison best isolates the benefit of explicit aspect selection over query-conditioned description generation with a downstream correctness reward?

  3. Reported rewards and ablations. Under Eq. (12), the three reward components lie in [0,1] and their weights sum to one, but the Total Reward curve in Figure 5 exceeds one. Is the figure showing an unweighted sum or a different quantity? Also, which AIR-Bench results are correct for the two SFT variants: 8.12/8.13 in Table 2 or 8.22/8.23 in Table 3? These discrepancies affect the interpretation of the training results.

  4. Description faithfulness. Is there evidence that the generated descriptions are more accurate with respect to the audio, independent of whether the Reasoner answers correctly? A small listening-based audit or an audio-shuffling control would help address this concern; a new benchmark is not necessary.

The reported improvements are promising, but the evaluation scope and the evidence for the proposed mechanism leave me below the acceptance threshold. I would reconsider this recommendation if the response clarifies the protocol and reporting discrepancies and provides stronger support for the speech-specific contribution.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the detailed feedback and constructive questions. Below, we provide detailed responses to each of your questions and suggestions.


Q1. For each of the three MCQ benchmarks, does the Reasoner receive answer options in the reported open-ended experiments? Please reconcile Section 4.3 with Appendix B.3. If the options are retained, the claims about fully open-ended understanding should be narrowed accordingly.

A1. Thank you for pointing out this ambiguity. We apologize for the imprecise terminology in the original manuscript.

In the original open-ended experiments on the three MCQ benchmarks, the Perceiver does not have access to the answer options, whereas the Reasoner does receive them. This design allows us to fairly compare with previous methods and evaluate the Perceiver's option-unaware perception capability while retaining accuracy as an objective and intuitive evaluation metric.

However, we agree that describing this setting as open-ended is not entirely accurate, since the Reasoner still has access to the answer options. To address this concern, we have made two changes in the revised manuscript:

1. Terminology clarification: We have renamed the original "open-ended setting" to the "perceiver-blind setting" as suggested by Reviewer XUxK, explicitly indicating that the Perceiver cannot access the answer options, while the Reasoner can.

2. Additional fully open-ended experiments: We have conducted additional experiments under a genuinely open-ended setting, in which neither the Perceiver nor the Reasoner has access to the answer options. Following the evaluation protocol used in AIR-Bench, we evaluate the system's free-form responses using an LLM judge, which assigns a score ranging from 1 to 10.

The results under the fully open-ended setting are summarized in the table below.

ModelMMAU Test-miniMMSU Per.MMSU Rea.MMSU AvgMMARAIR-Bench
Audio-Reasoner5.414.626.795.675.236.54
Audio Flamingo 35.114.776.565.645.107.35
Baichuan-Omni5.064.326.085.184.987.27
DIFFA4.283.433.273.353.625.94
GAMA2.272.112.442.272.774.22
Kimi-Audio5.074.305.995.124.826.81
Mellow2.483.002.292.663.022.51
Qwen2-Audio-Instruct4.484.505.975.214.307.61
Qwen2.5-Omni-3B4.944.286.355.284.867.27
Qwen2.5-Omni-7B5.684.756.455.575.097.79
SALMONN1.561.322.101.702.433.00
PERE (Ours)6.205.006.985.965.628.32

We have added these in the table 1 of the updated paper. We have also revised Sections 4.3 and Appendix B.3 to ensure consistency.


Q2. What training does the “Fine-tuned” baseline in Table 2 receive, including its SFT and RL data and budgets? Which comparison best isolates the benefit of explicit aspect selection over query-conditioned description generation with a downstream correctness reward?

A2. The “Fine-tuned” baseline in Table 2 is trained to directly generate answers using both D^weak_PERE and D^strong_PERE, with the same number of training update steps as PERE. No reinforcement learning is involved in this variant. This controlled setting is designed to rule out the potential effect of additional training data used in PERE. We have revised the main text to provide a clearer description of this variant.

To better isolate the benefit of explicit aspect selection over query-conditioned description generation with a downstream correctness reward, we conducted an additional ablation experiment with the following modifications. Specifically, during the dual-objective SFT stage, we removed the aspect-related output from the perception-only SFT objective, requiring the model to generate only a speech description without explicitly specifying the aspect from which the description is derived. We denote this variant as Dual-objective SFT w/o asp. Starting from the resulting checkpoint, we then performed QRPO training without the aspect selection reward. The experimental results are summarized in the table below.

VariantMMAU Test-miniMMAU TestMMSU Per.MMSU Rea.MMSU AvgMMARAIR-Bench
Qwen2.5-Omni-7B + Reasoner65.7765.8745.5974.3659.5950.348.06
 + Dual-objective SFT w/o asp71.4769.1046.4979.0962.2857.828.21
  + QRPO w/o R_asp72.6769.6746.3679.3062.3062.248.25
Qwen2.5-Omni-7B + Reasoner65.7765.8745.5974.3659.5950.348.06
 + Dual-objective SFT71.1769.3347.0178.3662.1659.528.23
  + QRPO (PERE)74.4769.7348.0180.3663.1664.938.32

Continued in the next comment...

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

The experimental results show that removing explicit aspect selection has little impact on performance after SFT. However, it leads to noticeable performance degradation following RL training, with scores dropping from 74.47 to 72.67 on MMAU Test-mini, from 63.16 to 62.30 on MMSU, and from 64.93 to 62.24 on MMAR. These results suggest that explicit aspect selection provides useful intermediate supervision that can be leveraged by the aspect selection reward during RL optimization, thereby enabling more targeted optimization of the model's query-conditioned speech perception.

We have incorporated these additional experiments and the corresponding analysis into the revised manuscript.


Q3. Under Eq. (12), the three reward components lie in [0,1] and their weights sum to one, but the Total Reward curve in Figure 5 exceeds one. Is the figure showing an unweighted sum or a different quantity? Also, which AIR-Bench results are correct for the two SFT variants: 8.12/8.13 in Table 2 or 8.22/8.23 in Table 3? These discrepancies affect the interpretation of the training results.

A3. Thank you for pointing out these inconsistencies. Regarding Figure 5, the plotted Total Reward is calculated as the unweighted sum of the three individual reward components. Therefore, its value can exceed 1. We have clarified the figure description accordingly in the updated version.

Regarding the AIR-Bench results of the two SFT variants, the correct scores are 8.22 and 8.23, as reported in Table 3. The values of 8.12 and 8.13 in Table 2 were due to typographical errors, which have now been corrected. We confirm that this correction does not affect our experimental analysis or the conclusions drawn from these results.


Q4. Is there evidence that the generated descriptions are more accurate with respect to the audio, independent of whether the Reasoner answers correctly? A small listening-based audit or an audio-shuffling control would help address this concern; a new benchmark is not necessary.

A4. Thank you for this helpful suggestion. To evaluate the audio faithfulness of the generated descriptions independently of the Reasoner's answer accuracy, we conducted both human and multimodal model-based evaluations.

For the human evaluation, we randomly sampled 10 audio clips from the benchmarks and generated descriptions for each clip using both Qwen2.5-Omni-7B and PERE, resulting in 20 descriptions in total. We invited 10 human annotators to independently rate each description on a 1–5 scale, with higher scores indicating greater accuracy with respect to the audio. Importantly, the evaluation focuses exclusively on the factual accuracy of the described content, rather than its comprehensiveness, allowing us to assess description faithfulness without conflating it with coverage. In addition, we conducted a multimodal model-based evaluation, using Qwen3-Omni-30B-A3B-Instruct to assess descriptions for all samples across the benchmarks under the same evaluation criterion. The results are presented in the tables below.

Human Evaluation

ModelHuman Evaluation
Qwen2.5-Omni-7B3.26
PERE (Ours)3.98

Model-based Evaluation

ModelMMAUMMSUMMARAIR-Bench
Qwen2.5-Omni-7B3.833.733.524.40
PERE (Ours)4.054.074.154.54

As shown, PERE consistently achieves higher description accuracy than the Qwen2.5-Omni-7B baseline in both human and multimodal model-based evaluations. These findings provide independent evidence that PERE generates descriptions that are more faithful to the input audio, rather than merely producing descriptions that are simply easier for the Reasoner to map to an answer.

We have added the above results and analysis to the revised manuscript.


We sincerely thank the reviewer for your time and expertise in providing comments to improve our manuscript. We have tried our best to address all concerns and suggestions to improve our manuscript. Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Summary of Updates

We would like to thank all reviewers for providing careful reviews and constructive feedback. In response to the reviewers' valuable comments, we have made the following updates to the manuscript:

1. Clearer evaluation protocol (Section 4.3, Section 5.1, Table 1): We rename the original open-ended setting as perceiver-blind setting and introduce additional experiments under a strictly open-ended setting, in which candidate answer options are withheld from the entire system.

2. Ablation of QRPO without dual-objective SFT (Section 5.2, Table 2): We add a variant that applies QRPO directly to the base Perceiver, skipping dual-objective SFT stage. This variant measures how much that SFT stage contributes before QRPO.

3. Ablation of explicit aspect selection (Section 5.2, Table 2): To evaluate the effectiveness of explicit aspect selection, we add a path that removes aspect outputs from dual-objective SFT, so the model writes only a speech description, and then runs QRPO without the aspect-selection reward R_asp.

4. Unified MCQ comparison (Appendix C): To separate PERE’s training from the choice of Reasoner and training data, we re-evaluate all MCQ baselines with the same Reasoner, prompts, and scoring as PERE. The comparison also includes Qwen2.5-Omni-7B fine-tuned on the same data.

5. Description faithfulness (Appendix H): Beyond answer accuracy, we evaluate whether generated descriptions are faithful to the audio via both human and multimodal model-based evaluations.

6. Latency and tokens per query (Appendix J): We compare PERE's inference latency and token consumption per query against those of an end-to-end SLM and a query-agnostic modular baseline.

7. Clarification of the fine-tuned baseline (Section 5.2): We clarify that the fine-tuned implicit-perception baseline in Table 2 is trained to answer directly on both D^weak_PERE and D^strong_PERE, with the same number of update steps as PERE and without RL.

8. Related work clarification (Section 2.2): We broaden the discussion of explicit speech perception to encompass both cascaded and in-model description generation, and introduce RAPID as a vision-domain counterpart.

9. Minor corrections (Table 2, Figure 5): We correct a typo in Table 2 and clarify the plotting settings of the reward curves in Figure 5.

We hope the revised manuscript could help address the reviewers' concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.