- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PERE, a framework for speech question answering. It consists of a perceiver and a reasoner. The perceiver's input is speech query and optionally answer options. Its outputs are perception aspects and a description of evidence. The reasoner is an LLM that answers the question from the description. The framework is trained with a special RL recipe that is based on GRPO. The training reward consists of format validity, F1, and reasoner answer correctness. The framework is evaluated over MMAU, MMSU, MMAR, and AIR-Bench and gains over other speech LMs are reported.
Critical strengths
- The gains seem convincing especially on the large benchmark MMSU (Table 1).
- The authors provide enough ablation to show what matters in the framework. Table 2 includes multiple ways of producing the intermediate text and shows that a lot of other ways do not help or even hurts.
Critical weaknesses
- The framework is in essence a cascade: a model that first turns speech into text and then another model understands the text. The novelty is limited and the paper should be positioned among prior cascade works like Rao et al. “Do as I Mean, Not as I Say: Sequence Loss Training for Spoken Language Understanding.” which also trains a pipeline ASR model with RL using reward from downstream task accuracy.
- It's not shown that indeed the framework decouples perception from reasoning. The answer-preserving SFT objective also encourages the perceiver to answer the query, so it's less clear whether the perceiver's description already encodes the answer.
Questions for authors
- Have the authors checked whether the descriptions outputted by the perceiver are an honest description of the audio?
- What are the end-to-end latency of the proposed framework compared with a single speech language model?
- The “Qwen2.5-Omni-7B (Fine-tuned)” baseline in Table 2 is described only as being fine-tuned “on the same training data of PERE”. Was it trained with SFT only, or also with RL?
3 comments
Official Comment by Authors (2/3)
Q3. The “Qwen2.5-Omni-7B (Fine-tuned)” baseline in Table 2 is described only as being fine-tuned “on the same training data of PERE”. Was it trained with SFT only, or also with RL?
A3. The “Fine-tuned” baseline in Table 2 is trained with SFT only. No RL is involved in this variant. This variant is designed to rule out the potential effect of additional training data used in PERE. We have revised the main text to provide a clearer description of this variant.
Q4 (weakness 1). The framework is in essence a cascade: a model that first turns speech into text and then another model understands the text. The novelty is limited and the paper should be positioned among prior cascade works like Rao et al. “Do as I Mean, Not as I Say: Sequence Loss Training for Spoken Language Understanding.” which also trains a pipeline ASR model with RL using reward from downstream task accuracy.
A4. Thank you for this important comment. We agree that, at a high level, PERE can be viewed as a cascade framework that first converts speech into a textual representation and then uses a separate model for downstream understanding. In fact, our original manuscript already discusses existing cascade-based approaches, including explicit speech perception models, in Related Work (Section 2.2). As discussed there, a key limitation of most existing approaches is that their intermediate textual representations are query-agnostic, regardless of the downstream task. This observation motivates the central idea of PERE: query-oriented and reasoning-aware perception.
Compared with prior work such as Rao et al., which primarily focuses on speech content or intent for downstream spoken language understanding, PERE addresses a substantially broader perception problem. Understanding speech may require capturing not only linguistic content but also various paralinguistic and acoustic characteristics, such as emotion, prosody, rhythm, fluency, articulation, and speaker characteristics. More importantly, PERE investigates a fundamental question that is largely overlooked in conventional cascade pipelines: whether the speech-to-text mapping itself can be made query-oriented and reasoning-aware. Given a user query, PERE explicitly considers both what information should be perceived to answer the query (what to perceive) and how the relevant evidence should be described to facilitate downstream reasoning (how to describe).
Therefore, PERE goes beyond simply cascading a speech model with a downstream language model. Its key contribution lies in making the intermediate perception process explicitly conditioned on the user's query and the requirements of downstream reasoning. We have clarified this distinction and strengthened the discussion of related cascade-based approaches in Section 2.2 of the revised manuscript.
Continued in the next comment...
Official Comment by Authors (3/3)
Q5 (weakness 2). It's not shown that indeed the framework decouples perception from reasoning. The answer-preserving SFT objective also encourages the perceiver to answer the query, so it's less clear whether the perceiver's description already encodes the answer.
A5. Thank you for raising this important concern. We would like to clarify that the answer-preserving SFT objective serves only as a gentle initialization strategy. In the perception-only SFT and the subsequent QRPO stages, the Perceiver is explicitly trained to output only the relevant aspects and their descriptions.
We also note that whether a description contains information sufficient to identify the correct answer depends naturally on the nature of the query. For perception-oriented questions, the perceived audio content itself may directly provide the answer, and it is therefore reasonable for the description to contain information corresponding to the correct option. In contrast, for questions requiring more complex reasoning, the Perceiver is expected to provide relevant perceptual evidence, leaving the more challenging reasoning process to the Reasoner, which is generally a more powerful LLM with good text reasoning capability but lacking audio perception capability.
To further investigate whether the Perceiver's descriptions already encode the answers, we analyzed the frequency with which descriptions generated under the MCQ setting contain text from the correct answer option. Note that the options are provided to the Perceiver as a list of their textual contents, rather than as letter labels (A/B/C/D). Therefore, we measure whether the generated descriptions contain text corresponding to the correct answer option (see the Perceiver prompt in the Appendix for details). The results are reported below.
| MMAU Test-mini | MMSU Per. | MMSU Rea. | MMSU Avg | MMAR | |
|---|---|---|---|---|---|
| PERE | 16.5% | 21.0% | 7.9% | 14.7% | 17.0% |
In MMSU, the perception subset evaluates what the model can directly hear from speech, such as basic acoustic and linguistic cues, whereas the reasoning subset evaluates what it can further infer from those perceived cues. As shown in the table, the probability of containing the answer text varies substantially depending on the nature of the query. For perception-oriented questions, descriptions are more likely to contain text from the correct option, with this occurring in 21.0% of cases on the perception subset of MMSU. In contrast, for reasoning-oriented questions, this probability drops to only 7.9% on the reasoning subset of MMSU.
We also hope to draw the reviewer’s attention to the case studies in the Appendix for concrete examples of the descriptions generated by the Perceiver. As illustrated by these examples, the Perceiver's descriptions primarily provide perceptual information from selected aspects of the audio, rather than attempting to directly answer the query. When a query concerns information that can be directly obtained through perception, the description may naturally contain text corresponding to the correct answer. Otherwise, there is no need for the Perceiver to explicitly encode the answer in its description, as the subsequent, more capable Reasoner is responsible for performing the necessary reasoning based on the provided perceptual evidence.
We sincerely thank the reviewer for your time and expertise in providing comments to improve our manuscript. We have tried our best to address all concerns and suggestions to improve our manuscript. Thank you!
Official Comment by Authors (1/3)
We thank the reviewer for the detailed feedback and constructive questions. Below, we provide detailed responses to each of your questions and suggestions.
Q1. Have the authors checked whether the descriptions outputted by the perceiver are an honest description of the audio?
A1. Thank you for raising this important question. To verify whether the descriptions generated by the Perceiver faithfully reflect the actual audio content, we conducted both human and model-based evaluations. Specifically, we randomly sampled 10 audio clips from the benchmarks and generated descriptions for each using both Qwen2.5-Omni-7B and PERE, resulting in 20 descriptions in total. Ten human annotators independently rated these descriptions on a 1–5 scale, with higher scores indicating greater factual accuracy with respect to the corresponding audio. Importantly, the evaluation focused exclusively on the accuracy of the described content, rather than its comprehensiveness. In addition, we employed Qwen3-Omni-30B-A3B-Instruct to evaluate descriptions for all samples across the benchmarks using the same criterion.
Human Evaluation
Model-based Evaluation
As shown in the results, PERE consistently achieves higher description accuracy than Qwen2.5-Omni-7B in both human and model-based evaluations. These findings provide empirical evidence that the descriptions generated by PERE faithfully reflect the input audio content.
We have added these results and the corresponding discussion to the revised manuscript.
Q2. What is the end-to-end latency of the proposed framework compared with a single speech language model?
A2. To evaluate the end-to-end inference efficiency of PERE compared with a single speech language model, we conducted additional measurements of both inference latency and token consumption per query. We compared PERE with an end-to-end model and a modular baseline. The modular baseline uses Qwen3-Captioner to generate a query-agnostic caption, which is then fed to the same Reasoner used in PERE. All measurements are conducted on a single NVIDIA H100 GPU with a batch size of 1 using MMAU samples.
Latency (Seconds / Query)
Tokens / Query
The end-to-end baseline is the most efficient, requiring only 0.17 s and 5.53 tokens per query, as it generates answers in a single decoding pass. Compared with the query-agnostic baseline, PERE is substantially more efficient: Qwen3-Captioner generates verbose captions averaging 508.47 tokens, resulting in a Perceiver latency of 20.28 s/query, whereas PERE generates concise, query-oriented descriptions averaging only 99.50 tokens, reducing Perceiver latency to 1.28 s/query. The shorter descriptions also reduce Reasoner latency from 0.42 s to 0.29 s. These results demonstrate that PERE achieves a favorable efficiency trade-off compared with the query-agnostic modular pipeline, although it remains slower than the end-to-end baseline.
We have added these results and the corresponding discussion to the appendix of the revised manuscript.
Continued in the next comment...