- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether audio language models can improve spoken question answering while preserving emotion recognition. It proposes ReMG, which groups recordings with two different contents and two emotions, and tests both tasks on each recording. The authors build training and test sets using synthesized questions and human emotional speech, and fine-tune models through content and emotion distillation. Experiments evaluate existing models, compare fine-tuning methods, and test generalization to other benchmarks.
Critical strengths
-
Evaluating content understanding and emotion recognition on the same recordings is a useful diagnostic design. It reveals task-interference failures that separate evaluations may overlook.
-
The evaluation combines synthesized reasoning speech, human emotional recordings, and existing benchmarks, providing complementary evidence about model behavior.
Critical weaknesses
-
The presentation is hard to follow. The term “ReMG protocol” covers evaluation, the construction of training and test sets, and the OPD-based method. The contributions of these components should be explained separately. Figures 1 and 2 are also too dense and difficult to understand.
-
The role of quadruples (or the created dataset) in training is unclear. Equations (8)–(9) use per-example losses without explicitly using the relationships within each quadruple. It is unclear what quadruple pairing adds beyond training both tasks on each audio.
-
The source of the gains is not fully established. Table 4 shows that most gains appear after adding privileged teacher instructions. A comparison with vanilla OPD using the same instructions is missing.
-
The paired metrics need clarification. Equations (3)–(4) measure joint correctness, which should not exceed average accuracy. However, Table 2 reports 68.37% content invariability versus 52.27% content accuracy.
-
A smaller gap partly comes from worse text performance. Table 2 implies that Phi-4’s text accuracy drops by 6.26 points, explaining more than half of its 10.80-point gap reduction. Text accuracy should be reported directly.
-
The baseline comparisons have limits. Several important components of prior methods are removed or replaced. Results from these modified versions do not fully support the broad claims about the original methods.
Questions for authors
-
How are the relationships within each quadruple used during training? What changes in the training procedure when adding the “ReMG protocol” to vanilla OPD?
-
How does vanilla OPD perform with the same privileged teacher instructions? This comparison would help separate the effects of the protocol and teacher prompting.
-
Could you clarify how the paired metrics are calculated? Under Equations (3)–(4), how can content invariability reach 68.37% when content accuracy is only 52.27% in Table 2?
-
Could you report text accuracy before and after fine-tuning? How much of the gap reduction comes from improved audio accuracy, and how much comes from decreased text accuracy?
2 comments
Thank you again for reviewing our work. As the end of the discussion phase is approaching, we would appreciate hearing your feedback and knowing whether there are any remaining concerns :)
Response to Reviewer uwdQ
Thank you for your time in reviewing our manuscript, and also for valuing our diagnostic design in revealing previously overlooked failures and providing complementary evidence. We hope the following response could address your concerns.
W2&Q1: Eq. 8&9 use per-example losses without explicitly using the relationships within each quadruple. How does it work beyond training both tasks on each audio? What changes in the training procedure when adding the “ReMG protocol” to vanilla OPD?
Thanks for the question. It works by injecting the quadruple relationships as data priors across the whole fine-tuning set (Lines 95-102), since each audio there is constructed under the ReMG protocol and uniquely comes from a single quadruple. DNNs are not so weak that only when we explicitly write something into a loss function can it learn that. Instead, the existence of such a relationship within the data distribution is enough. Similarly, adding the "ReMG protocol" does not need to change the standard training procedure of vanilla OPD. Just like its literal meaning, everything that happens here is an additional "protocol" (i.e., the quadruple relationships under ReMG requirements) in data organization.
Of course, we are not to deny that an explicit quadruple-aware loss might be useful. Then why not? Well, this is directly tied to the positioning of our work. Our main contribution is the ReMG protocol itself, instead of any specific methods to realize it (Lines 484-485), while both per-example and quadruple-aware losses belong to "specific methods". To emphasize our positioning, we deliberately keep other specific realizations simple, while more advanced realizations might be better left for follow-up works (Lines 502-505).
W3&Q2: Table 4 shows that most gains appear after adding privileged teacher instructions. A comparison with vanilla OPD using the same instructions is missing.
Thanks for the comment. Please kindly refer to [S1 in the "Response to Shared Questions"] above for our response.
W1. The term “ReMG protocol” covers evaluation, the construction of training and test sets, and the OPD-based method. The contributions of these components should be explained separately.
Thank you for the comment. Following the above discussions (W2&Q1, W3&Q2), we can now respond to it better. Specifically, the “ReMG protocol” strictly refers to the crafted data prior (i.e., the 2x2 data-query organization in Fig. 1 and Sec 2.1), upon which the three downstream components are clearly separated: 1) constructing data that satisfy the ReMG protocol (Sec 2.2), 2) evaluating models on such data with basically standard metrics and their pair/unit variants (Sec 2.3), and 3) fine-tuning models on such data via basically vanilla OPD (Sec 3). The core contribution of our work is to propose the “ReMG protocol” itself, while all other designs demonstrate one specific realization of it. So we deliberately used one term to emphasize this. Still, thanks to your kind suggestion, we have reviewed all positions where the term appears and refined our presentation to eliminate potential barriers to understanding in the revised PDF (Lines 24, 26, 29, 116, 127, 128, 184, 309, 350, 411, 454, 470, 475).
W4&Q3: Eq. 3&4 measure joint correctness, which should not exceed average accuracy. How can content invariability reach 68.37% when content accuracy is only 52.27% in Table 2?
Thank you for carefully pointing this out. Please kindly refer to [S2 in the "Response to Shared Questions"] above for our response.
W5&Q4: A smaller gap partly comes from worse text performance. Could you report text accuracy before and after fine-tuning?
Thank you for the comment. Please kindly refer to [S3 in the "Response to Shared Questions"] above for our response.
W6: Several important components of prior methods are removed or replaced. Results from these modified versions do not fully support the broad claims about the original methods.
Thanks for the comment, but we respectfully disagree. Please kindly refer to [S4 in the "Response to Shared Questions"] above for our response.