- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces OmniOPSD, a rationale-privileged on-policy self-distillation framework for multimodal affective understanding, covering emotion recognition, sentiment analysis, and intent recognition. Existing task annotations guide frontier models to generate multimodal descriptions of affective cues offline. These descriptions are paired with reference answers to form privileged context available only to the teacher during distillation. The student generates responses from the original multimodal inputs and task instructions, while a local teacher sharing its architecture and updated through an exponential moving average provides dense next-token distributions along the same student-generated prefixes. This design allows the student to learn along its own evolving affective reasoning trajectories, with the teacher supplying fine-grained, rationale-conditioned supervision. The distillation stage requires neither frontier-model logits nor online frontier-model calls, and inference uses only the student with the original task input. Experiments on MER-UniBench and MME-Emotion, together with controlled comparisons against SFT and GRPO at the 3B and 7B scales, provide evidence of the framework’s effectiveness across in-domain and out-of-domain affective tasks.
Critical strengths
-
A well-motivated framework for dense supervision of multimodal affective reasoning. OmniOPSD makes a clear methodological contribution by using multimodal affective rationales as teacher-only privileged information to supervise the student’s own reasoning trajectories. The distillation stage requires neither frontier-model logits nor online frontier-model queries. Sharing the architecture and tokenizer between the local teacher and student avoids the logit-access and tokenizer-alignment barriers of direct frontier-model distillation. EMA smooths the teacher parameters, while the reported time-to-target comparison demonstrates faster local training than OPD in the evaluated configuration.
-
Broad experimental coverage and informative analyses support the core design choices. The experiments examine both the effectiveness of OmniOPSD and how its supervision operates on affective reasoning.
-
The Label-only versus Label + Rationale comparison is particularly persuasive. Both teachers receive the correct answer, allowing the comparison to assess the additional value of affective evidence beyond answer conditioning alone.
-
This paper is clearly-written and well-organized.
Critical weaknesses
While the paper reports competitive recognition results across multiple benchmarks, the evaluation of reasoning quality relies primarily on automated metrics and model-based judgments. A systematic, independent assessment by affective computing experts would strengthen the comparisons with SFT, GRPO, and representative open- and closed-source models. In particular, the current evidence leaves some uncertainty about whether the performance gains are accompanied by better interpretation of auditory and visual cues, cross-modal integration, and affective inference.
Questions for authors
-
Evidence of improved cue interpretation. Beyond the token-level diagnostics, are expert evaluations or other independent assessments available showing that OmniOPSD reduces errors in visual and acoustic cue descriptions relative to SFT, GRPO, or Label-only OPSD?
-
Training comparability and data separation. Could the authors clarify the differences between the main-experiment and ablation protocols, particularly the training data, initialization, and training budgets? Which factors are controlled within each comparison, and what checks were performed to identify and address overlapping or near-duplicate samples across training and evaluation sources?
-
Uncertainty in the ablation results. Are results from multiple training seeds available for the key comparisons, particularly Label-only versus Label+Rationale and OmniOPSD versus GRPO?
-
Scope of the efficiency comparison. For Figure 4, could the authors specify the OPD teacher model and size, hardware allocation, batch and sequence settings, and timing boundaries? In particular, were comparable compute resources used, and does the 5.6-hour versus 17.7-hour comparison exclude offline rationale generation? These details are needed to interpret the reported approximately 3.2× speedup.
6 comments
Q3: Multi-Seed Results and Uncertainty
A3 (Multi-Seed Reliability). We assess variation across training seeds through repeated runs of the main method comparison and the reasoning-mode ablation.
Main method comparison (Table 3).
| Method | ID Avg. | OOD Avg. |
|---|---|---|
| SFT | 39.89 ± 0.83 | 23.27 ± 0.51 |
| GRPO | 38.16 ± 1.05 | 23.56 ± 0.20 |
| OmniOPSD | 40.67 ± 0.64 | 24.67 ± 0.08 |
OmniOPSD achieves the highest mean on both ID and OOD evaluations. Compared with GRPO, the improvements are 2.51 points on ID and 1.11 points on OOD; compared with SFT, they are 0.78 and 1.40 points, respectively.
Reasoning-mode comparison (Table 4).
| Method | Reasoning mode | ID Avg. | OOD Avg. |
|---|---|---|---|
| GRPO | w/o Think | 38.55 ± 1.45 | 23.67 ± 0.28 |
| GRPO | w/ Think | 38.20 ± 1.12 | 23.10 ± 0.46 |
| OmniOPSD | w/o Think | 39.13 ± 2.02 | 23.57 ± 0.80 |
| OmniOPSD | w/ Think | 40.62 ± 0.58 | 24.66 ± 0.08 |
With reasoning enabled, OmniOPSD exceeds GRPO by 2.42 points on ID and 1.56 points on OOD. Within OmniOPSD, enabling reasoning improves the corresponding means by 1.49 and 1.09 points, respectively.
These multi-seed results are consistent with the experimental trends and conclusions reported in the paper and further suggest that OmniOPSD’s advantage is reproducible across independent training runs.
Q4: Scope and Configuration of the Efficiency Comparison
A4 (Efficiency Comparison). The approximately 3.2× speedup in Figure 4 concerns post-training wall-clock time to the reported target score, with rationales cached in advance. The student-side GPU allocation is matched, while the total GPU allocation differs because OPD uses an external teacher service.
Models and resource allocation. Both runs start from Qwen2.5-Omni-3B Base and use the same student-side settings, including four 80 GB GPUs and a per-GPU batch size of 1. OPD obtains top-64 logits from Qwen3-Omni-30B-A3B-Instruct, deployed on four additional 80 GB GPUs.
| Configuration | OmniOPSD | OPD |
|---|---|---|
| Initial student | Qwen2.5-Omni-3B Base | Qwen2.5-Omni-3B Base |
| Teacher | EMA self-teacher | Qwen3-Omni-30B-A3B-Instruct |
| Student-side GPU allocation | 4 × 80 GB | 4 × 80 GB |
| Additional teacher-service GPUs | None | 4 × 80 GB |
| Total GPU allocation | 4 GPUs | 8 GPUs |
| Per-GPU student batch size | 1 | 1 |
Target score and timing boundaries. OmniOPSD first reaches the OPD step-1,000 reference score at step 550; both scores round to 0.546 macro-F1. The corresponding plotted times are 5.6 hours for OmniOPSD and 17.7 hours for OPD, giving a wall-clock time ratio of .
Offline rationale generation is excluded: OmniOPSD begins the measured post-training stage with the rationales already cached.
Figure 4 therefore supports a reduction in measured post-training time to the reported target score under the stated resource allocations.
I thank the authors for their response and additional results, e.g., the explanation-level evaluation and the results from multiple independent training runs provide valuable evidence, as well as the clarification of Figure 4, which substantially improves the interpretability of the efficiency claim. I keep my positive ratings, and want the authors to further solve my following concerns:
- Although the additional results further support the effectiveness of OmniOPSD, the multi-seed comparison between Label-only and Label + Rationale that I requested is still missing.
- While the ratings address my primary concern, further details of the rating procedure are needed: the background and number of the raters, whether they were blind to method identity.
We thank the reviewer for the constructive feedback. Our responses are as follows.
Q1: Evaluation of Multimodal Cue Interpretation and Affective Reasoning
A1 (Multimodal Interpretation and Reasoning). To assess multimodal evidence interpretation beyond recognition performance and token-level diagnostics, we directly evaluate the explanations generated by OmniOPSD, SFT, and GRPO on the same videos.
Evaluation protocol. We sample 40 videos from each of the eight MME-Emotion tasks, yielding 320 videos. Each explanation is assessed on six criteria: audio coverage, audio factual accuracy, visual coverage, visual factual accuracy, cross-modal cue integration, and reasoning linkage, which measures how well the cited evidence supports the affective conclusion. Each criterion is rated on a 0–2 scale, with higher scores indicating better performance.
For each explanation, we additionally compute the audio and visual cue scores as the minimum of the corresponding coverage and factual accuracy scores:
\qquad s_{\mathrm{visual}} = \min(c_{\mathrm{visual}}, a_{\mathrm{visual}}).$$ This rule requires both sufficient coverage and factual correctness for a high cue score. We compute the minimum for each explanation before aggregation and report results as mean ± standard deviation. | Dimension | OmniOPSD | SFT (7B) | GRPO (7B) | |---|---:|---:|---:| | Audio coverage | **1.54 ± 0.55** | 1.40 ± 0.57 | 1.36 ± 0.59 | | Audio factual accuracy | **1.28 ± 0.74** | 1.02 ± 0.78 | 1.13 ± 0.77 | | Audio cue score | **1.19 ± 0.70** | 0.90 ± 0.71 | 0.98 ± 0.70 | | Visual coverage | **1.50 ± 0.53** | 1.26 ± 0.60 | 1.13 ± 0.64 | | Visual factual accuracy | 1.14 ± 0.75 | 1.13 ± 0.78 | **1.22 ± 0.75** | | Visual cue score | **1.05 ± 0.71** | 0.91 ± 0.71 | 0.89 ± 0.68 | | Cross-modal cue integration | **1.25 ± 0.62** | 1.04 ± 0.61 | 1.07 ± 0.56 | | Reasoning linkage | **0.93 ± 0.73** | 0.78 ± 0.66 | 0.85 ± 0.64 | **Results and interpretation.** OmniOPSD achieves the highest mean in **seven of the eight dimensions**, including both derived cue scores. Audio factual accuracy improves by 0.26 over SFT and 0.15 over GRPO. The audio and visual cue scores also improve over both baselines, indicating better joint coverage and accuracy under the minimum-based scoring rule. Higher cross-modal integration and reasoning-linkage scores further support more effective use of multimodal evidence in the generated explanations. **Q2: Training Protocol Comparability and Train–Evaluation Data Separation** **A2 (Training Comparability and Data Separation).** We distinguish the main benchmark evaluation from the controlled comparisons, as these experiments use different training protocols and serve different purposes. **Main benchmark evaluation.** As described in Appendix E.2, the training data comprise the training splits of MELD, MIntRec 2.0, IEMOCAP, MC-EIU, CH-SIMS_v2_s, MAFW, MER2023, DFEW, CAER, BoLD, CMU-MOSI, and CMU-MOSEI. We first perform SFT cold-start training at a learning rate of $10^{-5}$. We then initialize OmniOPSD from the **SFT step-1,000 checkpoint** and train it for **one epoch, corresponding to 2,181 optimizer steps**, at a learning rate of $10^{-6}$. We evaluate the same resulting 7B checkpoint on both MER-UniBench and MME-Emotion. **Controlled comparisons.** As described in Appendix E.1, the controlled experiments examine both 3B and 7B models trained jointly on MELD, MIntRec 2.0, and IEMOCAP, using a common learning rate of $10^{-6}$. These experiments use the released training splits of MELD and MIntRec 2.0 for training and their test splits for evaluation. For IEMOCAP, Sessions 1–4 are used for training, while Session 5 is held out for evaluation. **Train–evaluation separation.** Both MER-UniBench and MME-Emotion construct their evaluation sets from the test or validation partitions of existing public datasets, whereas our training uses the training partitions. MER-UniBench follows the evaluation subsets listed in Appendix D.2. We also account for known overlap across dataset releases when selecting the training sources. Specifically, the MER2024 training set contains examples from the MER2023 test set that are included in MER-UniBench and MME-Emotion. **We therefore exclude MER2024 entirely from training** to prevent these evaluation examples from entering the training data. With these split assignments and dataset exclusions, the training samples and the reported evaluation samples are disjoint.