- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes OVGER for open-vocabulary group emotion recognition using Qwen2.5-Omni-7B. It combines automatically generated multimodal annotations, supervised fine-tuning, and two-stage GRPO with rewards for classification, formatting, evidence alignment, and consistency. Experiments on the VGAF validation set report 74.67% accuracy, 74.50% macro-F1, and 85.51% open-vocabulary emotion soft-F1.
Critical strengths
- The method extends conventional three-class group emotion recognition with fine-grained emotion descriptors and evidence-linked textual outputs while retaining compatibility with the original VGAF labels.
- The comparison of progressive and fixed GRPO reward schedules under 200 optimizer updates provides initial evidence that the ordering of reward components can affect recognition performance.
Critical weaknesses
- The method combines established SFT, GRPO, semantic evidence matching, structured generation, and reward scheduling. AffectGPT (ICML'25) already addresses open-vocabulary multimodal emotion understanding. More importantly, OPPO (ICML'26) introduces multimodal emotion reasoning with explicit visual and acoustic evidence matching and hallucination control. RewardMap (ICLR'26) also studies multi-stage reinforcement learning with specialized rewards. OVGER does not sufficiently distinguish its mathematical or algorithmic contributions from these methods.
- Section 3.2 and Appendix A describe a fully automated annotation pipeline in which the fusion annotator receives the ground-truth coarse label and generates both training targets and hidden evaluation references. The verifier is another language model rather than an independent human annotator. The paper provides no quantitative human assessment of annotation accuracy, evidence correctness, or inter-annotator agreement. Consequently, the reliability of the proposed OV-VGAF annotations remains unestablished.
- Section 3.4 and Appendix D use BGE-M3 semantic matching for reward computation and evaluation. The evidence-grounding and consistency diagnostics directly reuse the corresponding reward definitions. The reported 85.51% open-vocabulary emotion soft-F1 also measures agreement with automatically generated reference emotions. These metrics therefore do not independently establish emotional accuracy or factual grounding in the original recordings.
- Table 2 shows that evidence grounding decreases from 88.82% after SFT to 85.74% in the final model. More importantly, the forbidden-claim rate increases from 4.44% to 18.02%. Thus, the final model performs substantially worse on these diagnostics despite introducing evidence-specific rewards. The experiments do not establish that progressive GRPO improves evidence grounding relative to the SFT checkpoint.
- Equations 10 and 11 compute each evidence-bank score using only the three highest reference matches. A model can therefore receive the maximum bank score while ignoring most reference evidence items. The matching is also many-to-one, allowing one generated sentence to satisfy multiple references. This formulation does not measure comprehensive evidence coverage and can reward selective reproduction of annotation text.
- The proposed consistency reward in Equations 13–15 measures semantic agreement between generated text sections and polarity anchors. It cannot determine whether a visual or acoustic statement is supported by the corresponding input modality. Internally consistent but factually incorrect explanations can receive high scores. Unlike OPPO (ICML'26), the paper provides no modality-masking or counterfactual evaluation to establish whether generated explanations depend on the evidence they describe.
- Table 1 compares the fully trained OVGER primarily against zero-shot MLLMs. The resulting 10.44-point accuracy improvement over AffectGPT does not isolate the proposed reward design from the effects of task-specific fine-tuning and additional supervision. A Qwen2.5-Omni-7B baseline trained directly on VGAF labels with comparable supervision and optimization is missing. The experiments do not establish the added value of the proposed framework over conventional supervised adaptation.
- Table 3 removes either the observe or think section but does not evaluate an answer-only training target. The reported changes are also relatively small, particularly the 0.13-point accuracy difference when removing observe. The experiments therefore do not establish that the complete observe-think-answer structure is necessary for recognition performance or that its intermediate reasoning contributes beyond additional supervised text.
- Table 4 compares S1-to-S1, S2-to-S2, and S1-to-S2 schedules using one seed. It omits a constant reward mixture with matched average weights and the reversed S2-to-S1 schedule. The observed 2.22-point improvement over S1-to-S1 does not isolate the effect of reward ordering from the overall reward composition. The result supports a particular training configuration but not a general advantage of the proposed progression.
- The semantic similarity thresholds, evidence-bank coefficients, top-3 aggregation, forbidden-claim penalties, and polarity-anchor scaling are manually specified without systematic validation. Equation 7 additionally applies a second nonlinear transformation to the already normalized emotion-matching score. Appendix C reports a sensitivity study for an earlier objective containing a temporal reward that is absent from the final method. It does not establish the robustness of the final reward configuration.
- The method produces free-form emotion descriptors, but the evaluation does not establish recognition of previously unseen emotion concepts. There is no analysis of vocabulary diversity, unseen descriptors, semantic granularity, or generalization beyond the automatically generated reference distributions. The results demonstrate semantic agreement with reference phrases rather than a verified ability to recognize novel fine-grained group emotions.
- All experiments use a single VGAF validation split of 766 clips and one reported random seed. The paper provides no confidence intervals, repeated training runs, paired significance tests, or independent cross-dataset evaluation. The reported 2.22-point advantage of progressive GRPO corresponds to approximately 17 additional correct predictions. Without uncertainty estimates, the stability and generality of this improvement remain unclear.
- Appendix D confirms that Base/SFT and GRPO evaluations use different prompt formulations. Tables 1 and 6 also include results obtained with lenient parsers, including an audio-only accuracy of 57.83% compared with a strict-parser accuracy of 31.46%. These differences complicate interpretation of baseline and modality comparisons. Moreover, the complete annotation records, reward references, and implementation code are not available for independent verification at submission time.
- Figure 4 labels annotation examples as human-verified, whereas its caption describes automatically verified records and Appendix A identifies an automated verifier. The subsequent Appendix G descriptions also do not match the displayed examples. The positive example is described as a three-person talk show rather than the crowd shown in Figure 4, while the neutral example describes children with a toy stroller rather than the displayed panel discussion. These inconsistencies undermine confidence in the documented annotation cases. Although the text in Figures 1 and 2 remains readable, both figures contain numerous visual artifacts and graphical inconsistencies suggestive of AI-generated illustrations. The figures should be recreated as clean, consistent vector graphics rather than presented as raster images.
Questions for authors
- Can the authors provide independent human evaluations of factual evidence grounding for the SFT and final GRPO checkpoints, and explain why the forbidden-claim rate increases from 4.44% to 18.02% after reinforcement learning?
- Can the authors compare OVGER's evidence reward with OPPO (ICML'26) under matched training conditions and quantify the additional benefit of top-3 evidence matching and forbidden-claim penalties?
- Can the authors report results for Qwen2.5-Omni-7B trained directly on VGAF labels and for an answer-only SFT model, using the same clips, modalities, and comparable training budgets as OVGER?
- Can the authors evaluate a constant reward mixture with matched average weights and a reversed S2-to-S1 schedule across multiple random seeds, reporting statistical uncertainty for the observed improvements?
- Can the authors provide human-validated annotation quality measurements, including evidence correctness and agreement between independent annotators, and clarify whether the examples labeled as human-verified in Figure 4 underwent actual human review?
- Can the authors provide controlled audio and video masking or swapping experiments to establish whether the generated evidence statements actually depend on the corresponding input modalities rather than learned textual correlations?
- Can the authors evaluate recognition of previously unseen fine-grained emotion concepts using independently annotated references and report whether OVGER improves semantic accuracy rather than merely reproducing the vocabulary of its generated supervision?
- Can the authors recompute all baseline, training-stage, and modality comparisons using identical prompts and strict parsing rules, including the audio-only setting where reported accuracy changes from 57.83% to 31.46%?
- Can the authors clarify whether the 766 validation clips, their generated evidence references, or their evaluation diagnostics were used to select reward coefficients, training schedules, prompts, or checkpoints, and identify which results were obtained without such selection?