acceptodds
Under review as a conference paper at ICLR 2027

OVGER: Evidence-Grounded Open-Vocabulary Group Emotion Recognition via Blind Annotation and Progressive Reward Injection

Abstract

Group Emotion Recognition (GER) infers collective affect from multi-person audio-visual recordings for social-interaction analysis, multimedia understanding, and human-centered computing. Most existing GER systems formulate the task as closed-set classification over coarse affect categories. Although effective for benchmark comparison, this discriminative interface provides limited fine-grained affect description and little explicit evidence for the prediction. Motivated by this gap, we formulate open-vocabulary GER as a benchmark-compatible extension beyond conventional closed-set prediction and instantiate it with OVGER. The model retains the coarse group-affect decision while adding more specific free-form emotion descriptors and an evidence-linked structured response. The central contribution is the supervision-and-reward interface that makes these outputs jointly trainable and auditable: OV-VGAF augments the Video-level Group AFfect (VGAF) benchmark with label-blind modality observations, structured supervision, and hidden evidence references for evaluating coarse labels together with open-vocabulary affect and evidence alignment. This design couples benchmark-compatible group decisions with fine-grained affect descriptions whose supporting multimodal cues can be inspected. Extensive experiments show competitive coarse-label performance relative to representative task-specific GER systems and clear gains over the evaluated zero-shot MLLMs on the same 766-clip validation split, reaching 74.67% accuracy, 74.50% macro-F1, and 85.51% open-vocabulary emotion soft-F1.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
group emotion recognition, multimodal large language models, open-vocabulary emotion recognition, affective computing, evidence-grounded reasoning, reinforcement learning, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aYGrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes OVGER for open-vocabulary group emotion recognition using Qwen2.5-Omni-7B. It combines automatically generated multimodal annotations, supervised fine-tuning, and two-stage GRPO with rewards for classification, formatting, evidence alignment, and consistency. Experiments on the VGAF validation set report 74.67% accuracy, 74.50% macro-F1, and 85.51% open-vocabulary emotion soft-F1.

Critical strengths
  1. The method extends conventional three-class group emotion recognition with fine-grained emotion descriptors and evidence-linked textual outputs while retaining compatibility with the original VGAF labels.
  2. The comparison of progressive and fixed GRPO reward schedules under 200 optimizer updates provides initial evidence that the ordering of reward components can affect recognition performance.
Critical weaknesses
  1. The method combines established SFT, GRPO, semantic evidence matching, structured generation, and reward scheduling. AffectGPT (ICML'25) already addresses open-vocabulary multimodal emotion understanding. More importantly, OPPO (ICML'26) introduces multimodal emotion reasoning with explicit visual and acoustic evidence matching and hallucination control. RewardMap (ICLR'26) also studies multi-stage reinforcement learning with specialized rewards. OVGER does not sufficiently distinguish its mathematical or algorithmic contributions from these methods.
  2. Section 3.2 and Appendix A describe a fully automated annotation pipeline in which the fusion annotator receives the ground-truth coarse label and generates both training targets and hidden evaluation references. The verifier is another language model rather than an independent human annotator. The paper provides no quantitative human assessment of annotation accuracy, evidence correctness, or inter-annotator agreement. Consequently, the reliability of the proposed OV-VGAF annotations remains unestablished.
  3. Section 3.4 and Appendix D use BGE-M3 semantic matching for reward computation and evaluation. The evidence-grounding and consistency diagnostics directly reuse the corresponding reward definitions. The reported 85.51% open-vocabulary emotion soft-F1 also measures agreement with automatically generated reference emotions. These metrics therefore do not independently establish emotional accuracy or factual grounding in the original recordings.
  4. Table 2 shows that evidence grounding decreases from 88.82% after SFT to 85.74% in the final model. More importantly, the forbidden-claim rate increases from 4.44% to 18.02%. Thus, the final model performs substantially worse on these diagnostics despite introducing evidence-specific rewards. The experiments do not establish that progressive GRPO improves evidence grounding relative to the SFT checkpoint.
  5. Equations 10 and 11 compute each evidence-bank score using only the three highest reference matches. A model can therefore receive the maximum bank score while ignoring most reference evidence items. The matching is also many-to-one, allowing one generated sentence to satisfy multiple references. This formulation does not measure comprehensive evidence coverage and can reward selective reproduction of annotation text.
  6. The proposed consistency reward in Equations 13–15 measures semantic agreement between generated text sections and polarity anchors. It cannot determine whether a visual or acoustic statement is supported by the corresponding input modality. Internally consistent but factually incorrect explanations can receive high scores. Unlike OPPO (ICML'26), the paper provides no modality-masking or counterfactual evaluation to establish whether generated explanations depend on the evidence they describe.
  7. Table 1 compares the fully trained OVGER primarily against zero-shot MLLMs. The resulting 10.44-point accuracy improvement over AffectGPT does not isolate the proposed reward design from the effects of task-specific fine-tuning and additional supervision. A Qwen2.5-Omni-7B baseline trained directly on VGAF labels with comparable supervision and optimization is missing. The experiments do not establish the added value of the proposed framework over conventional supervised adaptation.
  8. Table 3 removes either the observe or think section but does not evaluate an answer-only training target. The reported changes are also relatively small, particularly the 0.13-point accuracy difference when removing observe. The experiments therefore do not establish that the complete observe-think-answer structure is necessary for recognition performance or that its intermediate reasoning contributes beyond additional supervised text.
  9. Table 4 compares S1-to-S1, S2-to-S2, and S1-to-S2 schedules using one seed. It omits a constant reward mixture with matched average weights and the reversed S2-to-S1 schedule. The observed 2.22-point improvement over S1-to-S1 does not isolate the effect of reward ordering from the overall reward composition. The result supports a particular training configuration but not a general advantage of the proposed progression.
  10. The semantic similarity thresholds, evidence-bank coefficients, top-3 aggregation, forbidden-claim penalties, and polarity-anchor scaling are manually specified without systematic validation. Equation 7 additionally applies a second nonlinear transformation to the already normalized emotion-matching score. Appendix C reports a sensitivity study for an earlier objective containing a temporal reward that is absent from the final method. It does not establish the robustness of the final reward configuration.
  11. The method produces free-form emotion descriptors, but the evaluation does not establish recognition of previously unseen emotion concepts. There is no analysis of vocabulary diversity, unseen descriptors, semantic granularity, or generalization beyond the automatically generated reference distributions. The results demonstrate semantic agreement with reference phrases rather than a verified ability to recognize novel fine-grained group emotions.
  12. All experiments use a single VGAF validation split of 766 clips and one reported random seed. The paper provides no confidence intervals, repeated training runs, paired significance tests, or independent cross-dataset evaluation. The reported 2.22-point advantage of progressive GRPO corresponds to approximately 17 additional correct predictions. Without uncertainty estimates, the stability and generality of this improvement remain unclear.
  13. Appendix D confirms that Base/SFT and GRPO evaluations use different prompt formulations. Tables 1 and 6 also include results obtained with lenient parsers, including an audio-only accuracy of 57.83% compared with a strict-parser accuracy of 31.46%. These differences complicate interpretation of baseline and modality comparisons. Moreover, the complete annotation records, reward references, and implementation code are not available for independent verification at submission time.
  14. Figure 4 labels annotation examples as human-verified, whereas its caption describes automatically verified records and Appendix A identifies an automated verifier. The subsequent Appendix G descriptions also do not match the displayed examples. The positive example is described as a three-person talk show rather than the crowd shown in Figure 4, while the neutral example describes children with a toy stroller rather than the displayed panel discussion. These inconsistencies undermine confidence in the documented annotation cases. Although the text in Figures 1 and 2 remains readable, both figures contain numerous visual artifacts and graphical inconsistencies suggestive of AI-generated illustrations. The figures should be recreated as clean, consistent vector graphics rather than presented as raster images.
Questions for authors
  1. Can the authors provide independent human evaluations of factual evidence grounding for the SFT and final GRPO checkpoints, and explain why the forbidden-claim rate increases from 4.44% to 18.02% after reinforcement learning?
  2. Can the authors compare OVGER's evidence reward with OPPO (ICML'26) under matched training conditions and quantify the additional benefit of top-3 evidence matching and forbidden-claim penalties?
  3. Can the authors report results for Qwen2.5-Omni-7B trained directly on VGAF labels and for an answer-only SFT model, using the same clips, modalities, and comparable training budgets as OVGER?
  4. Can the authors evaluate a constant reward mixture with matched average weights and a reversed S2-to-S1 schedule across multiple random seeds, reporting statistical uncertainty for the observed improvements?
  5. Can the authors provide human-validated annotation quality measurements, including evidence correctness and agreement between independent annotators, and clarify whether the examples labeled as human-verified in Figure 4 underwent actual human review?
  6. Can the authors provide controlled audio and video masking or swapping experiments to establish whether the generated evidence statements actually depend on the corresponding input modalities rather than learned textual correlations?
  7. Can the authors evaluate recognition of previously unseen fine-grained emotion concepts using independently annotated references and report whether OVGER improves semantic accuracy rather than merely reproducing the vocabulary of its generated supervision?
  8. Can the authors recompute all baseline, training-stage, and modality comparisons using identical prompts and strict parsing rules, including the audio-only setting where reported accuracy changes from 57.83% to 31.46%?
  9. Can the authors clarify whether the 766 validation clips, their generated evidence references, or their evaluation diagnostics were used to select reward coefficients, training schedules, prompts, or checkpoints, and identify which results were obtained without such selection?
Reviewer_pFrDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies open-vocabulary group emotion recognition. It retains the three VGAF emotion classes while adding fine-grained emotion descriptions and a structured observe–think–answer response. To construct OV-VGAF, separate audio and visual observers first collect cues without access to the target label. A fusion model then uses these observations and the coarse label to generate training targets and evaluation references. The authors train Qwen2.5-Omni-7B through supervised fine-tuning followed by two GRPO stages, first optimizing answer correctness and format, then introducing evidence and consistency rewards. The final model achieves 74.67% accuracy and 74.50% macro-F1 on 766 validation clips. The experiments examine training stages, reward components, response structure, input modalities, and reward schedules under the same number of optimizer updates.

Critical strengths
  1. The proposed output format is a useful extension of conventional group emotion recognition. It preserves the coarse labels needed for existing benchmark evaluation while adding emotion descriptions and observations that readers can inspect. Collecting audio and visual observations before exposing the fusion model to the target label also reduces the risk of constructing the initial observations around a known answer. Together, the dataset and training procedure provide a concrete starting point for studying more descriptive group emotion recognition.

  2. The benefit of the progressive reward schedule is supported by controlled comparisons. In Table 4, all three schedules use 200 actual optimizer updates. The progressive schedule reaches 74.67% accuracy, compared with 72.45% for continued S1 training and 71.41% for using the full S2 reward from the start. These results support the value of reward ordering in this setting. The component ablations also show that the evidence and consistency rewards affect their corresponding diagnostics.

Critical weaknesses
  1. The correctness of the fine-grained emotion descriptions and evidence statements has not been sufficiently evaluated independently of the annotation and reward pipeline. The emotion references, evidence banks, and forbidden claims are generated automatically, and the fusion stage has access to the coarse label. Table 7 lists the fused record as the verifier’s input, without clearly stating whether the verifier also checks the original audio and video. Appendix D further states that the open-vocabulary and evidence metrics reuse the BGE-M3 model and matching rules used in training. The authors appropriately describe these as reference-agreement diagnostics. However, the scores alone cannot establish whether the outputs describe the clips more accurately or mainly follow the wording and judgments of the automatic references. This matters because the additional descriptions and evidence statements are central to the paper’s contribution.

  2. The final model improves classification accuracy over SFT, but its evidence-related diagnostics become worse. In Table 2, accuracy increases from 69.32% to 74.67%, while the evidence score falls from 88.82 to 85.74 and the Forbidden metric rises from 4.44% to 18.02%. The latter measures semantic matches to predefined forbidden claims and should not be interpreted as a human-verified error rate. Nevertheless, the direction of the change needs discussion. Table 5 shows that the evidence reward mitigates the problem within the RL procedure, but it does not explain the deterioration relative to SFT. The missing S1 diagnostics also make it difficult to locate when this change occurs. A comparison only against the base model leaves this trade-off between classification performance and explanation quality insufficiently examined.

Questions for authors
  1. Figure 4 uses the label “human-verified annotation,” whereas the annotation procedure mainly describes automatic verification. What human checks were performed? Were they limited to the illustrated examples, and did the annotators inspect the original audio and video? If random samples were checked, please report the findings for the fine-grained emotions and evidence statements.

  2. How do you explain the lower evidence score and higher Forbidden metric of the final model relative to SFT? Could you provide the corresponding S1 diagnostics and examine whether the changes arise from longer responses, different wording, or additional unsupported interpretations?

  3. The evidence reward uses the highest-matching reference items and allows one generated sentence to match multiple references. How well does it detect responses that contain some correct cues alongside unsupported details? The sanity check in Appendix A changes only the coarse label. Do you have checks that keep the answer fixed while modifying the evidence, negation, or temporal order?

Reviewer_oM6sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OVGER, extending closed-set Group Emotion Recognition (GER) into an open-vocabulary, evidence-grounded generation task. Given an audio-visual clip, the model outputs a structured <observe>-<think>-<answer> response that pairs the standard coarse label with fine-grained emotion descriptors and multimodal rationales. To prevent label leakage during annotation, the authors construct OV-VGAF (3,427 clips) by combining label-blind audio/visual observations with label-conditioned fusion to generate training targets and hidden reward references. Built on Qwen2.5-Omni-7B, OVGER is trained via LoRA SFT followed by two-stage progressive GRPO, optimizing answer and format rewards before injecting BGE-M3-based evidence and consistency rewards. On the VGAF validation split, OVGER achieves 74.67% accuracy and 85.51% open-vocabulary soft-F1, outperforming zero-shot MLLMs and matching task-specific baselines.

Critical strengths
  1. It is easy to read and follow. The main text and supplementary materials provide sufficient methodological and experimental details.

  2. This paper reformulates closed-set GER into a structured <observe>-<think>-<answer> task that maintains compatibility with standard coarse-label benchmarks while adding fine-grained emotion descriptors and checkable rationales. In addition, it extends VGAF benchmark to OV-VGAF dataset.

  3. This paper provides an effective training pipeline. It adapts structured reasoning SFT and progressive GRPO post-training to audio-visual GER, with controlled equal-step ablations verifying the benefit of the two-stage reward schedule.

Critical weaknesses
  1. Marginal Gains over Task-Specific Baselines and Asymmetric MLLM Comparisons: In Table 1, the coarse-grained accuracy of OVGER (74.67%) offers only a marginal improvement (+0.39%) over traditional task-specific GER methods such as Hybrid Networks (Liu et al., 2020, 74.28%). While OVGER substantially outperforms the evaluated MLLM baselines (e.g., AffectGPT at 64.23% and Emotion-LLaMA at 62.27%), all competing MLLMs are evaluated strictly in a zero-shot setting without in-domain adaptation. Comparing a model that has undergone extensive in-domain SFT and GRPO post-training against zero-shot MLLMs is an asymmetric and unfair comparison, which overstates the empirical advantage of the proposed framework.

  2. Circular Evaluation and Lack of Human-Verified Ground Truth in OV-VGAF: Although the authors construct the OV-VGAF benchmark extension, the validation reference fields—including fine_emotions_ref, multimodal evidence banks, and forbidden claims—are entirely synthesized by automated LLM pipelines (gemini-3.1-flash-lite, qwen3.5-397b-a17b, and gpt-5.4) without independent human annotation or expert verification. Crucially, BGE-M3 cosine similarity against these synthetic references serves both as the optimization reward during GRPO training and as the evaluation metric (OVE soft-F1, Evidence, and Consistency) on the validation split. Because the training objective and evaluation metric share the exact same synthetic source and embedding model, the evaluation suffers from reward-metric collusion (circularity). Consequently, high scores in OVE (85.51%) and Evidence (85.74%) merely reflect how well Qwen2.5-Omni-7B distills the preferences of gpt-5.4 and BGE-M3, rather than demonstrating faithful, human-validated evidence grounding.

  3. Degradation of Evidence Grounding and Surge in Forbidden Claims during GRPO: Table 2 reveals a concerning trade-off between coarse-label accuracy and evidence faithfulness during reinforcement learning. While the SFT checkpoint achieves 88.82% on Evidence and a low Forbidden-claim rate of 4.44% (lower is better), applying two-stage GRPO (GRPO-S2) causes Evidence grounding to drop to 85.74% and the Forbidden-claim rate to surge more than fourfold to 18.02%. This regression strongly suggests that RL optimization sacrifices grounding fidelity and constraint compliance to chase higher coarse-grained classification accuracy (74.67%), indicative of potential reward hacking. Furthermore, the diagnostic metrics (Evidence, Consistency, and Forbidden) for the intermediate GRPO-S1 checkpoint are omitted in both Table 2 and Table 4, obscuring where this degradation originates.

  4. Missing Answer-Only Fine-Tuning Baseline in Ablation Studies: In Table 3, the response-structure ablation only evaluates removing <think> (w/o think) or removing <observe> (w/o observe) individually. It lacks a fundamental baseline where the backbone (Qwen2.5-Omni-7B) is fine-tuned solely for coarse-grained classification (or <answer>-only generation) without the <observe>-<think>-<answer> scaffold. Without this direct discriminative/answer-only baseline, it is unclear whether the complex structured supervision pipeline genuinely improves coarse-level group affect recognition or merely maintains baseline performance while adding text generation.

  5. Limited Algorithmic Novelty and Over-Engineered Heuristic Rewards: From a methodological standpoint, the framework is primarily an engineering integration of established components—LoRA SFT, standard GRPO, and BGE-M3 embedding matching—offering limited core algorithmic novelty for the ICLR community. Moreover, the reward formulation in Section 3.4 relies heavily on ad-hoc heuristics and dozens of hard-coded hyperparameters (e.g., similarity clipping bounds [0.40,0.75][0.40, 0.75] in Eq. 3, fine-emotion compensation weight 0.30.3 and answer weights 0.8/0.20.8/0.2 in Eq. 7–8, six separate penalty coefficients in RformatR_{\text{format}} in Eq. 9, bank weights 0.35/0.25/0.25/0.150.35/0.25/0.25/0.15 and penalty thresholds [0.75,0.85][0.75, 0.85] in Eq. 11–12, and consistency sub-weights in Eq. 13–15). Such extensive rule-stitching raises serious concerns regarding hyperparameter sensitivity and generalizability to other benchmarks.

Questions for authors

1.Fair MLLM & Vanilla Fine-Tuning Baselines: What is the performance (Accuracy and Macro-F1) of Qwen2.5-Omni-7B on the 766-clip validation split when fine-tuned purely on the ground-truth coarse labels (i.e., without <observe>, <think>, or open-vocabulary descriptors)? Additionally, how do strong open-source MLLM baselines (such as InternVL3.5-8B or Qwen3-VL-8B) perform when fine-tuned on the exact same OV-VGAF SFT dataset?

  1. Missing GRPO-S1 Diagnostics and Forbidden-Claim Surge: Could the authors report the missing Evidence (%), Consistency (%), and Forbidden (%) metrics for the + GRPO-S1 row in Table 2 and the S1-only continuation row in Table 4? Furthermore, how do the authors explain the drastic increase in the Forbidden-claim rate from 4.44% (SFT) to 18.02% (GRPO-S2) alongside the drop in Evidence grounding (88.82% →\rightarrow 85.74%)?

  2. Human Evaluation to Break Circularity: Given that both the RL rewards and validation diagnostics rely on BGE-M3 matching against gpt-5.4-generated references, can the authors conduct a blind human evaluation on a statistically meaningful subset of the validation clips to independently verify (a) the accuracy of the generated open-vocabulary emotions and (b) the factual grounding of <observe>/<think> claims, and report the correlation between human judgments and the automated metrics?

  3. Justification and Sensitivity of Reward Hyperparameters: Section 3.4 defines over 20 distinct thresholds, clipping intervals, and component weights across Equations (3)–(17) (while Appendix C only tests three values of an unused temporal weight). How were these specific hyperparameter values determined, and how sensitive is the progressive GRPO training to variations in these heuristic thresholds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.