- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper combines frame-by-frame AU supervision, keyframe visual reconstruction, and GRPO to enhance the use of facial evidence in open-vocabulary multimodal emotion recognition. The method has a clear motivation and the experiments are relatively complete.
Critical strengths
-
By supplementing textual supervision with AU prediction and continuous visual representation, a clear learning objective is provided for preserving fine-grained facial information.
-
Experiments cover multiple input modalities and backbone models, and provide a comparison with the same backbone to support the effectiveness of the method under different configurations.
-
The keyframe selection, AU granularity, and component ablation are presented systematically, but failure cases and methodological limitations are also reported.
Critical weaknesses
-
Since keyframes serve as both input and reconstruction targets, it is still difficult to distinguish whether the gains of ALR come from visual recoding or recursive computation. It is recommended to add a non-recursive comparison with the same budget.
-
The AU reward measures the consistency with OpenFace. It cannot yet prove that emotional judgment uses this evidence correctly. It is recommended to verify it by combining independent annotation and local action intervention.
-
OpenFace's AU estimate has been obtained during inference, but the model still needs to generate a complete AU sequence. It is recommended to compare it with directly inputting the same AU information to clarify the additional value of the generation step.
-
Comparisons of GRPO and continued SFT used different training data, and it is recommended to compare on the same MER2025-OV data to distinguish the contribution of the data and the optimization method.
Questions for authors
Please refer to the Critical Weaknesses section.
5 comments
Response to Reviewer qVaf(2/3)
Q2: Can independent annotation and local action intervention verify whether emotional judgment uses the AU evidence correctly?
A2: Adding the L1 AU reward improves emotional judgment: the nine-dataset mean rises by 2.74 points (77.55 to 80.29) and OV-MERD+ by 1.05 points (62.47 to 63.52; Table 3(a)). This comparison evaluates emotion predictions while fixing the SFT checkpoint, training data, visual inputs, ALR length and all other reward terms.
| Setting | Sentiment | Basic | OV-MERD+ | Mean |
|---|---|---|---|---|
| (a) AU reward after GRPO | ||||
| None | 84.61 | 74.27 | 62.47 | 77.55 |
| L1 reward | 86.89 | 77.89 | 63.52 | 80.29 |
| L2 reward | 84.41 | 73.98 | 62.16 | 77.30 |
For the independent annotation you requested, human judgments show that ALR raises facial-claim support from 77.6% to 86.0% and visible-event coverage from 53.0% to 60.6% (Table 15). AURORA and its matched no-ALR control use identical AU supervision, rewards, training data and visual inputs. We sampled 100 OV-MERD+ clips within four visibility strata before inspecting explanations, then froze both models' explanations for the same clips and original eight frames. Two trained coders independently inventoried visible facial events and their supporting frames before assessing randomized atomic claims. Model identities, detector traces, emotion predictions, audio and subtitles were hidden; subject and temporal qualifiers were retained, and disagreements were adjudicated. This creates a visual reference independent of OpenFace's training annotations.
\hline \text{Model} & \text{Claims} & S/(S+C) & U/\mathrm{claims} & \text{Unique matches / events} \\\\ \hline \rlap{\text{Overall comparison}} & & & & \\\\ \text{AURORA} & 362 & 283/329\\;(86.0\\%) & 33/362\\;(9.1\\%) & 255/421\\;(60.6\\%) \\\\ \text{AURORA w/o ALR} & 357 & 250/322\\;(77.6\\%) & 35/357\\;(9.8\\%) & 223/421\\;(53.0\\%) \\\\ \hline \rlap{\text{Clear: 30 clips and 143 visible events}} & & & & \\\\ \text{AURORA} & 111 & 100/109\\;(91.7\\%) & 2/111\\;(1.8\\%) & 94/143\\;(65.7\\%) \\\\ \text{AURORA w/o ALR} & 111 & 94/109\\;(86.2\\%) & 2/111\\;(1.8\\%) & 87/143\\;(60.8\\%) \\\\ \hline \rlap{\text{Occlusion: 30 clips and 128 visible events}} & & & & \\\\ \text{AURORA} & 109 & 84/98\\;(85.7\\%) & 11/109\\;(10.1\\%) & 76/128\\;(59.4\\%) \\\\ \text{AURORA w/o ALR} & 107 & 73/97\\;(75.3\\%) & 10/107\\;(9.3\\%) & 64/128\\;(50.0\\%) \\\\ \hline \rlap{\text{Large pose: 20 clips and 86 visible events}} & & & & \\\\ \text{AURORA} & 72 & 55/65\\;(84.6\\%) & 7/72\\;(9.7\\%) & 50/86\\;(58.1\\%) \\\\ \text{AURORA w/o ALR} & 72 & 49/64\\;(76.6\\%) & 8/72\\;(11.1\\%) & 44/86\\;(51.2\\%) \\\\ \hline \rlap{\text{Both: 20 clips and 64 visible events}} & & & & \\\\ \text{AURORA} & 70 & 44/57\\;(77.2\\%) & 13/70\\;(18.6\\%) & 35/64\\;(54.7\\%) \\\\ \text{AURORA w/o ALR} & 67 & 34/52\\;(65.4\\%) & 15/67\\;(22.4\\%) & 28/64\\;(43.8\\%) \\\\ \hline \end{array}$$ Here, $S$, $C$ and $U$ denote supported, contradicted and unjudgeable claims. Claims equals $S+C+U$; each pair of model rows shares the same clips and visible-event inventory. Overall, AURORA supports 283/329 judgeable claims versus 250/322 for the control, and covers 255/421 distinct visible events versus 223/421. At nearly identical description volumes of 3.62 and 3.57 claims per clip, it correctly describes 32 additional events with only five additional claims. Support and coverage improve across all four visibility strata. Table 3(a) establishes the recognition gain from the AU reward; Table 15 establishes the gain from ALR in describing independently annotated facial actions. These comparisons evaluate emotion-recognition accuracy and the accuracy and coverage of the facial evidence reported in explanations. The results reported for local facial actions are independent annotations and observations of the original videos. The frame-based analyses let you compare the local facial actions named in each explanation with the corresponding video frames. Appendix F.1, Fig. 3 shows the paired OpenFace/LibreFace AU activation patterns. Appendix I.1–I.4, Figs. 4–7 present `sample_00001433`, `sample_00001175`, `sample_00001334` and `sample_00002448`, respectively, with eight timestamped frames and model-response excerpts. In `sample_00001175` (Appendix I.2, Fig. 5), the visible smile under upper-face occlusion accompanies AURORA's prediction of joy and satisfaction, consistent with the reference emotions. The timestamped smile and the corresponding emotion prediction provide a concrete action-level example of the relationship assessed by the recognition and annotation results above.Response to Reviewer qVaf(3/3)
Q3: What is the additional value of generating a complete AU sequence over directly inputting the same AU information?
A3: Training AURORA to generate the complete AU sequence yields a 1.47-point mean-score advantage after SFT and a 1.68-point advantage after RL over training it to receive the same OpenFace AU information directly, with a final 1.58-point OV-MERD+ advantage (Table 21). Both models start from the same Qwen2.5-VL-7B backbone and share training clips, original eight-frame inputs, key frames, 32-step ALR, reconstruction weight, optimizer settings, batch/group sizes and update counts. Full AURORA learns to generate the complete AU sequence using OpenFace estimates as supervision; the direct-input model receives those estimates in its prompt throughout SFT, RL and evaluation. It retains rationale/answer and visual-reconstruction supervision, removes AU-output supervision and AU reward, and adapts the format reward to the remaining output structure; other reward coefficients are unchanged. Each RL run starts from its own SFT model and uses that model as its KL reference. Both are evaluated on the same complete benchmark splits with deterministic decoding.
| Stage | Model | ↑ | Mean ↑ | Latency (s) ↓ |
|---|---|---|---|---|
| SFT | Full AURORA | 62.37 | 77.65 | 16.73 |
| Direct AU input | 60.46 | 76.18 | 10.04 | |
| SFT+RL | Full AURORA | 63.52 | 80.29 | 16.28 |
| Direct AU input | 61.94 | 78.61 | 9.68 |
Latency is measured on OV-MERD+ using one 80GB H800, batch size 1 and a warmed-up runtime. Each clip includes video/face preprocessing, one OpenFace pass, prompt processing and complete response generation. Direct input reduces final median latency by 40.5%. Thus, directly inputting the same AU information improves inference speed, while learning to generate the complete sequence achieves higher emotion-recognition scores at both training stages. Because the direct-input alternative is trained with these estimates and retains the same visual inputs and ALR architecture, the comparison measures the added value of the AU-generation training objective and its AU reward alongside their decoding cost.
Q4: What do the training data and optimization method each contribute when GRPO and continued SFT use the same MER2025-OV data?
A4: GRPO reaches 80.29 versus 78.08 for continued SFT on the same MER2025-OV clips, a 2.21-point advantage under the same 96 GPU-hour budget (Table 10). Both start from the same AURORA-SFT checkpoint at 77.65, exclude overlap with OV-MERD+ and retain the same inference architecture and 32 latent steps. Each additional run uses eight 80GB H800 GPUs for 12 hours, including rollout generation for GRPO. Table 10 also includes continued SFT on the original MER-Caption+ data under the same additional budget.
| Additional optimization | Training data | GPU-hours | Mean | Gain over SFT |
|---|---|---|---|---|
| None | — | 0 | 77.65 | — |
| Continued SFT | MER-Caption+ | 96 | 77.82 | +0.17 |
| Continued SFT | MER2025-OV | 96 | 78.08 | +0.43 |
| GRPO with AU-aware reward | MER2025-OV | 96 | 80.29 | +2.64 |
With continued SFT and the additional budget fixed, changing its training data from MER-Caption+ to MER2025-OV raises the mean from 77.82 to 78.08, a 0.26-point increase. With the MER2025-OV clips fixed, continued SFT improves over the shared checkpoint by 0.43 points and GRPO by 2.64 points. Their 2.21-point difference measures the benefit of GRPO with AU-aware rewards at matched data and computation. These two comparisons separately quantify the change in training data and the change in optimization you asked about.
Response to Reviewer qVaf(1/3)
Thank you for your suggestions on separating visual recoding from recursive computation and examining how AU information contributes to emotional judgment. In response, we added a non-recursive comparison with the same budget, independent human annotations and video observations of local facial actions, a trained direct-AU-input model with recognition and timing measurements, and continued SFT on the same MER2025-OV data as GRPO. We discuss these results in relation to your four concerns below.
A1: With the same reconstruction loss and computation budget, recursive ALR scores 77.65 versus 75.73 for the non-recursive comparison, a 1.92-point advantage (Table 8). The non-recursive comparison directly appends the same frozen key-frame features at 32 additional decoder positions; we call this patch replay in Section 5.4. At position ℓ, it receives patch feature kℓ, whereas ALR receives the preceding context-conditioned hidden state hℓ−1. Here, “non-recursive” specifies the absence of this hidden-state input to the next position: both methods retain full causal attention and their KV caches. Both also retain the original key-frame input, Qwen2.5-VL-7B backbone, multimodal context, AU/text supervision and training budget. When enabled, reconstruction applies the same loss to their decoder outputs with weight 0.1.
\hline \text{Method} & \text{Mean after SFT}\uparrow \\\\ \begin{array}{l} \vphantom{\text{w/o reconstruction}} \\\\ \hline \text{Patch replay} \\\\ \textbf{ALR} \\\\ \end{array} & \begin{array}{cc} \text{w/o reconstruction} & \text{w/ reconstruction} \\\\ \hline 75.18 & 75.73 \\\\ 76.26 & \mathbf{77.65} \\\\ \end{array} \\\\ \hline \end{array}$$ The 1.92-point comparison holds the key-frame features and reconstruction loss fixed and measures the benefit of recursively updating the representation used at successive positions. Without reconstruction, ALR also exceeds the non-recursive comparison by 1.08 points (76.26 vs. 75.18). Adding reconstruction to ALR then gives a further 1.39 points (77.65 vs. 76.26). Thus, recursive computation improves recognition at both reconstruction settings, and visual reconstruction adds a complementary gain within the same recursive architecture. These comparisons distinguish the two contributions while matching key-frame access and the number of extra decoder positions.