acceptodds
Under review as a conference paper at ICLR 2027

AURORA:Action Unit-Reinforced Latent Reasoning for Multimodal Emotion Recognition

Abstract

Open-vocabulary multimodal emotion recognition (OV-MER) aims to predict flexible emotion expressions from video, audio, and language signals. Despite recent advances, existing OV-MER methods remain predominantly text-supervised: models are optimized to generate rationales and emotion labels, while reinforcement learning rewards only the final textual outputs. Such output-level supervision provides limited guidance for preserving fine-grained facial evidence, causing subtle muscle activations, temporal expression dynamics, and co-occurring facial movements to be diluted during language generation. We propose AURORA, a latent reasoning framework that explicitly grounds OV-MER in facial Action Units (AUs). AURORA combines frame-wise AU prediction with a continuous latent reasoning stage supervised to reconstruct the projected visual tokens of an AU-selected facial frame. This reconstruction objective encourages the latent states to retain facial evidence before emotion prediction. A second stage uses standard Group-Relative Policy Optimization (GRPO) with emotion-wheel, output-format, and AU-consistency rewards for task-level alignment. Experiments on OV-MERD+ and MER-UniBench demonstrate that AURORA achieves state-of-the-art performance, reaching a mean score of 84.87 on MER-UniBench when built on Qwen3-VL-8B.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal large language models, emotion recognition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qVafOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines frame-by-frame AU supervision, keyframe visual reconstruction, and GRPO to enhance the use of facial evidence in open-vocabulary multimodal emotion recognition. The method has a clear motivation and the experiments are relatively complete.

Critical strengths
  1. By supplementing textual supervision with AU prediction and continuous visual representation, a clear learning objective is provided for preserving fine-grained facial information.

  2. Experiments cover multiple input modalities and backbone models, and provide a comparison with the same backbone to support the effectiveness of the method under different configurations.

  3. The keyframe selection, AU granularity, and component ablation are presented systematically, but failure cases and methodological limitations are also reported.

Critical weaknesses
  1. Since keyframes serve as both input and reconstruction targets, it is still difficult to distinguish whether the gains of ALR come from visual recoding or recursive computation. It is recommended to add a non-recursive comparison with the same budget.

  2. The AU reward measures the consistency with OpenFace. It cannot yet prove that emotional judgment uses this evidence correctly. It is recommended to verify it by combining independent annotation and local action intervention.

  3. OpenFace's AU estimate has been obtained during inference, but the model still needs to generate a complete AU sequence. It is recommended to compare it with directly inputting the same AU information to clarify the additional value of the generation step.

  4. Comparisons of GRPO and continued SFT used different training data, and it is recommended to compare on the same MER2025-OV data to distinguish the contribution of the data and the optimization method.

Questions for authors

Please refer to the Critical Weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer qVaf(1/3)

Thank you for your suggestions on separating visual recoding from recursive computation and examining how AU information contributes to emotional judgment. In response, we added a non-recursive comparison with the same budget, independent human annotations and video observations of local facial actions, a trained direct-AU-input model with recognition and timing measurements, and continued SFT on the same MER2025-OV data as GRPO. We discuss these results in relation to your four concerns below.

Q1: Do ALR's gains come from visual recoding or recursive computation?

A1: With the same reconstruction loss and computation budget, recursive ALR scores 77.65 versus 75.73 for the non-recursive comparison, a 1.92-point advantage (Table 8). The non-recursive comparison directly appends the same frozen key-frame features at 32 additional decoder positions; we call this patch replay in Section 5.4. At position ℓ\ell, it receives patch feature kℓk_\ell, whereas ALR receives the preceding context-conditioned hidden state hℓ−1h_{\ell-1}. Here, “non-recursive” specifies the absence of this hidden-state input to the next position: both methods retain full causal attention and their KV caches. Both also retain the original key-frame input, Qwen2.5-VL-7B backbone, multimodal context, AU/text supervision and training budget. When enabled, reconstruction applies the same loss to their decoder outputs with weight 0.1.

\hline \text{Method} & \text{Mean after SFT}\uparrow \\\\ \begin{array}{l} \vphantom{\text{w/o reconstruction}} \\\\ \hline \text{Patch replay} \\\\ \textbf{ALR} \\\\ \end{array} & \begin{array}{cc} \text{w/o reconstruction} & \text{w/ reconstruction} \\\\ \hline 75.18 & 75.73 \\\\ 76.26 & \mathbf{77.65} \\\\ \end{array} \\\\ \hline \end{array}$$ The 1.92-point comparison holds the key-frame features and reconstruction loss fixed and measures the benefit of recursively updating the representation used at successive positions. Without reconstruction, ALR also exceeds the non-recursive comparison by 1.08 points (76.26 vs. 75.18). Adding reconstruction to ALR then gives a further 1.39 points (77.65 vs. 76.26). Thus, recursive computation improves recognition at both reconstruction settings, and visual reconstruction adds a complementary gain within the same recursive architecture. These comparisons distinguish the two contributions while matching key-frame access and the number of extra decoder positions.
AuthorsOpenReview ↗

Response to Reviewer qVaf(2/3)

Q2: Can independent annotation and local action intervention verify whether emotional judgment uses the AU evidence correctly?

A2: Adding the L1 AU reward improves emotional judgment: the nine-dataset mean rises by 2.74 points (77.55 to 80.29) and OV-MERD+ FsF_s by 1.05 points (62.47 to 63.52; Table 3(a)). This comparison evaluates emotion predictions while fixing the SFT checkpoint, training data, visual inputs, ALR length and all other reward terms.

SettingSentimentBasicOV-MERD+Mean
(a) AU reward after GRPO
None84.6174.2762.4777.55
L1 reward86.8977.8963.5280.29
L2 reward84.4173.9862.1677.30

For the independent annotation you requested, human judgments show that ALR raises facial-claim support from 77.6% to 86.0% and visible-event coverage from 53.0% to 60.6% (Table 15). AURORA and its matched no-ALR control use identical AU supervision, rewards, training data and visual inputs. We sampled 100 OV-MERD+ clips within four visibility strata before inspecting explanations, then froze both models' explanations for the same clips and original eight frames. Two trained coders independently inventoried visible facial events and their supporting frames before assessing randomized atomic claims. Model identities, detector traces, emotion predictions, audio and subtitles were hidden; subject and temporal qualifiers were retained, and disagreements were adjudicated. This creates a visual reference independent of OpenFace's training annotations.

\hline \text{Model} & \text{Claims} & S/(S+C) & U/\mathrm{claims} & \text{Unique matches / events} \\\\ \hline \rlap{\text{Overall comparison}} & & & & \\\\ \text{AURORA} & 362 & 283/329\\;(86.0\\%) & 33/362\\;(9.1\\%) & 255/421\\;(60.6\\%) \\\\ \text{AURORA w/o ALR} & 357 & 250/322\\;(77.6\\%) & 35/357\\;(9.8\\%) & 223/421\\;(53.0\\%) \\\\ \hline \rlap{\text{Clear: 30 clips and 143 visible events}} & & & & \\\\ \text{AURORA} & 111 & 100/109\\;(91.7\\%) & 2/111\\;(1.8\\%) & 94/143\\;(65.7\\%) \\\\ \text{AURORA w/o ALR} & 111 & 94/109\\;(86.2\\%) & 2/111\\;(1.8\\%) & 87/143\\;(60.8\\%) \\\\ \hline \rlap{\text{Occlusion: 30 clips and 128 visible events}} & & & & \\\\ \text{AURORA} & 109 & 84/98\\;(85.7\\%) & 11/109\\;(10.1\\%) & 76/128\\;(59.4\\%) \\\\ \text{AURORA w/o ALR} & 107 & 73/97\\;(75.3\\%) & 10/107\\;(9.3\\%) & 64/128\\;(50.0\\%) \\\\ \hline \rlap{\text{Large pose: 20 clips and 86 visible events}} & & & & \\\\ \text{AURORA} & 72 & 55/65\\;(84.6\\%) & 7/72\\;(9.7\\%) & 50/86\\;(58.1\\%) \\\\ \text{AURORA w/o ALR} & 72 & 49/64\\;(76.6\\%) & 8/72\\;(11.1\\%) & 44/86\\;(51.2\\%) \\\\ \hline \rlap{\text{Both: 20 clips and 64 visible events}} & & & & \\\\ \text{AURORA} & 70 & 44/57\\;(77.2\\%) & 13/70\\;(18.6\\%) & 35/64\\;(54.7\\%) \\\\ \text{AURORA w/o ALR} & 67 & 34/52\\;(65.4\\%) & 15/67\\;(22.4\\%) & 28/64\\;(43.8\\%) \\\\ \hline \end{array}$$ Here, $S$, $C$ and $U$ denote supported, contradicted and unjudgeable claims. Claims equals $S+C+U$; each pair of model rows shares the same clips and visible-event inventory. Overall, AURORA supports 283/329 judgeable claims versus 250/322 for the control, and covers 255/421 distinct visible events versus 223/421. At nearly identical description volumes of 3.62 and 3.57 claims per clip, it correctly describes 32 additional events with only five additional claims. Support and coverage improve across all four visibility strata. Table 3(a) establishes the recognition gain from the AU reward; Table 15 establishes the gain from ALR in describing independently annotated facial actions. These comparisons evaluate emotion-recognition accuracy and the accuracy and coverage of the facial evidence reported in explanations. The results reported for local facial actions are independent annotations and observations of the original videos. The frame-based analyses let you compare the local facial actions named in each explanation with the corresponding video frames. Appendix F.1, Fig. 3 shows the paired OpenFace/LibreFace AU activation patterns. Appendix I.1–I.4, Figs. 4–7 present `sample_00001433`, `sample_00001175`, `sample_00001334` and `sample_00002448`, respectively, with eight timestamped frames and model-response excerpts. In `sample_00001175` (Appendix I.2, Fig. 5), the visible smile under upper-face occlusion accompanies AURORA's prediction of joy and satisfaction, consistent with the reference emotions. The timestamped smile and the corresponding emotion prediction provide a concrete action-level example of the relationship assessed by the recognition and annotation results above.
AuthorsOpenReview ↗

Response to Reviewer qVaf(3/3)

Q3: What is the additional value of generating a complete AU sequence over directly inputting the same AU information?

A3: Training AURORA to generate the complete AU sequence yields a 1.47-point mean-score advantage after SFT and a 1.68-point advantage after RL over training it to receive the same OpenFace AU information directly, with a final 1.58-point OV-MERD+ FsF_s advantage (Table 21). Both models start from the same Qwen2.5-VL-7B backbone and share training clips, original eight-frame inputs, key frames, 32-step ALR, reconstruction weight, optimizer settings, batch/group sizes and update counts. Full AURORA learns to generate the complete 8×178\times17 AU sequence using OpenFace estimates as supervision; the direct-input model receives those estimates in its prompt throughout SFT, RL and evaluation. It retains rationale/answer and visual-reconstruction supervision, removes AU-output supervision and AU reward, and adapts the format reward to the remaining output structure; other reward coefficients are unchanged. Each RL run starts from its own SFT model and uses that model as its KL reference. Both are evaluated on the same complete benchmark splits with deterministic decoding.

StageModelFsF_s ↑Mean ↑Latency (s) ↓
SFTFull AURORA62.3777.6516.73
Direct AU input60.4676.1810.04
SFT+RLFull AURORA63.5280.2916.28
Direct AU input61.9478.619.68

Latency is measured on OV-MERD+ using one 80GB H800, batch size 1 and a warmed-up runtime. Each clip includes video/face preprocessing, one OpenFace pass, prompt processing and complete response generation. Direct input reduces final median latency by 40.5%. Thus, directly inputting the same AU information improves inference speed, while learning to generate the complete sequence achieves higher emotion-recognition scores at both training stages. Because the direct-input alternative is trained with these estimates and retains the same visual inputs and ALR architecture, the comparison measures the added value of the AU-generation training objective and its AU reward alongside their decoding cost.

Q4: What do the training data and optimization method each contribute when GRPO and continued SFT use the same MER2025-OV data?

A4: GRPO reaches 80.29 versus 78.08 for continued SFT on the same MER2025-OV clips, a 2.21-point advantage under the same 96 GPU-hour budget (Table 10). Both start from the same AURORA-SFT checkpoint at 77.65, exclude overlap with OV-MERD+ and retain the same inference architecture and 32 latent steps. Each additional run uses eight 80GB H800 GPUs for 12 hours, including rollout generation for GRPO. Table 10 also includes continued SFT on the original MER-Caption+ data under the same additional budget.

Additional optimizationTraining dataGPU-hoursMeanGain over SFT
None—077.65—
Continued SFTMER-Caption+9677.82+0.17
Continued SFTMER2025-OV9678.08+0.43
GRPO with AU-aware rewardMER2025-OV9680.29+2.64

With continued SFT and the additional budget fixed, changing its training data from MER-Caption+ to MER2025-OV raises the mean from 77.82 to 78.08, a 0.26-point increase. With the MER2025-OV clips fixed, continued SFT improves over the shared checkpoint by 0.43 points and GRPO by 2.64 points. Their 2.21-point difference measures the benefit of GRPO with AU-aware rewards at matched data and computation. These two comparisons separately quantify the change in training data and the change in optimization you asked about.

Reviewer_Rgm2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AURORA addresses open-vocabulary multimodal emotion recognition by combining frame-wise facial Action Unit traces with Affective Latent Reasoning (ALR). During Supervised Fine-tuning (SFT), the model is trained to reconstruct the projected patch tokens of an AU-salient key frame using continuous hidden states. Standard Group-Relative Policy Optimization (GRPO) then uses emotion-wheel, output-format, and AU-consistency rewards for task-level alignment. With Qwen3-VL-8B and audio-video-text inputs, AURORA reaches a mean score of 84.87 on MER-UniBench and 69.87 on OV-MERD+.

Critical strengths
  1. In Table 6, the score is 74.18 without AU traces or ALR. It rises to 77.48 with AU traces alone and 77.73 with ALR alone. Using both gives the highest score as 80.29. These results show that using the two components together would obtain the best performance.
  2. Against the standard SFT/RL baseline, AURORA improves the mean score by 2.95 on Qwen3-VL-4B and 3.12 on Qwen-VL-8B. These comparisons and the results on 9 datasets in Table 1 support the method across the tested backbones.
Critical weaknesses
  1. Reconstruction supervision is a key part of ALR. However, Table 6 removes both latent computation and reconstruction supervision when ALR is disabled. Table 4 and 5 do not separate these effects either. The reported gains therefore do not establish whether the reconstruction loss adds value beyond the latent computation. To answer this question, the authors may need to keep the inputs, 32 latent steps, and training settings fixed and remove only the reconstruction loss during SFT.

  2. Table 12 checks facial-action claims against OpenFace, which also supplies AURORA’s AU supervision. As the authors acknowledge, this measures agreement with OpenFace. It does not independently show that the claims match the video. Table 7 does not resolve this concern: it reports how often answers change after interventions, without reporting whether those answers are correct. The recognition results support the full method, but these analyses do not establish that ALR improves emotion prediction by preserving fine-grained facial evidence.

Questions for authors
  1. What would happen when the SFT reconstruction loss is removed with the same inputs, 32 latent steps and training budget?
  2. Can the facial-action claims be checked against human annotations or an independent reference?
  3. For Table 7, what are the subset sizes and emotion recognition scores?
  4. Also for Table 7, how is an answer change defined specifically?
  5. In Table 9, continued SFT uses MER-Caption+, while GRPO uses MER2025-OV. Can you add a comparison that separates the effect of GRPO from the effect of changing the training data, while keeping the compute budget matched?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Rgm2(1/4)

Thank you for emphasizing the need to isolate reconstruction supervision and check both facial-action claims and emotion-recognition accuracy. Following your suggestions, we removed only the SFT reconstruction loss while fixing the inputs, 32 latent steps and training budget, added independent human annotations, reported intervention sample sizes and recognition scores, specified the answer-change calculation, and added continued SFT on GRPO's training data under the same compute budget. We address your five questions below.

Q1: What happens when only the SFT reconstruction loss is removed, with the same inputs, 32 latent steps, and training budget?

A1: Removing only the SFT reconstruction loss lowers the nine-dataset mean from 77.65 to 76.26 (−1.39), with the same inputs, all 32 latent steps, and the same training budget retained (Table 8). This comparison changes only λalr\lambda_{\mathrm{alr}} in Eq. (3), from 0.1 to 0; gradients from the subsequent rationale and answer outputs continue to train the latent steps. The backbone, input tokens including KK, AU/rationale/answer supervision, and training settings are fixed. Table 8 also includes two controls that replace the 32 latent steps with 32 frozen key-frame patch inputs, with and without the reconstruction loss. In the ALR row, each preceding hidden state supplies the next latent step; in the Patch replay row, the corresponding frozen key-frame patch supplies that position. All four conditions retain causal attention and KV caches.

\hline \text{Method} & \text{Mean after SFT}\uparrow \\\\ \begin{array}{l} \vphantom{\text{w/o reconstruction}} \\\\ \hline \text{Patch replay} \\\\ \textbf{ALR} \\\\ \end{array} & \begin{array}{cc} \text{w/o reconstruction} & \text{w/ reconstruction} \\\\ \hline 75.18 & 75.73 \\\\ 76.26 & \mathbf{77.65} \\\\ \end{array} \\\\ \hline \end{array}$$ The 1.39-point decrease directly answers your reconstruction-loss question: visual reconstruction adds recognition value when the latent computation is kept intact. The patch-input controls further show that retaining the 32 latent steps improves the mean by 1.08 points without reconstruction and by 1.92 points with reconstruction. Thus, reconstruction supervision improves the representations computed during the latent steps, and the full combination achieves the highest recognition score under the same inputs and training budget.
AuthorsOpenReview ↗

Response to Reviewer Rgm2(2/4)

Q2: Can human annotations verify that the facial-action claims match the video?

A2: In the blinded human audit, 86.0% of AURORA’s judgeable facial-action claims match the video, versus 77.6% without ALR; visible-event coverage is 60.6% versus 53.0%

Two coders first independently annotated visible actions in 100 OV-MERD+ clips using the videos and original eight frames, then assessed anonymized claims with model identities and detector outputs hidden. Disagreements were adjudicated. The models share training data, AU supervision, rewards and visual inputs.

\hline \text{Model} & \text{Claims} & S/(S+C) & U/\mathrm{claims} & \text{Unique matches / events} \\\\ \hline \rlap{\text{Overall comparison}} & & & & \\\\ \text{AURORA} & 362 & 283/329\\;(86.0\\%) & 33/362\\;(9.1\\%) & 255/421\\;(60.6\\%) \\\\ \text{AURORA w/o ALR} & 357 & 250/322\\;(77.6\\%) & 35/357\\;(9.8\\%) & 223/421\\;(53.0\\%) \\\\ \hline \rlap{\text{Clear: 30 clips and 143 visible events}} & & & & \\\\ \text{AURORA} & 111 & 100/109\\;(91.7\\%) & 2/111\\;(1.8\\%) & 94/143\\;(65.7\\%) \\\\ \text{AURORA w/o ALR} & 111 & 94/109\\;(86.2\\%) & 2/111\\;(1.8\\%) & 87/143\\;(60.8\\%) \\\\ \hline \rlap{\text{Occlusion: 30 clips and 128 visible events}} & & & & \\\\ \text{AURORA} & 109 & 84/98\\;(85.7\\%) & 11/109\\;(10.1\\%) & 76/128\\;(59.4\\%) \\\\ \text{AURORA w/o ALR} & 107 & 73/97\\;(75.3\\%) & 10/107\\;(9.3\\%) & 64/128\\;(50.0\\%) \\\\ \hline \rlap{\text{Large pose: 20 clips and 86 visible events}} & & & & \\\\ \text{AURORA} & 72 & 55/65\\;(84.6\\%) & 7/72\\;(9.7\\%) & 50/86\\;(58.1\\%) \\\\ \text{AURORA w/o ALR} & 72 & 49/64\\;(76.6\\%) & 8/72\\;(11.1\\%) & 44/86\\;(51.2\\%) \\\\ \hline \rlap{\text{Both: 20 clips and 64 visible events}} & & & & \\\\ \text{AURORA} & 70 & 44/57\\;(77.2\\%) & 13/70\\;(18.6\\%) & 35/64\\;(54.7\\%) \\\\ \text{AURORA w/o ALR} & 67 & 34/52\\;(65.4\\%) & 15/67\\;(22.4\\%) & 28/64\\;(43.8\\%) \\\\ \hline \end{array}$$ Here, $S$, $C$ and $U$ denote supported, contradicted and unjudgeable claims. Support is $S/(S+C)$; coverage is the fraction of annotated events correctly described, counting each event once. ALR correctly describes 32 more events with only five additional claims, with gains across visibility conditions. After SFT and GRPO, the two rows with AU trace enabled in Section 5.3, Table 6 show that adding ALR raises OV-MERD+ $F_s$ from 61.75 to 63.52 and the nine-dataset mean from 77.48 to 80.29: $$\begin{array}{ccccc} \hline \scriptstyle \text{Response Components} & \scriptstyle \text{Sentiment} & \scriptstyle \text{Basic Emotion} & \scriptstyle \text{Fine-grained} & \scriptstyle \text{Mean} \\\\ \begin{array}{cc} \scriptstyle \text{AU trace} & \scriptstyle \text{ALR} \\\\ \hline \scriptstyle \times & \scriptstyle \times \\\\ \scriptstyle \checkmark & \scriptstyle \times \\\\ \scriptstyle \times & \scriptstyle \checkmark \\\\ \scriptstyle \checkmark & \scriptstyle \checkmark \\\\ \end{array} & \begin{array}{rrrr} \scriptstyle \text{MOSI} & \scriptstyle \text{MOSEI} & \scriptstyle \text{SIMS} & \scriptstyle \text{SIMS v2} \\\\ \hline \scriptstyle 76.56 & \scriptstyle 78.65 & \scriptstyle 81.88 & \scriptstyle 83.00 \\\\ \scriptstyle 82.36 & \scriptstyle 83.77 & \scriptstyle 85.92 & \scriptstyle 86.83 \\\\ \scriptstyle 83.62 & \scriptstyle 83.53 & \scriptstyle 87.77 & \scriptstyle 86.19 \\\\ \scriptstyle \mathbf{84.82} & \scriptstyle \mathbf{85.21} & \scriptstyle \mathbf{88.97} & \scriptstyle \mathbf{88.56} \\\\ \end{array} & \begin{array}{rrrr} \scriptstyle \text{MER2023} & \scriptstyle \text{MER2024} & \scriptstyle \text{MELD} & \scriptstyle \text{IEMOCAP} \\\\ \hline \scriptstyle 77.23 & \scriptstyle 83.66 & \scriptstyle 60.25 & \scriptstyle 66.07 \\\\ \scriptstyle 79.54 & \scriptstyle 85.70 & \scriptstyle 63.42 & \scriptstyle 68.00 \\\\ \scriptstyle 78.12 & \scriptstyle 86.14 & \scriptstyle 64.92 & \scriptstyle 68.20 \\\\ \scriptstyle \mathbf{83.58} & \scriptstyle \mathbf{87.91} & \scriptstyle \mathbf{69.59} & \scriptstyle \mathbf{70.46} \\\\ \end{array} & \begin{array}{r} \scriptstyle \text{OV-MERD+} \\\\ \hline \scriptstyle 60.29 \\\\ \scriptstyle 61.75 \\\\ \scriptstyle 61.08 \\\\ \scriptstyle \mathbf{63.52} \\\\ \end{array} & \begin{array}{r} \scriptstyle \vphantom{\text{OV-MERD+}} \\\\ \hline \scriptstyle 74.18 \\\\ \scriptstyle 77.48 \\\\ \scriptstyle 77.73 \\\\ \scriptstyle \mathbf{80.29} \\\\ \end{array} \\\\ \hline \end{array}$$ These results connect the independently verified improvement in facial descriptions with the recognition gain of the same ALR/no-ALR configurations. Figures 4–7 in Appendix I provide timestamped video examples for checking the described actions.
AuthorsOpenReview ↗

Response to Reviewer Rgm2(3/4)

Q3: What are the subset sizes and emotion-recognition scores in Table 7?

A3: The subset size is 532 OV-MERD+ clips for every condition. Table 7 now reports emotion-recognition FsF_s for each intervention: 55.84 after zeroing the latent trajectory, 56.79 after substituting another video’s trajectory, and 60.46 after replacing the subtitle while retaining the original trajectory. With replacement subtitles and each model’s normal computation, AURORA scores 58.93 versus 54.72 without ALR. We use the main Qwen2.5-VL-7B vision-text models after SFT and GRPO, retaining the original eight frames, AU-selected key frame, prompts and decoding settings. Both models receive the same replacement subtitles. Recognition is scored against the original clips' labels with the normalization and emotion-wheel protocol in Appendix C.1. Table 7 is reproduced below in its original layout. The original-input reference scores from Table 6 are 63.52 for AURORA and 61.75 without ALR.

\hline \text{System} & \text{Subtitle} & \text{Latent trajectory} & \text{Changed (\\%)} & F_s\uparrow \\\\ \hline \rlap{\text{(a) Latent interventions and fixed-latent subtitle control}} & & & & \\\\ \text{AURORA} & \text{Original} & \text{Set to zero} & 56.3 & 55.84 \\\\ \text{AURORA} & \text{Original} & \text{From another video} & 51.8 & 56.79 \\\\ \text{AURORA} & \text{Replaced} & \text{Retained from original input} & 24.7 & 60.46 \\\\ \hline \rlap{\text{(b) Mismatched subtitles with each model's normal latent computation}} & & & & \\\\ \text{w/o ALR} & \text{Replaced} & \text{None} & 47.6 & 54.72 \\\\ \textbf{AURORA} & \text{Replaced} & \text{Regenerated} & 33.4 & \mathbf{58.93} \\\\ \hline \end{array}$$ With the original visual inputs retained, zeroing and donor substitution reduce $F_s$ from 63.52 by 7.68 and 6.73 points, respectively, linking the original latent trajectory to task performance. Keeping the trajectory formed from the original video and subtitle preserves a score of 60.46 after subtitle replacement, a loss of 3.06 points. When each model follows its normal computation on the modified input, ALR reduces the recognition loss from 7.03 to 4.59 points. The emotion-recognition scores therefore show both the contribution of the original latent trajectory and the smaller loss of recognition accuracy with ALR under subtitle replacement. Changed (%) reports the emotion-word edit rate defined in A4. > **Q4: How is an answer change defined in Table 7?** **A4:** Changed (%) pools emotion-word edits over all 532 clips: each addition, deletion or substitution counts once. We extract emotion words from the final `<answer>` span, retain literal word forms and occurrence counts, and ignore word order and list separators. Thus, `happy` to `happiness` counts as one substitution and adding `surprised` counts as one addition. Stemming, synonym merging and emotion-wheel grouping are excluded from this lexical statistic, as are edits confined to `<think>`. For clip $i$, let $m_i$ denote its original emotion-word count and $a_i,d_i,s_i$ its addition, deletion and substitution counts. The statistic is $$\mathrm{Changed}\\;(\\%)=100\\,\frac{\sum_{i=1}^{532}(a_i+d_i+s_i)}{\sum_{i=1}^{532}m_i}.$$ Each model's own original answers define the denominator: 532 is the number of videos, and the total original emotion-word count normalizes the rate. Table 7 reports recognition quality separately as $F_s$, using the reference-label scoring of Table 1 and Appendix C.1. The two metrics distinguish changes in wording from changes in emotion-recognition performance.
Reviewer_GqEAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AURORA augments open-vocabulary multimodal emotion recognition with frame-wise AU prediction and continuous latent states trained to reconstruct an AU-selected facial frame. It combines SFT with AU-aware GRPO before generating rationales and emotion labels. Gains are consistent across several backbones and benchmarks, although the distinct role of latent reasoning beyond feature reconstruction and the accuracy of its visual grounding need stronger evidence.

Critical strengths

The motivation is sensible. Explicit facial-action supervision provides a targeted way to preserve evidence that free-form rationales may overlook.

The gains are not solely due to stronger backbones. Within-backbone ablations and comparisons on Qwen3-VL and LLaMA-2 support the framework’s effectiveness.

The implementation is carefully described. The paper explains how gradients propagate through latent computation during GRPO and distinguishes the different intervention and corruption settings.

Critical weaknesses

The latent reasoning that preserves fine-grained visual evidence has not been sufficiently distinguished from simpler feature reuse.

Questions for authors

What specifically drives the ALR gain? The reconstruction target KK is already present in the input. Removing ALR entirely does not separate extra computation, feature copying, and reconstruction supervision. Could the authors retain latent steps without reconstruction loss and compare against directly appending the same key-frame features?

Can the grounding improvement be independently validated? OpenFace supplies supervision, selects key frames, and evaluates claim consistency. Lower inconsistency therefore primarily establishes agreement with OpenFace. A blinded human audit or independent AU annotations, especially for occlusion and large poses, would strengthen the claim.

Could training data be controlled in the GRPO comparison? Appendix D matches compute, but continued SFT uses MER-Caption+ while GRPO uses MER2025-OV. The 2.47-point gap cannot isolate the optimization method. A supervised adaptation control using the same additional data would help.

4 comments
AuthorsOpenReview ↗

Response to Reviewer GqEA(1/3)

Thank you for your suggestions on identifying what drives ALR's gains and independently validating facial grounding. Following your suggestions, we retained all 32 latent steps while removing the reconstruction loss, compared them with directly appending the same key-frame features, added a blinded human audit and an independent automatic AU reference, and included supervised adaptation on the same additional data as GRPO. We explain how these additions address your three questions below.

Q1: What drives the ALR gain when latent steps are retained without reconstruction loss, compared with directly appending the same key-frame features?

A1: Retaining all 32 latent steps and removing only the reconstruction loss gives 76.26, compared with 77.65 for full ALR; reconstruction therefore adds 1.39 points with the latent steps fixed (Table 8). Directly appending the same key-frame features gives 75.18 without reconstruction and 75.73 with the same reconstruction loss. These are the two comparisons you requested, evaluated with the same backbone, visual inputs including KK, AU/rationale/answer supervision, training budget, and 32 additional decoder positions.

The condition that directly appends the features is called “Patch replay” in Table 8: each additional position ℓ\ell receives the corresponding frozen key-frame patch feature kℓk_\ell. ALR retains its latent steps, with each step receiving the preceding hidden state hℓ−1h_{\ell-1}. Both preserve full causal attention and their KV caches. Reconstruction, when enabled, supervises each decoder output against the same kℓk_\ell with λalr=0.1\lambda_{\mathrm{alr}}=0.1; removing it sets this weight to zero.

\hline \text{Method} & \text{Mean after SFT}\uparrow \\\\ \begin{array}{l} \vphantom{\text{w/o reconstruction}} \\\\ \hline \text{Patch replay} \\\\ \textbf{ALR} \\\\ \end{array} & \begin{array}{cc} \text{w/o reconstruction} & \text{w/ reconstruction} \\\\ \hline 75.18 & 75.73 \\\\ 76.26 & \mathbf{77.65} \\\\ \end{array} \\\\ \hline \end{array}$$ Scores are the SFT mean over nine MER-UniBench datasets. With reconstruction removed from both conditions, retaining latent steps exceeds directly appending the same features by 1.08 points (76.26 versus 75.18). With reconstruction applied to both, the difference is 1.92 points (77.65 versus 75.73). The fixed 32 additional positions control extra computation, directly appending the same features controls feature copying, and the 76.26-to-77.65 comparison isolates reconstruction supervision within ALR. These comparisons show how the latent steps and their reconstruction supervision each contribute to the gain you asked about. Table 5 additionally examines your observation that $K$ is already present in the input: in the visual-input ablation after SFT+RL, using the original and aligned face sequences while retaining $K$ only as the reconstruction target yields 77.32. This separates access to $K$ as an input from its use as a training target.
AuthorsOpenReview ↗

Response to Reviewer GqEA(2/3)

Q2: Can a blinded human audit or independent AU annotations validate grounding improvements, especially under occlusion and large poses?

A2: The blinded human audit shows 86.0% claim support and 60.6% visible-event coverage for AURORA, versus 77.6% and 53.0% without ALR (Appendix F.3, Table 15).

We evaluated 100 OV-MERD+ clips sampled across clear, occluded, large-pose and combined conditions before inspecting explanations. Both models used the same clips and original eight frames, with matched training, AU supervision and rewards. Two coders first independently inventoried visible actions, then judged anonymized claims with model identities and detector outputs hidden; disagreements were adjudicated.

\hline \text{Model} & \text{Claims} & S/(S+C) & U/\mathrm{claims} & \text{Unique matches / events} \\\\ \hline \rlap{\text{Overall comparison}} & & & & \\\\ \text{AURORA} & 362 & 283/329\\;(86.0\\%) & 33/362\\;(9.1\\%) & 255/421\\;(60.6\\%) \\\\ \text{AURORA w/o ALR} & 357 & 250/322\\;(77.6\\%) & 35/357\\;(9.8\\%) & 223/421\\;(53.0\\%) \\\\ \hline \rlap{\text{Clear: 30 clips and 143 visible events}} & & & & \\\\ \text{AURORA} & 111 & 100/109\\;(91.7\\%) & 2/111\\;(1.8\\%) & 94/143\\;(65.7\\%) \\\\ \text{AURORA w/o ALR} & 111 & 94/109\\;(86.2\\%) & 2/111\\;(1.8\\%) & 87/143\\;(60.8\\%) \\\\ \hline \rlap{\text{Occlusion: 30 clips and 128 visible events}} & & & & \\\\ \text{AURORA} & 109 & 84/98\\;(85.7\\%) & 11/109\\;(10.1\\%) & 76/128\\;(59.4\\%) \\\\ \text{AURORA w/o ALR} & 107 & 73/97\\;(75.3\\%) & 10/107\\;(9.3\\%) & 64/128\\;(50.0\\%) \\\\ \hline \rlap{\text{Large pose: 20 clips and 86 visible events}} & & & & \\\\ \text{AURORA} & 72 & 55/65\\;(84.6\\%) & 7/72\\;(9.7\\%) & 50/86\\;(58.1\\%) \\\\ \text{AURORA w/o ALR} & 72 & 49/64\\;(76.6\\%) & 8/72\\;(11.1\\%) & 44/86\\;(51.2\\%) \\\\ \hline \rlap{\text{Both: 20 clips and 64 visible events}} & & & & \\\\ \text{AURORA} & 70 & 44/57\\;(77.2\\%) & 13/70\\;(18.6\\%) & 35/64\\;(54.7\\%) \\\\ \text{AURORA w/o ALR} & 67 & 34/52\\;(65.4\\%) & 15/67\\;(22.4\\%) & 28/64\\;(43.8\\%) \\\\ \hline \end{array}$$ Here, $S$, $C$ and $U$ denote supported, contradicted and unjudgeable claims. Support is $S/(S+C)$; coverage is the fraction of unique visible events correctly described. With occlusion and large pose combined, support and coverage reach 77.2% and 54.7%, versus 65.4% and 43.8%. Overall, ALR correctly describes 32 more events with five additional claims. For independent AU annotations, Appendix F.1, Table 13 uses independently trained LibreFace (LF) alongside OpenFace (OF). We freeze explanations, extracted claims, the phrase-to-AU mapping and the original eight frames. The paired comparison retains claims fully covered by the 12 shared AUs, with valid outputs from both detectors at all eight positions. A positive action claim is unsupported if none of its mapped AUs reaches intensity 0.5 in any frame. Both paired columns score identical retained claims. $$\begin{array}{lrrrr} \hline \text{Model} & \text{Original OF} & \text{Paired OF} & \text{Paired LF} & \text{Retained/mapped} \\\\ \hline \text{Emotion-LLaMA} & 35.6 & 34.6 & 36.2 & 762/965\\;(79.0\\%) \\\\ \text{Emotion-LLaMAv2} & 30.8 & 29.6 & 31.5 & 803/1014\\;(79.2\\%) \\\\ \text{AffectGPT} & 37.2 & 36.4 & 35.7 & 846/1092\\;(77.5\\%) \\\\ \textbf{AURORA} & \mathbf{14.6} & \mathbf{15.0} & \mathbf{18.1} & 831/1048\\;(79.3\\%) \\\\ \hline \end{array}$$ Inconsistency is reported in % (lower is better); Retained/mapped gives evaluated/all AU-mappable claims. AURORA has the lowest inconsistency under both references, with comparable claim retention, supporting the advantage across independent AU annotations. In Section 5.3, Table 3(a), we compare GRPO training without an AU reward and with L1 or L2 rewards, fixing the SFT checkpoint, data, visual inputs, ALR length and other reward terms. L1 improves the nine-dataset mean by 2.74 points and OV-MERD+ $F_s$ by 1.05 over no AU reward. $$\begin{array}{lrrrr} \hline \text{Setting} & \text{Sentiment} & \text{Basic} & \text{OV-MERD+} & \text{Mean} \\\\ \hline \rlap{\text{(a) AU reward after GRPO}} & & & & \\\\ \text{None} & 84.61 & 74.27 & 62.47 & 77.55 \\\\ \textbf{L1 reward} & \mathbf{86.89} & \mathbf{77.89} & \mathbf{63.52} & \mathbf{80.29} \\\\ \text{L2 reward} & 84.41 & 73.98 & 62.16 & 77.30 \\\\ \hline \end{array}$$ Together, these experiments connect independently verified facial descriptions with the recognition benefit of AU supervision. Figure 3 in Appendix F.1 visualizes paired AU patterns; Figures 4–7 in Appendix I.1–I.4 provide timestamped video evidence and model excerpts under occlusion and head turns.
AuthorsOpenReview ↗

Response to Reviewer GqEA(3/3)

Q3: How does GRPO compare with supervised adaptation on the same additional data?

A3: With supervised adaptation and GRPO using the same additional MER2025-OV clips, GRPO reaches 80.29 versus 78.08 for supervised adaptation, a 2.21-point gain at the same 96 GPU-hour budget (Table 10). The supervised adaptation control you requested is continued SFT for those same clips. Both runs start from the same AURORA-SFT checkpoint at 77.65, remove overlap with OV-MERD+, and retain the same architecture and 32 latent steps. Each uses eight 80GB H800 GPUs for 12 hours, including rollout generation for GRPO.

\hline \text{Additional optimization} & \text{Training data} & \text{GPU-hours} & \text{Mean} & \text{Gain over SFT} \\\\ \hline \text{None} & \text{—} & 0 & 77.65 & \text{—} \\\\ \text{Continued SFT} & \text{MER-Caption+} & 96 & 77.82 & +0.17 \\\\ \text{Continued SFT} & \text{MER2025-OV} & 96 & 78.08 & +0.43 \\\\ \text{GRPO with AU-aware reward} & \text{MER2025-OV} & 96 & \mathbf{80.29} & \mathbf{+2.64} \\\\ \hline \end{array}$$ Using the same additional clips raises the supervised adaptation gain from 0.17 to 0.43 points, while GRPO gains 2.64 points over the same starting checkpoint. The comparison that answers your training-data question is therefore 80.29 versus 78.08: GRPO with AU-aware rewards improves recognition by 2.21 points when both the additional data and compute are matched. This supports the benefit of optimizing facial perception and emotion prediction with AU-aware rewards after SFT. ## Summary of Paper Revisions - Section 5.4, Table 8: We added the controls you requested: retain all 32 latent steps without reconstruction loss, and directly append the same key-frame features, with and without the same reconstruction supervision. With key-frame inputs and training budgets fixed, the four conditions distinguish the effects of latent steps, feature copying and reconstruction supervision. - Appendix F.3, Table 15: We added the blinded human audit of 100 clips, with an independent reference of visible video events and matched AURORA/no-ALR explanations. The section documents blinded coding, adjudication, support and coverage, and separate results for occlusion, large poses and their combination, addressing the visibility conditions you highlighted. - Appendix F.1, Table 13 and Figure 3: We added independently trained LibreFace as a second automatic AU annotation source. The paired OpenFace/LibreFace evaluation fixes explanations, original eight frames and the 12 shared AUs, and reports retained-claim counts. The heatmap shows emotion-conditioned AU activation patterns under both detectors. - Appendix I.1–I.4, Figures 4–7: We added four video analyses of the occlusion and head-turn conditions raised in your question. Eight timestamped frames and verbatim model excerpts identify which visible actions support each description, together with the person and time interval. - Appendix D, Table 10: We added supervised adaptation on the same additional MER2025-OV clips used by GRPO. The comparison fixes the starting checkpoint and 96 GPU-hour budget, and documents teacher-generated SFT targets, overlap removal and GRPO rollout accounting. The resulting 2.21-point comparison directly addresses your request to control the additional training data.

Dear Reviewer GqEA,

Thank you again for taking the time to review our paper and share your feedback. As the discussion period draws to a close, we’d appreciate hearing whether our responses have addressed your concerns. If any questions remain, we’d be happy to clarify them.

Comments on the paper

1 comment

Dear Reviewers,

I hope this message finds you well. As the discussion period is nearing its end with less than two days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.