acceptodds
Under review as a conference paper at ICLR 2027

Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

Abstract

Understanding facial expressions in unconstrained videos requires modeling subtle and localized facial dynamics. Despite substantial advances enabled by large-scale self-supervised pretraining, recent Vision Transformer (ViT)-based video models often remain biased toward dominant global motions and coarse temporal dynamics, limiting their sensitivity to fine-grained facial variations. To address this limitation, we propose MiRA (Marginal-induced Attention Redistribution), a plug-in framework for ViT backbones that enhances spatio-temporal selectivity for fine-grained facial cues without introducing additional trainable parameters. In its principled exact mode, it derives frame-level confidence and intra-frame concentration from post-softmax attention, capturing each frame's aggregate attention mass and its spatial concentration, respectively. Together, these quantities form a framewise importance score, which is then transformed into a prior for reallocating attention mass across frames based on global salience and within-frame localization. To further improve efficiency, we introduce flashLite mode, a lightweight pre-softmax approximation that derives framewise logit modulation from key-based proxy statistics and incorporates it into FlashAttention kernels while preserving the effectiveness of the exact formulation. Extensive experiments demonstrate that MiRA achieves state-of-the-art performance among video-only methods while remaining competitive with multimodal approaches.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Facial Expression Recognition, Vision Transformers, Video Representation Learning, Unsupervised Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_B4Z6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MiRA (Marginal-induced Attention Redistribution), a framewise attention reweighting framework for video-based facial expression recognition that introduces no additional trainable parameters. Its Exact mode combines frame-level attention mass and intra-frame concentration to construct a prior for redistributing attention across frames.

To improve computational efficiency, the authors introduce FlashLite, which derives framewise modulation from key-energy statistics and incorporates additive logit biases into fused attention computation. The approach is evaluated on DFEW, MAFW, and FERV39k, with additional ablations, comparisons between the two modes, frozen-feature probing, qualitative visualizations, and efficiency measurements.

Critical strengths

Overall, the paper’s strengths lie in its simple architectural design, encouraging empirical results, and practical efficiency. MiRA introduces no additional trainable parameters, while FlashLite achieves comparable recognition performance to Exact with substantially lower runtime and memory usage.

S1. A simple and practically useful design. MiRA derives framewise modulation from existing representations and preserves the overall Transformer architecture without adding trainable parameters. This makes it an appealing approach for adapting video backbones to facial expression recognition.

S2. Encouraging empirical performance. Table 5 reports improvements over the evaluated general video baselines under the stated training conditions, and Table 6 shows competitive results across three facial expression benchmarks. The five-fold comparisons in Table 4 further demonstrate comparable average performance between Exact and FlashLite on DFEW and MAFW.

S3. Substantial efficiency improvements over Exact. For ViT-B finetuning, FlashLite reduces iteration time from 700.8 ms to 169.9 ms and peak memory from 71.4 GB to 10.2 GB relative to Exact, as reported in Table 9. These results support its practical value when explicit attention-map manipulation would otherwise be costly.

Critical weaknesses

The main limitations concern the interpretation of FlashLite’s approximation, the evidence for expression-specific selectivity, and the completeness of several experimental controls. These points do not negate the reported empirical benefits, but addressing them would strengthen the technical explanation and clarify the method’s applicability.

W1. The relationship between Exact and FlashLite could be clarified further. Figure 4 provides useful quantitative comparisons, and Appendix A.2 acknowledges the limitations of key magnitude as a proxy for attention mass. However, FlashLite also changes the prior construction and scaling rule. Controlled comparisons using identical weights and inputs, with components replaced sequentially, would help distinguish the effects of these changes and better characterize the approximation.

W2. Additional evidence would strengthen the expression-specific interpretation. The visualizations in Figures 7–8 are encouraging, but attention mass and concentration may also reflect head motion, occlusion, or salient boundaries. Targeted evaluations would help establish when the proposed statistics favor expression-relevant information. The paper could also distinguish direct framewise redistribution from spatial improvements learned during training: before smoothing and interpolation, uniform scaling within a frame preserves relative attention weights for a fixed query.

W3. Additional controls would improve the completeness of the evaluation. Confidence-only and concentration-only variants would clarify the contribution of each statistic beyond the weighting study in Table 2. Given the sensitivity to the power factor observed on DFEW in Table 8, a fuller description of the validation procedure would also be helpful. Finally, adding vanilla VideoMAE with FlashAttention under matched settings would quantify FlashLite’s incremental overhead, complementing the demonstrated efficiency improvements over Exact.

Questions for authors

Q1. Could the authors clarify whether Figure 4 compares the two modes using identical model weights and Q,K,VQ,K,V? Frame-ranking correlations or comparisons of the induced attention changes would further help interpret the reported agreement.

Q2. Are confidence-only and concentration-only results available? These would help clarify how the two statistics contribute to the observed gains.

Q3. Could the authors provide additional evidence regarding robustness to head motion, occlusion, or illumination changes, or discuss the situations in which the proposed statistics may be less reliable?

Q4. What is FlashLite’s runtime and memory overhead relative to vanilla VideoMAE using FlashAttention under identical experimental conditions?

Q5. Please clarify the numerical handling of equal frame scores in Equation (5), where the written power-normalization denominator becomes zero, and provide the clipping bounds, interpolation coefficient, EMA momentum, and validation protocol.

1 comment
Reviewer_B4Z6OpenReview ↗

Suggestions for Additional Related Work

We encourage the authors to consider related studies such as S2D [1], SSM [2], FineCLIPER [3], and CLAIP-Emo [4]. These works address static-to-dynamic adaptation, knowledge transfer between facial action units and expressions, fine-grained facial modeling, and efficient audiovisual adaptation, respectively. Discussing and comparing the most relevant studies would help position MiRA within the broader literature on dynamic facial expression recognition and clarify its contribution.

References

[1] Chen, Y., et al. From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos. IEEE Transactions on Affective Computing, 2025.

[2] Li, J., et al. Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets. IEEE Transactions on Affective Computing, 2026.

[3] Chen, H., et al. FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs. ACM Multimedia, 2024.

[4] Chen, Y., et al. CLAIP-Emo: Parameter-Efficient Adaptation of Language-Supervised Models for In-the-Wild Audiovisual Emotion Recognition. IEEE Signal Processing Letters, 2026.

Reviewer_Uv4LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MiRA, a framewise attention reweighting method for video-based facial expression recognition. The method combines frame-level attention confidence with intra-frame attention concentration to redistribute attention across video frames. Experiments on DFEW, MAFW, and FERV39k show improvements over VideoMAE-based baselines, while comparisons between the exact and approximate implementations demonstrate the potential for reducing computational overhead. The method is simple to integrate into existing Vision Transformer backbones and does not introduce additional trainable parameters.

Critical strengths

The proposed module is relatively simple to integrate. MiRA does not introduce additional trainable parameters, and the authors implement it within the VideoMAE attention blocks. The evaluation covers multiple facial expression recognition benchmarks. The authors evaluate on DFEW, MAFW, and FERV39k, with five-fold cross-validation on DFEW and MAFW. The paper reports both unweighted average recall (UAR) and weighted average recall (WAR), providing complementary measures of recognition performance

Critical weaknesses
  1. Table 2 examines different combinations of the two weighting coefficients, but these experiments do not directly compare the full method against variants using only frame-level confidence or only intra-frame concentration. For example, under κ=12\kappa=12 on DFEW, the balanced setting achieves 63.28 UAR, compared with 62.06 and 62.31 for the two imbalanced settings. These results support the choice of balanced weighting, but they do not establish whether both statistics are necessary or how much each contributes independently.

  2. The generality of the method across video transformer architectures has not been established. The experiments use VideoMAE-based ViT backbones at different scales, including ViT-B, ViT-L, and ViT-H. These results demonstrate scalability across model sizes within the adopted backbone family, but they do not show whether the proposed mechanism transfers to other video transformer architectures with different attention designs.

  3. The method is motivated by the need to emphasize localized facial dynamics over dominant global motion. However, the reported recognition results and attention analyses do not directly establish whether the improvements are concentrated on samples with subtle expressions, head movements, or partial occlusion. A targeted evaluation on such cases, or a comparison with annotations of facial motion, would help substantiate the proposed explanation for the performance gains

Questions for authors
  1. Can the authors provide component-wise ablations for frame-level confidence and intra-frame concentration? In particular, how does the method perform when either statistic is removed, with all other settings held constant? The current weighting sensitivity analysis in Table 2 does not directly answer this question.

  2. Has MiRA been evaluated with video transformer architectures other than VideoMAE? The current experiments cover different VideoMAE-based model sizes, but it remains unclear whether the mechanism transfers to other attention architectures without substantial modification.

  3. Can the authors provide a more controlled comparison with existing methods? In particular, can the contribution of MiRA be isolated using matched backbones, pretraining data, and fine-tuning configurations? This would help clarify whether the reported improvements arise from the reweighting mechanism itself.

  4. Can the authors directly evaluate the method on videos containing subtle facial expressions or substantial head motion? Since the central motivation is to improve sensitivity to localized facial dynamics, such experiments would provide stronger evidence for the proposed mechanism than aggregate recognition metrics alone.

  5. Why does full-depth reweighting perform better than some intermediate-depth configurations? Table 1 reports that κ=12\kappa=12 achieves the best results, while κ=6\kappa=6 and κ=10\kappa=10 perform worse in several settings. The authors suggest that intermediate depths may introduce inconsistencies across the attention hierarchy, but this explanation remains a hypothesis. Can further analysis establish the source of this performance pattern?

1 comment

The authors have provided a reasonable clarification of MiRA's intended scope and addressed my main concerns regarding its applicability. I am satisfied with the response and willing to increase my score.

Reviewer_1S18OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MiRA, a frame-marginal attention reweighting module for ViT-based video models. It derives frame-level confidence and intra-frame concentration from attention statistics, converts them into a frame prior, and redistributes attention toward localized facial dynamics rather than dominant global motion. Two modes are introduced: an exact post-softmax reweighting mode and a FlashLite pre-softmax approximation based on key-energy proxies that is compatible with Flash Attention. Experiments use VideoMAE pretrained on VoxCeleb2 and finetuned on DFEW, MAFW, and FERV39k, showing strong video-only FER results, efficiency gains, and analyses of exact vs. FlashLite behavior.

Critical strengths

Practical bridge between a principled mechanism and scalable attention kernels. The exact mode is conceptually clean, while FlashLite converts the same frame-marginal redistribution idea into logit biases using key-based proxy statistics. The paper demonstrates that this approximation closely matches exact-mode recognition performance while substantially reducing memory and runtime overhead. This is consequential because it makes the method usable with FlashAttention-style training rather than being a post-hoc attention-map manipulation that is impractical at scale.

Alignment-free, parameter-free plug-in with consistent empirical gains. MiRA adds no trainable parameters and avoids task-specific face alignment/cropping, yet improves over strong VideoMAE baselines and achieves state-of-the-art results among video-only FER methods on large benchmarks. The gains also extend from ViT-B to ViT-L/H, which supports the claim that the module is scalable and broadly applicable within the chosen setting.

Critical weaknesses

Limited empirical scope relative to the generality claims. The method is evaluated only on facial expression recognition with a VideoMAE/VoxCeleb2 pipeline. The paper frames MiRA as a plug-in for ViT backbones and video representation learning more generally, but there is no evidence on general video action recognition, other video pretraining objectives, or non-VideoMAE backbones. This limits confidence in the claimed generality and is the main concern for significance.

FlashLite’s proxy and hyperparameter choices are under-analyzed. The key-energy proxy is heuristic; the paper shows empirical closeness to exact mode but does not deeply establish when the proxy should correlate with attention-derived frame statistics. In addition, several important hyperparameters appear fixed. This weakens robustness/reproducibility evidence for the approximation.

Questions for authors

Can you provide sensitivity analyses or ablations , and the clipping bounds , and clarify whether they were fixed across all datasets and model scales?

Can you add at least one experiment outside facial expression recognition—e.g., a general video recognition benchmark or a different ViT video backbone—to substantiate the “plug-in for ViT backbones” and general video representation claims?

Under what conditions does the key-energy proxy correlate poorly with post-softmax frame-level attention statistics, and does FlashLite degrade gracefully in those cases?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You and a Gentle Follow-up

Dear Reviewers,

Thank you again for your thoughtful reviews and for your time and consideration. We have carefully addressed each of your comments in our responses and are currently incorporating them into the revised manuscript. We would be grateful if you could let us know whether our responses address your concerns, or if there is anything further we can clarify.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.