acceptodds
Under review as a conference paper at ICLR 2027

Occlusion-Aware Dual-View Masked Autoencoding for Facial Representation Learning

Abstract

Visual intelligence is increasingly shifting from world-centric perception toward human-centric understanding. However, the visual evidence required to interpret human behavior and affective states remains susceptible to occlusion-induced corruption. Existing masked pretraining paradigms leave this setting underexplored, as their objectives can be satisfied by recovering missing facial content from visible context without requiring the encoder to distinguish facial evidence from occluder-induced appearance. When corrupted regions remain visible, the pretraining objective does not explicitly constrain the encoder against treating occluder-specific patterns as valid facial evidence, allowing them to persist alongside behavior-relevant facial information in the learned representation. We propose a dual-view masked autoencoding framework that learns from paired clean and synthetically occluded facial views. In the occluded branch, corrupted patches remain visible and are guided to reconstruct their corresponding clean facial content rather than the occluder appearance, directly linking corrupted observations to the facial information they obscure. This encourages the encoder to preserve facial semantics while reducing reliance on occluder-specific cues. Experiments on facial action unit detection and facial expression recognition demonstrate the effectiveness of the learned representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Facial representation learning, Masked autoencoding, Facial occlusion, Self-supervised pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RzKgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines clean-image masked autoencoding with occluded-image reconstruction to improve facial analysis under occlusion.

Critical strengths

The method addresses a practical problem without requiring ground-truth occlusion annotations.

Critical weaknesses
  1. As shown in ablation studies, relative to its matched clean-only baseline, the full model gains are quite small. In addition, based on the results of FER, QCS achieves higher accuracy (56.67% vs. 54.75%) with fewer parameters (55.17M vs. 85.81M). How these results demonstrate that the framework is useful?

  2. The technical novelty is incremental: the method combines standard MAE and denoising reconstruction with shared branches and a learned occlusion indicator. Similar mechanism appears in the previous studies[1]. A direct experimental comparison with existing mechanisms is needed to demonstrate that the new complex mechanism is better than the previous simple one.

[1] Denoising Masked Autoencoders Help Robust Classification. ICLR, 2023.

Questions for authors

See weaknesses.

Reviewer_gouHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a method of distinguish facial evidence from occluder-induced appearance in the MAE framework to improve facial representation learning. In particular, the occlude is implemented with synthetic corruption and applied is utilized to mask facial region for MAE training. In addition, the pre-defined mask is used to induce two kinds of reconstruction losses, one is for inpainting and one is for recovery under occlusion. Experiments on facial action unit detection and facial expression recognition proved the effectiveness of the proposed method.

Critical strengths

(1) The motivation is well presented of leveraging synthetic corruption for MAE training to improve facial representation learning.

(2) The illustrations are mostly clear and intuitive of the synthetic corruption, the occlusion embedding and training objectives.

Critical weaknesses

(1) On the motivation of synthetic corruption, it is another kind of augmentation different from empty or noise. However, there is no intrinsic improvement over the MAE framework. In fact, different kinds of corruption are straight-forward for MAE training.

(2) The overall contribution is limited since the core idea of synthetic corruption is trivial to the MAE framework and the setting is prone to confusion on reconstruction of real occlusion and synthetic occlusion.

(3) On Page 3, Line 155-156, the authors define “clean” as source image before synthetic corruption and does not imply that the original image is free of occlusion. However, this may introduce confusion for MAE training if the original image contains occlusion.

(4) On Page 3, Line 159-160, the authors add a learnable occlusion embedding to masked patches. However, this can cause inaccurate semantics if the mask is not indicative to the exact occluded region. This problem can be serious if the blended image has boundary distortions.

(5) On Page 4, Figure 2, the occlusion indicators are pre-defined during the synthetic procedure for data-preparation convenience. However, native occlusions are not indicated and ignored during training. This is misleading for MAE reconstruction.

(6) On the experiments, FAUD and FER are evaluated but in a controlled setting which considers mainly occlusion-aware cases. There are no basic comparisons on standard benchmarks.

Questions for authors

No.

Reviewer_L4wsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ODMAFace, an occlusion aware, dual-view masked autoencoding framework for facial representation learning. The proposed method jointly learns from clean and synthetically occluded facial views to introduce corrupted visual evidence into facial representation pretraining with a shared ViT encoder. The encoder preserves informative facial cues with much less reliance on occluder-specific appearance. The pre-trained encoder is applied to downstream tasks including Facial Action Unit detection and facial expression recognition, that further validates the effectiveness of the learned facial representation.

Critical strengths
  1. The central idea of this paper is clear, easy to follow, and provide a valid solution for facial representation learning under occlusion, directly linking corrupted observations to the corresponding clean face image. Different from standard MAE pre-training, a clean branch and a occluded branch with shared encoder will encourage the encoder to identify the affected tokens during pretraining and preserve informative facial features rather than relying on the occluder appearance.
  2. The reported downstream task results surpass state-of-the-art methods.
Critical weaknesses
  1. The major weakness of this paper is limited technical novelty for a ICLR paper. Comparing to existing MAE and occlusion-aware self-supervised learning methods, the dual branch strategy does not make the paper stand out. I have seen similar dual view idea in large-pose facial recovery papers.

  2. The author emphasizes multiple times that the dual-view model reduces reliance on the occluder-specific appearance, but it is not fully proved by experimental results. The experimental section focus on AU recognition and facial expression recognition results, but feature-level analyses are missing. For example, the author can show feature difference among the variants in Table. 3, not just the F1 results.

  3. In Table 3, comparing to the clean-only baseline, the DISFA F1 improvements over different variants are small and statistical variance is not reported.

  4. ODMAFace is evaluated on DISFA and AffectNet, which contain limited occlusion cases. This raises my question of why the method is presented as occlusion-aware but is not evaluated on real-world occluded facial images. Instead, all occlusion experiments are based on synthetical occlusions.

Questions for authors

Please check the weakness section. I also have two additional questions:

(1) The motivation and the experiments are not well aligned. The model focus on the occlusion problem in facial representation learning and is targeting at identifying the affected tokens and preserve informative facial features rather than relying on the occluder appearance. The model is pre-trained only using image reconstruction losses. It is unclear why this occlusion-aware reconstruction objective would necessarily improve facial expression recognition and AU detection performance. Especially for AU recognition on DISFA, which primarily contains controlled, in-lab facial recordings without significant object occlusions. The main challenge in AU detection is recognizing subtle facial movements and distinguishing between similar AU activation features. So it is unclear why the proposed occlusion-aware pretraining strategy leads to significant improvements in AU detection task.

(2) Are the reported performance improvements primarily caused by the proposed occlusion-aware learning mechanism, or could they result from increased synthetic image augmentation and larger amount of images during pre-training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.