acceptodds
Withdrawn from ICLR 2027

OCA3D: Occlusion-Cause-Aware 3D Facial Evidence Completion for Facial Expression Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world facial occlusion arises from two fundamentally different causes: external objects, such as masks, glasses, hair, and hands, and pose-induced self-occlusion. Existing methods typically address external occlusion and self-occlusion either independently or as generic appearance corruption, without explicitly distinguishing their underlying visibility mechanisms or exploiting cause-specific 3D facial correspondences for evidence recovery. We propose OCA3D, an occlusion-cause-aware 3D facial evidence completion framework that explicitly models these two sources of missing facial evidence. From each input image, OCA3D predicts an external-occlusion mask and regresses FLAME parameters, based on which a mesh-bound 3D Gaussian representation is constructed over the fixed facial topology. Self-occluded regions are completed through topology-defined bilateral symmetry, while Gaussian attributes for externally occluded regions are recovered from reliable visible mesh faces through nearest-neighbor correspondence within fine-grained semantic regions. The completed representation is rendered from both the input and frontal views, and the resulting evidence is fused with original-image features for FER. We further introduce Occlu-FER-Mask, a manually annotated dataset providing pixel-level supervision for external-occlusion segmentation in real-world FER images. OCA3D achieves Top-1 accuracies of 93.06%, 92.15%, 90.19%, and 89.73% on RAF-DB, FERPlus, Occlusion-RAF-DB, and Occlusion-FERPlus, respectively, and remains robust under large-pose self-occlusion on Pose30/45-RAF-DB and Pose30/45-FERPlus (93.50%, 93.01%, 92.85%, 92.58%) as well as real-world external occlusions. These results demonstrate the effectiveness of modeling occlusion according to its cause and recovering expression evidence directly on the 3D facial surface.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Occluded Facial Expression Recognition, 3D Gaussian Splatting, FLAME, Geometric Completion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WG2UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses facial expression recognition (FER) under occlusion by proposing OCA3D, an occlusion-cause-aware 3D facial evidence completion framework. OCA3D explicitly models these two sources of missing facial evidence. From each input image, OCA3D predicts an external-occlusion mask and regresses FLAME parameters, based on which a mesh-bound 3D Gaussian representation is constructed over the fixed facial topology. Self-occluded regions are completed through topologydefined bilateral symmetry, while Gaussian attributes for externally occluded regions are recovered from reliable visible mesh faces through nearest-neighbor correspondence within fine-grained semantic regions. The completed representation is rendered from both the input and frontal views, and the resulting evidence is fused with original-image features for FER.

Critical strengths

The method achieves state-of-the-art results across four benchmarks, with particularly impressive gains on pose-variant datasets, which aligns with the method's intended strength in handling self-occlusion.

Critical weaknesses

The paper's positioning statement in the introduction claims that "Existing methods typically address external occlusion and self-occlusion either independently or as generic appearance corruption, without explicitly distinguishing their underlying visibility mechanisms or exploiting cause-specific 3D facial correspondences for evidence recovery." However, this claim overlooks several relevant prior works: 3D facial expression recognition has been studied since at least Soyel & Demirel (ICIAR 2007) [1], which uses 3D facial feature distances for expression classification, demonstrating that 3D geometry has been leveraged for FER well before this work.

3D face inpainting has been addressed by Bai et al. (ICASSP 2023) [2], which proposes fine-grained blind face inpainting using 3D face component disentanglement — directly relevant to the paper's claim of novel 3D evidence recovery. Face inpainting for FER has been explored by Xing et al. (ACM MM 2022) [3], which proposes co-completion of occluded faces specifically for expression recognition, closely aligning with the paper's overall pipeline of inpainting/completing occluded faces followed by FER. The proposed OCA3D method appears to be a combination of ideas from [2] (3D face inpainting/completion) and [3] (FER via face completion), with the cause-aware distinction (external vs. self-occlusion) serving as the main differentiator. The paper should more accurately position itself relative to these works and explicitly discuss what specific design choices go beyond a straightforward combination of prior 3D inpainting + FER methods. The Related Work section should be expanded to include and discuss these references, and the novelty claim should be tempered accordingly. References: [1] Soyel, H., Demirel, H. (2007). Facial Expression Recognition Using 3D Facial Feature Distances. In: Kamel, M., Campilho, A. (eds) Image Analysis and Recognition. ICIAR 2007. LNCS, vol 4633. Springer, Berlin, Heidelberg. [2] Bai, Y., He, R., Tan, W., Yan, B., Lin, Y. (2023). Fine-Grained Blind Face Inpainting with 3D Face Component Disentanglement. In ICASSP 2023. [3] Xing, Z., Tan, W., He, R., et al. (2022). Co-Completion for Occluded Facial Expression Recognition. Proceedings of the 30th ACM International Conference on Multimedia.

Questions for authors

The pipeline involves FLAME fitting, 3D Gaussian construction, rendering from two views, and feature fusion — significantly more complex than image-based baselines. The paper does not report inference time, GPU memory usage, or training time. A comparison with baselines on these metrics would help readers assess practical feasibility.

In Figure 3 (qualitative comparison under real-world occlusion), the completion results produced by OCA3D appear overly blurry and lack fine facial details. More importantly, it is difficult to discern whether the completed regions actually convey a "surprise" expression, the facial cues (e.g., raised eyebrows, widened eyes) that would distinguish surprise from other expressions are not clearly rendered. This raises concerns about the visual fidelity and expressiveness of the completed faces, which is particularly problematic for an FER task where subtle expression details are critical. The authors should either improve the sharpness and expressiveness of the completed regions or provide a more rigorous qualitative evaluation protocol (e.g., user studies) to substantiate the visual quality claims.

Reviewer_8wpdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a 3D facial expression recognition framework based on 3D facial evidence completion. The model focus on the occlusion challenge in FER, distinguishing external-occluded and self-occluded region in the input image. For identifying external occlusions in the image, the author constructed a manually annotated dataset "Occlu-FER-Mask", providing pixel-level masks for external facial occlusions.

Critical strengths
  1. The methodology is clearly written, easy to follow.
  2. The experimental results shows encouraging improvements over existing methods in large-pose FER dataset.
  3. The real-world occlusion annotations in the Occlu-FER-Mask dataset is beneficial to the FER & 3D Face Reconstruction community.
Critical weaknesses
  1. The novelty of this paper is limited. The major components proposed in this work—including occlusion segmentation, 3D face reconstruction, and occlusion-aware Gaussian attribute completion—primarily combine existing techniques. First, binding 3D Gaussian primitives to a predefined mesh template is not new in 3D face reconstruction area. Second, the symmetry constraints and the nearest-neighbor interpolation strategy for handing self-occlusion and external occlusion is not novel. Similar idea has been implemented in previous 3D face reconstruction studies such as [1][2][3]. The major difference is that this paper performs occlusion-aware reconstruction using 3D gaussian attributes.

[1]Dey, Rahul, and Vishnu Naresh Boddeti. "3DFaceFill: An analysis-by-synthesis approach to face completion." In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1224-1233. IEEE, 2022.

[2]Chen, Jinsong, Hu Han, and Shiguang Shan. "Towards high-fidelity face self-occlusion recovery via multi-view residual-based GAN inversion." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, pp. 294-302. 2022.

[3]Zeng, Xiaoxing, Zhelun Wu, Xiaojiang Peng, and Yu Qiao. "Joint 3D facial shape reconstruction and texture completion from a single image." Computational Visual Media 8, no. 2 (2022): 239-256.

  1. The validity of 3D face completion improves FER is questionable. For self-occlusion cases, since the symmetry constraint is introduced for large-pose images, the gaussian attributes of the visible side of the face should be sufficient for recognizing the expression. Completing the self-occluded part does not bring more 3D information for the FER task. Similar for external occlusion, if the essential face region is covered by objects, the expression classification should have low confidence / high entropy. The KNN completion can generate incorrect 3D geometry resulting in high-confidence FER prediction.

  2. Important ablation study is missing. The Appendix includes hyperparameter sensitivity study and comparisons with alternative restoration methods, but these do not prove the contribution of the occlusion-cause-aware mechanism. The following ablation studies should be considered by the author: (i) 3D reconstruction without gaussian splatting for FER (for example, the author can construct a FER baseline by adapting the EMOCA model on different dataset). (ii) 3D reconstruction directly without any completion for FER. (iii) 3D reconstruction with all symmetry completion for FER. (iv) 3D reconstruction with all semantic completion for FER. (v) 3D reconstruction ignoring self-occlusion part for FER. (vi) 3D reconstruction ignoring external-occlusion part for FER.

Questions for authors
  1. What is the technical advance beyond combining existing 3D facial reconstruction, Gaussian representations, symmetry-based completion, and feature fusion techniques?
  2. The details of the Occlu-FER-Mask dataset is missing in the main paper and the Appendix. Is the occlusion mask generated by manual annotation or segmentation tools?
  3. What is the computational efficiency of the model comparing to existing methods?
  4. In Eq.8, the L_render is masked by M_ext, is M_ext a binary mask with 1 values for the occluded region? If so, then the rendering loss is only applied to the occluded pixels.
Reviewer_gi2JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OCA3D, a framework for robust Facial Expression Recognition (FER) under occluded conditions. The authors focus on the fact that facial invisibility stems from two fundamentally different causes: external occlusion (e.g., masks, glasses) and pose-induced self-occlusion. To address this, OCA3D utilizes a 3D Gaussian representation bound to the FLAME topology to perform cause-specific completion. For external occlusions, it applies semantic-region-constrained nearest-neighbor correspondence, while for self-occlusions, it performs topology-defined bilateral symmetry completion to restore missing expression information. Extensive benchmark experiments demonstrate that the proposed method achieves superior robustness and high recognition accuracy in occluded FER scenarios.

Critical strengths

The paper effectively highlights the distinction between two physical mechanisms of occlusion: external object occlusion and pose-induced self-occlusion.

Introducing cause-specific recovery strategies—nearest-neighbor completion from visible surfaces within the same semantic region for external occlusion, and 3D topology-based bilateral symmetry completion for self-occlusion—is highly sound and innovative.

The method is evaluated on large-scale benchmarks such as RAF-DB and FERPlus, demonstrating that it achieves state-of-the-art (SOTA) accuracy.

By applying completion based on 3D structural constraints and the physical causes of occlusion, the framework successfully balances visual restoration with downstream emotion recognition accuracy.

Critical weaknesses

Because the framework involves multiple pipelines—including external occlusion segmentation, FLAME geometry estimation, 3D Gaussian attribute prediction/completion, and dual-view rendering—it is presumed to have a significantly higher computational cost compared to conventional 2D FER methods.

The framework appears highly dependent on the performance of pre-trained modules and hyperparameter settings. It may not be easy to tune for specific custom image sets, and the paper lacks sufficient discussion on this limitation.

.

Questions for authors

In Table 3, the results show that ORSANet outperforms the proposed method at a 30% occlusion ratio. Do you have any insights or discussions regarding why this happens?

In the ablation study, it is crucial to understand how each element affects the results and how sensitive the model is to each hyperparameter. Table 4 shows partial experimental results, but what are the results when each of the three λ\lambda values is set to 0?

Additionally, regarding Table 4, the granularity of the changes applied to the λ\lambda values is inconsistent, making it difficult to properly compare the model's sensitivity to these hyperparameters.

Is there a specific reason why the number of nearest neighbors, KK, is capped at 4?

What is the total processing time from inputting an image to outputting the expression recognition result? Is real-time processing feasible with this pipeline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.