acceptodds
Withdrawn from ICLR 2027

Shape-Bayes: Bayesian Inference of Structured Shapes under Visual Ambiguity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Perceiving structured shapes, such as human faces, from pixels is an inherently ambiguous task under real-world conditions. Yet, shape inference is largely posed as a deterministic regression task, predicting fixed spatial coordinates. We find that deterministic regression is brittle when visual evidence is ambiguous or incomplete; under severe occlusions, deterministic models exhibit structural collapse, predicting incoherent shapes or reverting to generic averages. To address this, we introduce Shape-Bayes, a probabilistic framework that couples uncertainty-aware visual perception with Bayesian shape reasoning. Rather than forcing point estimates, Shape-Bayes dynamically weights confident visual evidence against geometric priors to infer a structurally coherent shape posterior. Using human face shape regression as a rigorous testbed featuring complex non-rigid deformations and strict anatomical constraints, Shape-Bayes comprises: (1) a base model that predicts noisy landmarks and aleatoric uncertainties distilled from a teacher; (2) a lightweight Transformer that encodes these observations into an adaptive prior over a PCA shape manifold; and (3) a differentiable Bayesian solver that computes closed-form posteriors by balancing the noisy predictions against this prior. By guaranteeing complete structural integrity (100% In-Distribution Rate), Shape-Bayes achieves an absolute improvement of up to  34% over state-of-the-art deterministic models. Simultaneously, it yields highly calibrated uncertainty bounds and establishes a new state-of-the-art for robust 2D face shape regression under severe occlusion, reducing relative error by up to 12.5%.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Probabilistic Shape Modelling, Occlusion-Robust Shape Perception, Uncertainty Modelling, Bayesian Inference, Face Alignment, Face Landmark Regression

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_a8nTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the question of fitting an Active Shape Model to face images that show severe occlusion. The authors follow a Bayesian approach (Contribution I): Uncertainty of the landmarks is quantified (Contribution II) and then incorporated into a differentiable form that is amenable to end-to-end training, thanks to differentiability of the posterior (Contribution III) . The authors claim state of the art Robustness and Accuracy (Contribution IV)

In its final form, the posterior distribution is a Gaussian whose posterior can be written down in closed form with analytic mean and covariance. Given this setting, no training would be needed in principle. However, there are two extensions over classical active shape model fitting. The prior uncertainty of the shape coefficients, which in a classical probabilistic formulation of Active Shape Model fitting is fixed, is determined by a transformer with learnable parameters. These parameters are learned from data in a training step and help that the prior becomes tuned to the given occlusion pattern. In addition to the MSE, which matches how well the landmarks are predicted, the cost function includes a NLL term, to help calibration.

An important part of the pipeline consists in estimating the initial landmark position and uncertainty. This is done using a separate model that predicts from augmented views of the image computing the uncertainty to a given ground truth fit. Evaluation of Accuracy is done on three different benchmarks with faces in the wild and is shown to be competitive. It also tests the approach with different base models and shows that modelling uncertainty using Shape-Bayes consistently improves the results. The paper then proceeds to discuss calibration, which shows that the uncertainty estimates are reasonable. Finally, it shows an ablation study on the 300 faces in the wild dataset. As expected, it shows that the uncertainty has a hugely positive influence and the dynamic prior a smaller positive influence, whereas the combination of both is best.

Critical strengths

Predictions of landmark position that not only output a point estimate, but also output a meaningful and well calibrated uncertainty is currently an under researched topic. The paper is a clear contribution in that area. While the measures taken in the paper seem slightly ad-hoc from a Bayesian point of view, the paper shows that incorporating uncertainty can be done while obtaining competitive performance using a combination of neural networks and closed form models, such as ASM. The paper also shows how two integrate different components, uncertainty estimation, prior adaption and classical shape model, into a coherent system.

The experiments are solid and show that the methods works as designed and the results are competitive and on two of the benchmarks better than the current state of the art. It is also shown that modelling the uncertainty leads to improvements with different base models. The ablation study shows that both the dynamic prior and the uncertainty contribute to the methods performance. The paper is clearly written and each component of the system is well motivated.

Critical weaknesses

The probabilistic model used in the paper is somehow unsatisfactory from a generative modelling point of view. It seems to me that the occluded landmarks should be treated as missing data and not just landmark points with a (slightly) inflated variance. Its variance should be the posterior variance given the observed landmarks. A comparison of the estimated variances on the synthetically occluded images with the exact posterior variances, obtained by treating the occluded landmarks as missing data, would have been interesting. What is also unusual from a Bayesian point of view is that the prior variance of the shape coefficients depends on the observations. A good shape prior of the face should not depend on the uncertainty of the landmarks observed in a particular image. The need for such a prior suggests that the model itself is misspecified, for example because of the entanglement of the pose and shape in the 2D project or because Gaussianity itself is not the correct assumption here.

The result section are detailed and overall solid. However, all the figures compare with a baseline that does not account for occlusion. It is not surprising that this does not work on the heavily occluded faces shown in the images. It is expected that any treatment of uncertainty will lead to improved results. Another point concerns the reporting of minNME in the accuracy comparison. It is not possible to select the best of these 100 samples, hence minNME does not say anything about the accuracy that will be achieved, only that the posterior covers the target. Therefore, the statement "Furthermore, drawing 100 posterior samples yields SOTA accuracy (minNME@100), outperforming specialized models like OccFace and ViT-Pose" seems misleading. It would have also been good to see examples of typical failure cases to get a visual impression, under what circumstances BayesShape fails to give accurate results.

While the paper presents a clear contribution, and a detailed experimental evaluation it fails to fully convince for the reasons detailed above. In addition, it builds on well-established techniques, which limits its novelty for the ICLR community and would be a better fit for a pure vision conference where the application is more in focus. For me the paper is therefore slightly under the acceptance threshold.

Questions for authors
  • Can you comment on the reason why on the WFLW dataset ShapeBayes seems to be quite a bit worse than OccShape.
  • It does not become clear how accurate the uncertainty estimates for the landmarks are. Are they close to the expected uncertainty (posterior given the other landmarks, or is the uncertainty underestimated?
  • It would be interesting to learn about the most common failure cases. Can you comment on those? Are these expected?
Reviewer_jyFQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Shape-Bayes, a probabilistic model for determining a posterior distribution of facial landmark locations under a variety of synthetic occlusions. The posterior distribution of landmark locations is obtained through a Bayesian combination of two Gaussian distributions: A Gaussian uncertainty distribution that is output by the base model, and a Gaussian "prior" distribution that is determined by a transformer whose input is the output of the base model. The method is set up so that it can use any base model that predicts landmark locations from an image, and in fact the paper tests the proposed method using many different base models.

Critical strengths

The paper proposes a method that can predict the locations of facial landmarks of partially occluded faces with calibrated uncertainty estimates, even using a deterministic base model for landmark localization that was never designed to produce an uncertainty distributions. The paper does so by first training an initial regression estimator of aleatoric uncertainty, using the base model and the "principle of visual feature stability" (lines 142-173).

However, this initial uncertainty estimate provided by the base regression model may not match the true errors of the estimates very well. To correct for this, the paper refines the initial uncertainty estimate using what the paper calls a "structural shape prior", which is a transformer that is trained to take in a set of predicted landmark locations and uncertainties (like those output by the base regression model) and output an update on the locations and the precision (inverse variance) of the landmark uncertainty estimates. The result is a posterior estimate of landmark locations and uncertainties that outperforms previous methods' results on occluded face images.

I think the biggest strength of the proposed method is that it can take any new state-of-the art deterministic landmark regression model and use it to create a new state-of-the-art probabilistic landmark localization model, which is more robust to occlusions than the base regression model. The experiments demonstrate this using 6 previous landmark regression models, which is excellent.

Critical weaknesses
  1. I like this paper a lot. It proposes a sound method that is very effective at estimating landmark locations and uncertainties from partially occluded face images. However, I believe that the paper's use of the term "prior" for the transformer is incorrect. I would prefer that it were called a Bayesian update, rather than a prior. The reason is that in Bayesian measurement models, priors are typically obtained prior to measuring the current input (the image). Then the prior estimate (which is not conditioned on the input image) is updated using a measurement model that takes the image as input. I don't think that is what is going on in the proposed method. Since the transformer's input depends on the input image, I don't believe it is correct to call it a prior. I think that what the proposed models is actually doing is not quite captured by the equation for the Bayesian Update given in Figure 2: p(S∣S′)∝pW(S′∣S)⋅pΛ(S)p(S | S') \propto p_W(S' | S) \cdot p_\Lambda(S).

Instead, after thinking about it for a while, I believe the proposed method can be better described as the following Bayesian update. The proposed method uses an initial measurement model (the base regression model) to obtain initial estimates of landmark locations and their uncertainties given the input image, which can be written as follows: $p(S' | I) \sim N(S', \sigma') $. (Note that the base model's estimates of uncertainty are not very good because they were not trained using log likelihood of the ground-truth locations.) The transformer is then trained to update the estimates of landmark locations and uncertainties using the initial estimates provided by the base landmark regression model. The transformer thus estimates pΛ(S∣S′).p_\Lambda (S | S'). The full Bayesian update performed by the proposed method can then be expressed as: p(S∣I)∝p(S∣S′)⋅p(S′∣I).p(S | I) \propto p(S | S') \cdot p(S' | I). This way of writing the equations acknowledges that the the transformer p(S∣S′)p(S | S') is not a prior, because it depend on S', which in turn depends on the input image I. (Priors should come before, i.e. prior to, the measurement of the current image.) Instead, this way of writing the equations explains the transformer as a Bayesian update on the initial landmark location and the rough uncertainty estimates that were provided by the measurement model (the base regression model). The posterior distribution is obtained by updating the base regression model's estimated distribution using the transformer. The model and all of the experiments in the paper can all be used as is, and are very good. I just think the explanation should be changed so that the transformer is interpreted not as a prior model, but as an update to the measurement model.

This interpretation also explains why LUVLi's uncertainty estimates are not improved as much by the transformer: since LUVLi's uncertainty estimates were already trained using the log likelihood of the ground-truth locations, its uncertainty estimates do not need as much updating. Nonetheless, the proposed method is still able to improve upon the output of LUVLi because the transformer model is better than LUVLi's architecture at determining which other landmarks should be attended to when predicting the locations and uncertainties of each landmark.

Other weaknesses:

  1. The paper claims that the proposed method will work not only for faces but for "any other structured shape inference." I think this is overstating the truth. The proposed method will generalize to some other structured shapes, but not all. E.g., I don't think it will generalize to articulated objects such as human bodies, because those may require non-Gaussian uncertainty distributions. The paper could be more specific (or at least more honest) about which types of shapes the method will generalize to.

  2. The section on Posterior Uncertainty Calibration should be explained in more detail, either in the main body of the paper or in an Appendix. For example, the ECE measure is never defined or explained. Also, the three graphs in Figure 7 should all be explained in more detail, preferably including citations to previous works that have used and explained graphs like these.

  3. One thing that was not clear from the paper is exactly how the uncertainty estimates are represented. Line 141 says that the base regression model estimates uncertainties using σ′∈RN×2\sigma' \in R^{N \times 2}, which suggests that each landmark's 2D covariance matrix contains only diagonal elements, σ1\sigma_1 and σ2\sigma_2. In other words, these should be axis-aligned 2D Gaussians. However, the base-model shape uncertainty predictions in Figures 5, 9, 10, and 11 all look like circularly symmetric Gaussians: σ1=σ2,\sigma_1 = \sigma_2, which is also what appears to be shown in the matrix in Figure 3. Did the base regression model actually only predict a single radial sigma value, rather than two separate sigma values for x and y?

  4. Also, have you thought about using a full 2D covariance matrix for each landmark's Gaussian, similar to the uncertainty estimate in LUVLi? Thus, instead of estimating the uncertainty Σ\Sigma as a diagonal matrix, it would be a block diagonal matrix, with 2 x 2 covariance matrix blocks (or 2 x 2 precision matrix blocks) on the diagonal. I believe this would give you even better results. For example, using full 2x2 covariance matrices would enable the Bayesian shape posterior Gaussians in Figure 9 to be aligned with the outer edge of the face, rather than trying to approximate the direction of the face edge using Gaussians that are axis-aligned (diagonal covariance matrix). This should enable the predictions and the log likelihood to be better, especially for landmarks that lie on edges that are oblique (not horizontal or vertical).

  5. Finally, the authors should make their artificially occluded training and test sets publicly available. That way, future methods can compare their results to the current paper's using the same test set, which will result in more citations for the current paper and would be a useful contribution to the community.

  6. The section on Related Work (Section 5) should be moved to much earlier in the paper, e.g., between Sections 1 and 2.

  7. The paper should cite a source for why they call it aleatoric uncertainty.

  8. The equation in line 162 appears to be squaring the N x 2 matrix (Si−Starget′)(S_i - S'_{target}) and then take the square root. It is not clear how to square an N x 2 matrix (nor how to take the square root of the result).

  9. Minor errors:

    a. Line 73 "As Figure 1 shows": Figure 1 actually shows only the first case (topologically invalid geometries), not the second case (regress to a generic average shape).

    b. Line 92 is missing the result number: "decreasing NME by __ for HR-noSA"

    c. Line 88: delete "a" before "brittle"

    d. Line 151: fix spelling of "rotated"

    e. Line 155 refers to "Appendix A", but that's the wrong appendix. I believe it should be Appendix F.

Questions for authors

Please respond to the Weaknesses I identified above, particularly weaknesses 1, 4, 5, and 6.

Reviewer_SBTEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Shape-Bayes combines facial landmark predictions and estimated observation uncertainty with a PCA shape representation. A Transformer predicts input-dependent diagonal precision values for a zero-mean Gaussian prior over PCA coefficients. A closed-form Gaussian update produces a posterior mean and covariance. For base models without native uncertainty estimates, prediction variability across image transformations is distilled into uncertainty predictions. Evaluation covers three facial landmark datasets, six base architectures across the available dataset/model combinations, synthetic occlusions, and original test sets. The reported benefits include lower landmark error, improved calibration, and 100% in-distribution rate under a PCA-based validity test.

Critical strengths

The strongest positive evidence is the consistency of the aggregate localization improvements across different base architectures, including models that already provide native uncertainty estimates. For example, on dynamically occluded 300W, HR-noSA's occluded-landmark NME decreases from 9.03 to 7.91, and overall NME decreases from 6.31 to 5.90 (Table 1). Table 6 also reports improved calibration. Together with the inexpensive analytical solve, these results suggest that uncertainty-aware shape fitting can be a useful addition to existing landmark predictors. This is a meaningful empirical benefit, although it does not outweigh the concerns below.

Critical weaknesses
  1. The overall scientific and practical contribution is insufficiently established. The core inference combines a classical linear shape representation with Gaussian conditioning. The learned prior adjusts regularization strengths within a fixed PCA basis. Such an integration can be useful, but the paper does not establish enough new insight or practical benefit to support its broad framing. In Table 7, a static prior with distilled uncertainty already reduces occluded-landmark NME from 9.03 to 8.08; the full adaptive-prior method reaches 7.91. The strongest robustness results rely on synthetic masks and conspicuous pasted-object cutouts (Appendix H; Figures 9-11). These are controlled missing-information tests, but do not establish transfer to natural occlusion. The practical value of recovering a particular hidden expression, rather than producing a plausible completion or identifying uncertainty, also needs clearer justification. Original-dataset results are provided, including COFW, but the mean-prediction gains in Table 4 are small, AUC decreases on all three datasets, and WFLW failure rate increases from 2.0% to 2.2%. My concern is the combined lack of sufficient insight, demonstrated relevance, and convincing evidence of a substantial advance; it extends beyond methodological novelty alone.

  2. Several central claims are stronger than the formulation and evaluation support. Equations 2-3 constrain predictions to the PCA subspace, whereas the IDR test in Equation 6 requires membership in a bounded confidence ellipsoid. Gaussian regularization discourages extreme coefficients but does not enforce that bound. A nondegenerate Gaussian posterior also has support outside any finite ellipsoid. The reported 100% IDR is therefore empirical, rather than guaranteed by the stated equations. Moreover, conformity to the PCA representation imposed by the method is limited evidence of anatomical validity. Equation 6 and Appendix G's Equation 9 appear to use different coefficient normalizations and conditions, which need reconciliation. The state-of-the-art accuracy claim also relies in part on minNME@100, which selects the closest candidate using ground truth. This measures oracle candidate coverage rather than automatically selected prediction accuracy; Appendix J further specifies deterministic eigenvector offsets before random sampling. For example, Table 5 reports WFLW NME of 3.94 for the posterior mean versus 3.77 for OccFace; 3.72 is the oracle minimum over candidates. These issues require correcting or qualifying the claims. The underlying Gaussian update can still be valid, but its demonstrated contribution is narrower than advertised.

Questions for authors
  1. Is there a coefficient constraint or other implementation step, absent from Equations 2-3, that enforces the bounded IDR region? Please reconcile the definitions in Equation 6 and Appendix G's Equation 9 and state which was used for the reported results.

  2. Do the existing original-dataset evaluations show a substantial benefit specifically on naturally occluded landmarks, separately from visible landmarks, including uncertainty calibration? Such evidence would affect the concern about transfer beyond synthetic corruptions.

These requests concern the current method and evidence. Resolving them would not automatically resolve the broader concern about the paper's overall significance.

Reviewer_LUWwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Shape-Bayes, a probabilistic framework for robust facial landmark estimation under occlusion. The method first obtains landmark coordinates and uncertainty estimates from a base detector. A lightweight Transformer then predicts an input-dependent precision matrix for a PCA shape prior, and a closed-form Gaussian update combines the noisy landmark predictions with this prior to produce a shape posterior.

Experiments on 300W, WFLW, and COFW demonstrate improvements in landmark localization, structural consistency, and uncertainty calibration across several existing detectors.

Critical strengths
  • The proposed framework improves landmark localization under synthetic occlusion across several backbone architectures. Its ability to work with different base detectors, including models with native uncertainty estimates, is a useful practical property.
  • The use of uncertainty-weighted shape fitting provides an intuitive way to reduce the influence of unreliable landmarks. The closed-form solver is computationally inexpensive and offers a clear interpretation of how visual predictions are reconciled with geometric constraints.
Critical weaknesses
  • The proposed framework largely revisits classical uncertainty-weighted PCA shape fitting. The Transformer only processes predicted landmarks and uncertainties to estimate prior precision, without incorporating additional visual evidence. The paper does not explain why this separate correction stage is preferable to directly learning structurally constrained landmark predictions.
  • Table 7 suggests that most improvements come from introducing shape regularization. On occluded 300W, a static prior with uncertainty reduces NME from 9.03 to 8.08, while the complete model only further improves it to 7.91. The necessity of the adaptive Transformer prior and the proposed uncertainty distillation remains insufficiently demonstrated.
  • The reported 100% IDR is closely tied to the PCA shape representation and does not establish guaranteed anatomical validity. Moreover, the SOTA claims partly rely on minNME@100, which selects the best hypothesis using ground-truth error and therefore represents oracle-assisted evaluation rather than achievable prediction accuracy.
Questions for authors
  • Why is a separate Transformer-based prior estimation and Bayesian correction module preferable to directly predicting PCA shape coefficients or incorporating explicit shape constraints into the base landmark detector? Can the authors provide comparisons under comparable training and computational budgets?
  • Given the relatively small differences in Table 7, how much of the final improvement is specifically attributable to the Transformer-based adaptive prior? Would a simple MLP or a fixed learned prior achieve comparable performance?
  • How should the reported 100% IDR be interpreted as a theoretical guarantee, given that the evaluation criterion includes a bounded confidence region? Can the authors clarify the distinction between PCA-subspace membership and anatomical validity?
  • Can the authors separate achievable posterior-mean accuracy from oracle-assisted minNME@100 when making state-of-the-art comparisons? How well does the proposed uncertainty measure correlate with actual localization errors, especially on naturally occluded images?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.