- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates the question of fitting an Active Shape Model to face images that show severe occlusion. The authors follow a Bayesian approach (Contribution I): Uncertainty of the landmarks is quantified (Contribution II) and then incorporated into a differentiable form that is amenable to end-to-end training, thanks to differentiability of the posterior (Contribution III) . The authors claim state of the art Robustness and Accuracy (Contribution IV)
In its final form, the posterior distribution is a Gaussian whose posterior can be written down in closed form with analytic mean and covariance. Given this setting, no training would be needed in principle. However, there are two extensions over classical active shape model fitting. The prior uncertainty of the shape coefficients, which in a classical probabilistic formulation of Active Shape Model fitting is fixed, is determined by a transformer with learnable parameters. These parameters are learned from data in a training step and help that the prior becomes tuned to the given occlusion pattern. In addition to the MSE, which matches how well the landmarks are predicted, the cost function includes a NLL term, to help calibration.
An important part of the pipeline consists in estimating the initial landmark position and uncertainty. This is done using a separate model that predicts from augmented views of the image computing the uncertainty to a given ground truth fit. Evaluation of Accuracy is done on three different benchmarks with faces in the wild and is shown to be competitive. It also tests the approach with different base models and shows that modelling uncertainty using Shape-Bayes consistently improves the results. The paper then proceeds to discuss calibration, which shows that the uncertainty estimates are reasonable. Finally, it shows an ablation study on the 300 faces in the wild dataset. As expected, it shows that the uncertainty has a hugely positive influence and the dynamic prior a smaller positive influence, whereas the combination of both is best.
Critical strengths
Predictions of landmark position that not only output a point estimate, but also output a meaningful and well calibrated uncertainty is currently an under researched topic. The paper is a clear contribution in that area. While the measures taken in the paper seem slightly ad-hoc from a Bayesian point of view, the paper shows that incorporating uncertainty can be done while obtaining competitive performance using a combination of neural networks and closed form models, such as ASM. The paper also shows how two integrate different components, uncertainty estimation, prior adaption and classical shape model, into a coherent system.
The experiments are solid and show that the methods works as designed and the results are competitive and on two of the benchmarks better than the current state of the art. It is also shown that modelling the uncertainty leads to improvements with different base models. The ablation study shows that both the dynamic prior and the uncertainty contribute to the methods performance. The paper is clearly written and each component of the system is well motivated.
Critical weaknesses
The probabilistic model used in the paper is somehow unsatisfactory from a generative modelling point of view. It seems to me that the occluded landmarks should be treated as missing data and not just landmark points with a (slightly) inflated variance. Its variance should be the posterior variance given the observed landmarks. A comparison of the estimated variances on the synthetically occluded images with the exact posterior variances, obtained by treating the occluded landmarks as missing data, would have been interesting. What is also unusual from a Bayesian point of view is that the prior variance of the shape coefficients depends on the observations. A good shape prior of the face should not depend on the uncertainty of the landmarks observed in a particular image. The need for such a prior suggests that the model itself is misspecified, for example because of the entanglement of the pose and shape in the 2D project or because Gaussianity itself is not the correct assumption here.
The result section are detailed and overall solid. However, all the figures compare with a baseline that does not account for occlusion. It is not surprising that this does not work on the heavily occluded faces shown in the images. It is expected that any treatment of uncertainty will lead to improved results. Another point concerns the reporting of minNME in the accuracy comparison. It is not possible to select the best of these 100 samples, hence minNME does not say anything about the accuracy that will be achieved, only that the posterior covers the target. Therefore, the statement "Furthermore, drawing 100 posterior samples yields SOTA accuracy (minNME@100), outperforming specialized models like OccFace and ViT-Pose" seems misleading. It would have also been good to see examples of typical failure cases to get a visual impression, under what circumstances BayesShape fails to give accurate results.
While the paper presents a clear contribution, and a detailed experimental evaluation it fails to fully convince for the reasons detailed above. In addition, it builds on well-established techniques, which limits its novelty for the ICLR community and would be a better fit for a pure vision conference where the application is more in focus. For me the paper is therefore slightly under the acceptance threshold.
Questions for authors
- Can you comment on the reason why on the WFLW dataset ShapeBayes seems to be quite a bit worse than OccShape.
- It does not become clear how accurate the uncertainty estimates for the landmarks are. Are they close to the expected uncertainty (posterior given the other landmarks, or is the uncertainty underestimated?
- It would be interesting to learn about the most common failure cases. Can you comment on those? Are these expected?