- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a surface aligned residual representation for refining existing 3D face reconstruction from multiple calibration images. This method does not predict the correction along the camera light or directly update the facial model coefficients, but represents the correction of each sampled surface point as a signed displacement along its local normal. The image features extracted by the frozen DINOv2 encoder are projected onto the currently reconstructed surface, combined with the observed geometric shape, and aggregated between views using mean and standard deviation pools. Learn the decoder to predict the residual displacement field, and then convert it into parameterized face model coefficients through robust regularization fitting.The core contribution is a model independent calibration interface that separates residual prediction based on learning images from downstream parameterized geometric representations. Experiments on FaceScape show that surface normal correction is superior to camera ray correction of targets, especially in oblique viewing angles. Through five views, this method achieved a residual correlation of 0.738 and reduced the symmetric chamfer distance by approximately 4.5% from strong initialization and 8.5% from image prediction initialization.The author also evaluated several existing face reconstruction methods and demonstrated the transfer of cross dataset to Headspace without retraining the residual predictor.
Critical strengths
1.Clear and innovative principle expression, sufficient geometric foundation Anchoring corrective evidence onto surface normals is a simple but important contribution, rather than a simple engineering improvement. By transferring the correction space from the perspective dependent ray space and the coefficient space bound to the model to the surface relative space, this method achieves dual decoupling: parametric decoupling from camera perspective and facial models. The geometric analysis in Section 3.1 strictly explains the reason why the normal displacement condition is better, and the theoretical intuition is highly consistent with the experimental results. This insight from the representational level can be extended beyond facial refinement, and has reference significance for broader multi perspective 3D refinement research. 2.Strict experimental design, complete control and ablation experiments The experimental design of this article is rigorous and the control of confounding variables is very effective. The author controlled the initial reconstruction quality by interpolating the initial values, isolated the influence of the features themselves through parameter matching, and conducted comprehensive ablation experiments on variables such as encoder, fusion strategy, and model parameterization. Of particular importance, the paper established strong negative controls (overall mean field, random field, other subject images, etc.), ruling out other possible explanations. The zero sample transfer experiment across datasets also convincingly validated the reusability of the predictor.
Critical weaknesses
- Insufficient verification in real unrestricted scenarios This method is based on two strong premises: the camera has been accurately calibrated and the initial grid is well aligned. Although the appendix tested the effects of camera perturbations, it only covers independent single factor perturbations and assumes that the initial geometric shape is accurate. This paper did not evaluate performance in common real-world scenarios, such as uncalibrated cameras, large initial pose errors, or low-quality initial reconstructions with topological defects. Due to the dependence of the entire surface alignment framework on precise 2D-3D correspondence, the performance degradation under imperfect initial conditions is still unclear, which limits the applicability of this method in uncontrolled acquisition scenarios. 2.In cross dataset migration, the refinement effect of individual specificity is relatively weak On the headspace dataset, the vast majority of error reduction comes from correcting the overall level sharing bias of Pixel3DMM, while the additional gain brought by specific subject image evidence is very limited (about 0.03mm). Although the authors mentioned this phenomenon, they did not analyze its fundamental causes or demonstrate meaningful fine-grained individual geometric refinement effects across datasets. Therefore, the conclusion of "cross dataset generalization" is more suitable for correcting system reconstruction bias, rather than the more valuable and challenging refinement of individual geometric details in the scene, which weakens the practical impact of transferable conclusions.
Questions for authors
Q1. How does the proposed correction representation compare to the powerful matching grid alignment refinement alternative? Can the author evaluate a multi view mesh alignment refinement tool that predicts 3D or normal displacement of vertex directions based on the same projection features under matched training data, regularization, and optimization budget? This will help determine whether the proposed interface provides significant advantages beyond using established surface alignment features and geometric fitting.These pieces of evidence will greatly affect my evaluation of the novelty of this method. Q2. How sensitive are the results of external reconstructors to reference scan assisted alignment? Appendix F.2 states that prior to refinement, each external reconstruction is aligned with the FLAME fit exported from the scan. Can the author evaluate this method using initial alignment based solely on available images and reconstructed output estimates, without reference to scanning geometry? Or, can the author quantify the impact of initial placement errors? This will clarify to what extent the reported improvements have been translated into a realistic reconstruction pipeline. Q3. To what extent are the improvements on Headspace attributable to subject-specific image evidence rather than population-level correction? Considering that the overall average residual field performs almost as well as the predicted values of specific subjects, can the author analyze the improvement of specific subjects after considering this shared correction? Segmentation based on initialization errors, visible facial coverage, or anatomical regions will be particularly useful. The answer will clarify whether the benefits of cross datasets mainly represent system bias correction or personalized geometric refinement.
3 comments
Response to Reviewer 8c4F (2/2)
W1. Validation under imperfect cameras, large pose errors and low-quality starts.
Thank you for this concern. In short, the method needs calibrated cameras, and within that setting it is robust. We address the four parts of your comment in turn.
- Joint camera errors. Table 14 in Appendix E.3 now includes rows in which focal length, rotation and camera centre are perturbed together. At 1%, 0.1° and 1 c.u., the gain is +0.058 c.u. at α = 0.5 and +0.182 c.u. at α = 1. This error level moves the face by about 2.5 pixels in the 224-pixel crop, or about 20 pixels at full resolution, which is far above the reprojection error of a calibrated rig. At 5%, 0.25° and 2 c.u. the gain is still +0.038 c.u. at α = 1, and it turns negative at α = 0.5.
- Uncalibrated cameras. The method is designed for calibrated capture, and Section 6 states this requirement together with the tolerance above. Cameras estimated by VGGT are about 50 crop pixels off and lie outside this range.
- Large initial pose errors. The image-only placement in Q2 adds about 5° of rotation error and about 5 c.u. of position error, and the gains hold.
- Low-quality starts. The starts in Table 2 are 5.3 to 5.7 c.u. from the scan, compared with 2.0 c.u. for the α = 0.5 start. From these starts our method gains +1.41 to +1.55 c.u. and improves 37 to 39 of the 40 identities, as Tables 3 and 16 show. The α = 2 start in Table 7 is twice as far from the scan fit as the image prediction. From it, one round gains +0.4125 c.u. and improves all 40 identities.
Regarding topological defects, the predictor works per point and accepts any mesh. The fit uses the topology of the face model, so a defective mesh would first be fitted to the model.
In summary, calibration is a real requirement of the method. Within the accuracy of calibrated rigs, the method is robust to joint camera error, to pose errors of about 5° and to starts that are far from the scan.
W2 and Q3. Is the cross-dataset gain a systematic correction or a subject-specific refinement, and what is the cause?
Thank you for asking for this analysis. On Headspace the gain is mainly a correction of Pixel3DMM's systematic error. Section 6 states this, and the revised abstract now says it as well. Appendix F.5 now analyzes the cause, and the FaceScape results show clear subject-specific refinement.
Why the shared part dominates on Headspace:
- Pixel3DMM's own error there is mostly shared. Through the same fit, the mean ground-truth offset gives 0.255 mm, and each subject's own ground-truth offsets give 0.349 mm. So even perfect evidence is 73% shared.
- The rig offers little multi-view evidence. Two of the five pods see 73% and 75% of the face, and the other three see under 5%.
- Even so, Table 18 shows that a subject's own images give 0.030 mm more gain than another subject's images, with a 95% interval of 0.023 to 0.037 mm.
Subject-specific refinement on FaceScape, after the shared correction is removed:
- From the bias-corrected Pixel3DMM start, the subject's own field beats the shared correction applied a second time by +0.208 c.u. The 95% interval is +0.163 to +0.253, and 37 of 40 identities improve.
- By anatomical region, the own field exceeds the population-mean field by +0.193 c.u. on the chin, +0.166 c.u. on the lips, +0.086 c.u. on the nose and +0.071 c.u. on the eye region. Appendix E.2 reports this.
- By expression, it exceeds the population-mean field on all 20 expressions, by +0.065 to +0.150 c.u., and every 95% interval is above zero.
Therefore, the cross-dataset result shows that a predictor trained on one dataset transfers a useful systematic correction to another without retraining, and this correction is about half of what the mean ground-truth correction achieves. Individual geometric refinement is demonstrated within FaceScape, across five reconstructors and all facial regions. Its transfer across capture rigs is weaker, and the paper states this boundary in Section 6.
Thank you again for the supportive and precise review. We hope the new experiments resolve your questions on novelty, placement and transfer, and we would be glad to provide any further detail.
Follow-up to Reviewer 8c4F
Dear Reviewer 8c4F,
Thank you again for your positive and careful review. We understand that this is a busy period, so we briefly summarize how our response addresses your questions.
- Q1, a matched vertex-displacement refiner. It gains +0.0956 c.u. against +0.0902 c.u. for our scalar head, while the surface target is ahead of the ray target by +0.0205 c.u. Tables 4 and 9 report this.
- Q2, scan-assisted alignment. With all starts placed from the images alone, our gains change by at most 0.06 c.u. on the four reconstructors, and the ranking of methods is unchanged. Table 17 reports this.
- W1, imperfect cameras. Under joint errors of 1%, 0.1° and 1 c.u. in focal length, rotation and camera centre, the gain is +0.058 c.u. at α = 0.5 and +0.182 c.u. at α = 1. Table 14 reports this.
- W2 and Q3, the cross-dataset gain. Appendix F.5 analyzes its cause. On FaceScape, the subject's own field beats the shared correction by +0.208 c.u.
We would be glad to answer any remaining question. If our response has resolved your concerns, we would be grateful if you could consider updating your assessment.
Thank you very much for your time.
Response to Reviewer 8c4F (1/2)
Thank you very much for the positive and careful review. We are glad that you found the geometric principle clear and the controls rigorous. Your three questions were specific and helpful, and we ran a new experiment for each of them. Section, table and figure numbers below refer to the revised PDF, where the changes for your points are marked in green and purple. Gains are Chamfer reductions on FaceScape in canonical units, where 1 c.u. is about 0.82 mm.
Q1. How does the representation compare with a matched vertex-displacement refiner?
Thank you for suggesting this experiment. We trained exactly this alternative, and Appendix B.1 reports it. The refiner uses the same projected DINOv2 features, the same fusion, the same training data and the same fit as our five-view model. Only the head differs. It predicts a 3D vertex offset, and the parameter count changes by 0.05%. The result is that the two heads perform almost the same, while targets outside the surface-aligned space are clearly weaker.
These rows come from Tables 1, 4 and 9.
A vertex-displacement refiner of this kind is itself a surface-aligned correction, so this result supports the claim of the paper. What matters is that the evidence and the correction are expressed on the current surface, and the compact scalar is sufficient within that space. Therefore, beyond established surface-aligned features and fitting, the interface provides a correction space that is better than ray space and coefficient space under matched conditions, and one predictor can serve several face models and reconstructors.
Q2. How sensitive are the results for external reconstructors to the scan-assisted alignment?
Thank you for raising this point, which matters for practical use. The answer is that the results do not depend on the scan-based placement. Appendix F.4 and Table 17 report a new experiment in which all five starts of Table 2 are placed from the images alone. A rigid pose is fitted to landmarks detected in the same five calibrated views, and no scan information is used. The scoring applies no alignment to the output, so any placement error counts against the method.
So the improvements carry over to a pipeline in which the start is placed without any reference scan, including pose errors of about 5°.