- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the candidate selection problem in visualizing postoperative results of facial plastic surgery. A fixed FLUX.1-Fill generator generates multiple candidate images for each case by varying the random seed and editing intensity (25%/50%/80%), but it lacks postoperative ground truth at deployment, requiring the selection of one image for display. The authors propose Envision, a setwise ranker shared across three procedures: rhinoplasty, blepharoplasty, and facelift. It combines three types of information: procedure-specific 5D morphological measurement changes, case-level editing directions predicted after mean pooling of the candidate set, and appearance features encoded by a trainable ViT. During training, Huber regression is performed using the SurgicalScore with the postoperative ground truth as the teacher score; during inference, only the preoperative image, procedure labels, and candidate set are needed.
Critical strengths
The article honestly presents the raw results.
Critical weaknesses
-
The experimental results fail to substantiate the conclusions. This paper's primary results (n=125) aggregate 76 validation set samples, which had previously been used for checkpoint selection based on the same "oracle-gain" criterion with only 49 held-out test set samples. Consequently, the reported significance is not an independent estimate. When considering only the held-out test set, the performance gains are marginal and statistically insignificant (+0.029 with p=0.17 for K=15; +0.023 with p=0.23 for K=25), with "blepharoplasty" even showing a performance decline. Results from the ablation studies also fail to support the claimed benefits of the "geometry-aware" and "setwise" designs. Table 4 shows that removing set context or geometric information actually yields a higher score than the full model (0.633 vs. 0.618), and the end-to-end ablation results in Table 10 (using a single random seed and excluding set information) differ from the full model by only 0.001. Furthermore, a simple baseline model using "frozen DINO features + ridge regression" achieves a score of 0.601.
-
The results presented in the paper are inconsistent. There are contradictions among several key data. Section 4.4 reports a gain of +0.032 (p=0.023) over Envisage for a sample pool of K=5, whereas the data provided in Tables 2 and 6 indicate a gain of only +0.002 (p=0.43). The text claims that "rhytidectomy" recovered 67% of the performance headroom at K=25, but this figure actually corresponds to the value for K=15 in Table 3 (at K=25, the proportion is approximately 48%); regarding the p-value calculations for the primary results, Section 4.1 specifies B=5,000 iterations, whereas Appendix C describes a resampling scheme involving 100,000 iterations. Given such weak effect sizes, I cannot determine which results are correct.
Questions for authors
FLUX.1-Fill is not a state-of-the-art model. I suggest the authors try using a more advanced model as the backbone.
4 comments
Thank you for your response and the additional experiments. While they resolve some of my concerns, the poor performance on blepharoplasty remains unaddressed because no per-procedure results were provided for the new test sets. I will maintain my current score for now.
Per-procedure results on both held-out test cohorts
Thanks for following up, and sorry we left out the per-procedure results in our earlier reply. For blepharoplasty, Envision scored higher than the Envisage rule in both held-out cohorts at K=15 and K=25. The paired gains were +0.012 and +0.016 at K=15, and +0.025 and +0.026 at K=25, on the locked and external HDA cohorts, respectively.
The tables below report all three procedures for the same frozen checkpoint trained on 278 cases. Envision and the Envisage rule were evaluated on identical candidate pools. Δ is the paired mean SurgicalScore difference (Envision minus Rule). Confidence intervals use 3,000 case bootstrap resamples, and p-values use 5,000 resamples from a two-sided paired sign-flip test.
Locked multi-procedure test set (N=160)
| K | Procedure (n) | Envisage rule | Envision [95% CI] | Oracle | Paired Δ [95% CI] | p |
|---|---|---|---|---|---|---|
| 5 | Rhinoplasty (66) | 0.591 | 0.593 [0.556, 0.630] | 0.768 | +0.002 [−0.024, +0.028] | 0.44 |
| 5 | Blepharoplasty (48) | 0.582 | 0.579 [0.538, 0.620] | 0.674 | −0.003 [−0.031, +0.025] | 0.41 |
| 5 | Rhytidectomy (46) | 0.593 | 0.611 [0.571, 0.651] | 0.685 | +0.018 [−0.008, +0.044] | 0.14 |
| 15 | Rhinoplasty (66) | 0.589 | 0.618 [0.580, 0.656] | 0.834 | +0.029 [+0.002, +0.056] | 0.038 |
| 15 | Blepharoplasty (48) | 0.581 | 0.593 [0.551, 0.635] | 0.728 | +0.012 [−0.008, +0.032] | 0.24 |
| 15 | Rhytidectomy (46) | 0.594 | 0.665 [0.628, 0.702] | 0.736 | +0.071 [+0.037, +0.105] | <0.001 |
| 25 | Rhinoplasty (66) | 0.585 | 0.620 [0.581, 0.659] | 0.848 | +0.035 [+0.005, +0.065] | 0.024 |
| 25 | Blepharoplasty (48) | 0.569 | 0.594 [0.551, 0.637] | 0.758 | +0.025 [−0.003, +0.053] | 0.084 |
| 25 | Rhytidectomy (46) | 0.595 | 0.654 [0.616, 0.692] | 0.751 | +0.059 [+0.024, +0.094] | 0.0014 |
External HDA cross-procedure test set (N=57)
| K | Procedure (n) | Envisage rule | Envision [95% CI] | Oracle | Paired Δ [95% CI] | p |
|---|---|---|---|---|---|---|
| 15 | Rhinoplasty (21) | 0.584 | 0.618 [0.562, 0.674] | 0.812 | +0.034 [+0.001, +0.067] | 0.046 |
| 15 | Blepharoplasty (27) | 0.575 | 0.591 [0.541, 0.641] | 0.715 | +0.016 [−0.011, +0.043] | 0.22 |
| 15 | Rhytidectomy (9) | 0.592 | 0.673 [0.608, 0.738] | 0.723 | +0.081 [+0.022, +0.140] | 0.012 |
| 25 | Rhinoplasty (21) | 0.581 | 0.618 [0.561, 0.675] | 0.826 | +0.037 [+0.002, +0.072] | 0.039 |
| 25 | Blepharoplasty (27) | 0.570 | 0.596 [0.545, 0.647] | 0.741 | +0.026 [−0.008, +0.060] | 0.13 |
| 25 | Rhytidectomy (9) | 0.589 | 0.652 [0.585, 0.719] | 0.733 | +0.063 [+0.009, +0.117] | 0.024 |
In an exploratory analysis pooling all 75 held-out blepharoplasty cases, the paired gain was +0.013 at K=15 (95% CI [−0.004, +0.030], p=0.14) and +0.025 at K=25 (95% CI [+0.002, +0.048], nominal p=0.036). The two cohorts show the same pattern of point estimates, with a small positive gain at K=15 and a larger estimate at K=25.
The updated manuscript summarizes the blepharoplasty results in Section 4.2 and presents the full per-procedure breakdown in Appendix Table 11 (page 18).
Thank you for providing the detailed per-procedure breakdown. While the gains on blepharoplasty alone remain statistically insignificant in each individual cohort, the new results confirm that the model no longer degrades performance on this procedure and achieves significant improvements on both rhinoplasty and rhytidectomy. Since most of my primary experimental concerns have been addressed, I will raise my score to 3.
Reply — W1, W2, Q1, ethics: independent evidence and corrected protocol
We addressed your central experimental concern with untouched, subject-disjoint tests, and corrected all three numerical inconsistencies you identified. The standard 278-case checkpoint, fixed before these evaluations, yields:
The original 49-case test result remains in the appendix. The matched three-seed end-to-end ablation on the locked 160 cases gives full 0.624, no set context 0.610 (−0.014, p=0.038), no geometry 0.604 (−0.020, p=0.015), and no direction gate 0.601 (−0.023, p=0.009). Thus every component improves the model under a common protocol. The original Table 4 inversion arose under a different cached-feature, mixed-development setup, which cannot isolate its cause. On the same locked set, a frozen DINOv2-CLS + Ridge visual baseline scores 0.598 versus full Envision's 0.624 (paired p=0.008); its selected geometry scores are also lower (direction A 0.542 vs 0.582; magnitude B 0.378 vs 0.453). The matched ablations provide component-level evidence; Ridge supplies a complementary visual-only baseline.
We corrected the three numerical/protocol errors in both prose and tables:
We chose FLUX.1-Fill-dev for its mask-conditioned inpainting workflow and held both the generator and candidate pools fixed across Envision and the deploy Rule, following Envisage. This cleanly attributes the measured gain to selection. The near-constant outside-mask score (E≈0.994) is achieved by hard pixel compositing. The revision separately evaluates single-output external generators; matched multi-candidate pools would test selector transfer beyond FLUX.
Finally, revised Figures 1, 4 and 11 embed opaque eye-region redaction in every patient panel, including inputs and targets. The Ethics Statement names the source datasets and reuse limits, and the code package excludes patient portraits. The paper also states the residual privacy risk and confines consent or institutional claims to documented evidence.