- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces GestaltCLIP, a concept-enhanced vision–language framework designed to learn clinically structured facial representations for rare genetic disease analysis from heterogeneous phenotype supervision. By unifying concept-, patient-, and disease-level supervision through ontology-aware HPO learning and patient-conditioned weak alignment, the method eliminates the need for complete image–phenotype pairing at inference time while significantly improving disease recognition, phenotype grounding, and cross-dataset transfer.
Critical strengths
Successfully bridges the gap between scarce image-level annotations and abundant but unstructured clinical knowledge by hierarchically integrating HPO concepts, patient phenotype sets, and disease-level priors.
Demonstrates robust improvements across multiple benchmarks, yielding a 15.58% average relative improvement in disease recognition on the GestaltMatcher Database and substantial gains in unseen disease generalization and cross-dataset transfer on RD-Face.
Operates with images alone at inference time by internalizing structured phenotype knowledge purely into the visual representation during training, avoiding the need for cumbersome text or ontology queries in deployment.
Critical weaknesses
The framework's performance heavily depends on the hierarchical structure and semantic accuracy of the Human Phenotype Ontology. Noisy, incomplete, or incorrectly mapped clinical descriptions could propagate errors into the latent phenotype concept space.
The three-stage training pipeline (involving real-case pretraining, paired multi-granularity alignment, and disease-mediated weak alignment) introduces multiple loss balancing and hyperparameter tuning challenges, potentially affecting training stability and reproducibility.
While quantitative metrics like mAP for image-to-HPO retrieval improve, the paper could benefit from a more rigorous clinical validation showing whether the specific facial regions activated by the model clinically correspond to true anatomical phenotype manifestations rather than confounding background or stylistic biases.
Questions for authors
See Weakness
9 comments
Response to Reviewer VyfY — Part 2/3: W2 — Loss-weight sensitivity, Training stability, and Reproducibility
W2. Loss Balancing, Training Stability, and Reproducibility
We thank the reviewer for raising concerns about the optimization stability and reproducibility of our three-stage framework. We agree that combining multiple training objectives requires careful loss balancing. To directly address this concern, we conducted two additional analyses: (1) loss-weight sensitivity across Stages 2–3 and (2) a three-seed training stability and checkpoint-selection audit. We also clarify how these results complement the hyperparameter sensitivity and reproducibility analyses already provided in the manuscript.
1. Loss-weight sensitivity.
We conducted a one-factor-at-a-time sensitivity analysis by independently scaling five loss coefficients by 0.5× and 2× relative to their canonical values: concept-level and phenotype-set alignment in Stage 2, and atomic alignment, paired replay, and weak set alignment in Stage 3. All other settings were kept unchanged. Stage-2 variants were propagated through subsequent Stage-3 training, allowing us to assess their effects on the final representation.
We evaluated disease recognition and phenotype grounding on GMDB Common and Rare, covering balanced accuracy (BA), atomic HPO retrieval mAP, and phenotype-set retrieval mAP.
Table R14. Sensitivity to Stage-2 and Stage-3 loss weights on GMDB Common and Rare. Each coefficient is independently scaled by 0.5× or 2× relative to the canonical configuration. Results are from a single training seed and reported in percentages (%). Higher is better for all metrics.
| Stage | Loss coefficient | Factor | Common BA | Common atomic mAP | Common set mAP | Rare BA | Rare atomic mAP | Rare set mAP |
|---|---|---|---|---|---|---|---|---|
| — | Canonical | 1× | 25.20 | 12.91 | 5.67 | 28.31 | 12.01 | 21.23 |
| 2 | Concept | 0.5× | 25.36 | 12.73 | 5.61 | 28.38 | 11.88 | 21.36 |
| 2 | Concept | 2× | 25.53 | 13.09 | 5.57 | 27.82 | 12.12 | 21.93 |
| 2 | Set | 0.5× | 25.49 | 12.74 | 5.58 | 27.89 | 11.92 | 20.81 |
| 2 | Set | 2× | 25.31 | 12.87 | 5.67 | 28.31 | 12.04 | 21.88 |
| 3 | Atomic | 0.5× | 26.17 | 12.50 | 6.24 | 28.87 | 11.53 | 22.60 |
| 3 | Atomic | 2× | 25.81 | 13.63 | 5.43 | 28.31 | 11.90 | 20.86 |
| 3 | Paired replay | 0.5× | 25.58 | 13.40 | 5.34 | 28.31 | 12.46 | 20.80 |
| 3 | Paired replay | 2× | 26.07 | 12.39 | 6.21 | 28.87 | 11.42 | 22.61 |
| 3 | Weak set | 0.5× | 25.38 | 12.89 | 5.60 | 28.31 | 12.03 | 21.24 |
| 3 | Weak set | 2× | 25.20 | 12.91 | 5.61 | 28.31 | 12.00 | 21.20 |
As shown in Table R14, disease recognition remains relatively stable across all ten loss-weight variations. Common balanced accuracy does not decrease, and Rare balanced accuracy decreases by at most 0.49 pp. Phenotype retrieval exhibits objective-dependent trade-offs, particularly between atomic and phenotype-set alignment. These results suggest that disease-recognition performance is not highly sensitive to the tested loss coefficients, although phenotype-grounding objectives require balancing.
2. Training stability across three seeds.
Beyond loss-weight sensitivity, we audited the existing training logs for all three stages across seeds 42, 43, and 44. We examined completed training epochs, divergence events, and validation-based checkpoint selection. This analysis uses the original training runs and does not require additional retraining.
Table R15. Training stability and checkpoint selection across three seeds (42/43/44). Completed epochs denote the observed training-duration range. Selected epochs correspond to the checkpoints chosen independently for each seed using the specified validation criterion.
| Stage | Seeds | Completed epochs | Divergent runs | Checkpoint-selection criterion | Selected epochs (42/43/44) |
|---|---|---|---|---|---|
| Stage 1 | 3 | 22–37 | 0/3 | Minimum validation loss | 30 / 15 / 22 |
| Stage 2 | 3 | 32–40 | 0/3 | Maximum validation selection score | 24 / 24 / 32 |
| Stage 3 | 3 | 13–16 | 0/3 | Maximum validation selection score | 10 / 10 / 7 |
All nine stage-level runs completed without divergence (Table R15), with validation-based checkpoint selection consistently applied across three seeds. Stage-2 and Stage-3 selected validation scores lie within narrow ranges (0.1879–0.1915 and 0.1182–0.1192). Together with the sensitivity analyses in Appendix F.2 and implementation details in Appendix H, these results support stable optimization and reproducibility under the evaluated settings.
Response to Reviewer VyfY — Part 3/3: W3 — Anatomical correspondence and visual confounders
W3. Anatomical Correspondence and Robustness to Visual Confounders
Our original Figures 4–5 qualitatively illustrate phenotype grounding: Protruding Ear improves from rank 110 to 6 (cosine similarity 0.12 to 0.35), and the occlusion map shifts toward the ear. However, individual examples cannot establish systematic anatomical correspondence or exclude visual confounders.
We therefore conducted two additional analyses on 219 held-out patients: (1) phenotype-conditioned occlusion of anatomically corresponding regions, unrelated facial regions, and background, and (2) robustness evaluation under brightness and color perturbations.
1. Anatomical Correspondence.
We evaluated four facial phenotypes, including Long Philtrum, Protruding Ear, Thin Upper Lip Vermilion, and Wide Nasal Bridge, which cover distinct anatomical regions. The corresponding target regions were defined based on the official HPO concept definitions and ontology hierarchy, developed and curated by the HPO Consortium using clinical expertise and established medical literature (Gargano et al., 2024), rather than inferred from model attribution.
For each patient–phenotype pair, we measured changes in image–HPO cosine similarity after occluding the corresponding anatomical region, a non-target facial region, or the background, using matched occlusion settings. We report target-minus-non-target and target-minus-background contrasts, where higher values indicate stronger anatomical selectivity. Scores were averaged across three seeds (42/43/44) before patient-level bootstrap analysis.
Table R16. Anatomical occlusion contrasts on 219 held-out patients. Higher values indicate stronger phenotype-conditioned anatomical selectivity. Results include 95% patient-level bootstrap confidence intervals (CIs); paired differences compare GestaltCLIP with Adapted OpenCLIP on the same patients.
| Anatomical contrast | Adapted OpenCLIP (95% CI) | GestaltCLIP (95% CI) | Paired improvement (95% CI) |
|---|---|---|---|
| Target − non-target | 0.0020 [−0.0028, 0.0068] | 0.0180 [0.0089, 0.0272] | +0.0160 [0.0084, 0.0236] |
| Target − background | 0.0046 [0.0006, 0.0087] | 0.0266 [0.0201, 0.0332] | +0.0220 [0.0157, 0.0283] |
As shown in Table R16, GestaltCLIP exhibits substantially stronger anatomical selectivity than Adapted OpenCLIP. The paired improvements are +0.0160 for target-minus-non-target and +0.0220 for target-minus-background, with both 95% CIs excluding zero. These results provide quantitative evidence that GestaltCLIP's phenotype similarity scores are more sensitive to anatomically corresponding facial regions than to unrelated facial regions or background, extending the qualitative visualization in Fig. 5.
2. Robustness to Appearance Variations.
To examine whether phenotype representations are influenced by low-level stylistic cues, we additionally evaluated image–HPO similarity stability and HPO ranking consistency under brightness and color perturbations on the same 219 patients.
Table R17. Robustness of image–HPO representations under brightness and color perturbations. Lower mean absolute cosine similarity change indicates greater score stability (95% patient-bootstrap CIs); higher Spearman correlation indicates more consistent HPO rankings (mean ± SD across 12 HPO-seed groups).
| Perturbation | Model | Absolute cosine change ↓ (95% CI) | HPO ranking Spearman ↑ |
|---|---|---|---|
| Brightness | Adapted OpenCLIP | 0.0180 [0.0160, 0.0200] | 0.9492 ± 0.0120 |
| Brightness | GestaltCLIP | 0.0096 [0.0085, 0.0108] | 0.9570 ± 0.0051 |
| Color | Adapted OpenCLIP | 0.0272 [0.0239, 0.0306] | 0.9074 ± 0.0336 |
| Color | GestaltCLIP | 0.0158 [0.0140, 0.0177] | 0.9216 ± 0.0229 |
As shown in Table R17, GestaltCLIP reduces the mean absolute cosine similarity change by 46.4% under brightness perturbations and 41.7% under color perturbations compared with Adapted OpenCLIP. It also maintains higher HPO ranking consistency under both conditions (Spearman: 0.9570 vs. 0.9492 for brightness; 0.9216 vs. 0.9074 for color), suggesting improved stability under the evaluated appearance variations.
Together with Figures 4–5, these analyses provide complementary qualitative and quantitative evidence that GestaltCLIP exhibits stronger phenotype-conditioned anatomical selectivity and greater stability to the evaluated appearance changes than Adapted OpenCLIP. We acknowledge that this does not replace independent clinical adjudication of patient-specific phenotype visibility.
Response to Reviewer VyfY — Part 1/3: W1 — HPO corruption robustness
Dear Reviewer VyfY,
Thank you for your constructive feedback. We conducted end-to-end robustness tests under incorrect and missing HPO annotations (W1, Table R13), loss-weight sensitivity and training-stability analyses (W2, Tables R14–R15), and anatomical occlusion and appearance-perturbation studies on 219 held-out patients (W3, Tables R16–R17). These results show limited degradation under the tested HPO corruption levels, stable optimization, and stronger phenotype-conditioned anatomical selectivity. Our detailed and point-by-point responses follow in three parts below, beginning with W1.
W1. Robustness to noisy and incomplete clinical HPO supervision
We thank the reviewer for highlighting the potential propagation of clinical annotation errors into the learned phenotype concept space. To directly investigate this concern, we conducted an additional end-to-end Stage-1 HPO corruption study, introducing either incorrect or missing clinical HPO annotations at corruption rates of 10%, 20%, and 30%.
Importantly, we retrained the complete Stage 1 → Stage 2 → Stage 3 pipeline for each corruption condition, rather than perturbing annotations only at evaluation time. All experiments used the same backbone, data splits, and evaluation protocol. This design directly assesses whether corrupted upstream phenotype supervision affects downstream disease recognition and phenotype grounding.
Table R13. Robustness to Stage-1 clinical HPO annotation corruption on GMDB Common and Rare. For each corruption condition, the complete three-stage pipeline was retrained. Results are reported in percentages (%) from a single seed. Higher is better. BA stands for Balanced Accuracy.
As shown in Table R13, incorrect clinical HPO annotations induce gradual but limited performance degradation. Even at 30% incorrect annotations, Common and Rare balanced accuracy decrease by only 1.19 and 0.91 percentage points, respectively. Atomic HPO retrieval mAP decreases by 0.62 and 1.04 points, while phenotype-set retrieval mAP decreases by only 0.18 and 0.30 points.
The model is also comparatively tolerant to missing annotations. At 30% missingness, balanced accuracy decreases by 0.44 and 0.91 points on Common and Rare, respectively, with similarly limited changes in phenotype-set retrieval.
These results provide direct end-to-end evidence that moderate corruption of upstream clinical HPO supervision does not lead to severe degradation in the final representations. Nevertheless, annotation quality remains important, as reflected by the consistent decline in atomic phenotype retrieval under increasing incorrect-label corruption. We clarify that this experiment evaluates robustness to clinical annotation errors and incompleteness, rather than structural errors in the underlying HPO ontology itself.