acceptodds
Under review as a conference paper at ICLR 2027

GestaltCLIP: Concept-Enhanced Learning of Clinically Structured Facial Representation from Heterogeneous Phenotype Supervision

Abstract

Facial morphology provides important diagnostic cues for many rare genetic disorders, yet learning facial representations structured by clinically meaningful phenotypic concepts remains difficult because available datasets are small and supervision is highly fragmented. Rare disease facial representation learning is further limited by scarce image-level phenotype annotations, while richer but heterogeneous supervision exists across multiple levels, including Human Phenotype Ontology (HPO) concepts, patient phenotype sets, and disease level phenotype knowledge. We introduce **GestaltCLIP**, a concept-enhanced vision-language framework that integrates these heterogeneous phenotype signals to learn clinically structured facial representations without requiring complete image–phenotype pairing. GestaltCLIP unifies concept-, patient-, and disease-level phenotype supervision through ontology-aware HPO learning, phenotype-set modeling, direct image–phenotype grounding, and disease-mediated weak supervision. The resulting vision encoder operates with images alone at inference time. On GestaltMatcher Database, GestaltCLIP consistently improves disease recognition across four disease recognition metrics, yielding an average relative improvement of 15.58%, and achieves a 7.73-fold increase in mean average precision (mAP) for atomic image-to-HPO retrieval. This demonstrates substantially improved phenotype-concept grounding. Under strict external evaluation on RDFace, Top-1 accuracy improves from 8.93% to 15.18%. These results demonstrate that integrating heterogeneous phenotype supervision through explicit concept learning produces clinically structured, phenotype-grounded, and transferable facial representations for rare disease analysis.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Rare disease, Facial representation learning, Vision-language learning, Human Phenotype Ontology, Phenotype representation, Cross-modal alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VyfYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GestaltCLIP, a concept-enhanced vision–language framework designed to learn clinically structured facial representations for rare genetic disease analysis from heterogeneous phenotype supervision. By unifying concept-, patient-, and disease-level supervision through ontology-aware HPO learning and patient-conditioned weak alignment, the method eliminates the need for complete image–phenotype pairing at inference time while significantly improving disease recognition, phenotype grounding, and cross-dataset transfer.

Critical strengths

Successfully bridges the gap between scarce image-level annotations and abundant but unstructured clinical knowledge by hierarchically integrating HPO concepts, patient phenotype sets, and disease-level priors.

Demonstrates robust improvements across multiple benchmarks, yielding a 15.58% average relative improvement in disease recognition on the GestaltMatcher Database and substantial gains in unseen disease generalization and cross-dataset transfer on RD-Face.

Operates with images alone at inference time by internalizing structured phenotype knowledge purely into the visual representation during training, avoiding the need for cumbersome text or ontology queries in deployment.

Critical weaknesses

The framework's performance heavily depends on the hierarchical structure and semantic accuracy of the Human Phenotype Ontology. Noisy, incomplete, or incorrectly mapped clinical descriptions could propagate errors into the latent phenotype concept space.

The three-stage training pipeline (involving real-case pretraining, paired multi-granularity alignment, and disease-mediated weak alignment) introduces multiple loss balancing and hyperparameter tuning challenges, potentially affecting training stability and reproducibility.

While quantitative metrics like mAP for image-to-HPO retrieval improve, the paper could benefit from a more rigorous clinical validation showing whether the specific facial regions activated by the model clinically correspond to true anatomical phenotype manifestations rather than confounding background or stylistic biases.

Questions for authors

See Weakness

9 comments
AuthorsOpenReview ↗

Response to Reviewer VyfY — Part 1/3: W1 — HPO corruption robustness

Dear Reviewer VyfY,

Thank you for your constructive feedback. We conducted end-to-end robustness tests under incorrect and missing HPO annotations (W1, Table R13), loss-weight sensitivity and training-stability analyses (W2, Tables R14–R15), and anatomical occlusion and appearance-perturbation studies on 219 held-out patients (W3, Tables R16–R17). These results show limited degradation under the tested HPO corruption levels, stable optimization, and stronger phenotype-conditioned anatomical selectivity. Our detailed and point-by-point responses follow in three parts below, beginning with W1.


W1. Robustness to noisy and incomplete clinical HPO supervision

We thank the reviewer for highlighting the potential propagation of clinical annotation errors into the learned phenotype concept space. To directly investigate this concern, we conducted an additional end-to-end Stage-1 HPO corruption study, introducing either incorrect or missing clinical HPO annotations at corruption rates of 10%, 20%, and 30%.

Importantly, we retrained the complete Stage 1 → Stage 2 → Stage 3 pipeline for each corruption condition, rather than perturbing annotations only at evaluation time. All experiments used the same backbone, data splits, and evaluation protocol. This design directly assesses whether corrupted upstream phenotype supervision affects downstream disease recognition and phenotype grounding.

Table R13. Robustness to Stage-1 clinical HPO annotation corruption on GMDB Common and Rare. For each corruption condition, the complete three-stage pipeline was retrained. Results are reported in percentages (%) from a single seed. Higher is better. BA stands for Balanced Accuracy.

Stage-1 HPO conditionCommon BACommon atomic mAPCommon set mAPRare BARare atomic mAPRare set mAP
Clean25.2012.915.6728.3112.0121.23
Incorrect 10%24.3112.485.5527.4011.4921.07
Incorrect 20%24.2112.385.5327.4011.2621.05
Incorrect 30%24.0112.295.4927.4010.9720.93
Missing 10%25.0112.745.8028.8711.9221.24
Missing 20%24.8612.745.6128.8711.5321.19
Missing 30%24.7612.565.6027.4011.2421.07

As shown in Table R13, incorrect clinical HPO annotations induce gradual but limited performance degradation. Even at 30% incorrect annotations, Common and Rare balanced accuracy decrease by only 1.19 and 0.91 percentage points, respectively. Atomic HPO retrieval mAP decreases by 0.62 and 1.04 points, while phenotype-set retrieval mAP decreases by only 0.18 and 0.30 points.

The model is also comparatively tolerant to missing annotations. At 30% missingness, balanced accuracy decreases by 0.44 and 0.91 points on Common and Rare, respectively, with similarly limited changes in phenotype-set retrieval.

These results provide direct end-to-end evidence that moderate corruption of upstream clinical HPO supervision does not lead to severe degradation in the final representations. Nevertheless, annotation quality remains important, as reflected by the consistent decline in atomic phenotype retrieval under increasing incorrect-label corruption. We clarify that this experiment evaluates robustness to clinical annotation errors and incompleteness, rather than structural errors in the underlying HPO ontology itself.

AuthorsOpenReview ↗

Response to Reviewer VyfY — Part 2/3: W2 — Loss-weight sensitivity, Training stability, and Reproducibility

W2. Loss Balancing, Training Stability, and Reproducibility

We thank the reviewer for raising concerns about the optimization stability and reproducibility of our three-stage framework. We agree that combining multiple training objectives requires careful loss balancing. To directly address this concern, we conducted two additional analyses: (1) loss-weight sensitivity across Stages 2–3 and (2) a three-seed training stability and checkpoint-selection audit. We also clarify how these results complement the hyperparameter sensitivity and reproducibility analyses already provided in the manuscript.

1. Loss-weight sensitivity.

We conducted a one-factor-at-a-time sensitivity analysis by independently scaling five loss coefficients by 0.5× and 2× relative to their canonical values: concept-level and phenotype-set alignment in Stage 2, and atomic alignment, paired replay, and weak set alignment in Stage 3. All other settings were kept unchanged. Stage-2 variants were propagated through subsequent Stage-3 training, allowing us to assess their effects on the final representation.

We evaluated disease recognition and phenotype grounding on GMDB Common and Rare, covering balanced accuracy (BA), atomic HPO retrieval mAP, and phenotype-set retrieval mAP.

Table R14. Sensitivity to Stage-2 and Stage-3 loss weights on GMDB Common and Rare. Each coefficient is independently scaled by 0.5× or 2× relative to the canonical configuration. Results are from a single training seed and reported in percentages (%). Higher is better for all metrics.

StageLoss coefficientFactorCommon BACommon atomic mAPCommon set mAPRare BARare atomic mAPRare set mAP
—Canonical1×25.2012.915.6728.3112.0121.23
2Concept0.5×25.3612.735.6128.3811.8821.36
2Concept2×25.5313.095.5727.8212.1221.93
2Set0.5×25.4912.745.5827.8911.9220.81
2Set2×25.3112.875.6728.3112.0421.88
3Atomic0.5×26.1712.506.2428.8711.5322.60
3Atomic2×25.8113.635.4328.3111.9020.86
3Paired replay0.5×25.5813.405.3428.3112.4620.80
3Paired replay2×26.0712.396.2128.8711.4222.61
3Weak set0.5×25.3812.895.6028.3112.0321.24
3Weak set2×25.2012.915.6128.3112.0021.20

As shown in Table R14, disease recognition remains relatively stable across all ten loss-weight variations. Common balanced accuracy does not decrease, and Rare balanced accuracy decreases by at most 0.49 pp. Phenotype retrieval exhibits objective-dependent trade-offs, particularly between atomic and phenotype-set alignment. These results suggest that disease-recognition performance is not highly sensitive to the tested loss coefficients, although phenotype-grounding objectives require balancing.

2. Training stability across three seeds.

Beyond loss-weight sensitivity, we audited the existing training logs for all three stages across seeds 42, 43, and 44. We examined completed training epochs, divergence events, and validation-based checkpoint selection. This analysis uses the original training runs and does not require additional retraining.

Table R15. Training stability and checkpoint selection across three seeds (42/43/44). Completed epochs denote the observed training-duration range. Selected epochs correspond to the checkpoints chosen independently for each seed using the specified validation criterion.

StageSeedsCompleted epochsDivergent runsCheckpoint-selection criterionSelected epochs (42/43/44)
Stage 1322–370/3Minimum validation loss30 / 15 / 22
Stage 2332–400/3Maximum validation selection score24 / 24 / 32
Stage 3313–160/3Maximum validation selection score10 / 10 / 7

All nine stage-level runs completed without divergence (Table R15), with validation-based checkpoint selection consistently applied across three seeds. Stage-2 and Stage-3 selected validation scores lie within narrow ranges (0.1879–0.1915 and 0.1182–0.1192). Together with the sensitivity analyses in Appendix F.2 and implementation details in Appendix H, these results support stable optimization and reproducibility under the evaluated settings.

AuthorsOpenReview ↗

Response to Reviewer VyfY — Part 3/3: W3 — Anatomical correspondence and visual confounders

W3. Anatomical Correspondence and Robustness to Visual Confounders

Our original Figures 4–5 qualitatively illustrate phenotype grounding: Protruding Ear improves from rank 110 to 6 (cosine similarity 0.12 to 0.35), and the occlusion map shifts toward the ear. However, individual examples cannot establish systematic anatomical correspondence or exclude visual confounders.

We therefore conducted two additional analyses on 219 held-out patients: (1) phenotype-conditioned occlusion of anatomically corresponding regions, unrelated facial regions, and background, and (2) robustness evaluation under brightness and color perturbations.

1. Anatomical Correspondence.

We evaluated four facial phenotypes, including Long Philtrum, Protruding Ear, Thin Upper Lip Vermilion, and Wide Nasal Bridge, which cover distinct anatomical regions. The corresponding target regions were defined based on the official HPO concept definitions and ontology hierarchy, developed and curated by the HPO Consortium using clinical expertise and established medical literature (Gargano et al., 2024), rather than inferred from model attribution.

For each patient–phenotype pair, we measured changes in image–HPO cosine similarity after occluding the corresponding anatomical region, a non-target facial region, or the background, using matched occlusion settings. We report target-minus-non-target and target-minus-background contrasts, where higher values indicate stronger anatomical selectivity. Scores were averaged across three seeds (42/43/44) before patient-level bootstrap analysis.

Table R16. Anatomical occlusion contrasts on 219 held-out patients. Higher values indicate stronger phenotype-conditioned anatomical selectivity. Results include 95% patient-level bootstrap confidence intervals (CIs); paired differences compare GestaltCLIP with Adapted OpenCLIP on the same patients.

Anatomical contrastAdapted OpenCLIP (95% CI)GestaltCLIP (95% CI)Paired improvement (95% CI)
Target − non-target0.0020 [−0.0028, 0.0068]0.0180 [0.0089, 0.0272]+0.0160 [0.0084, 0.0236]
Target − background0.0046 [0.0006, 0.0087]0.0266 [0.0201, 0.0332]+0.0220 [0.0157, 0.0283]

As shown in Table R16, GestaltCLIP exhibits substantially stronger anatomical selectivity than Adapted OpenCLIP. The paired improvements are +0.0160 for target-minus-non-target and +0.0220 for target-minus-background, with both 95% CIs excluding zero. These results provide quantitative evidence that GestaltCLIP's phenotype similarity scores are more sensitive to anatomically corresponding facial regions than to unrelated facial regions or background, extending the qualitative visualization in Fig. 5.

2. Robustness to Appearance Variations.

To examine whether phenotype representations are influenced by low-level stylistic cues, we additionally evaluated image–HPO similarity stability and HPO ranking consistency under brightness and color perturbations on the same 219 patients.

Table R17. Robustness of image–HPO representations under brightness and color perturbations. Lower mean absolute cosine similarity change indicates greater score stability (95% patient-bootstrap CIs); higher Spearman correlation indicates more consistent HPO rankings (mean ± SD across 12 HPO-seed groups).

PerturbationModelAbsolute cosine change ↓ (95% CI)HPO ranking Spearman ↑
BrightnessAdapted OpenCLIP0.0180 [0.0160, 0.0200]0.9492 ± 0.0120
BrightnessGestaltCLIP0.0096 [0.0085, 0.0108]0.9570 ± 0.0051
ColorAdapted OpenCLIP0.0272 [0.0239, 0.0306]0.9074 ± 0.0336
ColorGestaltCLIP0.0158 [0.0140, 0.0177]0.9216 ± 0.0229

As shown in Table R17, GestaltCLIP reduces the mean absolute cosine similarity change by 46.4% under brightness perturbations and 41.7% under color perturbations compared with Adapted OpenCLIP. It also maintains higher HPO ranking consistency under both conditions (Spearman: 0.9570 vs. 0.9492 for brightness; 0.9216 vs. 0.9074 for color), suggesting improved stability under the evaluated appearance variations.

Together with Figures 4–5, these analyses provide complementary qualitative and quantitative evidence that GestaltCLIP exhibits stronger phenotype-conditioned anatomical selectivity and greater stability to the evaluated appearance changes than Adapted OpenCLIP. We acknowledge that this does not replace independent clinical adjudication of patient-specific phenotype visibility.

Reviewer_SYPVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GestaltCLIP aims to learn facial representations aligned with medical phenotype concepts by leveraging incomplete phenotypic information from multiple sources. In this task, a small number of photographs have patient-level HPO annotations, while more photographs have only disease labels. Although external clinical cases and knowledge bases provide rich phenotypic information, they cannot be directly paired with the photographs. To utilize this heterogeneous supervision, the paper proposes a three-stage training scheme that progressively establishes correspondences among phenotype concepts, patient phenotype sets, and facial images.

Critical strengths

The method is well motivated and addresses an important practical problem in this domain: how to organize data with varying levels of granularity, completeness, and correspondence to images for visual representation learning.

Critical weaknesses

My main concern is that the current experiments do not sufficiently distinguish the gains from additional data resources from the gains from the complex training design. The method involves three training stages and multiple auxiliary modules, while also using weakly annotated images and external clinical case information that some baselines do not use. Under these circumstances, a baseline that uses the same resources but follows a simple training procedure is needed to assess the contribution of each design choice more clearly. The existing component-level ablations can demonstrate the roles of certain components within the full framework, but they cannot fully replace this overall comparison. Specifically, I suggest that the authors first establish a baseline that uses only simple rules to organize the available data into a unified set of training samples, followed by direct representation learning. Below is a reference design for such a baseline experiment:

  1. Convert all images into a common image–HPO sample format. For photographs with patient-level HPO annotations, directly use the existing annotations. For photographs with only disease labels, directly use the HPO set associated with the disease.
  2. Use external clinical cases through simple aggregation. Aggregate the HPO terms from cases available for training by disease, and combine them with existing disease–HPO associations to construct disease phenotype sets, without additionally training a PhenoEncoder.
  3. Directly treat the resulting labels as complete annotations. Mark HPO terms in the set as positive and all remaining terms as negative, and train with a standard loss without introducing PU downweighting, confidence weighting, teacher-based filtering, or patient-conditioned refinement. Although this setting contains label noise, it can serve as a reference for evaluating the actual gains from these noise-handling mechanisms.
  4. Mix all images from the beginning of training. Place the original paired samples and the samples constructed through disease associations into the same training set, and directly optimize the visual representation with a unified objective, without staged training.

This baseline would provide a clear starting point: what performance can simple training achieve with the same data resources? Building on this baseline, progressively adding incomplete-label handling, phenotype-set modeling, and patient-conditioned supervision would allow the gains from each design choice to be quantified. Otherwise, the effectiveness of the paper’s proposed methodological design lacks clear validation.

Questions for authors

Please refer to the questions raised in the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SYPV — Part 1/2: W1 — Same-resource baseline and progressive results

Dear Reviewer SYPV,

Thank you for the detailed and actionable recommendations. We implemented your proposed same-resource naive unified baseline, then progressively introduced incomplete-label handling, phenotype-set modeling, image-conditioned supervision, and staged alignment (Tables R10–R11). We additionally audited student optimization, image exposure, and model selection (Table R12). Compared with naive unified learning, GestaltCLIP improves GMDB Rare Top-1 accuracy by 13.41 pp and concept-set mAP by 12.59 pp. The detailed response is organized into two parts below.


W1. Same-Resource Naive Unified Baseline

We implemented the four design choices suggested by the reviewer in Variant A (Naive Unified): (i) directly assigning patient-level HPO annotations to paired images; (ii) constructing disease-level phenotype profiles by aggregating clinical cases and disease–HPO associations, then assigning them to disease-labeled images; (iii) treating all unannotated HPO concepts as negatives under standard binary cross-entropy (BCE) supervision, without PU handling or image-conditioned filtering; and (iv) jointly training on paired and weakly annotated images from the beginning, without staged alignment.

We then progressively incorporated the proposed mechanisms through Variants B–E, as summarized in Table R10.

Table R10. Definition of the same-resource progressive training variants.

VariantTraining strategy
A: Naive unifiedPatient HPOs for paired images; aggregated disease HPOs for weak images; standard BCE with unannotated HPOs treated as negatives; joint training from the start
B: +PUA + incomplete-label handling
C: +SetB + pretrained, frozen PhenoEncoder; joint-from-start training
D: +PosteriorC + image-conditioned supervision from a frozen teacher; joint-from-start training
E: Full GestaltCLIPStaged paired alignment followed by disease-mediated weak alignment with paired replay

All variants use the same underlying supervision resources: 2,884 paired facial images with patient-level HPO annotations, 3,959 disease-labeled images without usable patient-level HPO annotations, and 7,234 external clinical phenotype cases (6,509 for training and 725 for validation). They also share the same disease–HPO knowledge resources, facial-HPO vocabulary, and OpenCLIP ViT-B/16 initialization.

Table R11. Same-resource progressive comparison on held-out GMDB and RDFace test sets. Results are mean ± SD across three independent seeds (42/43/44), reported in percentages (%). Bold denotes the highest or tied-highest reported mean in each row.

EvaluationA: NaiveB: +PUC: +SetD: +PosteriorE: Full
Common Top-131.81 ± 0.2332.68 ± 0.9634.98 ± 0.4635.85 ± 0.1635.85 ± 0.09
Common atomic HPO mAP10.94 ± 0.1011.36 ± 0.2010.83 ± 0.1313.07 ± 0.2013.07 ± 0.16
Common concept-set mAP1.31 ± 0.113.37 ± 0.095.54 ± 0.123.81 ± 0.106.05 ± 0.41
Rare Top-113.54 ± 2.3921.35 ± 1.4824.09 ± 1.9322.53 ± 1.5826.95 ± 1.03
Rare atomic HPO mAP10.66 ± 0.5810.92 ± 0.229.25 ± 1.0411.38 ± 0.6311.91 ± 0.16
Rare concept-set mAP8.10 ± 0.0613.41 ± 0.7115.64 ± 1.1813.43 ± 1.0220.69 ± 1.44
RDFace shared Bal. Acc.8.42 ± 0.726.53 ± 1.6211.64 ± 1.1111.09 ± 1.5015.78 ± 0.89
RDFace unseen Bal. Acc.2.14 ± 0.003.81 ± 0.415.83 ± 2.634.58 ± 1.345.89 ± 0.93

As shown in Table R11, GestaltCLIP outperforms the reviewer-specified Naive Unified baseline across all eight evaluations under the same supervision resources, including +13.41 pp in Rare Top-1 and +12.59 pp in Rare concept-set mAP. These results show that straightforward aggregation of heterogeneous supervision alone does not reproduce the gains of the proposed training design.

AuthorsOpenReview ↗

Response to Reviewer SYPV — Part 2/2: W1 — Contributions of individual design choices and Training-budget fairness and selection

W1. Contribution of Individual Design Choices

The progressive results in Table R11, together with our original component ablations (Section 4.5, Table 4; Appendix D), further clarify the contributions of the proposed design choices.

Incomplete-label handling (A→B). Replacing complete-label BCE with PU-aware supervision improves GMDB Rare Top-1 accuracy from 13.54% to 21.35% (+7.81 pp). This supports the importance of distinguishing unannotated phenotypes from confirmed negatives when learning from incomplete patient-level and disease-mediated HPO supervision.

Phenotype-set modeling (B→C). Introducing the pretrained phenotype-set encoder further increases Rare Top-1 accuracy from 21.35% to 24.09% (+2.74 pp) and Common concept-set retrieval mAP from 3.37% to 5.54% (+2.17 pp). These gains indicate that modeling dependencies among co-occurring phenotypes offers complementary information beyond independent atomic HPO alignment.

Image-conditioned supervision (C→D). Introducing the frozen teacher's image-conditioned posterior improves Common atomic HPO mAP from 10.83% to 13.07% (+2.24 pp) and Rare atomic HPO mAP from 9.25% to 11.38% (+2.13 pp). Although several disease-recognition and concept-set retrieval metrics decrease in this transition, the atomic retrieval improvements support the value of refining disease-derived phenotype supervision using image-specific visual evidence.

Staged alignment and paired replay (D→E). The complete framework improves Rare Top-1 accuracy from 22.53% to 26.95% (+4.42 pp), Rare concept-set mAP from 13.43% to 20.69% (+7.26 pp), and Common concept-set mAP from 3.81% to 6.05% (+2.24 pp). This transition evaluates the combined contribution of the staged training schedule and paired replay, rather than isolating either mechanism individually. The original replay ablation (Table 4; Appendix D) provides additional evidence that preserving paired phenotype alignment during Stage 3 is important for maintaining patient-level grounding.

Importantly, individual transitions do not improve every metric monotonically. Different mechanisms emphasize complementary aspects of representation learning, and their benefits are therefore task-dependent. Nevertheless, full GestaltCLIP achieves the highest or tied-highest reported mean across all eight evaluation metrics in Table R11, supporting the overall effectiveness of the coordinated training design.


W1. Training-Budget Fairness and Model Selection

To assess whether these improvements could be explained by differences in visual optimization capacity or model-selection opportunities, we additionally audited training and selection budgets across all variants.

Table R12. Training and model-selection budgets across seeds 42/43/44. A/B serves as the reference; Same indicates identical values or seed-wise schedules. Bold highlights differences relative to A/B.

Budget dimensionA/B (reference)CDE
Unique training images6,843SameSameSame
Peak trainable visual parameters14.96MSameSameSame
Upstream phenotype parameters—2.56M2.56M2.56M
Student visual optimizer updates6,864 / 6,864 / 6,851SameSameSame
Total optimizer updates6,864 / 6,864 / 6,85110,638 / 9,108 / 9,80913,550 / 12,020 / 13,44910,638 / 9,108 / 9,809
Image presentations219k / 219k / 219kSameSame282k / 282k / 269k
Visual validation evaluations48 / 48 / 53SameSameSame
Total selection evaluations48 / 48 / 5385 / 70 / 82117 / 102 / 12285 / 70 / 82

Table R12 confirms matched supervision access, peak trainable visual capacity, student visual optimizer updates, and visual validation counts across variants. However, end-to-end computation is not identical: C–E require upstream phenotype training, D includes additional teacher optimization, and E uses more image presentations through paired replay. Together with the progressive results (Table R11), these findings support the benefits of our training design beyond straightforward aggregation of supervision resources, while acknowledging differences in computational and image-exposure budgets.

Dear Reviewer SYPV, Thank you for your time and thoughtful review. As the discussion closes tomorrow, November 18 (AoE), we would appreciate hearing whether our rebuttal addresses your concerns. We would be happy to clarify any remaining points before the deadline.

Best, The authors

Dear Reviewer SYPV, Thank you for your thoughtful feedback. As the ICLR author–reviewer discussion closes today (November 18, AoE), we wanted to follow up on our responses. We have addressed each of your comments and completed all the analyses you requested.

We would be grateful if you could take these results into account as you finalize your assessment. If any questions or concerns remain, we would be happy to address them before the discussion closes.

Best, The Authors

Reviewer_pf3HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GestaltCLIP, a three-stage vision-language framework for learning facial representations structured by Human Phenotype Ontology concepts when phenotype supervision is available at different levels of granularity. First, an ontology-aware HPO concept space and phenotype-set encoder are trained using clinical phenotype cases. Second, paired facial images and patient HPO sets provide atomic and set-level alignment. Third, disease labels, disease-HPO priors, teacher-derived visual evidence, and phenotype-set structure generate weak image-conditioned supervision for images without patient-level annotations. The final image encoder operates without patient phenotype input at inference. GestaltCLIP generally improves over pretrained visual and biomedical vision-language baselines. Extensive ablations and sensitivity analyses examine the proposed stages.

Critical strengths
  1. The paper addresses a clinically relevant learning problem: combining sparse image-level annotations with richer ontology and disease-level phenotype knowledge rather than assuming complete image-text pairing.
  2. The staged design is coherent. Ontology refinement, phenotype-set modelling, paired visual grounding, and disease-mediated weak alignment correspond to distinct supervision granularities and are supported by detailed ablations.
  3. The evaluation covers disease classification, image retrieval, atomic HPO retrieval, phenotype probing, patient-profile retrieval, external transfer, and visually unseen diseases.
Critical weaknesses
  1. The headline comparisons do not use matched supervision. GestaltCLIP receives paired image-HPO data, additional disease-labelled images, disease-HPO priors, and external phenotype cases, whereas most baselines remain frozen and Adapted OpenCLIP uses only the paired subset. The gains therefore cannot be attributed specifically to the proposed architecture.
  2. The rare-disease split is insufficiently specified. Diseases contain only one to six patients, while 306 diseases occur in training and only 204 in validation and test. It is unclear how diseases lacking patient-disjoint coverage are handled.
  3. Statistical evaluation uses three seeds and fixed patient splits. Patient-sampling uncertainty is not quantified, which is important for the small RDFace subsets and for diseases represented by very few individuals. Paired tests or patient-level bootstrap intervals are absent.
Questions for authors
  1. Can the authors provide a supervision-budget table showing, for every method, the number of images, disease labels, phenotype cases, HPO annotations, trainable parameters, and optimisation steps used?

  2. How are diseases represented by only one or two patients assigned across the patient-disjoint splits, and which diseases are included when balanced accuracy and macro-F1 are calculated?

  3. How many evaluated HPO annotations are confirmed to be visible in the corresponding image, and can results be separated for directly and conditionally observable concepts?

  4. How many of the 56 RDFace unseen diseases occur in the Stage-1 phenotype corpus or disease-HPO resources? Can performance be separated into visually unseen and semantically unseen subsets?

  5. Why is human inter-rater agreement for facial-HPO observability only kappa = 0.34, and how were disagreements resolved? Can the main results be repeated with a human-adjudicated vocabulary?

8 comments
AuthorsOpenReview ↗

Response to Reviewer pf3H — Part 1/6: W1/Q1 — Matched supervision comparison and supervision budget

Dear Reviewer pf3H,

Thank you for your thoughtful and constructive feedback. We conducted a same-resource comparison and a supervision-budget audit (W1/Q1, Tables R1–R3), patient-level bootstrap analysis (W3, Table R4), observability-stratified and image-level HPO retrieval analyses (Q3, Tables R5–R6), Data split and an RDFace semantic-exposure audit (W2/Q2, Q4, Table R7), and a human-consensus vocabulary evaluation (Q5, Table R8). We also evaluated backbone-adapted baselines (EQ2, Table R9) and clarified our methodological novelty and related work (EQ1, EQ3). These results strengthen the supervision-fairness and phenotype-grounding evidence while explicitly distinguishing clinical annotations from confirmed image visibility. Our point-by-point responses follow in 6 parts below, beginning with W1/Q1.


W1/Q1. Matched Supervision and Per-Method Supervision-Budget Analysis

We agree that the original frozen baselines and paired-only Adapted OpenCLIP do not fully control for differences in supervision access. To address this, we introduce a same-resource Naive Unified Baseline and provide a per-method supervision-budget audit.

Same-resource Naive Unified Baseline. The baseline uses the same OpenCLIP ViT-B/16 initialization and supervision resources as GestaltCLIP: 2,884 paired image–HPO samples, 3,959 disease-labeled images, 7,234 external clinical phenotype cases (6,509 training; 725 validation), and the same disease–HPO knowledge resources.

It directly uses patient-level HPO annotations for paired images and assigns aggregated disease-level HPO profiles as surrogate labels to weakly annotated images. All images are jointly trained using standard multi-label BCE, treating unannotated concepts as negatives.

Both methods have the same peak trainable visual capacity (14.96M parameters) and matched student visual optimizer updates (6,864 / 6,864 / 6,851 across seeds 42/43/44).

Table R1. Performance under matched supervision resources and visual adaptation budgets. Mean ± SD over three seeds (%). Improvements are percentage points (pp).

Evaluation taskNaive UnifiedGestaltCLIPImprovement
GMDB Common Top-131.81 ± 0.2335.85 ± 0.09+4.04 pp
GMDB Common atomic HPO mAP10.94 ± 0.1013.07 ± 0.16+2.13 pp
GMDB Common concept-set mAP1.31 ± 0.116.05 ± 0.41+4.74 pp
GMDB Rare Top-113.54 ± 2.3926.95 ± 1.03+13.41 pp
GMDB Rare atomic HPO mAP10.66 ± 0.5811.91 ± 0.16+1.25 pp
GMDB Rare concept-set mAP8.10 ± 0.0620.69 ± 1.44+12.59 pp
RDFace shared Bal. Acc.8.42 ± 0.7215.78 ± 0.89+7.36 pp
RDFace unseen Bal. Acc.2.14 ± 0.005.89 ± 0.93+3.75 pp

As shown in Table R1, GestaltCLIP outperforms the same-resource baseline across all eight settings, particularly on Rare Top-1 accuracy (+13.41 pp) and Rare concept-set retrieval (+12.59 pp). Thus, access to additional phenotype resources alone does not reproduce the gains of our structured heterogeneous-supervision approach.

Per-method supervision-budget audit. We additionally quantify the representation-learning resources used by each baseline in Table R2, including trainable parameters and optimizer updates.

Table R2. In-study representation-learning supervision and optimization budgets. Zero indicates no additional in-study representation training, not an absence of original pretraining.

MethodTraining imagesDisease-labeled imagesHPO-annotated imagesExternal phenotype casesTrainable representation paramsOptimizer updates (42/43/44)
OpenCLIP000000
BiomedCLIP000000
MedCLIP000000
PhenoLIP000000
FaRL000000
ViT (ImageNet)000000
GestaltMatcher-Arc000000
Adapted OpenCLIP2,88402,884014.57M visual3,640 / 3,640 / 3,640
Naive Unified6,8436,8432,8846,509 train + 725 val14.96M visual6,864 / 6,864 / 6,851
GestaltCLIP6,8436,8432,8846,509 train + 725 valS1: 2.56M; S2: 14.96M; S3: 0.393M10,638 / 9,108 / 9,809

All methods use identical downstream evaluation protocols, including disease-classification linear probes trained on the same 7,073 GMDB training images. These downstream probe budgets are excluded from Table R2. GestaltCLIP's total optimizer updates additionally include Stage-1 phenotype training; its student visual updates are matched to the naive baseline. The 725 external validation cases are used only for model selection. Adapted BiomedCLIP and Adapted FaRL use the same 2,884 paired images, visual optimizer-update schedule, and selection protocol as Adapted OpenCLIP (EQ2); trainable parameter counts may differ by architecture.

AuthorsOpenReview ↗

Response to Reviewer pf3H — Part 2/6: W1/Q1(finished), W2/Q2, W3 — Annotation-level budgets, Data split protocol, and Statistical Uncertainty

W1/Q1 (continued).

To quantify supervision beyond sample counts, Table R3 reports the underlying HPO annotation and disease–phenotype association budgets.

Table R3. Annotation-level supervision budgets for the full-resource methods.

Annotation categoryNaive UnifiedGestaltCLIP
Patient-level HPO-annotated images2,8842,884
Patient-level positive HPO assignments17,92617,926
Clinical-case HPO assignments used for training28,90826,000 / 26,060 / 25,968
Disease–HPO associations in filtered HPOA resource42,12142,121
Effective disease–HPO edges for weak-image diseases6,3215,975

Table R3 further confirms that the naive baseline is not disadvantaged in annotation quantity: both methods use the same patient-level HPO annotations and underlying disease–HPO resource, while the naive baseline uses more clinical-case assignments and effective disease–HPO edges. Although student visual optimizer updates are matched, end-to-end computation and image exposure differ due to upstream phenotype training and paired replay. We will clarify these distinctions in the revised manuscript.


W2/Q2. Rare Disease Split Protocol

As detailed in Appendix B.1 (Table B.1), the GMDB Rare partition contains 306 diseases, each with 1–6 patients, and all splits are patient-disjoint. Diseases with 1–2 patients are assigned exclusively to training, while those with 3–6 patients are distributed across training, validation, and test, with at least one distinct patient per split. Thus, all 306 diseases appear in training, 204 are represented in both validation and test, and the remaining 102 are training-only.

Importantly, the downstream linear probe performs 306-way classification, without restricting predictions to the 204 test-supported diseases. Balanced accuracy and macro-F1 are computed over the 204 diseases represented in the test set; predictions of the 102 training-only classes are treated as incorrect predictions, not excluded from evaluation.

We will clarify these split rules and evaluation definitions in the revised manuscript.


W3. Statistical Uncertainty

We thank the reviewer for highlighting patient-sampling uncertainty, particularly in small external cohorts. Beyond the three-seed mean ± SD reported in the manuscript, we conducted disease-stratified patient-level bootstrap analysis on GMDB Common and RDFace Shared, resampling patients within diseases to estimate 95% confidence intervals.

Table R4. Patient-level bootstrap 95% confidence intervals for GestaltCLIP. Each cell reports the mean performance (%) followed by its 95% CI. GMDB Common contains 1,880 images from 1,457 patients across 275 diseases; RDFace Shared contains 112 patients across 25 diseases.

EvaluationTop-1 (%) [95% CI]Top-5 (%) [95% CI]Bal. Acc. (%) [95% CI]Macro-F1 (%) [95% CI]
GMDB Common35.85 [33.87, 37.86]60.20 [58.13, 62.29]25.93 [24.43, 27.80]26.78 [24.17, 27.59]
RDFace Shared15.18 [11.60, 18.75]29.17 [23.51, 34.52]15.78 [12.38, 19.25]20.73 [16.45, 23.57]

The intervals quantify patient-sampling uncertainty beyond training-seed variation, particularly for the smaller RDFace Shared cohort.

For GMDB Rare, the test set contains 204 patients across 204 diseases. As each disease has only one test patient, within-disease patient bootstrap resampling is degenerate and cannot estimate within-disease variability. We therefore retain the three-seed mean ± SD reported in the manuscript rather than presenting uninformative bootstrap intervals.

AuthorsOpenReview ↗

Response to Reviewer pf3H — Part 3/6: Q3 — Photographic observability and image-level recovery

Q3. HPO Photographic Observability and Image-Level Annotation Recovery

We thank the reviewer for highlighting the distinction between concept-level photographic observability and image-specific phenotype visibility. To address this concern, we provide two complementary analyses: (1) observability-stratified HPO retrieval and (2) image-level clinical annotation recovery.

Concept-level photographic observability. As described in Appendix A.1, our vocabulary construction filters 12,528 HPO concepts into 1,127 photographically assessable concepts, comprising 285 DIRECT and 842 CONDITIONAL concepts. DIRECT concepts are generally assessable from facial photographs, whereas CONDITIONAL concepts depend on factors such as viewing angle, anatomical visibility, or imaging conditions.

We first evaluate image-to-HPO retrieval separately within these two concept groups.

Table R5. Image-to-HPO retrieval mAP stratified by photographic observability on GMDB Common and Rare. Mean ± SD across three seeds (%). Retrieval is evaluated separately within the DIRECT (285-concept) and CONDITIONAL (842-concept) candidate vocabularies, using identical candidate sets across methods. These subgroup-restricted mAP values are not directly comparable to the full 1,127-concept retrieval results in the manuscript.

EvaluationOpenCLIPAdapted OpenCLIPGestaltCLIP
Common — DIRECT2.99 ± 0.0013.31 ± 0.1116.52 ± 0.25
Common — CONDITIONAL2.38 ± 0.0012.99 ± 0.1114.18 ± 0.54
Rare — DIRECT3.03 ± 0.0010.42 ± 0.0712.69 ± 0.50
Rare — CONDITIONAL3.16 ± 0.008.98 ± 0.1811.96 ± 0.30

As shown in Table R5, GestaltCLIP outperforms Adapted OpenCLIP in all four settings. The improvements are +3.21/+1.19 pp for Common DIRECT/CONDITIONAL and +2.27/+2.98 pp for Rare DIRECT/CONDITIONAL, indicating that the retrieval improvements extend to both observability groups.

Image-level clinical annotation recovery. To examine phenotype grounding at the individual-image level, we construct an HPO-by-image annotation matrix using patient-level clinical annotations from 798 GMDB Common and 73 GMDB Rare held-out images. We compare this matrix against image-only HPO predictions, ranking the full 1,127-concept vocabulary for each image and counting clinically annotated HPO terms recovered among the Top-10 and Top-20 predictions. Recall is micro-averaged over annotated image–HPO pairs within each group.

Table R6. Image-level recovery of clinically annotated HPO concepts on GMDB Common and Rare. Annotation counts are fixed across models. Each model cell reports micro annotation Recall@10 / Recall@20 (%), averaged over three seeds. Unlike Table R5, retrieval uses the full 1,127-concept candidate vocabulary.

EvaluationImagesAnnotated pairsOpenCLIP R@10 / R@20Adapted OpenCLIP R@10 / R@20GestaltCLIP R@10 / R@20
Common — DIRECT7982,9280.51 / 1.3012.47 / 18.8115.14 / 23.00
Common — CONDITIONAL7981,9083.14 / 6.4511.08 / 15.1312.21 / 17.56
Rare — DIRECT732780.36 / 0.727.07 / 13.7912.11 / 18.71
Rare — CONDITIONAL731855.41 / 8.1113.29 / 17.6815.68 / 19.46

As shown in Table R6, GestaltCLIP improves annotation recovery over Adapted OpenCLIP in all four groups at both retrieval cutoffs. At Recall@20, improvements are +4.19 pp (Common DIRECT), +2.43 pp (Common CONDITIONAL), +4.92 pp (Rare DIRECT), and +1.78 pp (Rare CONDITIONAL). For example, on Common DIRECT, GestaltCLIP recovers approximately 673 of 2,928 clinically annotated image–HPO pairs within its Top-20 predictions, averaged across seeds.

Scope and limitations. Table R5 assesses retrieval quality within concept-specific vocabularies, whereas Table R6 quantifies image-level annotation recovery under the full vocabulary. Together, they provide complementary evidence of phenotype grounding across both photographic-observability groups. However, clinical HPO annotations do not independently establish that a phenotype is visible in its corresponding photograph. Therefore, the recovered pairs should not be interpreted as expert-confirmed visible manifestations. Such confirmation would require patient-specific image-level clinical adjudication, which is not available in the present evaluation.

We will incorporate these analyses and clarify the distinction between concept-level photographic assessability, image-level annotation recovery, and clinically confirmed visibility in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.