acceptodds
Withdrawn from ICLR 2027

Representation: Geometry Supervised Representation Learning of Phenotypes via Counterfactual Reasoning for Medical VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Medical vision-language models (VLMs) have shown increasing potential for radiological image interpretation. Medical VLMs encode radiological images into visual representations that capture both anatomical and phenotypic information for diagnosis. Existing approaches improve pathological phenotype representations through semantic-guided representation alignment. However, pathological phenotypes arise as lesion-specific visual changes superimposed on underlying normal anatomy. Such semantic alignment approaches fail to model the phenotype-specific increment relative to the corresponding normal anatomical representation.To address this gap, we propose Representation, a visual phenotype representation learning framework based on counterfactual reasoning for medical VLMs. It comprises BaseAnatomy, a geometry-supervised representation learning module, and Phenotype, a counterfactual incremental representation learning module. BaseAnatomy provides fine-grained geometric supervision through spatial relationships across and within anatomical structures. Phenotype computes the representation increment between lesion representations and their corresponding normal anatomical representations, and supervises increments associated with the same phenotype to cluster in the representation space. Experiments on ReXGroundingCT and LIDC-IDRI demonstrate that Representation effectively structures pathological phenotype representations and improves lesion grounding and phenotype characterization accuracy in medical VLMs. Code is available at https://anonymous.4open.science/r/deltarep-CF6D.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
pathologicial phenotypes, representation learning, geometry-supervision, counterfactual reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1aKROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ΔRepresentation, a representation-learning framework for medical vision-language models that characterizes pathological phenotypes as representation increments relative to estimated counterfactual normal anatomy. The paper introduces BaseAnatomy to organize anatomical representations using geometric supervision and ΔPhenotype to learn lesion-specific changes relative to these anatomical references. The overall perspective is constructive and intuitive, and I believe the high-level idea is promising.

However, the paper’s current level of completion is limited. Many important implementation details remain insufficiently explained, and the logical development of the method is not always rigorous. The visual presentation is also below the standard expected of an academic paper. Several figures are low-resolution and contain poorly standardized visual elements. Figure 1, in particular, appears to contain obviously AI-generated illustrative assets with numerous imprecise details. I do not object to the use of generative models for figure creation in principle, but the resulting figures must still satisfy academic standards. Similarly, the method figure lacks careful alignment and even basic horizontal and vertical organization (It appears that the authors have not received standardized training in academic literacy). Finally, the empirical analysis relies heavily on scatter plots and low-dimensional projections, which are not always appropriate or sufficient to substantiate the claimed representation properties (but not a big deal).

Given the space constraint, I focus below on one critical strength and two critical weaknesses.

Critical strengths

The paper’s central perspective, "representing a pathological phenotype as a deviation from its corresponding normal anatomical representation," is both intuitive and potentially valuable. It offers a conceptually meaningful alternative to learning pathology representations solely through absolute visual features or image-text alignment. The proposed direction is broadly consistent with clinical reasoning, in which abnormalities are often recognized as changes relative to expected anatomy. Although the present implementation requires substantial improvement, the underlying research direction is promising.

Critical weaknesses

1: Mismatch Between the 3D Methodological Motivation and the 2D Evaluation

The paper motivates the method in terms of volumetric anatomical geometry, three-dimensional spatial continuity, and reasoning over 3D CT anatomy. However, the actual benchmark treats an individual axial CT slice as the unit of analysis. The reported downstream tasks operate on resized 2D slices, and lesion grounding is reduced to selecting a cell from a (4 x 4) two-dimensional grid. Slice-position encoding does not, by itself, constitute a convincing evaluation of volumetric reasoning.

Consequently, the experiments do not demonstrate that the proposed representation preserves meaningful 3D anatomical relationships, supports cross-slice reasoning, or improves volume-level lesion localization and characterization. The observed gains could instead arise from learning 2D spatial patterns or slice-position priors. To support the paper’s broader claims, the authors should evaluate on complete CT volumes using genuinely 3D tasks and baselines, for example, volume-level lesion localization or segmentation, cross-slice consistency, and volumetric report generation, and should ablate the contribution of slice-position information.

2: Heavy Dependence on Localization Priors and Dense Annotations

The proposed framework depends heavily on strong localization priors during representation learning. BaseAnatomy requires anatomical annotations and lesion masks, while ΔPhenotype uses known pathological patches, anatomically matched mask-negative neighbors, lesion-level aggregation, and explicit localization supervision. Moreover, the phenotype VQA setup directly provides the target grid cell in the question. These design choices substantially simplify the learning problem and make it difficult to determine whether the improvements originate from the proposed counterfactual representation mechanism or from the extensive spatial supervision provided to the model.

Although the paper states that masks are not required during inference, this does not eliminate the framework’s substantial training-time dependence on accurately localized anatomical and pathological regions. This dependence may limit scalability to datasets or clinical settings without dense annotations and raises concerns about robustness when annotations are noisy or produced by imperfect automatic segmentation models (such issue always happens). The paper needs stronger ablations that remove or weaken these priors, comparisons against baselines receiving equivalent localization supervision, and experiments using predicted or noisy masks. Without such evidence, the practical generalizability of the proposed approach remains unclear.

Questions for authors

See weaknesses.

Reviewer_wc4dOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The central idea of ΔRepresentation is to learn lesion-specific changes relative to an anatomical reference. BaseAnatomy first organizes normal anatomical representations through geometric supervision. ΔPhenotype then estimates local normal references and learns representation increments that are consistent within a phenotype and separated across phenotypes.

The framework is evaluated on ReXGroundingCT and LIDC-IDRI through lesion-grounding and phenotype questions and report generation, using several medical VLM backbones.

Critical strengths

Anatomically grounded formulation. Modeling a lesion relative to its underlying anatomical context is a well-motivated approach. The combination of anatomical geometry and phenotype increments provides a concrete way to move beyond generic image–text alignment and makes the representation-learning objective easy to relate to the medical imaging problem.

Evidence across backbones. Evaluation across several VLM backbones and two datasets provides useful evidence that the approach is not tied to a single model. The paper also examines representation organization and the effect of increasing lesion-training data, which complements the downstream results.

The approach is promising and the reported improvements are encouraging. My recommendation is weak acceptance.

Critical weaknesses

Two aspects would benefit from a clearer explanation: the assumption behind the estimated normal reference, and the comparison protocol. In particular, the contribution of increment learning should be distinguished from the benefit of additional supervision and adaptation. These are qualifications to an otherwise positive assessment.

Questions for authors

Normal-reference construction What conditions make a nearby mask-negative patch a reliable normal reference? Please explain how the construction handles diffuse or multifocal abnormalities and cases with too few suitable neighbors. It would also help to distinguish an estimated representation reference from a verified healthy counterfactual.

Supervision and adaptation Could you provide a compact account of the supervision and adaptation available to each baseline, including anatomical/lesion labels and multimodal-projector training? This would help readers interpret which comparisons isolate the proposed representation formulation.

Evaluation scope and parsing Please make the evaluation scope explicit: single axial images, grid-based VQA grounding, and attribute-based report evaluation. For the report parser, how were ambiguous or multiple findings handled, and was its extraction consistency checked? I am asking for a clearer explanation of the current protocol rather than a new evaluation suite.

Reviewer_PpJ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes anatomy-referenced representation learning for medical VLMs. BaseAnatomy organizes visual representations using anatomical geometry, while ΔPhenotype learns lesion-specific increments relative to estimated normal references. Experiments report improvements on lesion grounding and phenotype characterization across two CT datasets and six VLM backbones.

Critical strengths
  1. Separating anatomical context from pathological variation is a well-motivated direction for medical representation learning.
  2. The evaluation covers multiple backbones and patient-disjoint splits, and the reported improvements are consistent across both datasets (Tables 1–2).
Critical weaknesses
  1. Tables 1–3 do not include matched-supervision controls that separate the effects of anatomical/lesion annotations, additional training, and counterfactual subtraction. A BaseAnatomy-only control and a phenotype-learning control without reference subtraction would help establish the specific contribution of ΔPhenotype.
  2. Equation 2 estimates a reference by interpolating nearby mask-negative patches. Figure 4 does not adequately explain how the corresponding real normal representations are obtained. The compactness and abnormal-to-reference distances in Table 5 establish local consistency, but not necessarily the accuracy of the unobserved normal counterpart. This limits the interpretation of the learned difference as a phenotype-specific change.
Questions for authors
  1. Under identical annotations, projector training, and optimization budgets, how do BaseAnatomy alone and phenotype learning without subtraction compare with the full method?
  2. How is the normal-reference ground truth in Figure 4 obtained? Can reference estimation be evaluated quantitatively on held-out normal patches, with a simple alternative-reference baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.