acceptodds
Under review as a conference paper at ICLR 2027

Geometry-Supervised Visual Representation Learning for Multi-Phenotype Lesion Interpretation in Medical VLMs

17 views

Abstract

Medical vision-language models (VLMs) have shown increasing potential for clinical image interpretation. However, these models still struggle to interpret multi-phenotype lesions whose diagnosis requires the joint assessment of multiple pathological phenotypes. Existing vision–language alignment methods produce visual representations that fail to preserve anatomical hierarchies and relationships among phenotypic subclasses. This stems from their reliance on semantic supervision, which lacks geometric constraints to preserve these relationships in the visual embedding space. Moreover, the sparsity of lesion-related anatomical and phenotypic representations makes it difficult for medical VLMs to capture important diagnostic evidence. To address these limitations, we propose PureVision, a geometry-supervised visual representation learning framework for multi-phenotype lesion interpretation in medical VLMs. It combines a geometry-supervised representation learning module, PureEyes, and an anatomy-guided evidence aggregation module, PureNeurons. PureEyes provides geometric supervision through ideal spatial distributions that encode anatomical hierarchies and phenotypic subclass relationships. PureNeurons projects visual representations into the learned latent space, using their positions to selectively aggregate lesion-specific anatomical and phenotypic evidence. Experiments on LIDC-IDRI, CBIS-DDSM, and 3DReasonKnee demonstrate that PureVision improves lesion grounding and phenotype characterization in visual question answering and radiology report generation. Code is available at: https://anonymous.4open.science/r/purevision-06C2.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
representation learning, medical VLM, geometry-supervision, phenotype interpretation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EMBAOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates the visual representation challenges faced by medical vision-language models in understanding lesions with multiple phenotypes. The authors note that existing semantic alignment-based methods struggle to preserve the hierarchical relationships of anatomical structures and the continuous relationships between different pathological phenotypes within the visual embedding space; furthermore, sparse lesion information is easily diluted by the abundance of normal tissue patches. To address this, the authors propose "PureVision," which comprises two main modules: "PureEyes," which supervises anatomical and phenotypic visual representations using ideal geometric distributions, and "PureNeurons," which selects lesion-relevant patches from the structured visual space and fuses anatomical and phenotypic evidence before passing it to the language model. Experiments conducted on the LIDC-IDRI, CBIS-DDSM, and 3DReasonKnee datasets, covering tasks such as VQA and radiology report generation, demonstrate that this method enhances capabilities in lesion localization and the description of pathological phenotypes.

Critical strengths
  1. The issues the paper focuses on are more practical. The author did not just continue to strengthen vision-text alignment, but further considered whether the visual feature space itself retains the medical anatomical hierarchy and pathological phenotype relationships. This starting point has certain significance.

  2. The designs of PureEyes and PureNeurons in Section 3.2 and Section 3.3 have a clear complementary relationship: the former is responsible for building a structured visual representation space, and the latter is responsible for filtering and delivering sparse lesion evidence.

  3. The experiment covered three different imaging modalities: CT, mammography and MRI. Multiple medical VLM backbones were tested at the same time and verified on two tasks: VQA and report generation. The scope of the experiment was relatively comprehensive.

Critical weaknesses
  1. I feel the paper's framing regarding "multi-phenotype lesion interpretation" is slightly overstated. The 3DReasonKnee dataset described in Section 4.1 actually involves only a single phenotype dimension; thus, it serves more to validate cross-modal and cross-disease generalization rather than strictly testing a multi-phenotype setting.

  2. In Appendix A, the methods for calculating phenotype accuracy differ between the VQA and RRG tasks, yet Tables 1 and 2 use identical metric names. This could mislead readers into thinking the phenotype accuracy metrics for the two tasks are identical and directly comparable; I suggest the authors provide further clarification.

  3. The terminology used to describe core concepts in the paper is somewhat inconsistent; terms such as "ideal spatial distribution," "ideal visual semantic distribution," and "ideal geometry" are all used to refer to similar supervision targets. While the authors' intent is generally clear, for a paper whose primary contribution is geometry supervision, it would be best to standardize the terminology to avoid confusing readers about the specific objects of the constraints.

Questions for authors

Could the authors clarify what aspect of the multi-phenotype lesion interpretation setting is intended to be validated by 3DReasonKnee, given that Section 4.1 includes only one phenotype dimension for this dataset?

Reviewer_vtBjOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PureVision, a geometry-supervised visual representation learning framework for improving lesion localization and phenotype characterization in medical VLMs. It consists of two modules: PureEyes, which uses separate anatomical and phenotypic encoders with geometry-aware constraints to preserve anatomical hierarchies and phenotype relationships, and PureNeurons, which selects lesion-relevant patches and integrates their evidence into the language model through soft semantic fusion. Experiments on three medical imaging datasets demonstrate consistent improvements across multiple medical VLMs in VQA and radiology report generation.

Critical strengths
  1. Meaningful geometric supervision. PureEyes introduces geometry-aware constraints for different medical attributes, preserving continuity, ordinal relationships, and anatomical hierarchies in visual representations.
  2. Comprehensive experiments. The method is evaluated on three medical imaging datasets and six medical VLMs, showing consistent improvements in VQA and radiology report generation.
  3. Useful visualizations. The t-SNE visualizations illustrate the improved feature distributions. However, some plots are visually dense, making the differences difficult to distinguish.
Critical weaknesses
  1. Insufficient validation of geometric supervision. Missing comparisons with standard losses make its independent contribution unclear.
  2. Limited novelty of PureNeurons. Its Top-K selection and semantic fusion lack comparisons with conventional alternatives.
  3. Dependence on dataset-specific annotations. PureEyes requires anatomical and phenotype labels, raising concerns about scalability and transferability.
  4. Unclear computational overhead. The dual-encoder design lacks efficiency analysis and comparisons with a single-encoder baseline.
Questions for authors
  1. Scalability and Transferability: How might PureEyes extend to more complex anatomical settings, such as whole-brain MRI, particularly regarding new geometric constraints and annotation requirements? I would appreciate a discussion of this issue rather than additional experiments.
  2. Novelty of PureNeurons: What fundamentally distinguishes PureNeurons from existing region-guided or concept-alignment methods (e.g., RadAlign, MCA-RG, Reg2RG)?

Clarifying these points would help assess the method's novelty and generalizability and may influence my final rating.

Reviewer_TnEZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address two key limitations of medical Vision-Language Models (VLMs) in interpreting "multi-phenotypic lesions": the lack of anatomical hierarchy and phenotypic sub-category structure in visual representations, and the extreme sparsity of lesion evidence, which often gets overwhelmed by normal tissue. This paper proposes the "PureVision" framework, consisting of two modules. The first, "PureEyes," employs two decoupled visual encoders—one dedicated to anatomy and the other to phenotypes—that bypass text-embedding supervision in favor of explicitly defining the expected structure of the visual space. The second, "PureNeurons," isolates lesion-related patches and feeds them into the decoder as probability-weighted "soft tokens," incorporating both anatomical/phenotypic evidence and estimated lesion locations. Experiments on the LIDC-IDRI, CBIS-DDSM, and 3DReasonKnee datasets demonstrate performance improvements across both Visual Question Answering (VQA) and report generation tasks.

Critical strengths
  1. The problem diagnosis is concrete, and the research direction, focusing on "geometric rather than semantic supervision", is one that has not been seriously explored before. Most fine-grained alignment methods rely on text embeddings as the supervisory signal; the authors point out that this fails to preserve anatomical hierarchies or the continuity between phenotypes. This critique is well-founded and leads to actionable design choices, moving beyond vague claims that "representations are insufficient." The authors tailor their objectives to the nature of the phenotypes (continuous, ordered, or categorical), incorporate a distillation term to prevent the degradation of features in "normal" regions, and include a specific term to disentangle confusing structures, such as distinguishing blood vessels from nodules. These details suggest that the authors have genuinely grappled with practical challenges, rather than simply cobbling together a solution based on intuition alone.

  2. The evidence provided is more robust than what is typically found in such papers; it does not rely solely on final performance metrics. The consistent improvements observed across six different backbones, with the most significant gains seen in the weakest models—are far more convincing than results achieved by "chasing" higher scores on already powerful models. Crucially, the authors provide evidence independent of downstream task metrics: t-SNE visualizations reveal a clear, progressive trajectory based on size and density, and the anatomical-textual similarity score improves from 0.36 to 0.53. Furthermore, ablation studies show that removing the "PureNeurons" component still yields performance figures of 32.50/36.57, confirming that both modules contribute meaningfully to the overall framework. The evaluation protocol is also rigorous, employing a patient-disjoint data split.

Critical weaknesses
  1. The "grounding" metric is potentially circular, and the baseline falls below random-chance levels, suggesting that the reported performance gains are likely overestimated. The paper itself states that "PureNeurons" injects the "estimated lesion location" into the decoder's input; since the grounding accuracy metric evaluates whether the output location is correct, it is essentially measuring the precision of the internal localization module rather than the model's actual grounding capability. A compelling piece of supporting evidence appears in the ablation study: removing "PureNeurons" and replacing it with "lesion grounding" results in an 8-point increase in grounding accuracy, while the phenotype prediction performance remains virtually unchanged, a pattern strongly suggestive of simply "feeding the location in and reading it back out."
  2. The claim that "geometric supervision is the key" lacks proper ablation; the observed gains are confounded by the inclusion of additional annotations and the increased capacity of the dual-tower architecture. The paper provides no ablation of the "PureEyes" components, nor does it include a control experiment using the same annotations but applying standard supervision (such as supervised contrastive learning or direct classification/regression). It is highly probable that such an approach could replicate most, if not all, of the reported gains—which would invalidate the claim that "ideal geometry" is the driving factor, even though this claim serves as the central thesis of the paper. Furthermore, there is no sensitivity analysis regarding the manually set constants (the scale factor of 1.5, the margin of 0.1, and the temperature parameter of 0.125).
Questions for authors
  1. Are the estimated lesion locations incorporated into both the VQA and report-generation tasks? If so, could you provide a baseline comparison where these location inputs are omitted—relying solely on visual tokens? Additionally, what is the standalone accuracy of the localization module within PureNeurons? Furthermore, to what extent does the reported "grounding" accuracy simply reflect the model echoing the input?
  2. How exactly are the VQA answers extracted and matched? Do the baselines that fall below 25% accuracy primarily fail due to issues with output formatting? It would be helpful to report "parsability rate" and "accuracy among parsable outputs" separately.
  3. Have you conducted a comparison between "geometric supervision" and "standard supervision"? This could be evaluated by running a supervised contrastive learning experiment using the same annotations. This is the question I am most interested in.
  4. Could you provide an ablation study for the components of PureEyes? Specifically, what is the individual contribution of each element: the three geometric types (continuous/ordered/categorical), anatomical hierarchy, distillation, and the separation of confusing tissue types?
  5. What are the actual parameter count and inference overhead after adding the two visual encoders? Is there a single-encoder version available? How does the model perform compared to an approach that uses ground-truth masks to crop the regions of interest (ROIs)?

Comments on the paper

2 comments
Reviewer_EMBAOpenReview ↗

Response to the Program Chairs

To provide greater clarity about my review process, I have included my original review comments below for reference:

1, Med Caption 任务现在临床上都不用,不知道为啥还有继续坚持这些任务,性能再好有什么意义呢,如果注入模型之后 反过来改善数学或者coding 甚至意义更大一些;

2,Med + VLM 这类文章最大的问题就是他到底在解决医学问题还是在解决AI问题,这文章关注multi-phenotype lesion,说白了就是长得不一样的病灶,对医学影响比较大,对智能的增益 我觉得没啥用,关键是他测的也不是multi-phenotype lesion,3DReasonKnee 也就一种款式啊?

3,写的有点乱,感觉前后不一致问题比较大, ideal spatial distribution ,ideal visual semantic distribution,感觉怪怪的,像中式英语,不是专有的学术表达,感觉指标的计算上 对比的setting也有点问题;

To avoid having my review flagged as AI-generated, I revised the wording and then translated it using Google Translate. However, it was still flagged as AI-generated.

Dear Reviewers,

The authors have posted responses. As the discussion period is ending soon, could you please take a look and reply shortly, or at least acknowledge that you have read it?

Thank you!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.