acceptodds
Withdrawn from ICLR 2027

AnatoBridge: Strengthening and Transferring Anatomical Representations for 3D Chest CT Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models remain challenged in understanding 3D chest computed tomography (CT), where clinically relevant evidence is volumetric, spatially localized, and often subtle. Existing models inadequately exploit the anatomical structure needed to ground such evidence: i) anatomical guidance is weakly integrated into visual encoding, ii) lesion supervision remains underused in representation learning, and iii) explicit anatomical outputs are not directly exposed to the LLM. In this paper, we present AnatoBridge, an anatomy-aware 3D chest CT VLM that incorporates anatomical information throughout visual encoding, representation learning, and language alignment to address these limitations. AnatoBridge strengthens the visual representations by integrating region-constrained anatomy queries throughout the transformer and using lesion-guided pretraining to enrich organ features and construct directly supervised disease-specific representations. Moreover, a dual-channel interface carries this information into language modeling to provide the LLM with complementary fine-grained visual information and interpretable anatomical evidence, where the feature channel projects the learned representations into continuous anatomy tokens and the prompt channel serializes explicit anatomical outputs as disease findings, organ geometry, and lesion grounding. Experimental results show that AnatoBridge outperforms the strongest baseline by 1.1 in AUC (85.5 vs. 84.4) in zero-shot classification on CT-RATE and yields a 0.7 AUC gain (76.1 vs. 75.4) on the external Rad-ChestCT cohort. Remarkably, it also exceeds RadSight-8B by an average 3.33 points across the three 3D-RAD tasks, by 1.80 points on CT-Spatial-VQA, and by 6.23 points in report-generation clinical efficacy F1.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Medical Imaging, Vision-Language Models, Anatomical Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_faDHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a method for radiology report generation for 3D Chest CTs. To improve the performance, the authors enforce anatomical, lesion, and disease guidance through a specialized image encoder architecture with anatomy-, lesion-, and disease-aware pretraining. Furthermore, they also pass information about organ-properties, lesion-, and disease- findings via textual prompts to the vision-language model. They evaluate both the vision encoder (using linear probes) and the resulting vision-language model (on report generation and VQA).

Critical strengths

1. Analysis and Ablation Studies on both Vision Features and End-to-End Vision-Language Modeling

The authors analyze the performance of both their proposed vision encoder (using linear probes) and their resulting end-2-end vision-language model (on report generation and VQA).

Furthermore, they provide several ablation studies on most of their design decisions, thereby validating the overall benefits of their proposed contributions. (also see Weakness 2 for more discussion on the analyses)

Critical weaknesses

1. Limited discussion of contributions in comparison to prior works

While the authors discuss related work in Sec. 2, they do not always clearly specify how their contributions are related to these existing works (e.g. do they reuse ideas or build on some of them). Most notably, the contribution listing on p. 2 remains vague, making it hard to identify which contributions are new ideas, reuse existing ideas, or extend them. This makes it hard to judge the novelty and relevance of their contributions.

Specifically, the authors’ describe the following aspects of their method:

(a) Anatomy-aware encoding with organ-level tokens (anatomy queries) and organ-constrained attention

(b) Voxel-level anatomy- and lesion supervision (i.e. segmentation loss)

(c) Organ-level features enriched by associated lesion-tokens, which are computed by aggregating voxel tokens using lesion segmentation masks

(d) Constrative image-text alignment on the organ- and disease-level, where disease-level tokens are aggregated from the lesion tokens

(e) Textual prompts provided as context for the LLM including disease findings, organ geometry, and lesion grounding, derived from predictions of (b) and (d).

For (a), the authors describe that they extend CT-DiagVLM, claiming (in the abstract, introduction, and method section) that “anatomical guidance is weakly integrated into visual encoding”. In the method section, they write: “Such queries extract regional features only after encoding completes, leaving the visual encoding anatomically uninformed.” (p. 4, l. 171). I disagree with this statement, as these queries are based on information from the visual encoding (explicitly steered using organ masks and organ-level textual descriptions). Furthermore, the authors’ proposed contribution restricts attention between anatomy queries and does not allow vision tokens to attend to anatomy tokens. Therefore, the proposed architecture acts more like a multi-layer attention pooling per anatomy than an integrated processing of anatomy tokens. This makes it conceptually equal to CT-DiagVLM’s pooling but extended to a multi-layer approach. Note that the ablation study in Tab. 4 (second row) also demonstrates only minuscule benefits of their approach. Therefore, (a) remains a minor contribution.

For (b), the anatomy-segmentation is derived from CT-DiagVLM (which the authors also state). However, it is unclear whether lesion-level segmentation was already used by existing methods or is a newly proposed extension.

For (c): this seems to be a new contribution by the authors. However, it should be set in context with existing literature.

For (d): the disease-level tokens seem to be a new extension. But again, it should be set in context with existing literature.

Overall, the proposed contributions appear to be straightforward extensions to existing work. Authors should therefore clearly discuss (e.g., in their method or related work section) how these relate to existing literature and what exactly they contribute. Furthermore, the contributions should be described more fairly in the abstract and introduction (which currently slightly overstates them).

2. Limited Depth of Analysis

While the authors provide a set of ablation studies that validate the benefits of their contributions / design decisions (see strength 1), the analysis focuses mostly on “downstream performance”, i.e. classification, report generation, VQA. However, the authors introduce several specialized tokens and feature vectors (q_o, t_l, f_o, f_d), which are designed (via aggregation or supervision) to capture specific information about the image, e.g. anatomy-, lesion-, or disease-specific features. As these tokens/vectors are central parts of the authors’ contributions and hypotheses, the authors should validate that these vectors actually capture the expected information. For example,

  • Linear probes could be used to verify that organ-specific vectors contain more information about as specific organ than the other organ vectors, e.g. by comparing distances to organ-level report sentences or comparing classification performance on organ-specific lesions/diseases. Similarly for lesion- and disease-specific vectors
  • and/or some organ- or lesion-specific vectors (e.g., q_o or t_l) could be masked before computing F_organ, and the resulting generated reports could be studied. Alternatively, these vectors could be replaced by vectors from other patients with differing diagnoses. If Masking/modifying these leads to reports with missing or misreported information about the specific lesion/organ, then the relevance of those tokens would be validated.

In your rebuttal, please carefully consider which analyses could validate your hypotheses (you may or may not use the examples above).

Questions for authors
  • How is zero-shot classification evaluated, i.e., which tokens are used and how are they aggregated?
Reviewer_f1arOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AnatoBridge is a 3D chest CT vision–language framework that incorporates anatomical information into visual encoding, representation learning, and language alignment. It introduces organ queries with region-constrained attention throughout the visual transformer, uses lesion segmentation supervision to construct enhanced organ and disease representations, and connects these representations to an LLM through both continuous anatomy tokens and textual descriptions of predicted findings and spatial information.

The authors evaluate the pretrained encoder on CT-RATE and external Rad-ChestCT classification, and the resulting VLM on medical VQA, spatial understanding, and report generation. Reported improvements include AUC and ACC gains of 1.1 and 0.7 points over CT-DiagVLM, respectively, and a 7.11 point report-generation CE F1 improvement over RadSight-8B. Component ablations and qualitative failure analyses provide additional evidence about the proposed design.

Critical strengths
  1. A coherent integration of anatomical supervision across the model. The method connects region-constrained organ queries, lesion-aware feature pooling, disease-specific representation learning, and the language interface. In particular, transferring both continuous features and explicit predictions is a useful design: the former preserves information beyond a fixed vocabulary, while the latter exposes structured findings to the LLM. Tables 4–6 provide meaningful evidence about these components rather than relying solely on a final-model comparison.

  2. Broad evaluation with useful controls and transparent failure analysis. The experiments cover encoder classification, multiple VQA settings, spatial understanding, and report generation. The external Rad-ChestCT results support some cross-cohort transfer, while matching RadSight’s instruction-tuning data improves the relevance of the downstream comparison. The paper also acknowledges weaker performance on DeepChestVQA visual reasoning and reports segmentation failures and misleading disease prompts. This transparency strengthens the empirical contribution and helps identify where the approach succeeds or fails.

Critical weaknesses
  1. The benefit of the proposed lesion hierarchy is not fully separated from the benefit of additional lesion annotations. AnatoBridge introduces voxel-level supervision from 2,992 ReXGroundingCT training scans. In Table 4, the lesion-guided configuration combines this additional supervision with lesion-aware representations, without a clearly reported control that adds the same segmentation loss and annotations to the baseline while omitting the proposed hierarchy. Table 5 is helpful, but it does not specify whether the configurations without lesion fusion retain identical lesion supervision during pretraining. Consequently, the experiments do not fully establish how much improvement comes from the architectural contribution versus the additional dense labels. This distinction matters particularly because the overall encoder gains are modest, and uncertainty estimates are absent.

  2. The reliability of the explicit anatomical evidence is insufficiently validated. Predicted organ and lesion masks influence both feature aggregation and the spatial descriptions supplied to the LLM. However, the paper reports only four qualitative segmentation examples, without quantitative held-out segmentation, lesion-detection, or spatial-description accuracy. Improved downstream scores alone do not establish that these intermediate predictions are spatially correct. Table 12 provides valuable error analysis, but it also limits the stronger reliability interpretation: using its reported counts and accuracies, I calculate approximately 46% accuracy on positive existence questions, compared with approximately 40% for the volume-only baseline, despite an overall accuracy of 87.38%. Thus, positive detection improves, but the aggregate score conceals substantial remaining failures. Direct grounding metrics and class-specific evaluation are needed to substantiate the anatomical-evidence claims.the empirical contribution and helps identify where the approach succeeds or fails.

Questions for authors
  1. Do the “without lesion fusion” configurations in Table 5 retain the same lesion head, segmentation loss, annotated scans, and pretraining budget? How does AnatoBridge compare with a baseline receiving identical lesion supervision but without the proposed lesion-token hierarchy?

  2. What are the quantitative lesion-detection and segmentation results on held-out ReXGroundingCT annotations? Beyond the disease-score stratification in Table 12, how does downstream performance vary with the correctness of the spatial predictions?

  3. Do the internal/external AUC improvements and the smaller component-ablation gains remain reliable across training seeds or under patient-level paired bootstrap confidence intervals?

  4. What are the exact averaging conventions for F1 and precision in Table 1? If both are positive-class macro averages over the same diseases, precision of 46.3% implies an upper bound of approximately 63.3% on macro F1, below the reported 81.0%. Does F1 instead include class-weighted averaging over positive and negative labels? The same pattern appears in the cited baseline, so this requires clarification rather than establishing that the predictions are incorrect. Please specify how classification thresholds were selected.

Reviewer_SjgVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AnatoBridge, a framework for incorporating anatomical information into 3D chest CT vision–language models. The authors propose to first uses region-constrained organ queries to aggregate information across transformer layers. They then pool visual features using predicted lesion masks; this allows lesion information to enrich organ representations and support disease-specific contrastive learning. To transfer this information to the language model, the framework provides both continuous anatomy features and structured descriptions of predicted findings, organ geometry, and lesion locations. Experiments in classification, medical question answering, and report generation shows the advantage of the proposed method. Further ablations examine how the representations and anatomical prompts contribute to these gains.

Critical strengths
  1. Useful integration of anatomical information. I found the way the authors incorporated the anatomical information useful and well motivated to some extend.
  2. The paper is generally easy to follow and read.
Critical weaknesses
  1. Missing comparison with equal lesion supervision. The proposed AnatoBridge uses lesion annotations from 2,992 training scans, but the baseline does not receive equivalent supervision. Consequently, the ablations do not fully separate the benefit of additional labels from the proposed architecture. I think the authors should compare against a baseline using the same lesion labels through an auxiliary segmentation loss, with training and downstream tuning held fixed.
  2. Limited recovery evidence for incorrect prompts. The prompt-only model performs close to the full system, making the reliability of predicted anatomical descriptions especially important. Although the error analysis shows that incorrect findings can reduce performance, it does not establish whether image features can override these mistakes. The authors are encouraged to introduce controlled prompt errors while keeping the CT fixed and compare the full, feature-only, and prompt-only models.
  3. Missing uncertainty estimates for small gains. Several improvements are small, including the 0.7-point external AUC gain, but the paper reports only point estimates. Since multiple questions or examinations can come from the same scan or patient, uncertainty estimates should account for these dependencies. Normally I wouldn't budge about missing intervals but in this case I encourage the authors to provide paired patient level and/or scan level bootstrap intervals. More seeds are also encouraged to distinguish the incremental improvements.
Questions for authors

Please see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.