- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a method for radiology report generation for 3D Chest CTs. To improve the performance, the authors enforce anatomical, lesion, and disease guidance through a specialized image encoder architecture with anatomy-, lesion-, and disease-aware pretraining. Furthermore, they also pass information about organ-properties, lesion-, and disease- findings via textual prompts to the vision-language model. They evaluate both the vision encoder (using linear probes) and the resulting vision-language model (on report generation and VQA).
Critical strengths
1. Analysis and Ablation Studies on both Vision Features and End-to-End Vision-Language Modeling
The authors analyze the performance of both their proposed vision encoder (using linear probes) and their resulting end-2-end vision-language model (on report generation and VQA).
Furthermore, they provide several ablation studies on most of their design decisions, thereby validating the overall benefits of their proposed contributions. (also see Weakness 2 for more discussion on the analyses)
Critical weaknesses
1. Limited discussion of contributions in comparison to prior works
While the authors discuss related work in Sec. 2, they do not always clearly specify how their contributions are related to these existing works (e.g. do they reuse ideas or build on some of them). Most notably, the contribution listing on p. 2 remains vague, making it hard to identify which contributions are new ideas, reuse existing ideas, or extend them. This makes it hard to judge the novelty and relevance of their contributions.
Specifically, the authors’ describe the following aspects of their method:
(a) Anatomy-aware encoding with organ-level tokens (anatomy queries) and organ-constrained attention
(b) Voxel-level anatomy- and lesion supervision (i.e. segmentation loss)
(c) Organ-level features enriched by associated lesion-tokens, which are computed by aggregating voxel tokens using lesion segmentation masks
(d) Constrative image-text alignment on the organ- and disease-level, where disease-level tokens are aggregated from the lesion tokens
(e) Textual prompts provided as context for the LLM including disease findings, organ geometry, and lesion grounding, derived from predictions of (b) and (d).
For (a), the authors describe that they extend CT-DiagVLM, claiming (in the abstract, introduction, and method section) that “anatomical guidance is weakly integrated into visual encoding”. In the method section, they write: “Such queries extract regional features only after encoding completes, leaving the visual encoding anatomically uninformed.” (p. 4, l. 171). I disagree with this statement, as these queries are based on information from the visual encoding (explicitly steered using organ masks and organ-level textual descriptions). Furthermore, the authors’ proposed contribution restricts attention between anatomy queries and does not allow vision tokens to attend to anatomy tokens. Therefore, the proposed architecture acts more like a multi-layer attention pooling per anatomy than an integrated processing of anatomy tokens. This makes it conceptually equal to CT-DiagVLM’s pooling but extended to a multi-layer approach. Note that the ablation study in Tab. 4 (second row) also demonstrates only minuscule benefits of their approach. Therefore, (a) remains a minor contribution.
For (b), the anatomy-segmentation is derived from CT-DiagVLM (which the authors also state). However, it is unclear whether lesion-level segmentation was already used by existing methods or is a newly proposed extension.
For (c): this seems to be a new contribution by the authors. However, it should be set in context with existing literature.
For (d): the disease-level tokens seem to be a new extension. But again, it should be set in context with existing literature.
Overall, the proposed contributions appear to be straightforward extensions to existing work. Authors should therefore clearly discuss (e.g., in their method or related work section) how these relate to existing literature and what exactly they contribute. Furthermore, the contributions should be described more fairly in the abstract and introduction (which currently slightly overstates them).
2. Limited Depth of Analysis
While the authors provide a set of ablation studies that validate the benefits of their contributions / design decisions (see strength 1), the analysis focuses mostly on “downstream performance”, i.e. classification, report generation, VQA. However, the authors introduce several specialized tokens and feature vectors (q_o, t_l, f_o, f_d), which are designed (via aggregation or supervision) to capture specific information about the image, e.g. anatomy-, lesion-, or disease-specific features. As these tokens/vectors are central parts of the authors’ contributions and hypotheses, the authors should validate that these vectors actually capture the expected information. For example,
- Linear probes could be used to verify that organ-specific vectors contain more information about as specific organ than the other organ vectors, e.g. by comparing distances to organ-level report sentences or comparing classification performance on organ-specific lesions/diseases. Similarly for lesion- and disease-specific vectors
- and/or some organ- or lesion-specific vectors (e.g., q_o or t_l) could be masked before computing F_organ, and the resulting generated reports could be studied. Alternatively, these vectors could be replaced by vectors from other patients with differing diagnoses. If Masking/modifying these leads to reports with missing or misreported information about the specific lesion/organ, then the relevance of those tokens would be validated.
In your rebuttal, please carefully consider which analyses could validate your hypotheses (you may or may not use the examples above).
Questions for authors
- How is zero-shot classification evaluated, i.e., which tokens are used and how are they aggregated?