- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Spectrum to address a limitation of existing 3D CT vision–language pretraining: it primarily learns semantic correspondences between images and text, without adequately modeling differences in severity within the same disease. The authors use finding category labels, measurements, severity descriptions, and progression information from radiology reports to construct weak ordering relations across patients through a rule-based scorer. They propose Burden-Direction Alignment (BDA) to align the directions of change in disease burden between images and text in both global and organ-level representation spaces.
To reduce interference from differences across patients, the authors further introduce a target-conditioned alignment module and strengthen the ordering supervision through direction discrimination, endpoint consistency, and triplet composition constraints. The model first undergoes visual warm-up supervised by disease labels, after which the visual backbone is frozen for vision–language training. The auxiliary BDA modules are removed after training. Experiments on CT-RATE and RAD-ChestCT cover zero-shot disease recognition, linear probing, image and report retrieval, and ablations of different loss terms and alignment modules. The results show that the method improves the discriminative quality of downstream vision–language representations. However, direct validation of whether the final representations accurately encode disease burden within a given disease remains lacking.
Critical strengths
S1: The paper proposes an approach to learning disease burden across patients with clear medical relevance.
The paper extends the objective of medical vision–language learning from disease category recognition to modeling severity within the same disease, which has clear research value. Compared with approaches that derive supervision on changes from repeated follow-up scans of the same patient, this work attempts to directly use existing descriptions of size, count, and severity in cross-sectional reports to construct weak ordering relations across patients. This reduces reliance on longitudinal data and offers a new way to derive supervision from existing clinical reports.
In its method design, the paper combines disease-conditioned information with global and organ-level visual and language representations, explicitly incorporating relations in disease burden through alignment of image–text change directions, direction discrimination, and consistency constraints. Compared with conventional objectives that focus only on whether images and reports match, this direction has potential methodological novelty and provides a useful basis for further research on severity-aware medical representation learning.
S2: The downstream performance improvements are substantial and supported by clear internal ablations.
The authors report strong disease recognition results on both CT-RATE and RAD-ChestCT, as well as performance advantages in frozen linear probing, image retrieval, and report retrieval. Beyond comparisons with external baselines, the paper includes a correspondence-only control using the same architecture, indicating that target-conditioned information provides some benefit for downstream prediction. This supports BDA as an effective training objective and constitutes an important experimental contribution.
Critical weaknesses
W1: Direct, independent quantitative validation of within-disease burden information in the final representations is missing.
The central research objective is to learn orderings of disease burden within the same disease, but the main experiments currently focus on disease category recognition, image retrieval, and image–text matching. These tasks measure discriminative ability and the quality of cross-modal correspondence, but do not sufficiently establish that the final representations can correctly compare severity between two patients with the same disease. For example, a model may improve AUROC by learning stronger disease category features without reliably distinguishing severity differences within a disease.
The main text and supplementary materials provide directional consistency analyses for some patient triplets, but these results rely on ordering relations generated by report-based rules and lack independent clinical severity references. The supplementary materials also explicitly acknowledge that the displayed cases were selected and cannot be used to estimate the actual ordering accuracy. Since the auxiliary BDA modules are removed after training, directional differences in those modules cannot directly substitute for severity evaluation of the final frozen visual representations.
The current results therefore establish more clearly that BDA improves downstream task performance, while leaving the burden sensitivity emphasized in the title and main contributions without direct validation. This gap directly affects confidence in the paper's central scientific conclusion and is the most important concern in this review.
W2: The clinical meaning and reliability of the weak ordering relations remain unclear, and the need for the complex directional alignment mechanism is insufficiently validated.
The paper automatically generates cross-patient ordering relations using a rule-based scorer. However, the supplementary materials explicitly state that, once a target finding keyword is present, the scorer accumulates severity modifiers, counts, and measurements from the full report and the relevant organ text without strictly establishing semantic links between these cues and the target finding. Descriptions of other findings may therefore be incorrectly included in the burden score for the current disease. In addition, the scorer combines current lesion size with descriptions of progression relative to previous examinations, although current disease burden and longitudinal change are not identical clinical concepts. The paper does not sufficiently explain what clinical quantity the combined score represents, nor does it provide systematic manual audit results to assess the accuracy of the weak ordering relations.
BDA also incorporates target-conditioned alignment, direction discrimination, endpoint consistency, and triplet composition, but has not been directly compared with a simpler within-disease ranking objective using the same weak ordering relations and training budget. The cumulative ablations show that most of the performance gain comes from the initial introduction of feature-difference alignment, while subsequent constraints provide further improvements. The advantages of the complex directional alignment structure over simple ranking supervision therefore remain to be clarified.
Furthermore, the target-conditioned aligner is intended to reduce irrelevant differences across patients, such as body habitus, scanning protocols, and anatomy. However, the existing controls mainly report classification AUROC and do not directly verify whether these confounding factors have been effectively disentangled. The current evidence more clearly supports the module's ability to improve downstream prediction, while providing limited direct support for the proposed mechanism of removing irrelevant cross-patient variation.
Questions for authors
- Independent evaluation of the central claim about disease burden
Could the authors directly evaluate the ranking ability of the final frozen visual representations using disease burden references that are independent of the training scorer? For example, diseases with explicit size, volume, or clinical severity annotations could be selected, and Pairwise Ranking Accuracy, Spearman correlation, or Kendall's Tau could be reported. Since the auxiliary BDA modules are removed after training, please clarify whether the final retained representations, rather than only the directional variables used during training, actually encode disease burden information.
- Clinical meaning and reliability of the weak ordering scores
Does the defined burden score represent current disease extent, disease severity, or a combination that also includes progression relative to previous examinations? How do the current rules handle cases in which size, severity, and progression cues imply different orderings? Given that measurements and modifiers are not bound to the target finding, could the authors audit a random sample of the mined relations and report agreement with independent burden assessments, together with the main types of errors?
- The need for the complex directional alignment mechanism
Could the authors compare full BDA with a simple within-disease Pairwise Ranking Loss or Margin Ranking Loss using the same visual warm-up checkpoint, weak ordering relations, and training budget? This comparison should be distinguished from the ranking constraint between disease-positive and disease-negative examples used during warm-up. It would help determine whether the gains come from the additional within-disease severity supervision or from BDA's specific directional alignment design.
- The proposed mechanism of the target-conditioned aligner
Table 5 supports the classification benefits of the target-conditioned aligner. Do the authors have additional analyses showing that this module reduces non-target differences, such as body habitus and scanning protocols, while preserving disease burden information? If the available evidence mainly comes from classification ablations, please distinguish the observed performance gains from the interpretation of the module's mechanism in the main text.
4 comments
Response 2/2: BDA comparisons and aligner validation
W2 and Q3: Complete BDA versus within-disease ranking. Thank you for suggesting a comparison that examines the learning objective. Spectrum (Ours) achieves 85.6% CT-RATE AUROC, compared with 82.3% for within-disease ranking, a difference of 3.3 percentage points.
The ranking control replaces the entire BDA objective with a pairwise logistic loss that learns a pathology-specific scalar score from mined relations. Both methods use the same training and evaluation settings and the same mined within-pathology relations. It learns ordering among cases with the same pathology, distinct from the disease-positive versus disease-negative ranking used during warm-up.
Complete BDA aligns pathology-conditioned image differences with the corresponding report differences. The report difference supervises what varies between cases alongside which case has higher burden. Forward–reverse discrimination, endpoint consistency and triplet composition provide complementary constraints.
Submitted Table 4 reports the cumulative BDA loss-component ablations:
| BDA objective in Table 4 | CT-RATE AUROC (%) |
|---|---|
| Delta alignment | 83.9 |
| + Forward–reverse discrimination | 84.3 |
| + Endpoint consistency | 84.5 |
| + Triplet composition, Spectrum (Ours) | 85.6 |
The gain over within-disease ranking supports BDA’s use of report-described differences alongside burden order. Table 4 separately evaluates the complementary terms within BDA. The cumulative pattern supports difference alignment as the core objective and shows that the added constraints further improve diagnostic performance. The supplementary material now reports the ranking comparison, complementing the cumulative ablation in the main paper.
W2 and Q4: Target-conditioned alignment and its interpretation. Thank you for suggesting a direct analysis of non-target variation. We complement the diagnostic ablation with controlled CT perturbations.
The report-guided aligner transforms the source visual representation conditioned on the comparison target. Its visual difference is trained to match the pathology-conditioned report difference through . The report branch bypasses the aligner, providing a direct semantic reference. Forward–reverse discrimination, endpoint consistency and triplet composition add complementary constraints.
Related work on change captioning, such as SCORER [3], uses cross-view matching and representation reconstruction to learn stable semantic differences under viewpoint variation.
| Visual aligner | CT-RATE AUROC (%) |
|---|---|
| Identity | 84.2 |
| Target-blind learned | 84.5 |
| Target-conditioned, Spectrum (Ours) | 85.6 |
With complete BDA fixed, Table 5 shows gains of 1.1 percentage points over target-blind alignment and 1.4 over identity alignment, supporting a transformation specific to each comparison.
We evaluated controlled 3D scaling and resolution perturbations in LIDC-IDRI [1] with comparison scans matched by annotated nodule count. In the BDA projection space, the distance ratio is the mean distance from perturbed scans to their own originals divided by the mean distance to original scans from other patients with the same nodule count. Both comparisons used the same frozen model weights, projection and normalization.
| LIDC-IDRI perturbation | Distance ratio, direct subtraction | Distance ratio, target-conditioned alignment |
|---|---|---|
| 10% 3D downscaling | 0.341 | 0.281 |
| 10% 3D upscaling | 0.203 | 0.161 |
| 1.5× through-plane sampling spacing | 0.307 | 0.250 |
| 1.5× in-plane sampling spacing | 0.242 | 0.189 |
Target-conditioned alignment lowers this ratio by approximately 18–22% across all four conditions. A common multiplicative rescaling of all distances would leave the ratio unchanged. These results support greater relative stability of same-case representations under the tested image-space perturbations.
Together, Table 5 demonstrates the diagnostic benefit of target conditioning, and the perturbation analysis provides direct evidence of greater relative stability in the BDA projection space. The main paper now presents this analysis, and the supplementary material details the perturbation protocol and results.
References
[1] S. G. Armato III et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans. Medical Physics 38(2), 915–931 (2011).
[3] Tu Y et al. Self-supervised cross-view representation reconstruction for change captioning. ICCV, 2023: 2793–2803.
Thanks for the additional experiments. The nodule-count evaluation of the frozen representations, especially the comparison without BDA, addresses my main concern. I will raise my recommendation to weak accept.
Thank you for your follow-up and for taking the time to review our additional analyses. We sincerely appreciate your thoughtful feedback and careful consideration of our work.
Response 1/2: Independent evaluation of BDA-trained representations
Thank you for recognizing cross-patient burden learning. We address burden evaluation and relation validation here. Response 2/2 covers BDA and the aligner under W2 and Q3–Q4.
W1 and Q1: Burden information in the final frozen representations. LIDC-IDRI [1] provides radiologist-annotated nodule counts independent of our report-based training scorer. Held-out nodule-positive patients include 125 with 1–3 nodules and 48 with ≥4, counting nodules ≥3 mm. This assesses within-pathology burden beyond disease presence.
After BDA pretraining, we freeze the full image representation pathway. A linear probe fitted on count-derived training labels and selected on validation cases scores the global CT embeddings by the predicted probability of ≥4 nodules for held-out test evaluation.
We report pairwise ranking accuracy (PRA).
PRA exceeds 50% chance. Positive rank correlations support agreement with annotated counts, showing linearly accessible graded burden in frozen final CT representations learned through BDA.
W2 and Q2: Meaning of the burden score and relation validation. Our weak ordering score combines report-derived extent, multiplicity, measurements and progression relative to prior examinations. The supplementary material specifies the fixed additive weights that combine conflicting cues. After a pathology keyword matches, cues and measurements accumulate over the organ snippet and whole report.
Manual review of 50 randomly sampled cross-patient pairs found agreement with the mined direction in 42 pairs (84%). To reduce annotation effort at scale, Spectrum (Ours) learns from existing CT–report data and pathology labels without added burden-relation annotations or patient-matched longitudinal scans. Supplementary Table 2 and Figs. 1–5 give model-derived numerical direction scores and qualitative CT examples across nine pathologies, including cue-consistent pairwise orderings within noisy mined chains.
LIDC-IDRI validation against radiologist annotations. On 993 CT examinations, we rotated four reading sessions. One supplied a templated nodule description per examination to the unchanged miner. The median count from the other three supplied the reference. Ordering agreement uses selected pairs with positive, unequal reference counts.
At batch 24, agreement reaches 78.9% against annotations not used in the input description. The larger batch yields more evaluable pairs with similar agreement.
Disagreement was 22.3% (224/1006) at batch 4 and 21.1% (677/3205) at batch 24. The following percentages describe the distribution among disagreements in this LIDC-IDRI analysis.
BDA aligns pathology-conditioned image and report differences at whole-study and organ scopes using weak supervision. Forward–reverse, endpoint and composition constraints add complementary supervision.
We separately disabled progression scoring and replaced mining-time organ snippets with sentences selected by fixed pathology keywords. The latter retains whole-report scoring. Both retain relation mining, complete BDA and encoder text inputs while adding and excluding pairs and triplets used for burden learning.
Keyword-selected snippets approach the original CT-RATE AUROC. Both variants retain within-pathology burden ranking above chance on LIDC-IDRI.
With 20% text-endpoint swapping, fixed image order and independent composition-target reversal at the same rate, the model retains 64.9% LIDC-IDRI PRA versus 67.6% for Spectrum (Ours). Complete BDA and image–report correspondence learning are retained. PRA remains 14.9 percentage points above chance, supporting burden learning from weak supervision despite directional noise. These evaluations are now in the main paper and supplement. We will refine cue integration to reduce mining errors and strengthen burden learning.
References
[1] S. G. Armato III et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans. Medical Physics 38(2), 915–931 (2011).