acceptodds
Under review as a conference paper at ICLR 2027

Learning How Much, Not Just What: Cross-Patient Burden Order for CT Vision-Language Pretraining

Abstract

Volumetric CT vision–language pretraining learns 3D representations from scan–report pairs, but global and anatomy-aware objectives supervise only correspondence: they establish what is present and leave how much unconstrained. Nothing separates a mild from an extensive case of the same finding along a consistent direction, so the graded burden language in reports collapses into a present/absent signal. Longitudinal supervision would supply this order, but patient-matched CT pairs are scarce at scale; cross-sectional cohorts already encode weak burden cues across different patients. We introduce Spectrum, an anatomy-conditioned framework that represents each study at whole-study and organ scopes. For each organ-mapped pathology, a rule-based scorer mines confidence-filtered lower-to-higher pairs of different patients, and Burden-Direction Alignment (BDA) aligns the pathology-conditioned image delta with the report delta at each scope, separating that direction from its reverse. Because the endpoints are different people, a target-conditioned aligner first makes them comparable, so the delta reflects burden rather than between-patient variation. BDA further separates the selected direction from its reverse, anchors it to the observed higher-burden endpoint, and enforces consistency across ordered triplets. Since every pair is drawn within a single pathology, BDA is designed to constrain intra-class structure that image–report contrast alone never touches. Spectrum attains 85.6 zero-shot AUROC on CT-RATE and 72.7 on external RAD-ChestCT, with consistent gains in linear probing and retrieval. Weak cross-patient order is thus a scalable complement to anatomy-aware correspondence, yielding burden-aware CT representations without longitudinal data.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Medical Vision-Language Learning, 3D CT Representation Learning, Vision-Language Pretraining, Radiology

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_e9dYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Spectrum to address a limitation of existing 3D CT vision–language pretraining: it primarily learns semantic correspondences between images and text, without adequately modeling differences in severity within the same disease. The authors use finding category labels, measurements, severity descriptions, and progression information from radiology reports to construct weak ordering relations across patients through a rule-based scorer. They propose Burden-Direction Alignment (BDA) to align the directions of change in disease burden between images and text in both global and organ-level representation spaces.

To reduce interference from differences across patients, the authors further introduce a target-conditioned alignment module and strengthen the ordering supervision through direction discrimination, endpoint consistency, and triplet composition constraints. The model first undergoes visual warm-up supervised by disease labels, after which the visual backbone is frozen for vision–language training. The auxiliary BDA modules are removed after training. Experiments on CT-RATE and RAD-ChestCT cover zero-shot disease recognition, linear probing, image and report retrieval, and ablations of different loss terms and alignment modules. The results show that the method improves the discriminative quality of downstream vision–language representations. However, direct validation of whether the final representations accurately encode disease burden within a given disease remains lacking.

Critical strengths

S1: The paper proposes an approach to learning disease burden across patients with clear medical relevance.

The paper extends the objective of medical vision–language learning from disease category recognition to modeling severity within the same disease, which has clear research value. Compared with approaches that derive supervision on changes from repeated follow-up scans of the same patient, this work attempts to directly use existing descriptions of size, count, and severity in cross-sectional reports to construct weak ordering relations across patients. This reduces reliance on longitudinal data and offers a new way to derive supervision from existing clinical reports.

In its method design, the paper combines disease-conditioned information with global and organ-level visual and language representations, explicitly incorporating relations in disease burden through alignment of image–text change directions, direction discrimination, and consistency constraints. Compared with conventional objectives that focus only on whether images and reports match, this direction has potential methodological novelty and provides a useful basis for further research on severity-aware medical representation learning.

S2: The downstream performance improvements are substantial and supported by clear internal ablations.

The authors report strong disease recognition results on both CT-RATE and RAD-ChestCT, as well as performance advantages in frozen linear probing, image retrieval, and report retrieval. Beyond comparisons with external baselines, the paper includes a correspondence-only control using the same architecture, indicating that target-conditioned information provides some benefit for downstream prediction. This supports BDA as an effective training objective and constitutes an important experimental contribution.

Critical weaknesses

W1: Direct, independent quantitative validation of within-disease burden information in the final representations is missing.

The central research objective is to learn orderings of disease burden within the same disease, but the main experiments currently focus on disease category recognition, image retrieval, and image–text matching. These tasks measure discriminative ability and the quality of cross-modal correspondence, but do not sufficiently establish that the final representations can correctly compare severity between two patients with the same disease. For example, a model may improve AUROC by learning stronger disease category features without reliably distinguishing severity differences within a disease.

The main text and supplementary materials provide directional consistency analyses for some patient triplets, but these results rely on ordering relations generated by report-based rules and lack independent clinical severity references. The supplementary materials also explicitly acknowledge that the displayed cases were selected and cannot be used to estimate the actual ordering accuracy. Since the auxiliary BDA modules are removed after training, directional differences in those modules cannot directly substitute for severity evaluation of the final frozen visual representations.

The current results therefore establish more clearly that BDA improves downstream task performance, while leaving the burden sensitivity emphasized in the title and main contributions without direct validation. This gap directly affects confidence in the paper's central scientific conclusion and is the most important concern in this review.

W2: The clinical meaning and reliability of the weak ordering relations remain unclear, and the need for the complex directional alignment mechanism is insufficiently validated.

The paper automatically generates cross-patient ordering relations using a rule-based scorer. However, the supplementary materials explicitly state that, once a target finding keyword is present, the scorer accumulates severity modifiers, counts, and measurements from the full report and the relevant organ text without strictly establishing semantic links between these cues and the target finding. Descriptions of other findings may therefore be incorrectly included in the burden score for the current disease. In addition, the scorer combines current lesion size with descriptions of progression relative to previous examinations, although current disease burden and longitudinal change are not identical clinical concepts. The paper does not sufficiently explain what clinical quantity the combined score represents, nor does it provide systematic manual audit results to assess the accuracy of the weak ordering relations.

BDA also incorporates target-conditioned alignment, direction discrimination, endpoint consistency, and triplet composition, but has not been directly compared with a simpler within-disease ranking objective using the same weak ordering relations and training budget. The cumulative ablations show that most of the performance gain comes from the initial introduction of feature-difference alignment, while subsequent constraints provide further improvements. The advantages of the complex directional alignment structure over simple ranking supervision therefore remain to be clarified.

Furthermore, the target-conditioned aligner is intended to reduce irrelevant differences across patients, such as body habitus, scanning protocols, and anatomy. However, the existing controls mainly report classification AUROC and do not directly verify whether these confounding factors have been effectively disentangled. The current evidence more clearly supports the module's ability to improve downstream prediction, while providing limited direct support for the proposed mechanism of removing irrelevant cross-patient variation.

Questions for authors
  1. Independent evaluation of the central claim about disease burden

Could the authors directly evaluate the ranking ability of the final frozen visual representations using disease burden references that are independent of the training scorer? For example, diseases with explicit size, volume, or clinical severity annotations could be selected, and Pairwise Ranking Accuracy, Spearman correlation, or Kendall's Tau could be reported. Since the auxiliary BDA modules are removed after training, please clarify whether the final retained representations, rather than only the directional variables used during training, actually encode disease burden information.

  1. Clinical meaning and reliability of the weak ordering scores

Does the defined burden score represent current disease extent, disease severity, or a combination that also includes progression relative to previous examinations? How do the current rules handle cases in which size, severity, and progression cues imply different orderings? Given that measurements and modifiers are not bound to the target finding, could the authors audit a random sample of the mined relations and report agreement with independent burden assessments, together with the main types of errors?

  1. The need for the complex directional alignment mechanism

Could the authors compare full BDA with a simple within-disease Pairwise Ranking Loss or Margin Ranking Loss using the same visual warm-up checkpoint, weak ordering relations, and training budget? This comparison should be distinguished from the ranking constraint between disease-positive and disease-negative examples used during warm-up. It would help determine whether the gains come from the additional within-disease severity supervision or from BDA's specific directional alignment design.

  1. The proposed mechanism of the target-conditioned aligner

Table 5 supports the classification benefits of the target-conditioned aligner. Do the authors have additional analyses showing that this module reduces non-target differences, such as body habitus and scanning protocols, while preserving disease burden information? If the available evidence mainly comes from classification ablations, please distinguish the observed performance gains from the interpretation of the module's mechanism in the main text.

4 comments
AuthorsOpenReview ↗

Response 1/2: Independent evaluation of BDA-trained representations

Thank you for recognizing cross-patient burden learning. We address burden evaluation and relation validation here. Response 2/2 covers BDA and the aligner under W2 and Q3–Q4.

W1 and Q1: Burden information in the final frozen representations. LIDC-IDRI [1] provides radiologist-annotated nodule counts independent of our report-based training scorer. Held-out nodule-positive patients include 125 with 1–3 nodules and 48 with ≥4, counting nodules ≥3 mm. This assesses within-pathology burden beyond disease presence.

After BDA pretraining, we freeze the full image representation pathway. A linear probe fitted on count-derived training labels and selected on validation cases scores the global CT embeddings by the predicted probability of ≥4 nodules for held-out test evaluation.

We report pairwise ranking accuracy (PRA).

LIDC-IDRI metricSpectrum (Ours)
AUROC (%)71.4
PRA (%)67.6
Spearman ρ0.414
Kendall τ-b0.313

PRA exceeds 50% chance. Positive rank correlations support agreement with annotated counts, showing linearly accessible graded burden in frozen final CT representations learned through BDA.

W2 and Q2: Meaning of the burden score and relation validation. Our weak ordering score combines report-derived extent, multiplicity, measurements and progression relative to prior examinations. The supplementary material specifies the fixed additive weights that combine conflicting cues. After a pathology keyword matches, cues and measurements accumulate over the organ snippet and whole report.

Manual review of 50 randomly sampled cross-patient pairs found agreement with the mined direction in 42 pairs (84%). To reduce annotation effort at scale, Spectrum (Ours) learns from existing CT–report data and pathology labels without added burden-relation annotations or patient-matched longitudinal scans. Supplementary Table 2 and Figs. 1–5 give model-derived numerical direction scores and qualitative CT examples across nine pathologies, including cue-consistent pairwise orderings within noisy mined chains.

LIDC-IDRI validation against radiologist annotations. On 993 CT examinations, we rotated four reading sessions. One supplied a templated nodule description per examination to the unchanged miner. The median count from the other three supplied the reference. Ordering agreement uses selected pairs with positive, unequal reference counts.

Batch sizeCorrect / evaluable pairsOrdering agreement (%)
4782 / 100677.7
242528 / 320578.9

At batch 24, agreement reaches 78.9% against annotations not used in the input description. The larger batch yields more evaluable pairs with similar agreement.

Disagreement was 22.3% (224/1006) at batch 4 and 21.1% (677/3205) at batch 24. The following percentages describe the distribution among disagreements in this LIDC-IDRI analysis.

LIDC-IDRI disagreement categoryBatch 4 (%)Batch 24 (%)
Size–count discordance57.671.2
Size contributions outweighing multiplicity cues20.54.4
Input-reading versus held-out-reader count ordering21.924.4

BDA aligns pathology-conditioned image and report differences at whole-study and organ scopes using weak supervision. Forward–reverse, endpoint and composition constraints add complementary supervision.

We separately disabled progression scoring and replaced mining-time organ snippets with sentences selected by fixed pathology keywords. The latter retains whole-report scoring. Both retain relation mining, complete BDA and encoder text inputs while adding and excluding pairs and triplets used for burden learning.

Model or scoring variantCT-RATE AUROC (%)LIDC-IDRI PRA (%)CT-RATE triplet-order retention (%)
Spectrum (Ours)85.667.6—
No progression scoring84.466.455.3
Deterministic mining snippets85.165.697.0

Keyword-selected snippets approach the original CT-RATE AUROC. Both variants retain within-pathology burden ranking above chance on LIDC-IDRI.

With 20% text-endpoint swapping, fixed image order and independent composition-target reversal at the same rate, the model retains 64.9% LIDC-IDRI PRA versus 67.6% for Spectrum (Ours). Complete BDA and image–report correspondence learning are retained. PRA remains 14.9 percentage points above chance, supporting burden learning from weak supervision despite directional noise. These evaluations are now in the main paper and supplement. We will refine cue integration to reduce mining errors and strengthen burden learning.

References

[1] S. G. Armato III et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans. Medical Physics 38(2), 915–931 (2011).

AuthorsOpenReview ↗

Response 2/2: BDA comparisons and aligner validation

W2 and Q3: Complete BDA versus within-disease ranking. Thank you for suggesting a comparison that examines the learning objective. Spectrum (Ours) achieves 85.6% CT-RATE AUROC, compared with 82.3% for within-disease ranking, a difference of 3.3 percentage points.

The ranking control replaces the entire BDA objective with a pairwise logistic loss that learns a pathology-specific scalar score from mined relations. Both methods use the same training and evaluation settings and the same mined within-pathology relations. It learns ordering among cases with the same pathology, distinct from the disease-positive versus disease-negative ranking used during warm-up.

Complete BDA aligns pathology-conditioned image differences with the corresponding report differences. The report difference supervises what varies between cases alongside which case has higher burden. Forward–reverse discrimination, endpoint consistency and triplet composition provide complementary constraints.

Submitted Table 4 reports the cumulative BDA loss-component ablations:

BDA objective in Table 4CT-RATE AUROC (%)
Delta alignment83.9
+ Forward–reverse discrimination84.3
+ Endpoint consistency84.5
+ Triplet composition, Spectrum (Ours)85.6

The gain over within-disease ranking supports BDA’s use of report-described differences alongside burden order. Table 4 separately evaluates the complementary terms within BDA. The cumulative pattern supports difference alignment as the core objective and shows that the added constraints further improve diagnostic performance. The supplementary material now reports the ranking comparison, complementing the cumulative ablation in the main paper.

W2 and Q4: Target-conditioned alignment and its interpretation. Thank you for suggesting a direct analysis of non-target variation. We complement the diagnostic ablation with controlled CT perturbations.

The report-guided aligner transforms the source visual representation conditioned on the comparison target. Its visual difference is trained to match the pathology-conditioned report difference through LΔL_{\Delta}. The report branch bypasses the aligner, providing a direct semantic reference. Forward–reverse discrimination, endpoint consistency and triplet composition add complementary constraints.

Related work on change captioning, such as SCORER [3], uses cross-view matching and representation reconstruction to learn stable semantic differences under viewpoint variation.

Visual alignerCT-RATE AUROC (%)
Identity84.2
Target-blind learned84.5
Target-conditioned, Spectrum (Ours)85.6

With complete BDA fixed, Table 5 shows gains of 1.1 percentage points over target-blind alignment and 1.4 over identity alignment, supporting a transformation specific to each comparison.

We evaluated controlled 3D scaling and resolution perturbations in LIDC-IDRI [1] with comparison scans matched by annotated nodule count. In the BDA projection space, the distance ratio is the mean distance from perturbed scans to their own originals divided by the mean distance to original scans from other patients with the same nodule count. Both comparisons used the same frozen model weights, projection and normalization.

LIDC-IDRI perturbationDistance ratio, direct subtractionDistance ratio, target-conditioned alignment
10% 3D downscaling0.3410.281
10% 3D upscaling0.2030.161
1.5× through-plane sampling spacing0.3070.250
1.5× in-plane sampling spacing0.2420.189

Target-conditioned alignment lowers this ratio by approximately 18–22% across all four conditions. A common multiplicative rescaling of all distances would leave the ratio unchanged. These results support greater relative stability of same-case representations under the tested image-space perturbations.

Together, Table 5 demonstrates the diagnostic benefit of target conditioning, and the perturbation analysis provides direct evidence of greater relative stability in the BDA projection space. The main paper now presents this analysis, and the supplementary material details the perturbation protocol and results.

References

[1] S. G. Armato III et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans. Medical Physics 38(2), 915–931 (2011).

[3] Tu Y et al. Self-supervised cross-view representation reconstruction for change captioning. ICCV, 2023: 2793–2803.

Thanks for the additional experiments. The nodule-count evaluation of the frozen representations, especially the comparison without BDA, addresses my main concern. I will raise my recommendation to weak accept.

Thank you for your follow-up and for taking the time to review our additional analyses. We sincerely appreciate your thoughtful feedback and careful consideration of our work.

Reviewer_84kcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Spectrum, an anatomy-conditioned framework for CT vision-language pretraining. It introduces Burden-Direction Alignment (BDA) to learn pathology severity ordering from cross-patient image-report pairs. Experiments on CT-RATE and RAD-ChestCT show improvements in classification, linear probing, and retrieval.

Critical strengths
  • Interesting Idea: Learning pathology burden ordering beyond image-text correspondence is a meaningful direction for medical VLMs.
  • Strong Results: The method achieves substantial improvements across different evaluation tasks and two CT datasets.
  • Detailed Ablations: The paper includes component ablations and aligner comparisons to support the proposed design.
Critical weaknesses
  • Limited Burden Evaluation: The main experiments focus on abnormality classification and retrieval rather than severity estimation. Only one qualitative example is provided for burden ordering (Figure 5). More quantitative evaluation is needed to support the main contribution.
  • Unclear Zero-Shot Setting: The visual encoder is pretrained with pathology label supervision (Section 3.4), including labels used in downstream evaluation. This raises concerns about the zero-shot comparison with existing methods.
  • Baseline Comparison: Most baseline results are taken from published papers with different training settings and supervision. More controlled comparisons are needed to establish the benefits of the proposed method.
  • Reliability of Burden Ordering: The method relies on rule-based scoring of radiology reports to construct cross-patient pairs. The accuracy and reliability of these automatically generated severity relationships are not quantitatively evaluated.
Questions for authors
  1. Can the authors provide a quantitative evaluation of pathology severity prediction or burden ranking?
  2. How does label-supervised pretraining affect the zero-shot comparison with existing methods?
  3. Can the authors provide comparisons under matched training data and supervision?
  4. How accurate are the automatically mined burden relationships across different pathologies?

I am willing to adjust my score according to the rebuttal.

7 comments
AuthorsOpenReview ↗

Response 1/2: Quantitative burden evaluation and relation validation

Thank you for the constructive questions. We address burden evaluation and relation validation here, followed by pretraining and comparisons in Response 2/2. W1–W4 and Q1–Q4 follow your review order.

W1 and Q1: Does the retained representation encode burden beyond disease presence? We selected LIDC-IDRI [1] for radiologist-annotated nodule counts independent of our training scorer. These assess within-pathology burden, while CT-RATE assesses diagnosis using disease-presence labels.

All evaluated patients have nodules ≥3 mm: 125 have 1–3 and 48 have at least 4, assessing burden variation within the same pathology.

After BDA pretraining, we freeze the full image representation pathway. A linear probe fitted on count-derived training labels and selected on validation cases scores the global CT embeddings by the probability of at least four nodules for held-out testing.

We report pairwise ranking accuracy (PRA).

LIDC-IDRI metricSpectrum (Ours)
AUROC (%)71.4
PRA (%)67.6
Spearman ρ0.414
Kendall τ-b0.313

AUROC measures count-group separation. PRA exceeds 50% chance, and positive Spearman and Kendall correlations support agreement with annotated counts. These results show that nodule burden is linearly accessible in the frozen representations learned with BDA.

W4 and Q4: Validation of automatically mined relations. In a manual review of 50 randomly sampled cross-patient pairs, the assessed burden ordering agreed with the mined direction in 42 pairs (84%). To reduce annotation effort at scale, Spectrum (Ours) learns from existing CT–report data and pathology labels without additional burden-relation annotations or patient-matched longitudinal scans.

LIDC-IDRI validation against radiologist annotations. We assessed the miner on 993 LIDC-IDRI CT examinations. One reading session per examination was converted into a templated nodule description for the unchanged miner. The median nodule count from the other three sessions served as the reference. We rotated the input across all four sessions. Ordering agreement was evaluated among selected pairs with positive, unequal reference counts.

Batch sizeCorrect / evaluable pairsOrdering agreement (%)
4782 / 100677.7
242528 / 320578.9

Agreement reaches 78.9% at batch size 24 against annotations excluded from the input. The larger batch provides more evaluable pairs with similar agreement.

Analysis of LIDC-IDRI ordering disagreements. Disagreement rates were 22.3% (224/1006) at batch size 4 and 21.1% (677/3205) at batch size 24. Each category below is expressed as a percentage of disagreements in that batch setting.

LIDC-IDRI disagreement categoryBatch 4 (%)Batch 24 (%)
Size–count discordance57.671.2
Size contributions outweighing multiplicity cues20.54.4
Input-reading versus held-out-reader count ordering21.924.4

Size–count discordance was the largest category at both batch sizes. BDA uses report-derived relations as weak supervision for image–report difference alignment. The separate perturbation experiment below supports learning useful burden representations despite directional noise. We will investigate better integration of size and multiplicity cues to reduce mining errors and further strengthen burden learning.

Supplementary Table 2 and Figs. 1–5 provide complementary numerical model-derived direction scores and qualitative CT examples across nine pathologies, including learned pairwise orderings that follow the displayed burden cues within noisy mined chains.

BDA aligns pathology-conditioned image and report differences at whole-study and organ scopes. Report differences supervise what varies between cases alongside their burden order, with forward–reverse discrimination, endpoint consistency and triplet composition as complementary constraints.

We also trained a model with a 20% probability of swapping text endpoints while retaining image order. Composition targets were independently reversed with the same probability. Complete BDA and global and organ-level image–report correspondence learning were retained.

Training configurationLIDC-IDRI PRA (%)
Spectrum (Ours)67.6
20% text-direction corruption64.9

The perturbed model remains 14.9 percentage points above chance, supporting burden learning under the tested directional noise. Together, validation and perturbation results support report-derived weak supervision for burden learning. These results are now in the main paper and supplement.

References

[1] S. G. Armato III et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans. Medical Physics 38(2), 915–931 (2011).

AuthorsOpenReview ↗

Response 2/2: Pretraining supervision and model comparisons

W2 and Q2: Pretraining and zero-shot evaluation. Thank you for asking how pretraining supervision affects the comparison. We trained CT-CLIP and Merlin [2] with the same training data and warm-up strategy and evaluated them under the same CT-RATE protocol as Spectrum (Ours).

ModelWarm-upCT-RATE AUROC (%)
CT-CLIP✓72.8
Merlin✓72.9
Spectrum (Ours)✓85.6

These results show that Spectrum (Ours) retains its diagnostic advantage when the compared baselines also receive pathology-label pretraining. BDA aligns pathology-conditioned image and report differences among patients sharing the same pathology, alongside image–report correspondence learning. Its supervision therefore includes the semantic content of burden variation within a disease category.

Here, zero-shot denotes prompt-based diagnosis without downstream task-specific fine-tuning: no downstream task-specific classifier is trained and no model parameter is updated during evaluation. Predictions use the final global CT embedding and fixed pathology prompts. This usage is consistent with Merlin [2], which evaluates zero-shot findings classification and cross-modal retrieval without model adaptation after pretraining with diagnosis-code labels and radiology reports.

W3 and Q3: External comparisons and the BDA objective. The external comparisons above use matched training data, warm-up strategy and CT-RATE evaluation settings. We also evaluated within-disease ranking to examine how the mined relations are used for learning.

MethodCT-RATE AUROC (%)Δ AUROC (percentage points)
Spectrum (Ours)85.6—
Within-disease ranking82.3+3.3

Δ AUROC = Spectrum (Ours) − control on CT-RATE.

The ranking control replaces the entire BDA objective with a pairwise logistic loss that learns a pathology-specific scalar score from mined relations. Both methods use the same training and evaluation settings and the same mined within-pathology relations. A scalar ranking objective supervises which case has higher burden. Complete BDA uses the corresponding report difference as a semantic reference for the visual difference between those cases. Thus, learning is guided by the report-described content of the difference as well as its direction. Forward–reverse discrimination, endpoint consistency and triplet composition add complementary constraints.

The retained correspondence objective continues to train whole-study and organ-level image–report matching. The ranking comparison examines how burden relations provide additional supervision.

Spectrum (Ours) exceeds this ranking control by 3.3 percentage points. The external baseline results show its diagnostic advantage with added label-based pretraining for the compared methods. The internal ranking result supports learning the semantic content of cross-patient differences through BDA. Both comparisons are now reported in the supplementary material. The supplementary material details them and the pretraining and prompt-based evaluation protocol.

We appreciate your questions, which help us present the contribution and its supporting evidence more clearly.

References

[2] L. Blankemeier, A. Kumar, J. P. Cohen et al. Merlin: a computed tomography vision–language foundation model and dataset. Nature 652, 1318–1328 (2026).

Thanks for the rebuttal. W4 is addressed and the ranking baseline in W3 is helpful, though a few issues still remain:

  • The LIDC-IDRI table only compares Spectrum against 50% random chance without any baselines. Without Spectrum w/o BDA and external baselines, this still does not show that severity awareness comes from BDA.

  • Results without label supervision are still missing, and it is unclear whether the mined pairs only include positive cases.

  • All the new controlled comparisons are only reported on CT-RATE, whose labels were already used during training. External results on RAD-ChestCT are still missing for these baselines.

Reviewer_zzp8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Spectrum, a CT vision–language pretraining framework that incorporates weak cross-patient ordering of pathology burden. It mines lower-to-higher burden relations from abnormality labels and report descriptions, then aligns pathology-conditioned image and text differences at global and organ-specific scopes. Experiments are performed on CT-RATE and RAD-ChestCT with reasonable performance improvements.

Critical strengths
  • The observation is interesting and the idea is well motivated. Moving beyond image–report correspondence to within-pathology burden relationships could help further exploit clinically meaningful information.
  • The paper is generally well organized. The overall framework and the motivation behind its main components are easy to follow.
  • The approach has potential for scaling up at relatively low annotation cost, requiring no additional fine-grained annotations.
  • The experimental results support the method design.
Critical weaknesses
  • The coverage and selection bias of burden supervision need further elaboration. What proportion of studies cannot be assigned reliable burden evidence, and how do these studies contribute to training?
  • The operational definition and robustness of burden require further justification. The scorer combines several cues, but these signals may not always imply a consistent ordering. The paper would benefit from independent validation of the resulting rankings and sensitivity analyses across scoring rules and report formulations. What if LLM-based processing is used to construct organ-specific snippets, its influence on the mined relations should also be assessed.
  • Evaluation stratified by burden would better connect the results to the motivation.
  • The justification for the visual aligner remains incomplete. Although its ablation supports its empirical usefulness, the architecture does not guarantee that it modifies only nuisance factors.
  • Global Top-K mining may introduce pathology-dependent sampling bias, differences in scoring ranges or pathology prevalence could cause certain diseases to dominate relation selection.
  • The downstream results demonstrate that the framework is useful, but do not establish that its improvements depend on accurate burden ordering. The paper needs a more detailed assessment of mined-order accuracy, tolerance to ordering noise, and the relationship between ordering quality and downstream performance.
Questions for authors
  • With architecture, training samples, and computational budget held constant, how do random relations, shuffled burden cues, and direct pairwise ranking perform? How much additional benefit specifically comes from meaningful burden ordering?
  • Do all baselines use consistent settings? Are the reported results averages over multiple independent runs, it might be better to provide confidence intervals or standard deviations.
  • What is the agreement between mined relations and independent expert judgments? How sensitive is performance to incorrect or ambiguous orderings?
3 comments
AuthorsOpenReview ↗

Response 1/3: Independent burden evaluation and relation validation

Thank you for the thoughtful review and for recognizing the motivation of learning pathology burden. W1–W6 and Q1–Q3 follow the order of your Weaknesses and Questions for Authors.

W3: Independent evaluation of burden. We selected LIDC-IDRI [1] because its radiologist-annotated nodule counts provide a burden reference independent of our report-based training scorer. We evaluate nodule-positive patients, including 125 with 1–3 nodules and 48 with at least 4, counting nodules measuring at least 3 mm. This tests variation within a pathology beyond disease presence.

After BDA pretraining, we freeze the full image representation pathway. A linear probe fitted on count-derived training labels and selected on validation cases scores the resulting global CT embeddings by the predicted probability of at least four nodules for held-out test evaluation.

We report pairwise ranking accuracy (PRA).

LIDC-IDRI metricSpectrum (Ours)
AUROC (%)71.4
PRA (%)67.6
Spearman ρ0.414
Kendall τ-b0.313

PRA exceeds 50% chance, and positive rank correlations support agreement with annotated counts. Graded burden information is linearly accessible in the final representations learned with BDA.

W2, W6 and Q3: Mined relations and directional noise. In a manual review of 50 randomly sampled cross-patient pairs, the assessed burden ordering agreed with the mined direction in 42 pairs (84%). To reduce annotation effort at scale, Spectrum (Ours) learns from existing CT–report data and pathology labels without additional burden-relation annotations or patient-matched longitudinal scans. Supplementary Table 2 and Figs. 1–5 provide complementary numerical model-derived direction scores and qualitative CT examples across nine pathologies, including cue-consistent pairwise orderings within noisy mined chains.

LIDC-IDRI validation against radiologist annotations. We assessed the miner on 993 LIDC-IDRI CT examinations. One reading session per examination was converted into a templated nodule description for the unchanged miner. The median nodule count from the other three sessions served as the reference. We rotated the input across all four sessions. Ordering agreement was evaluated among selected pairs with positive, unequal reference counts.

Batch sizeCorrect / evaluable pairsOrdering agreement (%)
4782 / 100677.7
242528 / 320578.9

At batch size 24, ordering agreement reaches 78.9% against radiologist annotations not used to construct the input description. The larger batch yields more evaluable pairs while preserving similar agreement.

BDA aligns pathology-conditioned image and report differences at whole-study and organ scopes. Report differences supervise what varies between cases alongside their burden order. Forward–reverse discrimination, endpoint consistency and triplet composition add complementary constraints.

To test sensitivity, we trained with a 20% probability of swapping text endpoints while retaining image order. Composition targets were independently reversed with the same probability. Complete BDA and global and organ-level image–report correspondence learning were retained.

Training configurationLIDC-IDRI PRA (%)
Spectrum (Ours)67.6
20% text-direction corruption64.9

The perturbed model remains 14.9 percentage points above 50% chance, demonstrating retained burden discrimination under the tested directional noise.

W2: Scoring rules and report formulations. Fixed additive rules combine extent, multiplicity, measurements and progression as within-pathology weak supervision. The supplementary material provides the fixed cue weights and score calculation. We separately disabled progression-word scoring and replaced mining-time organ snippets with report sentences selected by fixed pathology keywords. The latter retains the full report as an additional scoring input. Both retain the complete BDA objective and the original encoder text inputs. Both change pair and triplet selection, adding and excluding relations used for burden learning.

Model or scoring variantCT-RATE AUROC (%)LIDC-IDRI PRA (%)CT-RATE triplet-order retention (%)
Spectrum (Ours)85.667.6—
No progression scoring84.466.455.3
Deterministic mining snippets85.165.697.0

Keyword-selected snippets approach the original CT-RATE AUROC. Both variants retain within-pathology burden ranking above chance on LIDC-IDRI.

These evaluations and mining details are now in the main paper and supplement.

References

[1] S. G. Armato III et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans. Medical Physics 38(2), 915–931 (2011).

AuthorsOpenReview ↗

Response 2/3: Supervision coverage and pathology-balanced mining

Thank you for asking how relation selection affects study participation and pathology representation.

W1: Coverage and contribution of training studies. All training studies contribute to global and organ-level image–report correspondence where the corresponding organ input is available. BDA adds pathology-specific relation supervision when report evidence passes the mining criteria. The confidence score, endpoint evidence score and between-endpoint score margin must each be at least 0.75.

CT-RATE training coverageStudies (%)
Used in global BDA relations98.7
No global BDA relation used1.3

These percentages measure participation in filtered relations. Relation filtering controls the additional BDA supervision, while studies without selected relations continue image–report learning.

W5: Pathology representation under Global Top-K. The original miner ranks eligible relations jointly across pathologies. Score distributions and the number of eligible candidates therefore affect each pathology’s share. We measured the selected distribution during CT-RATE training and evaluated an alternative that interleaves eligible relations across pathologies.

CT-RATE training pathologyTop-K pair share (%)Balanced pair share (%)
Pleural effusion38.19.5
Lung nodule29.89.9
Pericardial effusion14.79.8

These values describe shares of selected training pairs. Combined shares are calculated before rounding individual entries. Pathology-balanced mining reduces the combined share of the three dominant pathologies from 82.5% to 29.3%.

Model configurationCT-RATE AUROC (%)
Spectrum (Ours)85.6
Balanced mining85.4

The more evenly distributed relation pool preserves diagnostic performance close to the original model. This supports the effectiveness of BDA under different sampling distributions. Balancing operates over eligible relations, while all studies continue to contribute to image–report correspondence learning. The supplementary material now includes the coverage and balanced-mining analyses.

AuthorsOpenReview ↗

Response 3/3: Aligner evidence, BDA controls and reproducibility

W4: Role of the visual aligner. Thank you for distinguishing the aligner’s empirical benefit from its mechanism. The aligner is a task-guided, target-conditioned transformation for burden-relevant comparison. It transforms the source visual representation conditioned on the comparison target. The resulting visual difference is trained to match the pathology-conditioned report difference through LΔL_{\Delta}.

The report branch bypasses the aligner as a semantic reference. The residual is trained to capture report-described differences with forward–reverse, endpoint and composition constraints.

Related work on change captioning, such as SCORER [3], uses cross-view matching and representation reconstruction to learn stable semantic differences under viewpoint variation.

Visual alignerCT-RATE AUROC (%)
Identity84.2
Target-blind learned84.5
Target-conditioned, Spectrum (Ours)85.6

With the complete BDA objective fixed, Table 5 shows gains of 1.1 percentage points over target-blind alignment and 1.4 over direct subtraction. These results support learning a transformation specific to each comparison. We now describe this demonstrated role, with nuisance reduction as the design motivation.

We evaluated controlled 3D scaling and resolution perturbations in LIDC-IDRI [1] with comparison scans matched by annotated nodule count. In the BDA projection space, the distance ratio is the mean distance from perturbed scans to their own originals divided by the mean distance to original scans from other patients with the same nodule count. Both comparisons used the same frozen model weights, projection and normalization.

LIDC-IDRI perturbationDistance ratio, direct subtractionDistance ratio, target-conditioned alignment
10% 3D downscaling0.3410.281
10% 3D upscaling0.2030.161
1.5× through-plane sampling spacing0.3070.250
1.5× in-plane sampling spacing0.2420.189

Target-conditioned alignment lowers this ratio by approximately 18–22% across all four conditions. A common multiplicative rescaling of all distances would leave the ratio unchanged. These results support greater relative stability of same-case representations in the BDA projection space under the tested perturbations. This analysis is now in the main paper and supplement.

Q1: Relation construction and the learning objective.

MethodCT-RATE AUROC (%)Δ AUROC (percentage points)
Spectrum (Ours)85.6—
Within-disease ranking82.3+3.3
Random relations83.1+2.5

Δ AUROC = Spectrum (Ours) − control on CT-RATE.

The ranking control replaces the entire BDA objective with a pairwise logistic loss that learns a pathology-specific scalar score from mined relations. Both methods use the same training and evaluation settings and the same mined within-pathology relations. Complete BDA aligns pathology-conditioned image and report differences. Its gain over ranking supports learning report-described differences alongside which case has higher burden. Forward–reverse discrimination, endpoint consistency and triplet composition provide complementary constraints.

The random control resamples both pairs and triplets from eligible cases within each pathology and retains complete BDA. BDA’s core image–report difference alignment remains active in this control. Each selected image pair remains associated with its own reports, so the report difference provides supervision for that same source–target pair. Random sampling removes the prescribed lower-to-higher ordering of pairs and triplets. Its 83.1% AUROC is consistent with retaining useful training signals, while Spectrum (Ours)’s 85.6% supports the complete design combining burden-guided relations, difference alignment and complementary constraints.

Q2: Comparison settings, confidence intervals and repeated runs. CT-CLIP and Merlin use the same training data, warm-up strategy and CT-RATE evaluation protocol, achieving 72.8% and 72.9% AUROC versus 85.6% for Spectrum (Ours).

ModelDataset and metricEstimate (95% CI)
Spectrum (Ours)CT-RATE AUROC (%)85.6 [84.8, 86.3]

The CT-RATE interval was calculated from fixed predictions using 2000 bootstrap resamples. Across repeated runs of the main training with three seeds, CT-RATE AUROC is 85.40% ± 0.26% (mean ± SD). The small variation across these three runs supports consistent diagnostic performance. The supplement now reports these results.

References

[1] S. G. Armato III et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans. Medical Physics 38(2), 915–931 (2011).

[3] Tu Y et al. Self-supervised cross-view representation reconstruction for change captioning. ICCV, 2023: 2793–2803.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their constructive feedback. The reviewers recognized the motivation for learning within-pathology burden from cross-patient CT–report data (zzp8, 84kc, e9dY), the strong downstream results (84kc, e9dY), and the supporting component ablations (84kc, e9dY).

Building on this feedback, we have added independent burden evaluation using radiologist annotations, validation of mined relations, matched baseline and ranking comparisons, and controlled perturbation analyses. These additions provide direct evidence for burden information in the representations learned with BDA and clarify the contributions of the learning objective and target-conditioned aligner. The revised main paper and supplementary material present these results and their evaluation protocols.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.