- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SIA-CT, a new method to learn from 3D chest CT scans and corresponding radiology reports. It builds on top of ItemizedCLIP and aims to distinguish not only different findings, but also their respective states (anatomy, polarity, laterality, severity, uncertainty). It consists of two components. The S component uses these states to determine which training pairs should be matched or separated. The R component links image features of organs to report statements that describe those organs. On a set of 750 report-derived questions, SIA-CT improves four-axis accuracy from 63.3% (matched baseline) to 66.8%. Additional analyzes include results by finding, prompt sensitivity, retrieval, and a worked example.
Critical strengths
Evaluation of finding states: Recognizing a finding does not mean that a model can correctly distinguish its side, severity, presence, or uncertainty. Evaluating these details separately gives a more informative picture of model performance, which is valuable.
Careful use of report information: The method distinguishes missing attributes from explicitly stated values and avoids separating statements that describe the same or related findings. The controlled comparisons show gains from adding S and then R, while organ masks are needed only during training.
Critical weaknesses
Limited technical novelty: The method builds on ItemizedCLIP and combines existing ideas (supervised contrastive positives, false-negative masking, organ-pooled alignment) through state-based training rules. Its advantage over simpler alternatives needs clearer evidence.
Small, unverified gains: Disease recognition gains over ItemizedCLIP are marginal (+0.1/+0.6 AUROC), and the +3.5 pp state gain comes from a single run on 150 questions per axis, with no seeds, confidence intervals or significance tests. The three changes within S are not ablated separately, and R is never evaluated alone, so the source and stability of the gains remain unclear. In addition, the generated benchmark questions lack independent radiologist review, and chance-level performance is not reported. These gaps make the gains harder to interpret.
Questions for authors
- Separate effects of each part: The limitations section mentions this as future work, but do you have results for each of the three changes in S on its own (item separation, cross-patient positives, false-negative masking)? Which one helps most? Have you also tried R without S, to see if R helps by itself and if using both together adds more?
- Stability of the results: Can you share results from several training runs (different seeds), or confidence intervals? A simple statistical test comparing Baseline and SIA-CT on the 750 state questions would also help show the gains are real and not due to chance.
- Same parser for training and testing: Are the state labels used for training S made by the same pipeline that built the test questions? If so, how do you make sure the model is learning to read the CT image, and not just learning to match the parser's style?
- Chance performance per task: Appendix B notes candidate counts vary, so a single chance level doesn't apply. Could you report the number of answer choices per task and the resulting random-guess accuracy for each?
Thank you for the questions about component attribution, stability, and evaluation. We provide direct uncertainty estimates for the paper's aggregate state results and the candidate-count reference, while specifying what each analysis supports.
Q1. What the component comparisons establish
The controlled comparisons share training data, report itemization, evaluation prompts and candidates, the epoch-60 endpoint, and TCS readout. The ablations assess S jointly and the increment of R given S. For the four-axis mean, the increments are +2.00 points for Baseline+S versus Baseline, 95% CI [-0.67, +4.71], and +1.50 points for SIA-CT versus Baseline+S, CI [-1.57, +4.65]. These are exploratory pointwise intervals without multiplicity correction. Positive increments are consistent with complementary contributions, but do not establish each component's statistical significance.
The current table does not isolate S's three operations or include R-only training. The separate S/R masking intervention tests a different mechanism: removing S-positive/R-negative candidates from R. Under a matched, single-seed 40-epoch budget, its overall difference is +0.13 points, 95% CI [-1.79, +1.96]. It provides direct evidence about that rule change, without substituting for the requested component ablations.
Q2. Statistical support and stability
For the original epoch-60 TCS checkpoints, SIA-CT achieves 66.83% four-axis macro accuracy versus 63.33% for the matched baseline: +3.50 percentage points, 95% CI [+0.69, +6.41]. These 600 questions cover 372 patients and 435 scans. In each of 100,000 paired patient-cluster resamples, we sample patients with replacement, retain all their questions, apply the same sample across models, recompute each axis accuracy, and average the four axes. A second direct patient-index implementation confirms the interval-based conclusion.
Applying the same bootstrap procedure to all five tasks gives the main 750-question comparison (434 patients, 515 scans): 64.00% versus 60.93%, +3.07 percentage points, 95% CI [+0.52, +5.69]. Accuracy is total correct answers divided by total sampled questions. This overall interval and the four-axis interval address different aggregates and patient sets.
Point estimates favor SIA-CT on all five tasks. Each task has only 150 questions, yielding wider intervals and less precision for small differences than the aggregate analysis. All task-specific SIA-CT-minus-baseline intervals include or touch zero. We therefore base the statistical claim on the aggregate matched-baseline comparisons, whose intervals exclude zero, and treat task-level results as exploratory. Patient resampling addresses shared-patient dependence conditional on fixed checkpoints; it does not measure training-seed variability. We subsequently evaluated two further seed pairs, giving five training seeds in total. We compared SIA-CT with the matched Baseline at epoch 60 for each seed. For the four-axis mean raised in your review, the original seed showed a gain of 3.50 percentage points. Across all five seeds, the mean paired difference was +0.27 percentage points. On all 750 questions, mean accuracy was 63.63% for SIA-CT and 63.31% for Baseline.
Q3. Scope of report-derived evaluation
Our central claim is improved report-consistent state discrimination under matched conditions. Each question holds finding identity fixed and tests alternative state descriptions, with identical candidates across models. This evaluates the distinction motivating S; broader image-based clinical understanding requires independent radiologist-adjudicated evaluation.
The historical matching audit found that removing unobserved values leaves all 49,556,482 potential directed cross-patient positive pairs unchanged. This supports historical pair-construction consistency, without validating clinical labels. Explicit normalization for arbitrary unobserved-value invariance and documentation of the implemented temporal_change constraint remain pending. Shared candidate construction also does not exclude parser-related regularities. Independent label validation and assessment of those regularities remain necessary for stronger clinical claims.
Q4. Candidate counts and uniform-guess reference
Each task contains 150 questions. Expected accuracy under uniform guessing is mean(1/K), computed separately for each question's candidate count K:
The overall uniform-guess reference is 44.74%. All models rank the same candidates per question. This reference accounts for each question's candidate count; exceeding it alone does not establish image grounding or resolve the annotation-validity questions above.