acceptodds
Under review as a conference paper at ICLR 2027

SIA-CT:State-Structured Item Alignment for Open-Vocabulary 3D Chest CT Representation Learning

Abstract

Radiology reports describe both findings and the states that determine their meaning: where a finding occurs, whether it is present, and how severe or certain it is. Learning from these reports requires supervision that preserves these distinctions. We introduce SIA-CT, a framework for state-structured item alignment in 3D chest CT. Its central principle is that cross-patient item alignment requires agreement in observed states, whereas within-examination item separation requires evidence of distinct findings. We instantiate this principle through coordinated item separation, cross-patient alignment, and false-negative masking. A complementary regional objective aligns organ-supported visual features with report descriptions through shared transformations. Together, these objectives train a representation queried by natural language using only CT and text at inference. On 750 report-derived state questions, SIA-CT reaches 66.8% four-axis macro accuracy, exceeding a matched itemized baseline by 3.5 percentage points. Global and textconditioned score fusion yields AUROCs of 83.3% on CT-RATE and 75.3% on RadChestCT, while text-conditioned scoring gives stronger aggregate state discrimination. These results establish observed-state relations as a useful source of supervision and show why finding recognition and state discrimination should be evaluated together.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
CT, Representation learing, VLM, clip

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GRDvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SIA-CT, a new method to learn from 3D chest CT scans and corresponding radiology reports. It builds on top of ItemizedCLIP and aims to distinguish not only different findings, but also their respective states (anatomy, polarity, laterality, severity, uncertainty). It consists of two components. The S component uses these states to determine which training pairs should be matched or separated. The R component links image features of organs to report statements that describe those organs. On a set of 750 report-derived questions, SIA-CT improves four-axis accuracy from 63.3% (matched baseline) to 66.8%. Additional analyzes include results by finding, prompt sensitivity, retrieval, and a worked example.

Critical strengths

Evaluation of finding states: Recognizing a finding does not mean that a model can correctly distinguish its side, severity, presence, or uncertainty. Evaluating these details separately gives a more informative picture of model performance, which is valuable.

Careful use of report information: The method distinguishes missing attributes from explicitly stated values and avoids separating statements that describe the same or related findings. The controlled comparisons show gains from adding S and then R, while organ masks are needed only during training.

Critical weaknesses

Limited technical novelty: The method builds on ItemizedCLIP and combines existing ideas (supervised contrastive positives, false-negative masking, organ-pooled alignment) through state-based training rules. Its advantage over simpler alternatives needs clearer evidence.

Small, unverified gains: Disease recognition gains over ItemizedCLIP are marginal (+0.1/+0.6 AUROC), and the +3.5 pp state gain comes from a single run on 150 questions per axis, with no seeds, confidence intervals or significance tests. The three changes within S are not ablated separately, and R is never evaluated alone, so the source and stability of the gains remain unclear. In addition, the generated benchmark questions lack independent radiologist review, and chance-level performance is not reported. These gaps make the gains harder to interpret.

Questions for authors
  1. Separate effects of each part: The limitations section mentions this as future work, but do you have results for each of the three changes in S on its own (item separation, cross-patient positives, false-negative masking)? Which one helps most? Have you also tried R without S, to see if R helps by itself and if using both together adds more?
  2. Stability of the results: Can you share results from several training runs (different seeds), or confidence intervals? A simple statistical test comparing Baseline and SIA-CT on the 750 state questions would also help show the gains are real and not due to chance.
  3. Same parser for training and testing: Are the state labels used for training S made by the same pipeline that built the test questions? If so, how do you make sure the model is learning to read the CT image, and not just learning to match the parser's style?
  4. Chance performance per task: Appendix B notes candidate counts vary, so a single chance level doesn't apply. Could you report the number of answer choices per task and the resulting random-guess accuracy for each?
1 comment

Thank you for the questions about component attribution, stability, and evaluation. We provide direct uncertainty estimates for the paper's aggregate state results and the candidate-count reference, while specifying what each analysis supports.

Q1. What the component comparisons establish

The controlled comparisons share training data, report itemization, evaluation prompts and candidates, the epoch-60 endpoint, and TCS readout. The ablations assess S jointly and the increment of R given S. For the four-axis mean, the increments are +2.00 points for Baseline+S versus Baseline, 95% CI [-0.67, +4.71], and +1.50 points for SIA-CT versus Baseline+S, CI [-1.57, +4.65]. These are exploratory pointwise intervals without multiplicity correction. Positive increments are consistent with complementary contributions, but do not establish each component's statistical significance.

The current table does not isolate S's three operations or include R-only training. The separate S/R masking intervention tests a different mechanism: removing S-positive/R-negative candidates from R. Under a matched, single-seed 40-epoch budget, its overall difference is +0.13 points, 95% CI [-1.79, +1.96]. It provides direct evidence about that rule change, without substituting for the requested component ablations.

Q2. Statistical support and stability

For the original epoch-60 TCS checkpoints, SIA-CT achieves 66.83% four-axis macro accuracy versus 63.33% for the matched baseline: +3.50 percentage points, 95% CI [+0.69, +6.41]. These 600 questions cover 372 patients and 435 scans. In each of 100,000 paired patient-cluster resamples, we sample patients with replacement, retain all their questions, apply the same sample across models, recompute each axis accuracy, and average the four axes. A second direct patient-index implementation confirms the interval-based conclusion.

Applying the same bootstrap procedure to all five tasks gives the main 750-question comparison (434 patients, 515 scans): 64.00% versus 60.93%, +3.07 percentage points, 95% CI [+0.52, +5.69]. Accuracy is total correct answers divided by total sampled questions. This overall interval and the four-axis interval address different aggregates and patient sets.

Point estimates favor SIA-CT on all five tasks. Each task has only 150 questions, yielding wider intervals and less precision for small differences than the aggregate analysis. All task-specific SIA-CT-minus-baseline intervals include or touch zero. We therefore base the statistical claim on the aggregate matched-baseline comparisons, whose intervals exclude zero, and treat task-level results as exploratory. Patient resampling addresses shared-patient dependence conditional on fixed checkpoints; it does not measure training-seed variability. We subsequently evaluated two further seed pairs, giving five training seeds in total. We compared SIA-CT with the matched Baseline at epoch 60 for each seed. For the four-axis mean raised in your review, the original seed showed a gain of 3.50 percentage points. Across all five seeds, the mean paired difference was +0.27 percentage points. On all 750 questions, mean accuracy was 63.63% for SIA-CT and 63.31% for Baseline.

Q3. Scope of report-derived evaluation

Our central claim is improved report-consistent state discrimination under matched conditions. Each question holds finding identity fixed and tests alternative state descriptions, with identical candidates across models. This evaluates the distinction motivating S; broader image-based clinical understanding requires independent radiologist-adjudicated evaluation.

The historical matching audit found that removing unobserved values leaves all 49,556,482 potential directed cross-patient positive pairs unchanged. This supports historical pair-construction consistency, without validating clinical labels. Explicit normalization for arbitrary unobserved-value invariance and documentation of the implemented temporal_change constraint remain pending. Shared candidate construction also does not exclude parser-related regularities. Independent label validation and assessment of those regularities remain necessary for stronger clinical claims.

Q4. Candidate counts and uniform-guess reference

Each task contains 150 questions. Expected accuracy under uniform guessing is mean(1/K), computed separately for each question's candidate count K:

  • Polarity: K=2 for 40 questions; K=3 for 110. Uniform guess: 37.78%.
  • Laterality, severity, and uncertainty: K=2 for all 150 questions in each task. Uniform guess: 50.00% each.
  • Conflict selection: K=2 for 37 questions; K=3 for 86; K=4 for 26; K=5 for one. Uniform guess: 35.91%.

The overall uniform-guess reference is 44.74%. All models rank the same candidates per question. This reference accounts for each question's candidate count; exceeding it alone does not establish image grounding or resolve the annotation-validity questions above.

Reviewer_DShKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper suggests an item alignment framework in 3D chest CT, called SIA-CT, utilizing state relations (S) and organ alignment (R). Based on ItemizedCLIP, the S component treats report items from different patients as positive matches when their finding identities, observed attributes, and corresponding values agree. It also revised the alignment and separation objectives to reduce false negatives and avoid unnecessary separation between semantically dependent findings. The R aligns visual features pooled using organ masks with report items describing the corresponding anatomy during training. Using this modules, they shows the improvement in four-axis mean accuracy from 63.3% on baseline to 66.8%. Also, they reports the 83.3% AUROC on CT-RATE and 75.3% AUROC on RadChestCT when they utilized Global+TCS.

Critical strengths
  • The paper clearly recognizes existing problems with training on radiology reports with finding-specific sentences: sentences that describe the same finding using different wording may be pushed apart, and examples from different patients may be treated as negatives even when they share the same finding. The proposed approach addresses these issues in a simple and straightforward manner by decomposing findings into units called states.
  • Organ alignment (R) is used during training but is not required at inference time. This provides the benefits of organ-level supervision without requiring additional masks during inference.
Critical weaknesses
  • In the “Candidate and positive sets” discussion for R in Section 3.3, the authors state, “Completely repeated text from a different patient is also positive within that candidate support.” However, sentences with different wording but the same meaning appear to be treated as positives when using states, while being treated as negatives under R. This seems to create a conflict between the two objectives. For example, in Table 8 and Equation 11, the A1–B pair is treated as positive under S but as a non-positive contrastive candidate under R.
  • Under the same TCS condition, the improvements in the four-axis mean from 63.3 to 66.8 and the five-task mean from 60.9 to 64.0 are encouraging. However, Table 3 shows only marginal differences compared with ItemizedCLIP, on which this work builds: AUROC is higher by 0.1 points on CT-RATE and 0.6 points on RadChestCT, while wF1 is lower by 0.1 points and higher by 0.4 points, respectively. Differences of this magnitude could arise simply from changing the random seed and therefore do not establish that the proposed framework performs better. Furthermore, Figures 4 and 5 show improvements only for certain findings, while performance on other findings remains unchanged or declines.
  • In the image-to-image retrieval results in Figure 7, the proposed method underperforms both the baseline and ItemizedCLIP on the primary metrics, mAP@5 and mAP@10, suggesting no improvement on this task. Moreover, because the proposed method aims to align states between images and text, the evaluation should include image-to-text and text-to-image retrieval tasks that require distinguishing different states of the same finding.
  • The State-Readout evaluation does not include an assessment in which radiologists directly inspect the CT scans to determine whether the predicted states are correct.
Questions for authors
  • Could the authors use a statistical procedure such as paired bootstrapping to assess whether the proposed method outperforms other state-of-the-art methods and whether the differences are statistically significant? Please also analyze whether the proposed method is effective only for certain findings, and explain why it provides no benefit or causes performance degradation for others.
  • Some pairs treated as positives under S appear to be pushed apart under R. What is the rationale for this design, and can the authors provide evidence that this conflict does not adversely affect learning?
3 comments

Thank you for raising the statistical and S/R consistency questions. We provide patient-cluster uncertainty estimates for the original model comparison and a separate matched intervention testing the disputed pair assignments.

Q1 / Critical Weakness 2. Statistical support and heterogeneous effects

Our main statistical comparison is aggregate SIA-CT versus matched-baseline accuracy on the frozen 750-question benchmark (434 patients, 515 scans). The models share training data, report itemization, evaluation prompts and candidates, the epoch-60 endpoint, and TCS readout. SIA-CT obtains 64.00% versus 60.93%: +3.07 percentage points, 95% CI [+0.52, +5.69].

We use 100,000 paired patient-cluster bootstrap resamples, sampling patients with replacement, retaining all questions of each sampled patient, and using the same sample across models. Overall accuracy is total correct answers divided by total sampled questions. This accounts for questions sharing a patient while preserving question-level weighting.

The paper's four-axis macro accuracy is analyzed separately on its 600 questions (372 patients, 435 scans). Recomputing each axis accuracy within each patient resample and averaging the four axes gives 66.83% versus 63.33%: +3.50 percentage points, 95% CI [+0.69, +6.41]. A second direct patient-index implementation confirms the four-axis interval-based conclusion.

Point estimates favor SIA-CT on all five tasks. With only 150 questions per task, task-specific estimates have wider intervals and less precision for small differences than the aggregate comparison. The task-specific SIA-CT-minus-baseline intervals all include or touch zero. We therefore base the statistical claim on the aggregate comparisons, whose intervals exclude zero, and treat task-level results as exploratory.

Comparator-specific conclusions remain distinct. Against ItemizedCLIP, the state-accuracy differences are +1.47 points overall, 95% CI [-2.07, +4.99], and +2.33 points for the four-axis mean, CI [-1.41, +6.12]. These intervals, and the component-increment intervals, include zero. Secondary comparisons are pointwise without multiplicity correction.

Our claim is improved aggregate report-consistent state discrimination under matched conditions. Disease AUROC/wF1, retrieval, and per-finding degradations require endpoint-specific analyses; state-task intervals cannot explain finding-level changes. The current analysis conditions on fixed checkpoints and does not quantify training-seed variability or establish independent clinical validity.

A matched training queue for Baseline and SIA-CT at two additional seeds was launched on 15 November. Those results will be reported separately after validation and are not included in the intervals above.

Q2 / Critical Weakness 1. S-positive pairs becoming R negatives

We tested this concern directly by excluding S-positive/R-negative candidates from the R contrastive denominator from the first training batch, while preserving R positives. Both arms started from identical MAE/CXR-BERT base pretrained weights, used the same data order, batch size and learning-rate schedule, and completed 40 epochs with 3,716 successful optimizer updates. Thus, this comparison tests the rule change throughout training.

At the fixed epoch-40 endpoint, original-rule accuracy is 62.80% (471/750) and masked-rule accuracy is 62.93% (472/750): +0.13 percentage points, 95% paired patient-cluster CI [-1.79, +1.96], using 100,000 resamples of the 434 patients. Masking corrected 24 answers and changed 23 correct answers to incorrect ones. It removed 5.92% of accumulated R-negative pair occurrences; this is a pair-count statistic, not a gradient-conflict measure.

The design assigns complementary roles to S and R: S transfers supervision through cross-patient observed-state matches, while R aligns organ-supported visual features with examination-paired descriptions and exact text repetitions. These different positive sets can give a pair opposing assignments. The direct intervention found little overall accuracy change at this endpoint, but task changes were mixed, including polarity decreasing from 62.67% to 56.67%.

This single-seed experiment provides evidence about the observed effect of removing those assignments under a matched 40-epoch budget. Its interval allows improvements and declines, so it does not establish equivalence or absence of harm. It is separate from the original epoch-60 model comparisons above and does not establish effects across training seeds or longer training.

Thank you for the detailed responses and additional experiments. I appreciate the patient-cluster bootstrap analysis, which provides stronger support for the improvement over the matched baseline. The S/R masking experiment is also helpful in assessing my concern about conflicting supervision.

However, I still have some concerns. The improvement over ItemizedCLIP remains statistically uncertain, and the masking experiment was conducted with a single seed at 40 epochs, so it does not establish that the conflicting assignments are harmless in the final model. The state evaluation also relies on report-derived labels without independent image-based validation.

Overall, the rebuttal addresses part of my statistical concerns, but the remaining issues still limit my confidence in the main contribution. I will therefore maintain my rating of Weak Rejection (2).

Thank you for raising the concern about the final checkpoint. To address it, we repeated the masking ablation at the full 60-epoch training schedule for more seeds. For each seed, we excluded S-positive/R-negative pairs from the R denominator from the first training batch onward, and compared the result against the original SIA-CT trained with the same seed. We observed no evidence that masking meaningfully changes final accuracy; the effects were small, inconsistent in direction, and statistically indistinguishable from zero. The intervals do not rule out differences of up to about ±3 points.

Reviewer_pTLCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SIA-CT builds on ItemizedCLIP to learn chest CT features from structured findings in reports. It distinguishes missing information from explicit findings and uses these findings to decide which examples should be matched or separated during training. It also uses organ information to connect image regions with text. The experiments test selection of report findings, disease recognition, retrieval, and different ways of combining model outputs.

Critical strengths
  • The treatment of report findings is useful and well motivated. Missing information is kept separate from an explicit statement, and related findings are not automatically treated as independent. The examples and released component code make these choices easier to inspect. The model uses organ information during training without requiring organ masks at test time.
  • The controlled comparisons support the combined method on the task built from reports. Adding the proposed components improves selection of report findings under the same training and evaluation setup. Disease recognition is also competitive on external data. The paper usefully shows that combining outputs can improve disease recognition while hurting selection between conflicting findings, so the gains depend on the task.
Critical weaknesses
  • The main evaluation measures agreement with findings taken from reports. This is a reasonable task, but it does not fully establish broader clinical understanding. Some questions share the same scan, so the question count does not describe the number of independent patients. The paper should clarify the patient coverage and the answer choices used for each question. Disease recognition gains over published ItemizedCLIP are modest, and the grouped component tests do not show how much each operation contributes.
  • A possible difference between the method description and code needs clarification. The stated matching rule compares only observed attributes, while the code in relations.py compares the full stored state through _state_signature. These rules agree if missing fields always have the same default values. The released tests do not establish that condition. This is a question about consistency, not a demonstrated training error.
Questions for authors
  • How many distinct patients and scans are represented in the report-based questions, and how do the answer choices vary across questions? If available, please include uncertainty estimates that account for questions sharing a patient.
  • Does the parser always assign the same default values to unobserved fields? A test showing that changes to unobserved fields do not affect matching would help resolve the difference between the written rule and the code.
1 comment

Thank you for the constructive questions. Our audit confirms patient coverage and shows that unobserved defaults did not alter positive-pair membership in the historical training data.

Q1. Patient coverage, candidate sets, and uncertainty

The 750 questions cover 515 distinct scans and 434 patient identifiers, with 150 questions per task. Each question holds finding identity fixed and varies the state description. Candidates are deduplicated, and distractors also supported by the structured report are removed. Polarity can include a possible-presence alternative, so it is not uniformly binary. All models rank the same candidates per question.

The candidate-count distribution is:

  • Polarity: 40 two-choice and 110 three-choice questions.
  • Laterality, severity, and uncertainty: 150 two-choice questions each.
  • Conflict selection: 37 two-choice, 86 three-choice, 26 four-choice, and one five-choice question.

Our main statistical comparison is aggregate SIA-CT versus matched-baseline accuracy over all 750 questions. The models share training data, report itemization, evaluation prompts and candidates, the epoch-60 endpoint, and TCS readout. For these checkpoints, accuracy is 64.00% (480/750) versus 60.93% (457/750): +3.07 percentage points, 95% CI [+0.52, +5.69]. We use 100,000 paired patient-cluster bootstrap resamples: sample patients with replacement, retain all their questions, and apply the same patient sample to both models. Accuracy is total correct answers divided by total sampled questions.

We also analyze the paper's four-axis macro accuracy separately. Its 600 questions cover 435 scans and 372 patients. Recomputing each axis accuracy within each patient resample and averaging the four axes gives 66.83% versus 63.33%: +3.50 percentage points, 95% CI [+0.69, +6.41]. A second direct patient-index implementation confirms this interval-based conclusion.

Point estimates favor SIA-CT on all five tasks. Each task has only 150 questions, so the task-level estimates have wider intervals and less precision for small differences than the aggregate comparison. Their intervals all include or touch zero. We therefore base the statistical claim on the aggregate comparisons, whose intervals exclude zero, and treat task-level estimates as exploratory. These analyses account for patient clustering conditional on fixed checkpoints; they do not measure training-seed variability.

A matched training queue for Baseline and SIA-CT at two additional seeds was launched on 15 November. Those results will be reported separately after validation and are not included in the intervals above.

Our central claim is improved report-consistent state discrimination under matched conditions. Disease recognition provides complementary evidence about finding-level performance. Broader clinical interpretation requires independent image-based validation. The grouped ablations assess S jointly and R's increment given S, without isolating each S operation.

Q2. Unobserved values and matching consistency

The key historical-data result is that removing unobserved values leaves the positive-pair set completely unchanged: 49,556,482 potential directed cross-patient pairs under either representation. Finding identity, observation masks, and all remaining constraints were held fixed. Thus, unobserved defaults did not alter the audited historical matching relations underlying the reported runs. These are full-dataset potential relations, not counts of pairs encountered in training batches.

We reproduced all 100,724 S-eligible items' signatures and hashes using the actual generation function. Every unobserved field had one consistent stored default: anatomy=unknown, laterality=none, severity=none, uncertainty=definite, and temporal_change=unknown. No eligible item had unobserved polarity. The observation mask distinguishes these storage defaults from observed clinical values.

The perturbation test addresses implementation robustness beyond this verified data convention. All 317,296 single-field perturbations of unobserved values changed the hash because the function serializes the full stored state. This exposes a vulnerability if future parser outputs use inconsistent defaults; it does not change the verified historical pair-set equality. Explicit normalization and passing invariance regression tests remain pending.

Separately, the implemented signature includes temporal_change, observed in 13,476 eligible items. Removing this constraint increases potential directed positive pairs to 58,006,272. The reported checkpoints use this additional temporal constraint, which needs to be documented in the paper's five-attribute definition. This description correction is separate from the unobserved-default issue, for which the historical matching audit found no change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.