acceptodds
Under review as a conference paper at ICLR 2027

TriPoD: Triadic 3D CT Representation Learning via Physically Anchored Pretraining and Polarity-Debiased Text Alignment

Abstract

General-purpose foundation models for 3D computed tomography (CT) should transfer across global recognition, dense spatial prediction, and language-guided inference. Yet standardized resampling obscures scan-specific acquisition geometry, while report-derived supervision can admit textual shortcuts when concepts are associated with a dominant polarity. We present **TriPod**, a triadic framework combining physically anchored volumetric pretraining with polarity-debiased text alignment. Pretrained on more than 470,000 quality-controlled CT volumes from multiple medical centers, TriPod preserves native voxel lattices and voxel-to-world transforms instead of resampling entire scans isotropically. Affine-aware 3D rotary positional encoding maps patch centers into view-centered physical space, representing scale and orientation across acquisitions. For locked-image CT–report alignment, a patient-disjoint text-only audit reveals phrase-conditioned polarity leakage. Polarity-Debiased Opposite Sentence Learning (PD-OSL) addresses this shortcut by combining evidence-constrained atomic findings with polarity-debiased sampling. We evaluate the shared visual encoder across classification, segmentation, and internal and external zero-shot recognition. It transfers strongly across all three regimes. Controlled ablations show small, task-dependent effects of physical coordinates on global recognition and dataset-dependent effects on dense transfer, with a clear gain on one large multi-organ benchmark but negligible change on another. Under matched checkpoint and prompt conditions, PD-OSL reduces phrase-conditioned leakage and improves zero-shot recognition on both cohorts. These results highlight acquisition-aware geometry and shortcut-resistant language supervision as complementary design principles for transferable 3D CT representation learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
3D CT foundation models, self-supervised learning, physical-coordinate positional encoding, vision-language alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3haYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new CT pre-training approach and provides evidence for its pre-training performance (using linear probing) and downstream tasks (dense segmentation, multi-organ segmentation, and zero-shot recognition). The results are desent, and the proposed method is interesting. However, the method details are fairly insufficent for ICLR publication. Also the results were from single-run experiments.

Critical strengths
  1. The proposed method is useful for solving CT pre-training via a geometric view with RoPE.
  2. Figure 2 is clear and easy to understand.
  3. The scores are competitive on several tasks, like CT classification and segmentation.
Critical weaknesses
  1. Only classification results are provided for comparing foundation model baselines with the proposed method.
  2. Scores provided in Sections 4.2-4.5 are just single-run results. Please consider using at least 3 different random seeds and report average scores with standard deviations.
  3. Details of the optimization step (rr) is missing. Also, how to update λ\lambda with scheduling is not clear.
  4. According to Figure 2, Qwen is used for candidate extraction. However, nothing relevant to the approach was described in main text.
  5. The method is greatly inconsistent with the overview figure (Figure 2). For example, how to pre-process the reports was not described; how to get the atomic sentence pairs was not described; how to dilute the negative-dominant phrases was not described.
  6. The GPU usage is tremendous. Please compare the GPU hours with the baselines to further show the transparency of the proposed method.
Questions for authors
  1. Will the pre-training corpus be released after acceptance?
  2. Since main experiments used ViT-B, is it possible to provide controlled positional-encoding analysis results with ViT-B instead of ViT-S?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3haY (part1)

Thank you for your constructive suggestions.

W1: Foundation model comparisons beyond classification

A: We compare TriPoD with other foundation models on segmentation across 11 benchmarks (Section 4.3, Table 1) and on zero-shot recognition on CT-RATE and RAD-ChestCT (Section 4.5, Table 4), in addition to the classification comparison in Section 4.2 (Figure 3).

Each protocol tests a different aspect of transfer. For classification, we freeze the visual encoder and train a linear classifier on its features. For segmentation, we fine-tune each encoder end to end with its corresponding decoder. Table 1 therefore measures the performance of complete segmentation systems and does not isolate dense prediction from frozen features. For zero-shot recognition, we keep the visual encoder fixed during CT–text alignment and use text prompts to recognize findings without fitting a classifier to the evaluation labels. Together, these results show that TriPoD’s pretrained representation supports global classification, dense anatomical and lesion segmentation after fine-tuning, and language-aligned zero-shot recognition.

W2: Mean and standard deviation across at least three random seeds for Sections 4.2–4.5

A: For each downstream task, we compared all methods on the same dataset split so that they were evaluated on the same cases. Repeating every method on every dataset in Sections 4.2–4.5 for at least three seeds would exceed our available compute budget. We therefore repeated TriPoD fine-tuning on two representative segmentation tasks: SegTHOR for thoracic anatomy and MSD Hepatic Vessel for vessels and tumors.

We used three seeds to generate three data splits and kept all other training and evaluation settings fixed. We report Dice (%) for each split and the mean and sample standard deviation across the three runs.

DatasetFold 0Fold 1Fold 2Mean ± SD
SegTHOR90.7691.0689.2790.36 ± 0.96
MSD Hepatic Vessel66.8165.7564.9865.85 ± 0.92

These standard deviations quantify variation across data splits for TriPoD on the two tasks.

W3: Definition of the optimization step r and the loss-weight schedule λ(r)

A: After Eq. (8), we describe rr as the optimization step and refer readers to Appendix A.4, but we omit the explicit schedule and the direct reference to Table 7, where we report the target weight and warm-up length.

During CT–report alignment, we count optimizer updates starting at r=1r=1. At update rr, we multiply the PD-OSL loss in Eq. (8) by λ(r)=0.5min⁡(r/300,1)\lambda(r)=0.5\min(r/300,1). The weight is 0.5/3000.5/300 on the first update, reaches 0.50.5 on update 300, and stays at 0.50.5 through update 6,000. We will revise the text immediately after Eq. (8) to define the update count, state the formula, explain the warm-up in words, and cite Table 7 directly.

W4&W5: Qwen candidate extraction and construction of PD-OSL training examples from reports

A: W4. Qwen’s role. We used qwen3.8-max offline to organize report passages and extract candidate findings. During CT–report alignment, we used the resulting sentence pairs to train CXR-BERT. The two extraction prompts are summarized below.

PromptInstruction and output schema
StructureAssign each numbered source unit to exactly one of eight clinical sections, preserve its wording, and return every section, including empty ones. The output maps each section to a list of source IDs.
ShortenRead the original units grouped by section. Extract short statements of current findings, retain anatomical qualifiers, distinguish presence from explicit absence, and attach the supporting words. Each section contains positive_findings, negative_findings and normal_evidence. A candidate contains text, source_id and evidence. For Impression-only reports, we also require a verbatim supporting clause (assertion), retain the full source unit as evidence, and leave normal_evidence null.

W5. Correspondence between Figure 2(b) and Section 3.4. We respectfully disagree that the figure and the method are inconsistent. The operations labeled dilute and supplement are both defined in Section 3.4. Equation (4) gives the probability of retaining an explicit negative for a negative-dominant phrase. Equation (5) specifies how we sample weak counterfactual negatives for positive-dominant phrases. Section 3.4 also describes how we anchor atomic findings to report passages and construct their opposites. We agree that we gave too little detail on how we preprocess reports to obtain those findings.

AuthorsOpenReview ↗

Response to Reviewer 3haY (part2)

Report preprocessing and pair construction. We extracted candidates from the Findings section, using the Impression when Findings was unavailable. We checked the training manifests against the held-out split, grouped repeated reconstructions by patient, report text and processing mode, and assigned source IDs and character offsets before extraction. We validated each candidate against its cited source unit. We retained definite current findings, set aside uncertain, historical, resolved or comparison-only statements, and did not infer a specific absence from silence or a general statement of normality. We preserved laterality and anatomical scope when forming opposites. During reconstruction we shortened a location or modifier tail only when a single finding could still be identified safely. Ambiguous negation and unresolved multi-finding statements were set aside.

For a de-identified example from our report-recovery records, the source reads “Pleural effusion-thickening was not detected in both hemithorax.” The two supported pleural findings share the same source span. For effusion, the validated candidate is “No pleural effusion in either hemithorax.” Our local rule forms the pair (“Pleural effusion in either hemithorax.”, “No pleural effusion in either hemithorax.”) and assigns y=0y=0 because the report supports the second statement. The same procedure yields a separate pair for pleural thickening, also with y=0y=0. This preserves the bilateral scope and makes explicit which member of each pair the report supports.

W5. Dilution and supplementation. We built the counts and donor bank from the CT-RATE alignment training split (24,115 retained report groups covering 47,144 CT reconstructions). Frequencies count distinct patients. We sample slots during training with a pseudorandom generator keyed by the run seed, volume identifier and training microstep. The following numbers come from a replay with seed 17.

  • Dilution (Eq. 4). We retain an explicit negative for phrase uu with probability min⁡(1,(n~+(u)+1)/(n−(u)+1))\min(1,(\tilde n_+(u)+1)/(n_-(u)+1)). For thoracic aortic dilatation, the positive lookup returned 0 and 804 patients had an explicit negative. Its retention probability was 1/8051/805 (0.124%). The draw was 0.5147, so this negative was skipped. The positive lookup uses counts from section-specific phrase entries; when a phrase occurs in multiple sections, the stored entry supplies the count.
  • Supplementation (Eq. 5). We fill remaining negative slots from eligible section–phrase entries, with weight equal to the number of distinct positive donor patients. In the same replay, “Atelectasis.” had weight 184 and was sampled as (“Atelectasis.”, “No atelectasis.”) with y=0y=0. We require a known section status, exclude entries whose donor set includes the current patient, and reject exact normalized phrases already positive in the record or already selected. Synonymous wording can escape this check, so these labels carry weak supervision. Each CT contributes up to four positive and four negative slots, with retained explicit negatives taking priority and empty slots ignored.

We will expand Appendix A.5 with the extraction prompts, output schema, a worked report example and sampling pseudocode. We will also link the corresponding steps in Figure 2(b) directly to Section 3.4 and Eqs. (4)–(5).

W6: Training cost in GPU-hours and comparison with baselines

A: Across both stages of visual pretraining, we used approximately 14,066 NVIDIA L20Z GPU-hours. We summed the allocated GPU count multiplied by each job’s elapsed time across all recorded attempts, including initialization, restarts, and interrupted runs.

Visual pretraining stageGPUsCumulative job durationGPU-hours
Stage 148 L20Z217.35 h10,432.87
Stage 288 L20Z41.28 h3,633.05
Total14,065.92

Stage 1 uses complete monitoring logs. We estimated Stage 2 from distributed-training timestamps, with an uncertainty of several minutes in elapsed time. Counting only time inside the training loop gives 13,948.62 GPU-hours. These totals cover visual pretraining. We separately report the CT–report alignment configuration in Table 7 (8 GPUs, 6,000 updates). A measured GPU-hour total for alignment is unavailable here.

We did not re-pretrain the baselines. For the external models in our main comparisons, we therefore give the original authors’ figures or calculate GPU-hours where they provide both GPU count and elapsed time. “Not reported” means we found no pretraining total for the indicated model or checkpoint in the cited material.

AuthorsOpenReview ↗

Response to Reviewer 3haY (part3)

BaselinePretraining GPU-hoursScope and source
VISTA3D≈20,000The authors report total training on 64 V100 (32 GB) GPUs. He et al., Training Requirements
CT-FM≈2,304Calculated from 4 Quadro RTX 8000 GPUs × 24 days × 24 h/day. Pai et al., training methods
Merlin≈160The authors report approximately 160 hours on one A6000. Blankemeier et al.
CT-CLIP≈288Zero-shot model, calculated from 4 A100 GPUs × 3 days × 24 h/day. Hamamci et al., Supplementary Table 3
COLIPRI288 for the released modelThe authors’ model card reports 4 A100 GPUs × 72 h. Separate totals for the C, CM and CRM variants are not reported. Microsoft model card, Environmental impact
Swin UNETR≈6 GPU-days as reported (≈144 GPU-hours by unit conversion)Tang et al. give this figure for a 45k-step setting. Their main paper lists 450k iterations, and Li et al. separately cite 1,152 GPU-hours for the 5,050-scan comparator. The cost attributable to the evaluated checkpoint is therefore uncertain. Tang et al., supplement · Main paper, §5.2 · Li et al., Fig. 2
SuPreMNot reported for the 2,100-scan checkpointThe 40 GPU-hours in Fig. 2 refer to a supervised experiment on 21 CT volumes. Li et al.
VoCo (Swin-B and nnU-Net/VoComni)Not reported per checkpointWu et al. report 8 H800 GPUs for pretraining. Their figure of over 10,000 GPU-hours covers downstream evaluation. Wu et al., Experiment settings
SPECTRENot reportedThe paper specifies 24 B200 GPUs for the two pretraining phases. Appendix B.2
FlexiCTNot reportedTable S1 specifies 16 B200 GPUs per phase. Supplementary Table S1
DINO-Med3DNot reportedThe method starts from DINOv3 and describes two stages of medical adaptation. Hu et al.

The GPU types, training objectives, and treatment of failed runs differ across studies, which limits direct comparisons of training efficiency.

Q1: Availability of the pretraining corpus after acceptance

A: We cannot publicly release the clinical pretraining corpus after acceptance. As stated on page 10, the participating centers approved retrospective research use of the de-identified data in secure computing environments. Their data-governance and privacy restrictions preclude public redistribution. Upon publication, we will release the code, configuration files, and evaluation prompts listed in our Reproducibility Statement. We will also publicly release the pretrained TriPoD weights so that other researchers can evaluate and adapt the model.

AuthorsOpenReview ↗

Response to Reviewer 3haY (part4)

Q2: Controlled positional encoding ablation on ViT-B

A: We do not have a controlled ViT-B positional encoding ablation to report. To isolate the contribution of physical coordinates in the final ViT-B model, we would need to repeat both pretraining stages with normalized voxel-grid coordinates, holding the training data, objectives, and optimization schedule fixed, and then evaluate the resulting checkpoint under the same downstream protocols. Our original two-stage ViT-B visual pretraining used approximately 14,066 NVIDIA L20Z GPU-hours (see W6). We did not have the resources for a matched full-scale run within this study.

Within our available budget, we trained two ViT-S encoders for 30,000 updates on the same fixed 10% subset, varying only the coordinate parameterization (Section 4.4, Table 2). Physical coordinates improved macro Dice on TotalSegmentator v2 by 1.90 percentage points (paired 95% CI [+1.16, +2.73]), while the difference on AbdomenAtlas1.0Mini was −0.020 points (−0.087 to +0.047). These results establish a dataset-dependent effect at ViT-S scale. The independent contribution of positional encoding to the final ViT-B results remains unquantified, and we will make this limitation explicit alongside those results.

Reviewer_ye6cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TriPoD, a framework for general-purpose 3D CT pretraining. The authors conduct visual pretraining using 472,084 CT reconstruction series from nine centers and construct 3D RoPE from physical coordinates so that spatial encoding can incorporate scan scale and orientation information. During language alignment, the visual backbone is frozen, and the authors introduce PD-OSL, which adjusts the construction and sampling of positive and negative phrase examples to reduce the predictability of supervision polarity labels from the phrases alone. Experiments cover eight classification and eleven segmentation benchmarks, as well as zero-shot recognition on CT-RATE and RAD-ChestCT.

The contributions include transferable 3D representations, physical positional encoding, and adjustments to text polarity supervision. The results show that TriPoD transfers well across tasks, that physical positional encoding benefits some segmentation tasks, and that PD-OSL improves vision–language alignment performance.

Critical strengths

S1:Large-scale multicenter pretraining and broad evaluation across tasks provide substantial empirical support for the model's general-purpose representations.

The paper uses large-scale multicenter CT data for pretraining and validates the model's transfer performance on multiple classification, segmentation, and zero-shot recognition tasks. In particular, the unified frozen linear probing protocol uses consistent data splits and hyperparameter search spaces, making the representation quality of different visual encoders reasonably comparable. The authors also report frozen probing and complete fine-tuned system performance separately, avoiding the attribution of all downstream gains directly to the visual encoder. Although the model's performance may benefit jointly from data scale and the overall training approach, the broad task coverage and relatively clear distinctions between evaluation settings provide persuasive evidence for the effectiveness of general-purpose 3D CT representations and are an important basis for recommending acceptance.

S2:The paper makes polarity shortcuts in medical text supervision a quantifiable problem and validates the corresponding improvement through controlled experiments.

The authors show that balancing the overall proportions of positive and negative examples does not eliminate the association between specific medical phrases and supervision polarity, and they quantify this potential text shortcut using a patient-disjoint text-only probe. Building on this analysis, PD-OSL adjusts phrase construction and sampling strategies to reduce the predictability of polarity labels from text. The authors then compare OSL and PD-OSL using the same visual checkpoint, test prompts, and evaluation settings, with improvements in the average metrics on both test sets. This relatively complete progression from identifying bias and quantifying it to controlled validation provides direct evidence for the method's effectiveness and offers a useful approach to analyzing supervision bias in medical vision–language models.

Critical weaknesses

W1:The statistical stability of the performance gains from PD-OSL is insufficiently analyzed, and performance varies across diseases.

Through patient-disjoint text-only evaluation and controlled experiments using the same checkpoint, the authors demonstrate that PD-OSL reduces measurable polarity information leakage and improves average zero-shot recognition performance on both test sets. However, the alignment experiments lack repeated training results and corresponding confidence intervals, making it difficult to determine whether some of the smaller gains are stable. For example, in Table 4, PD-OSL improves AUROC on CT-RATE by only 0.56 percentage points over conventional OSL.

In addition, Table 12 in the appendix shows some heterogeneity across diseases under the same COLIPRI prompt setting: AUROC decreases by 3.78, 2.95, and 2.16 percentage points for pulmonary nodules, bronchiectasis, and coronary artery calcification, respectively, while increasing by 6.38 percentage points for mosaic attenuation. This indicates that an improvement in average performance does not imply consistent gains for all diseases. Although weak counterfactual negative examples may be affected by report omissions or ambiguity in clinical meaning, there is currently no evidence that they directly caused these performance differences.

The available results therefore support the overall effectiveness of PD-OSL under the evaluated settings, but provide insufficient evidence for the stability of its gains and its applicability across diseases. Providing paired statistical intervals or existing results from repeated experiments would help clarify the actual contribution of this module.

W2:The independent contribution of physical positional encoding to the final main model has not been directly quantified.

The existing ablations provide a well-controlled comparison of physical coordinates and normalized grid coordinates by fixing the architecture, training data, and optimization settings. On TotalSegmentator v2, physical positional encoding yields gains supported by paired statistical analysis. On AbdomenAtlas1.0Mini, the change is close to zero, and frozen classification results do not show consistent improvements either.

These results support the effectiveness of physical positional encoding in some settings. However, the ablations use ViT-S and 10% of the pretraining data, whereas the final transfer results come from ViT-B trained with the full training scheme. It therefore remains difficult to determine how much of the final model's performance advantage is attributable to physical positional encoding. The authors explicitly acknowledge this limitation in scope, but it still leaves uncertainty about the magnitude of this core component's contribution to the final system.

Questions for authors

1.Implementation details of physical coordinate normalization

How does the axis-wise coordinate normalization in Table 5 of the appendix correspond to Equations (2)–(3) in the main text? Please specify the units and scale used for normalization, whether its parameters vary across scans or training views, and how it relates to coordinates in millimeters and the computation of RoPE phases. This would help clarify whether the method preserves physical scale information across CT scans and improve reproducibility.

2.Statistical uncertainty in the performance gains from PD-OSL

In Table 4, PD-OSL improves AUROC on CT-RATE by 0.56 percentage points over OSL. Although the authors acknowledge in the Limitations section that the alignment experiments lack repeated runs and confidence intervals, this gain is relatively small. Could the authors provide patient-level paired confidence intervals based on existing predictions, or report any existing results from repeated training, to further clarify the uncertainty associated with this improvement?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ye6c (part1)

We thank the reviewer for recognizing the breadth of our transfer evaluation and our quantitative analysis of how reliably a finding’s wording alone predicts its polarity during training. Below, we address the reviewer’s questions about the PD-OSL gains, the scope of our positional encoding ablation, and how we compute RoPE phases from physical coordinates.

W1&Q2: Statistical stability of PD-OSL gains and variation across diseases

A: We compared PD-OSL with OSL using a paired patient-level bootstrap with 2,000 replicates. In each replicate, we sampled patients with replacement, retained all volumes from each sampled patient, and used the same draw for both methods. The table reports the change in macro AUROC for PD-OSL relative to OSL, in percentage points (pp).

DatasetVolumes / patientsΔ macro AUROC (pp)Paired 95% CI (pp)
CT-RATE1,564 / 1,304+0.55[+0.18, +0.91]
RAD-ChestCT3,630 / 3,072+1.28[+0.99, +1.57]

Both 95% confidence intervals lie above zero, supporting a positive macro AUROC difference on each benchmark. The intervals quantify uncertainty due to test-patient sampling for the fixed OSL and PD-OSL models.

Possible reasons for inconsistent gains across diseases. With the same COLIPRI prompt, PD-OSL improves AUROC over OSL for 12 of 18 CT-RATE findings (Table 12), including mosaic attenuation (+6.38 pp), with declines for lung nodules (−3.78 pp), bronchiectasis (−2.95 pp), and coronary artery calcification (−2.16 pp). PD-OSL rebalances negatives by phrase, reducing how reliably wording alone predicts polarity (LeakageAUC falls from 0.787 to 0.573 in Table 3). The benefit may vary by finding because downsampling can discard informative explicit negatives, and a finding omitted or described by a synonym can enter training as a weak negative. These mechanisms remain hypotheses for individual findings because the leakage audit pools phrases across diseases.

W2: Contribution of physical positional encoding to the final ViT-B model

A: We agree that the contribution of physical positional encoding (PE) to the final ViT-B model remains unquantified. In our controlled ablation (§4.4, Table 2), both ViT-S encoders trained for 30,000 steps on the same fixed 10% subset. Switching from normalized voxel coordinates to physical coordinates improved macro Dice by 1.90 percentage points on TotalSegmentator v2 (paired 95% CI [+1.16, +2.73]) and yielded a change of −0.020 points on AbdomenAtlas1.0Mini [−0.087, +0.047]. This establishes a dataset-dependent effect at ViT-S scale. To isolate the contribution at ViT-B scale, we would need to repeat both pretraining stages on the full corpus with Normal PE and evaluate the resulting encoder under the same downstream protocols. Our training logs put the original two-stage pretraining cost at approximately 14,066 NVIDIA L20Z GPU-hours, including restarts and interrupted jobs. The resources for a second full-scale run were unavailable within the scope of this study, so we limit our claim about the effect of PE to the controlled ViT-S comparison.

AuthorsOpenReview ↗

Response to Reviewer ye6c (part2)

Q1: Does axis-wise coordinate normalization preserve physical scale across scans?

A: Table 5's “axis-wise coordinate normalization” refers to the normalized voxel-grid baseline (Normal PE). The Physical PE implementation in Eqs. (2)–(3) uses millimeter displacements from the view center. We will correct Table 5 to make this distinction explicit.

For axis aa, let iai_a be the zero-based patch index, pap_a the patch size in voxels, and NaN_a the number of patches. The patch center is ua=(ia+1/2)pau_a=(i_a+1/2)p_a and the view center is ca=Napa/2c_a=N_a p_a/2. Equation (2) gives x=A(u−c)\mathbf{x}=A(\mathbf{u}-\mathbf{c}), where AA is the full 3×33\times3 linear part of the transformed voxel-to-world affine. Centering cancels the scanner-origin translation, while the off-diagonal terms retain the direction of oblique displacements. Equation (3) then gives θa,f=2πxa/Ta,f\theta_{a,f}=2\pi x_a/T_{a,f} radians. We verified that our ViT-B launch configurations specify 12, 10, and 10 geometrically spaced periods for the z,y,xz,y,x axes, respectively, ranging from 1 mm to 800, 500, and 500 mm, inclusive. These periods are fixed across scans, views, and batches. For example, along an axis-aligned depth direction, an eight-voxel displacement at 2.5 mm spacing spans 20 mm and gives a phase difference of π/20\pi/20 at the 800 mm period. At 5 mm spacing, the same voxel offset spans 40 mm and gives π/10\pi/10. Thus the phase responds to physical distance even when grid dimensions are identical.

The only stochastic scaling is the student's training-time coordinate jitter. On each forward pass, each view receives a per-axis multiplier exp⁡(ϵa)\exp(\epsilon_a) with ϵa∼Uniform(−ln⁡1.1,ln⁡1.1)\epsilon_a\sim\mathrm{Uniform}(-\ln 1.1,\ln 1.1) before phase computation. The teachers and evaluation use a multiplier of one. At evaluation, equal physical displacement vectors therefore give equal phases across scans at each shared period.

Thanks for the response.Your explanation has addressed my main concerns, and I will maintain my weak acceptance.

Reviewer_gCjcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses two challenges in training 3D CT foundation models: differences in voxel spacing and orientation across scans, and textual shortcuts that arise when certain medical phrases appear mostly as positive or negative findings. It proposes TriPoD with PD-OSL. TriPoD preserves native scan geometry and uses physical coordinates for patch positions, helping the model account for actual distances and orientations. PD-OSL adjusts how positive and negative finding descriptions are sampled to reduce these shortcuts. Experiments across multiple benchmarks and cohorts show that the pretrained CT encoder transfers well across tasks and support the effectiveness of TriPoD and PD-OSL.

Critical strengths
  1. The method is well motivated and clearly structured.
  2. The evaluation covers a broad range of downstream tasks, including classification, segmentation, and zero-shot recognition.
Critical weaknesses
  1. Segmentation performance still falls well behind the strongest listed baselines on several datasets, despite TriPoD having the highest reported mean Dice. These gaps suggest that the current setup may not be optimal for some segmentation tasks.
  2. The role of physical coordinates in downstream segmentation could be clearer, especially when scans already have standardized spacing. TotalSegmentator is evaluated at 1.5-mm isotropic spacing, so its improvement does not directly show robustness to spacing differences across test scans. More detail on downstream preprocessing would help explain when physical coordinates provide additional value.
  3. Resizing to fixed-size inputs may lose fine details in large CT volumes, such as whole-body scans, especially along the slice axis. Frozen classification resizes each volume to 64 x 256 x 256, leaving only 64 slices to represent the full scan. CT-text alignment uses 240 x 480 x 480; if the same size is used for zero-shot inference, this may also limit detail in large scans. The impact on small findings and scans covering larger anatomical regions remains unclear.
  4. The image-text evaluation is limited to thoracic findings: 18 labels on CT-RATE and 16 mapped labels on RAD-ChestCT. It remains unclear whether the alignment generalizes to other diseases or anatomical regions.
Questions for authors
  1. Could the authors provide the segmentation input spacing, crop sizes, and inference settings, including sliding-window size, overlap, and prediction merging if used? How are the affine and physical-coordinate RoPE updated during preprocessing and for each inference window?
  2. Is intensity normalization, including channel-wise mean/std, consistent across pretraining, CT-text alignment, and downstream evaluation? If not, why does it change, especially when the visual backbone is frozen?
  3. Report-derived labels and constructed weak negatives raise concerns about label and model quality. If already available, could the authors share radiologists' checks of these labels against the scans, or model results on an image-verified subset? I am not asking for a new clinical study.
5 comments
AuthorsOpenReview ↗

Response to Reviewer gCjc (part1)

We thank the reviewer for recognizing the motivation behind TriPoD and the breadth of our evaluation, and we address the points raised below.

W1: Segmentation performance varies across benchmarks

A: We agree that TriPoD’s highest mean Dice across the eleven tasks (73.92%) leaves substantial room for improvement on individual tasks. On MSD Colon, for example, TriPoD scored 42.89% compared with 49.14% for COLIPRI. The rankings differ on broader anatomical tasks, where TriPoD achieved the highest Dice on AbdomenAtlas1.0Mini (91.35%) and TotalSegmentator v2 (90.50%).

It is worth emphasizing that we kept all eleven downstream segmentation datasets separate from pretraining and used them for downstream fine-tuning and evaluation. Some baselines had prior exposure to data from the same benchmark sources. For example, VISTA3D’s training corpus included all five Medical Segmentation Decathlon (MSD) tasks in Table 1 [1]. This distinction matters on MSD Pancreas, where VISTA3D achieved 63.13% Dice and TriPoD achieved 58.67%. The published corpus description establishes overlap at the dataset level.

In addition, published comparisons offer context for the tumor results. Harsy et al. pretrained a 3D CNN and a ViT on the same 2.1 million radiological volumes using masked autoencoding. Under their evaluation protocol, the CNN achieved higher Dice on both tumor-only MSD tasks: 66.2% versus 61.6% on Lung and 44.0% versus 20.1% on Colon [2, Table F13]. Isensee et al. also found that standard nnU-Net outperformed each of four transformer-based or hybrid systems on LiTS, BraTS 2021, and KiTS 2023 under a common five-fold evaluation protocol [3, Table 1]. Across 108 downstream tasks, Harsy et al. found that CNNs performed best on segmentation and detection, while transformers excelled at classification, retrieval, and report generation from frozen features, suggesting that successful transfer depends on large-scale pretraining, an architecture suited to the task, and adaptation to each dataset [2]. In our Table 1, the encoder and decoder are fine-tuned together, and the systems differ in both architecture and pretraining data. While these findings support a CNN advantage on segmentation tasks involving small targets, our goal with TriPoD is to learn a general-purpose CT representation that performs consistently across diverse tasks and settings.

W2&Q1: Role of physical coordinates in standardized segmentation and how coordinates are updated

A: We designed Physical PE to give the encoder access to the physical distance between patches. For example, a ten-voxel displacement spans 10 mm on a 1-mm isotropic grid and 20 mm on a 2-mm isotropic grid, even though both grids have the same anisotropy ratio. On TotalSegmentatorV2, all 248 test scans have 1.5-mm isotropic spacing, so a given voxel displacement has the same physical length across cases. We agree that this benchmark tests how well a representation pretrained with physical coordinates transfers to a standardized grid. Robustness to changes in test spacing is a separate question.

To assess transfer when scans retain their native spacing, we extended the same Stage 1 ViT-S ablation (30k steps) to AbdomenAtlas and HepaticVessel. We compared both encodings on the same test cases within each dataset. Dice is averaged equally across cases, and the intervals are paired bootstrap 95% confidence intervals for the mean difference.

DatasetTest casesTest spacing (mm)Normal PE Dice (%)Physical PE Dice (%)ΔDice (pp) [95% CI]
TotalSegmentatorV22481.5 isotropic84.67386.573+1.900 [+1.157, +2.727]
AbdomenAtlas1,038 evaluableNative, z = 0.451–7.50090.82790.807−0.020 [−0.087, +0.047]
HepaticVessel61Native, z = 1.250–7.50064.97266.661+1.689 [−0.063, +3.562]

Physical PE improved Dice on the standardized TotalSegmentatorV2 grid. Across the two benchmarks with variable native spacing, the encodings performed nearly identically on AbdomenAtlas, while Physical PE had a higher mean Dice on HepaticVessel, with a confidence interval that included zero.

AuthorsOpenReview ↗

Response to Reviewer gCjc (part2)

Segmentation inputs and inference. In our current ViT-B/UPerNet configurations, we reorient volumes to SPL. Nine of the eleven tasks, including all three above, retain the supplied voxel spacing. AISD uses 1.5-mm isotropic spacing and INSTANCE uses (4.96, 0.451, 0.451) mm in depth–height–width order. Where resampling is enabled, we use linear interpolation for CT intensities and nearest-neighbor interpolation for labels. We crop to the foreground using a −1000 HU threshold and pad to at least the ROI size. Training uses four crops per volume with a 3:1 foreground-to-background sampling ratio. The training crop and inference window are both 64 × 256 × 256 voxels (depth × height × width), except on INSTANCE, where both are 32 × 320 × 320. During inference we process eight windows per batch with 50% overlap, merge raw logits by Gaussian-weighted averaging, then take the argmax and map the discrete prediction back to the original label grid by nearest-neighbor interpolation.

Physical coordinates for each window. We track the full voxel-to-world affine through reorientation, resampling, cropping, padding, and training flips and rotations. Physical RoPE uses the updated 3 × 3 linear part, A, to convert token-center offsets from the view center into millimeters. For a window starting at voxel s, with local token center u, local window center c, and affine translation t, the physical offset is [A(s + u) + t] − [A(s + c) + t] = A(u − c). Thus each window uses its own center and the volume’s updated linear affine. The window origin cancels, while voxel scale and orientation are retained. We keep the full 4 × 4 affine for mapping predictions back to the original label grid.

W3: Loss of detail in small lesions and scans with extensive anatomical coverage after resizing

A: For frozen classification, we resize each input volume to 64 × 256 × 256 (depth × height × width) and then normalize the channels. We use the same three HU windows as in pretraining for all downstream classification tasks except LUNA25 and COVID-CT, for which we use a single lung window (center −600 HU, width 1500 HU) across all three channels. For LUNA25, we also crop to the lung region using a lung mask before resizing. We thank the reviewer for raising this point and agree that our original submission could have explained these two exceptions more clearly.

Our classification benchmarks do not include whole-body scans. LUNA25 and COVID-CT come closest to the reviewer's concern about small findings because the pulmonary abnormalities relevant to classification can occupy a small fraction of the input volume. Our pretraining windows include no dedicated lung window, so we chose one for both tasks to emphasize contrast within the lung parenchyma. On LUNA25, where the finding of interest is a pulmonary nodule, cropping to the lungs before resizing devotes more of the fixed input grid to the region containing the nodule. SPECTRE performs well on these two tasks in our comparison. Its sliding-window approach processes local crops and may help preserve fine detail. SPECTRE implementation

We varied the intensity windows, field of view, in-plane size, and number of output slices. To isolate the effect of through-plane sampling, we compared 64 with 96 slices at 384 × 384 and 64 with 128 slices at 512 × 512, keeping the lung window and the input region fixed within each dataset. AUC and accuracy are percentages.

AuthorsOpenReview ↗

Response to Reviewer gCjc (part3)

Input regionChannelsDepth × height × widthLUNA25 AUCLUNA25 accuracyCOVID-CT AUCCOVID-CT accuracy
Full scanPretraining windows64 × 256 × 25662.6158.4989.2571.87
Full scanLung window64 × 256 × 25666.5658.9996.6781.25
Full scanLung window64 × 384 × 384——96.9778.12
Full scanLung window96 × 384 × 384——93.8470.31
Full scanLung window64 × 512 × 512——91.5073.43
Full scanLung window128 × 512 × 512——89.0673.43
Lung ROILung window64 × 256 × 25674.3360.50——
Lung ROILung window64 × 384 × 38469.4358.49——
Lung ROILung window96 × 384 × 38468.6849.50——
Lung ROILung window64 × 512 × 51267.0558.49——
Lung ROILung window128 × 512 × 51265.5456.00——

At 64 × 256 × 256, switching to the lung window on full scans improved AUC by 3.95 percentage points on LUNA25 and 7.42 on COVID-CT. Cropping to the lung ROI on LUNA25 added a further 7.77 points. These results support the reviewer's concern that preprocessing matters for localized findings.

At a fixed depth of 64 slices, increasing in-plane size reduced LUNA25 AUC at both larger sizes. On COVID-CT, AUC rose slightly from 96.67 to 96.97 at 384 × 384 while accuracy fell, and both metrics declined at 512 × 512. Increasing depth at a fixed in-plane size gave a more consistent result. At 384 × 384, increasing from 64 to 96 slices reduced AUC by 0.75 percentage points on LUNA25 and 3.13 on COVID-CT. At 512 × 512, increasing from 64 to 128 slices reduced AUC by 1.51 and 2.44 points, respectively. Accuracy fell in three of the four comparisons and was unchanged on COVID-CT at 512 × 512.

A plausible explanation is that sampling the same anatomical extent more densely can produce similar neighboring patch tokens. The frozen encoder must then aggregate a longer sequence into a scan-level representation, while the lesion still occupies roughly the same fraction of the field of view. Cropping the lung ROI, by contrast, increases the fraction of the input devoted to pulmonary anatomy. We hope these comparisons address the reviewer’s concern about how resizing affects localized findings. We will include the window settings, lung-mask cropping procedure, and input dimensions in the revised manuscript so that readers can see exactly how each dataset was processed and why.

W4: Generalization of CT–text alignment to findings outside the chest

A: We applied the same alignment method to abdominal CT using the Merlin dataset. Following the locked-image text tuning (LiT) strategy in our submission, we kept the pretrained TriPoD image encoder frozen and learned the alignment from abdominal CT–report pairs. We used the same procedure as on CT-RATE to extract positive and negative finding statements and evaluated on Merlin’s original held-out split.

Merlin learns its image and text representations jointly: it aligns CT scans with report findings using a contrastive loss, alternating between the full findings section and individual anatomical sections, and trains the image encoder to predict EHR phenotypes. At inference, it compares each scan with prompts describing the presence or absence of a finding. We include Merlin’s published result on the internal 30-finding benchmark for reference (Blankemeier et al., §4.2–4.3 and Appendix B.1, Table 4).

MethodMacro AUROCMacro AUPRCAverage F1 across findings
Merlin (published)——0.741 (95% CI 0.727–0.755)
TriPoD + CLIP0.74110.73050.6965
TriPoD + CLIP + OSL0.81290.80790.7450
TriPoD + CLIP + PD-OSL0.81790.81520.7637

Comments on the paper

2 comments

Dear Reviewers,

Thank you for the time and effort you have devoted to reviewing our work. As the discussion period draws to a close, we would greatly appreciate any feedback on our rebuttal and the additional results and clarifications we have provided. We also welcome any further questions or remaining concerns and would be happy to address them.

Thank you again for your thoughtful consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their thoughtful and constructive comments. We have revised the manuscript, with changes highlighted in blue for Reviewer gCjc, dark orange for Reviewer ye6c, and purple for Reviewer 3haY. Specifically:

  • @Reviewer gCjc, we have expanded the preprocessing and evaluation details (Appendix B.3), added label-source and training-negative audits (Appendix A.5), and supplemented input-size and spacing analyses and abdominal CT–report alignment results (Appendices D.1–D.3 and E.2).
  • @Reviewer ye6c, we have clarified the physical-coordinate positional encoding and ablation scope (Appendices A.2–A.3), explained the bootstrap estimation of Dice differences (Table 2), and added patient-level bootstrap results and discussion of category-specific performance changes (Appendix E.1).
  • @Reviewer 3haY, we have clarified the loss-weight schedule (Section 3.4), documented the offline extraction model and prompts (Appendix A.5), reported segmentation results across three data splits (Appendix C.3), and provided pretraining GPU-hours alongside available baseline costs (Appendix A.2).

Please let us know if you have any further questions. We welcome continued discussion and would be happy to provide further clarification.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.