acceptodds
Under review as a conference paper at ICLR 2027

VOLT: 3D Medical Foundation Model for Volumetric Latent Tokenization

19 views

Abstract

Medical foundation model (FM) development faces a critical trade-off between semantically aligned multimodal models constrained by paired data and scalable scan-only models with weaker clinical grounding, compounded by volumetric anisotropy and the scarcity of paired scan–text data. To bridge this gap, we introduce VOLT, a scan-only 3D medical FM for high-fidelity volumetric tokenization trained through a two-stage strategy. Stage-1 learns robust through-plane priors through Depth Shuffle reconstruction and discretizes latents into compact codewords using the proposed Additive Binary Spherical Quantization (ABSQ) module. Stage-2 aligns these representations with clinical semantics through patch-consistency matching and distillation from a frozen vision-language teacher, without requiring paired reports for the training scans. VOLT yields semantically structured representations that transfer effectively across segmentation, classification, visual question answering (VQA), report generation, and synthetic data generation. Compared with the strongest reported baseline for each metric, VOLT achieves relative improvements of 13.57% in PSNR and 6.22% in SSIM for volumetric reconstruction, 0.35% in Dice and a 10.98% reduction in HD95 for segmentation, and 1.84% in mean VQA accuracy. For report generation, it improves BLEU, METEOR, and BERTScore by 4.47%, 2.11%, and 0.52%, respectively. Code and dataset will be open-sourced.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundational Model, 3D Medical Imaging, 3D Medical Foundational Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wNFPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VOLT, a 3D medical foundation model for learning discrete volumetric representations from medical scans. The method uses a two-stage training strategy: Stage-1 combines Depth Shuffle pretraining with Additive Binary Spherical Quantization (ABSQ), while Stage-2 introduces patch consistency matching and semantic distillation from a frozen vision-language teacher. The authors pretrain VOLT on 73,648 CT and MRI volumes and evaluate it on reconstruction, synthesis, segmentation, classification, VQA, and report generation. The results show improvements over several existing medical tokenizers across these tasks.

Critical strengths
  1. Learning high-quality 3D representations from medical volumes is challenging due to volumetric anisotropy and limited paired scan-text data. The proposed framework is relevant to both medical image understanding and generation.

  2. VOLT achieves substantial improvements in reconstruction quality, particularly PSNR and SSIM. The evaluation also covers several downstream tasks, which helps demonstrate the usefulness of the learned representations beyond reconstruction.

Critical weaknesses
  1. Depth Shuffle is closely related to existing reconstruction-based pretraining strategies, while ABSQ extends binary spherical quantization using additive residual refinement. Stage-2 also relies on relatively standard contrastive learning and knowledge distillation objectives. Although their combination is effective, I find the methodological novelty somewhat limited.

  2. The paper reports better reconstruction quality than VQ, BSQ, LFQ, and RVQ. However, it does not clearly compare these methods under matched bitrates or discrete representation capacities.

  3. VOLT is initialized from a pretrained WAN2.2 VAE and uses a frozen M3D-CLIP teacher during Stage-2. While the authors acknowledge that the teacher has been trained using paired scan-text data, this makes it difficult to isolate the contribution of the proposed scan-only objectives from that of existing pretrained models.

Questions for authors
  1. Can the authors clarify the main technical novelty of ABSQ compared with existing residual quantization methods? How does ABSQ perform against BSQ, LFQ, and RVQ when the bitrate and representation capacity are matched?

  2. How much of VOLT's improvement comes from the WAN2.2 initialization and the pretrained M3D-CLIP teacher? Could the authors provide a controlled comparison using the same pretrained backbone without the proposed objectives?

  3. Since M3D-CLIP provides semantic supervision and M3D benchmarks are used for downstream evaluation, can the authors clarify how potential overlap between the teacher's training data and downstream evaluation data is handled?

Reviewer_kDrgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VOLT, a scan-only 3D medical foundation model centered on volumetric latent tokenization for CT/MRI. The method has been pretrained with two meticulously designed stages, and the pretrained tokenizer/backbone transfers well across different sub-stream tasks, such as reconstruction, 3D synthetic data generation, segmentation, VQA, and so on.

Critical strengths
  1. The decoupling of volumetric structural learning (Stage 1) and semantic alignment (Stage 2) is a highly practical and effective solution to the paired-data scarcity problem in 3D medical imaging. It allows the model to leverage massive amounts of scan-only data while still inheriting rich clinical semantics.
  2. Depth Shuffle is a clever and intuitive pretext task. Unlike standard masked image modeling (MIM) which often devolves into local texture interpolation, Depth Shuffle forces the encoder to learn global through-plane anatomical progression.
  3. The authors evaluate VOLT on a highly diverse set of downstream tasks such as reconstruction, generation, segmentation, VQA, and report generation.
Critical weaknesses
  1. While ABSQ avoids codebook collapse through iterative residual quantization (using N progressive BSQ blocks), this sequential process inherently introduces computational overhead during the forward and backward passes compared to a single-shot VQ lookup. The paper lacks a detailed theoretical or empirical breakdown of the memory and latency costs of ABSQ versus standard VQ or Residual VQ during training and inference. 2. The pretraining corpus is heavily skewed towards Chest CTs (as show in Figure 7). The paper does not thoroughly discuss how this skew affects performance on underrepresented modalities (e.g., Heart MRI). 3. The semantic distillation mechanism is under-justified. Stage 2 distills token-wise distributions from M3D-CLIP visual features to encoder features via KL after channel-softmax with temperature. A few issues remain unclear. First, why should a teacher optimized for cross-modal global discrimination produce spatially aligned token distributions appropriate for dense 3D tokenization? Second, how sensitive is the method to the feature layer chosen from the teacher? 4. The main segmentation experiment is only on BTCV. For a paper claiming broad, anatomy-agnostic volumetric representations, one dataset is weak evidence. Especially because the pretraining corpus heavily includes abdominal data, BTCV may be favorable. More validation on data involving other anatomical structures or modalities (e.g., brain, spine, MRI segmentation) would make the generality claim much stronger.
Questions for authors
  1. How robust are results across anatomies with less representation in the pretraining set? The corpus is highly chest-heavy. Does performance vary substantially across brain/spine/heart tasks?
  2. What is the inference cost of VOLT as a tokenizer compared with MedVAE and MedITok? Table 1 reports reconstruction time, but not tokenization/de-tokenization cost broken down by stage.
  3. Why is M3D-CLIP an appropriate teacher for token-level semantic distillation? Since M3D-CLIP is primarily trained for cross-modal alignment, it is unclear why its visual features should provide spatially aligned supervision for dense 3D tokenization.
Reviewer_sdTbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VOLT tokenizes 3D CT and MRI in two stages. Stage 1 permutes slices, trains the encoder to recover their order, and discretizes the latent with additive residual binary spherical quantization (ABSQ). Stage 2 freezes a WAN2.2-initialized autoencoder, trains LoRA and ABSQ, and adds same-scan patch InfoNCE plus KL distillation from a frozen M3D-CLIP vision encoder. Pretraining uses 73,648 volumes. Downstream tasks are reconstruction, BTCV synthesis and segmentation, M3D visual question answering and report generation, and CirrMRI600+ classification.

Critical strengths

Each loss targets a stated failure: slice-order shortcuts, quantizer collapse, and patch drift. In the authors' reconstruction suite, Depth Shuffle beats masking and generic augmentation; ABSQ beats VQ, BSQ, LFQ, and residual VQ; raising the number of BSQ blocks from 2 to 8 moves PSNR from 33.51 to 35.49. Table 1 reports 35.07 dB, SSIM 0.946, LPIPS 0.078, against MedVAE at 30.88 dB and MedITok at SSIM 0.890, with tighter run variance. Swapping only the tokenizer improves mFVD for every generator they try (TRACE 400 to 312, MAGViT-v2 302 to 222, VicTr 278 to 219). The language model is held fixed (MedGemma-1.5), which is the right tokenizer comparison: closed-ended VQA mean goes from 79.92 to 81.39, and BLEU from 36.89 to 38.54. A linear probe on CirrMRI600+ reaches 76.09, against 73.20 for M3D-CLIP and 72.88 for MedVAE. Two radiologists marked 22 of 25 reconstructions clinically acceptable.

Critical weaknesses

The abstract's percentages each use a different strongest baseline. Dice 84.87 versus nnFormer 84.57 is three tenths of a point over a fully supervised specialist; HD95 20.02 is compared with MedVAE's 22.49. Frozen-encoder, decoder-only BTCV training is a different experiment from nnU-Net and nnFormer. The report-generation prose calls Med3DVLM the strongest prior baseline, but MedVAE is ahead of it on ROUGE, METEOR, and BERTScore; the absolute gaps versus MedVAE are about 0.14 ROUGE and 0.46 BERTScore. The appendix states Stage 1 as 400 optimization steps at effective batch 384, and BTCV segmentation as 100 steps. Those budgets need to be corrected or the tables are not believable. The autoencoder is initialized from WAN2.2 and given an extra depth downsample to x16 compression, while MedVAE and MedITok use other backbones, so Table 1 mixes architecture with the proposed quantizer. "Scan-only" describes the 73,648 volumes. The clinical geometry is distilled from a text-supervised teacher. Chest contributes 58,318 of those volumes. CirrMRI600+ is both cited inside the pretraining corpus and used as the classification test; the patient-level split claim needs a per-dataset holdout statement. Classification is only in the appendix. VQA and report gains over MedVAE are about 1.5 points and 1.7 BLEU.

Questions for authors

See above. The tokenizer is carefully ablated and the reconstruction gap is large. Do the external comparisons and the stated training budget support the foundation-model claims?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.