acceptodds
Under review as a conference paper at ICLR 2027

mDINO: A sequential 2D–3D Latent-space Self-distillation Pre-training Framework for Medical Foundation Models

Abstract

Medical foundation models based on self-supervised learning (SSL) follow two paradigms: reconstruction-based methods capture the local texture at the expense of global semantics, and contrastive methods yield discriminative representations while losing fine-grained local details. Latent-space self-distillation offers a unified alternative to both paradigms, yet has not been adopted in medical foundation models, especially 3D domain, because of two main challenges: 1) Latent-space self-distillation methods demands large-scale pre-training corpora that far exceed the scale of available medical datasets. 2) Existing latent-space self-distillation models, such as DINOv3, are pre-trained for 2D natural images, leaving the domain gap of 3D medical foundation models unaddressed. In this paper, we propose mDINO, a sequential 2D-to-3D latent-space self-distillation pre-training framework for medical foundation models. To address data scarcity, we construct a large-scale CT corpus of 113M axial slices extracted from 1.13M volumes spanning diverse anatomical regions. mDINO first conducts 2D pre-training on 2D corpus, then leverages the spatial correspondence between 2D corpus and their source 3D volumes to transfer the learned representations as a structurally consistent initialization for 3D pre-training. To bridge the domain gap and enable latent-space self-distillation on 3D volumes, we further introduce 3D Hilbert flatten for locality-preserving token serialization and isotropy 3D-medical-RoPE for volumetric positional encoding. mDINO achieves superior performance over all reconstruction-based and contrastive-based medical foundation models on 40+ widely used 2D and 3D medical segmentation benchmarks. Code and pre-trained weights will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical foundation model, self-supervised learning, CT images

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tHhROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes mDINO, a self-supervised pretraining framework for medical imaging that follows a sequential 2D-to-3D training strategy. The model first learns from 113 million CT slices, then transfers its Transformer parameters to a 3D model for continued pretraining on 1.13 million CT volumes. To accommodate volumetric data, the authors adopt Hilbert ordering so that consecutive tokens correspond to spatially adjacent patches, together with 3D positional encoding that shares frequency settings across the three spatial axes. Experiments cover various 2D and 3D segmentation tasks, CT classification, and report generation, with analyses of initialization, token ordering, positional encoding, and backbone architecture.

Critical strengths
  1. The broad downstream evaluation provides useful evidence of transferability. The experiments cover different organs, imaging modalities, and tasks. Although pretrained exclusively on CT, the model also achieves good results after downstream fine-tuning on ultrasound, dermoscopy, and MRI segmentation. In Table 2, the average DSC across five datasets is 89.43%, compared with 88.56% for VoCo-SSL, with more substantial improvements on several MSD tumor segmentation tasks. These results constitute a valuable empirical contribution.

  2. This paper proposes mDINO, a self-supervised pretraining framework for medical imaging that follows a sequential 2D-to-3D training strategy. The model first learns from 113 million CT slices, then transfers its Transformer parameters to a 3D model for continued pretraining on 1.13 million CT volumes. To accommodate volumetric data, the authors adopt Hilbert ordering so that consecutive tokens correspond to spatially adjacent patches, together with 3D positional encoding that shares frequency settings across the three spatial axes. Experiments cover various 2D and 3D segmentation tasks, CT classification, and report generation, with analyses of initialization, token ordering, positional encoding, and backbone architecture.The paper provides reasonably thorough analyses of its main design choices. Table 6 examines 2D pretraining initialization, different token orderings, positional encodings, and backbone architectures. Given the substantially larger pretraining dataset used by mDINO, the inclusion of a VoCo control using the same data and ViT-S backbone is particularly useful. These ablations help explain the contributions of individual components, although the initialization settings still require clarification.

Critical weaknesses
  1. Limited methodological novelty The methodological advance remains unclear. 3DINO and TAP-CT already perform native 3D self-distillation, while SPECTRE uses DINOv3 with shared-frequency 3D RoPE. These directly relevant methods are neither discussed nor compared. The additional value of sequential 2D-to-3D training and Hilbert ordering is therefore insufficiently established.

  2. Unclear benefit from the larger pre-training scale mDINO uses roughly seven times as many pre-training volumes as VoCo, yet Table 2 reports only a 0.87-point average DSC gain. Its smaller backbone is a strength, but data and compute efficiency remain unclear. The data-matched ablation covers only one epoch on BTCV. Results across data subsets and fine-tuning seeds would help establish the value of this larger-scale training.

  3. Incomplete comparison with recent pre-training methods The VoCo comparison needs clarification. Table 2 reports 89.45 DSC on AMOS, while the extended VoCo paper reports 90.86 for Swin-B. The authors should explain checkpoint and supervision differences, particularly self-supervised versus omni-supervised training. Other relevant SSL baselines, such as MiM, are also missing.

Questions for authors
  1. What are mDINO’s main methodological and performance advantages over 3DINO, TAP-CT, and SPECTRE?

  2. The main text specifies five training epochs per stage, while Table 9 lists ten warm-up epochs and the initialization ablation in Table 6 uses only one epoch. Could the authors clarify the training schedules? Does the benefit of 2D initialization persist after the full five epochs of 3D pretraining, or mainly reflect faster early convergence?

  3. In Table 2, mDINO uses substantially more pretraining data than VoCo but improves average DSC by only 0.87 percentage points. How do the authors interpret this modest gain? Does it exceed variability across fine-tuning random seeds?

  4. Which VoCo checkpoints and supervision settings were used, and why do the results differ from the extended paper?

  5. Beyond the BLEU gains in Table 5, do the generated reports improve in clinical accuracy or completeness?

Reviewer_dyxJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a pre-training framework for medical foundation models that operate on CT images. First, a ViT is trained on 2D slices, and then those weights are used to initialize a 3D model which is further trained on the volumes themselves. The 3D model uses a different way of ordering the chunks which preserves locality (Hilbert ordering), as well as a 3D version of RoPE. In addition to the framework, they also construct a large dataset of 1.13M volumes from multiple institutions across Asia (113M 2D slices), which they use to pretrain their model. The corresponding 2D/3D encoder is placed within an existing pipeline to be evaluated on several 2D/3D medical benchmarks. The authors report improvements over prior methods, with ablations supporting their decisions.

Critical strengths
  1. The authors construct a very large dataset, much larger than the one used on one of the previously best-performing models for 3D medical pretraining. The evaluation also covers a vast array of tasks in the 2D/3D space.

  2. The modifications to the 3D model (Hilbert ordering and 3D RoPE) are simple and intuitive. The ablations point in the expected direction, though each is a single one epoch run.

Critical weaknesses
  1. It is quite unclear whether mDINO actually leads to better performance instead of other factors, such as data or experimental setup. This is due to several reasons. First, I noticed that most baseline rows are identical to those from previous papers (Mobile U-ViT and VoCo), despite the paper stating that “all baselines are integrated and fine-tuned under this same encoder-replacement protocol to ensure fair comparison”. The new additions are the VoCo-SSL row and the mDINO row, but it’s also not clear what VoCo-SSL is even after looking at the previous paper (and this is important considering VoCo-SSL is the main comparison against mDINO). There are also some unexplained table entries. For example, in table 4, all tasks seem generally close to each other across baselines except for task 3 and task 7, where only the new (non-copied) rows, VoCo-SSL and mDINO, are much higher than the rest of the entries. This leads me to believe the setup is different for the methods, but the authors do not discuss this at all. Further, VoCo’s Swin-B variant from the original paper seems to be 51.34 for task 7 while VoCo-SSL (Swin-B) is 68.20 for task 7 in this paper (again, it's unclear if these are the same method).

  2. There is another issue with the paper’s comparisons regarding pre-training data. While the dataset is presented as one of the novelties, mDINO does end up pre-training on much more data than VoCo and other baselines. The single ablation that’s supposed to answer this is extremely limited. It is only one epoch (models can converge at different rates) and only evaluated on BTCV. I don’t think this shows convincing evidence that the improvements are due to mDINO’s setup. Perhaps fully training mDINO on VoCo’s 160K set would be a better comparison.

  3. For all the metrics, there are no measures of uncertainty (confidence intervals) anywhere throughout the paper. Therefore it's unclear if the performance gain is due to Monte-Carlo noise (or the aforementioned differences in model set up), or if it's statistically significant.

Questions for authors

There are also some other table entries which seem to be mistakes. For example, table 2 lists the “Total” column delta as 1.38, when according to the table entries it should be 1.58. In table 3, a few deltas are colored blue (when they should be red to indicate negative/loss against VoCo?). These are just the ones I noticed, but I feel that the text and results were not carefully looked over.

Additionally, there has been a line of work that directly leverages frozen 2D ViT for (volumetric) 3D tasks (e.g., Raptor), which the paper does not discuss. Given they are directly comparable, they should be at least mentioned, and why the authors chose not to compare against them.

Several grammatical errors are present throughout the text. For example, “self-distillation methods demands”, “introduces substantially training complexity”, etc.

Reviewer_FRuLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes mDINO, a two-stage self-supervised pre-training framework for CT. Stage 1 trains a 2D ViT-S from scratch (mDINO-2D) on 113M axial CT slices using the full DINOv3 pipeline. Stage 2 transfers all ViT block weights to a 3D model with three changes: a 3D patch embedding, a 3D Hilbert curve that determines token serialization order (used for iBOT block masking), and a 3D RoPE with a frequency shared across the three axes (called isotropy 3D-medical-RoPE). Pre-training then continues on 1.13M CT volumes (mDINO-3D). The pre-training data are private, collected from 97 institutions in Asia, and will not be released. Downstream evaluation covers 4 2D segmentation tasks, 39 3D segmentation tasks (CT/MRI), report generation, and two classification tasks, using the Mobile U-ViT framework for 2D and the SwinUNETR/VoCo framework for 3D. The authors report that mDINO outperforms VoCo, SuPreM, and others on most tasks.

Critical strengths
  1. The scale of this work is substantial: 1.13M CT volumes and about 15k H200 GPU hours, far exceeding existing public 3D medical SSL efforts. If the weights and downstream code are released as promised, this will be of real practical value to the community.
  2. The downstream evaluation is very broad: 40+ benchmarks across CT, MRI, ultrasound, and dermoscopy, covering organs, tumors, and vessels, plus report generation and classification.
  3. The method is simple and direct, every component has a corresponding ablation, and the PCA and similarity visualizations show good anatomical semantic structure.
  4. The appendix includes a leakage and de-duplication analysis, detailed hyperparameters, and a release plan, which reflects a careful attitude.
Critical weaknesses
  1. Novelty is limited, and prior work is misrepresented. The paper states that latent-space self-distillation “has not been adopted in medical foundation models, especially 3D,” and that “no existing implementation supports volumetric self-distillation.” Both statements are inaccurate. For example, 3DINO (Xu et al., npj Digital Medicine 2025) already extends DINOv2 to 3D medical volumes. The paper cites MedDINOv3 and MM-DINOv2 but neither discusses nor compares with 3DINO, which is the most directly related work.
  2. All components are existing techniques: Hilbert serialization is widely used in point-cloud Transformers. Axis-blocked 3D RoPE is standard axial RoPE. The “isotropy” is just sharing the frequency across the three axes, which is already the default in axial RoPE. 2D→3D weight transfer is essentially the same as existing inflation and 2D→3D transfer approaches.
  3. The main contribution is data and compute: The core contribution is closer to “training DINOv3 on a large private CT corpus,” and since the data cannot be released, the work is hard to reproduce scientifically.
  4. The training configuration and compute reports are inconsistent. The final training runs only 5 epochs per stage and the ablations only 1 epoch, yet Table 9 specifies a 10-epoch warm-up and a teacher temperature ramp from 0.04 to 0.07 over 10 epochs. This means neither the learning-rate warm-up nor the temperature schedule ever completes in the final training, and the ablations run entirely within warm-up.
  5. Downstream comparisons are not clearly fair, and the conclusions are insufficiently supported. The authors claim all baselines were re-fine-tuned under the same encoder-replacement protocol, but many baseline results appear to be taken directly from VoCo and other papers. Please state which were rerun and which were cited.
  6. Key controls needed to support the central claims are missing. The central argument is that latent-space self-distillation beats reconstruction and contrastive methods, yet the only same-data comparison is a single epoch against VoCo. VoCo was designed for Swin, and a single epoch with ViT-S may not converge. Comparisons against reconstruction methods such as MAE and 3D MAE on the same data are also needed.
Questions for authors
  1. How does this work relate to 3DINO, and how does it compare with them?

  2. Is CT-2D strictly a subset of CT-3D, and how is the scan-length contradiction explained?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.