acceptodds
Withdrawn from ICLR 2027

3D MRI Image Pretraining through Controllable 2D Slice Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-supervised pretraining has become the mainstream approach for learning MRI representations from unlabeled scans. However, most existing objectives treat each scan as static volumes, which is inefficient use of the scans. We ask whether there exists an intrinsic self-supervision signal that is different from reconstructing the masked patches to assist improving representation quality? Through transforming the 3D volumes into controllable 2D dynamic sequences by rendering slices at continuous positions, orientations, and scales, we extract the action trajectories as the advantageous signal. We study this formulation with an action-conditioned pretraining objective, where a tokenizer encodes slice observations and a latent dynamics model predicts the evolution of latent features. Across representative anatomical and spatial downstream tasks, the proposed pretraining is evaluated against other strong 3D MRI self-supervision or supervision baselines, and ablation modules of ours. These results suggest that controllable MRI slice navigation provides a useful complementary pretraining interface for learning anatomical and spatial representations from large unlabeled MRI collections.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for health and biotechnology, Computer vision, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9xPeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised pretraining framework that reformulates 3D MRI volume representation learning through an interactive world-model lens. Using a volume rendering engine, static 3D MRI scans are re-rendered into dynamic 2D slice sequences parameterized by continuous 8D viewing controls. The method first trains a 2D MAE slice tokenizer to extract latent tokens, and subsequently trains a causal latent dynamics model to predict and denoise the current slice representation conditioned on past observations and viewing actions. Downstream performance is evaluated by fine-tuning lightweight task heads on frozen representations across healthy brain tissue segmentation, brain region parcellation, and 3D coordinate field prediction on the HCP dataset, alongside a miniature cross-domain cardiac evaluation.

Critical strengths
  1. The paper innovatively reformulates static 3D MRI volumes as dynamic observation sequences driven by controllable viewing actions, introducing a distinct medical world-model pretraining paradigm beyond standard 3D MAE and static slice reconstruction.
  2. The two-stage decoupled architecture, combining a slice-level MAE tokenizer with a causal latent dynamics model, effectively circumvents the prohibitive cost of pixel-level video generation while explicitly enforcing continuous spatial geometric transition constraints.
  3. The diagnostic action ablations and three-axis perturbation analysis provide encouraging evidence that the dynamics model uses action signals and exhibits smaller relative degradation under controlled rotations than static baselines, though absolute performance remains below S3D.
Critical weaknesses
  1. Pretraining is conducted exclusively on 148 healthy adult T1-weighted brain MRI volumes, which is critically insufficient in sample size and diversity for a self-supervised foundation model aiming to establish a general medical pretraining paradigm.
  2. While the pretraining stage focuses on learning slice transition dynamics across arbitrary 8D viewpoints, downstream 3D evaluation completely degrades into a unidirectional slice-by-slice traversal along the Z-axis, largely underutilizing and sidelining the learned high-dimensional controllable trajectory representation during actual inference.
  3. Comparing a lightweight head fine-tuned for only 5 epochs on frozen features against fully fine-tuned 3D baselines trained for hundreds to thousands of epochs conflates early adaptation efficiency with asymptotic representation capacity, while leaving the performance of an end-to-end fully fine-tuned version of the proposed backbone unaddressed.
  4. The external evaluation on the MSD Heart dataset relies on an unreliably small test set of only 3 volumes, and the proposed method's performance drops substantially behind the S3D baseline, indicating a severe bottleneck in generalizing across imaging modalities and unseen organs.
  5. Downstream evaluation predominantly depends on automated atlas labels and synthetic coordinate fields in highly standardized healthy brains, providing limited evidence of clinical efficacy on real pathological scans where lesions or anatomical disruptions invalidate structural continuity.
Questions for authors
  1. If both the tokenizer and dynamics backbone are fully unfrozen for end-to-end fine-tuning, does the proposed model demonstrate a measurable performance advantage over thoroughly converged S3D and UNETR baselines?
  2. Does the unidirectional Z-axis slice-by-slice inference introduce observable inter-slice discontinuities or staircase artifacts when 3D volumes are reconstructed and inspected along the coronal and sagittal planes?
  3. When evaluated on scans with large space-occupying lesions that break healthy anatomical priors, does the dynamics model suffer from error accumulation or hallucinations when predicting through lesion-bearing slices?
  4. What specific sampling and interpolation strategies were used to generate the continuous 8D action trajectories during pretraining, and how quantitatively sensitive is the denoising dynamics objective to the magnitude of the trajectory step size?
  5. Given that the pretraining phase enables arbitrary slice navigation, why is test-time multi-view/multi-trajectory ensemble inference not adopted to improve the spatial consistency of the final 3D prediction?
Reviewer_vycQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a self-supervised pretraining task for 3D brain MRI (magnetic resonance imaging). Each volume is turned into 2D slice videos that record the viewing controls, called actions. An action holds the slice position, the orientation and the in-plane scale. A masked autoencoder, called the tokenizer, encodes each slice. A dynamics model then denoises the latent code of the current slice, given the preceding slices and the actions. Both models stay frozen. Small heads are trained for 5 epochs on three probes: brain-region segmentation, tissue segmentation and dense coordinate prediction. The data are 184 T1 MRI volumes from the Human Connectome Project, split by subject into 148, 18 and 18. Dice is an overlap score between prediction and label, from 0 to 1. With a skip-connection (SC) decoder, the model reports 0.9073 tissue Dice, against 0.8964 for S3D, a 3D masked autoencoder. The idea is sound and useful for future work. But the results do not show that the method outperforms existing self-supervised methods. S3D is the only one compared, and it scores higher in most comparisons. The one row that scores higher than S3D is arithmetically inconsistent. The effect of the actions on downstream accuracy is not isolated.

Critical strengths
  1. The paper adds a new and cheap kind of self-supervision for volumes. Earlier geometry tasks ask the network to predict the relation between views. This paper gives the relation as input instead. The volume stays fixed, so an action changes the view and not the anatomy. Labels come from replaying the same controls on the label volume. Trajectories can also be generated with no manual input, at 3.5 s per volume.

  2. The diagnostic tests include the right controls. The action-use test in Sec. 4.4 trains with recorded actions, then probes with zero and shuffled actions. A zero-action model serves as a control. Under rotation, the tokenizer's mean Dice drops 6.48 points. Zero actions drop 2.21, shuffled actions 1.84 and recorded actions only 0.29. So correct actions add robustness. The paper limits this result to orientation shift, calling it "not a uniform in-domain downstream gain".

Critical weaknesses
  1. The results do not show that the method outperforms existing self-supervised methods. The abstract claims an evaluation "against other strong 3D MRI self-supervision or supervision baselines". Table 1 has only two baselines. S3D is a generative masked autoencoder. UNETR has no pretraining. S3D scores higher than the proposed model in most comparisons.
  • Table 1: S3D is higher than Ours with the normal decoder on all four metrics (tissue Dice 0.8964 against 0.8552). It is also higher than Ours SC on brain-region mIoU (0.6297 against 0.6105).

  • A.4.1: S3D has higher Dice at every rotation angle (0.9039 against 0.7810 at 0 degrees). The paper states that its rotation comparison "does not show superior absolute segmentation accuracy" (p16 l854-855).

  • Table 9, transfer to the Medical Segmentation Decathlon (MSD) Heart task: 0.7578 against 0.6752 Dice.

    In absolute accuracy, the proposed model scores higher than S3D only in the Ours SC row. Weakness 2 shows that this row is inconsistent.

    No contrastive or hybrid method is tested. The paper cites LVM-Med and 3DINO (p3 l133-135) and geometry-prediction methods (p3 l136-139), such as Zhang et al. (MedIA 2024), but compares none of them. Many pretrained models have released weights. The recent nnFoundation benchmark compares seven (Ulrich Harsy et al., arXiv 2026, Table 1).

  • Contrastive: VoCo (Wu et al., CVPR 2024) and CT-FM (Pai et al., arXiv 2025).

  • Hybrid: SwinUNETR, with inpainting, contrastive and rotation tasks (Tang et al., CVPR 2022).

  • Self-distillation: Curia, trained on CT and MRI (Dancette et al., arXiv 2025).

  • Volume fusion: MIS-FM (Wang et al., arXiv 2023).

  • Vision-language: Merlin (Blankemeier et al., Nature 2026).

  • Supervised: VISTA3D (He et al., CVPR 2025).

    I do not ask for a comparison with nnFoundation itself. It appeared three days before the deadline. I suggest:

  • Fine-tune at least one contrastive, one hybrid and one self-distillation model under the Table 1 protocol.

  • Pretrain one contrastive method on the same 148 MRI volumes, so data size and modality do not explain the result.

  • Report three-run means and spreads. Sec. 4.1 says all experiments were repeated at least three times, but Tables 1 and 2 show single values.

    If the proposed model does not outperform these models, the paper does not show that actions are a better pretraining signal.

  1. The Ours SC row of Table 1 is arithmetically inconsistent. Sec. 4.2 claims the highest Dice for this row. A.3 averages Dice and mIoU over the same foreground classes. For one class, IoU equals Dice / (2 minus Dice) (Eq. 17). This function is convex. Its curve bends upward. For a convex function, the average of its values is never below its value at the average input. So mIoU can never fall below this function applied to the mean Dice.

    For region Dice 0.8016, this lower bound is 0.8016 divided by 1.1984, which is 0.6689. The row reports a region mIoU of 0.6105, below the bound. So the region pair cannot come from one evaluation pass. The tissue pair passes this bound (0.8942 against 0.8303). Table 13 also disagrees with Table 1. Its five-epoch row repeats the tissue Dice of 0.8552 from the Ours row with the normal decoder. But it gives region Dice 0.5736, not 0.7380. I suggest recomputing the row from saved predictions, with Dice and IoU for each class. This is answerable in the rebuttal.

  2. No downstream ablation isolates the actions. The paper credits correct actions with "useful transition supervision beyond static tokenizer pretraining". Sec. 1 promises tokenizer-only, zero-action and shuffled-action comparisons, plus fixed-axis slice baselines (p3 l108-110). Tables 1 and 2 contain none of these rows. Fig. 5 shows that the dynamics model uses the actions during pretraining, not that they help downstream. Table 8 reports Dice drops under rotation, not Dice levels. The coordinate probe has a second problem. Its target is a fixed function of the slice geometry, and the backbone receives the action as input. So a head could recover the coordinates from the action alone. Under the Table 1 protocol with the SC decoder, I suggest:

  • Train heads on tokenizer-only, zero-action, shuffled-action and action-conditioned features, with three runs each.

  • Add an untrained backbone with the same decoder.

  • For the coordinate task, add a head that sees the action but no image.

    The Table 8 variants already exist, and the heads train in 5 epochs. This is answerable in the rebuttal. If the action-conditioned model does not score highest, the claim becomes narrower: rotation robustness only.

Questions for authors

For each number of the proposed model, state the split, trajectory and decoder. Also state the head epochs and the context length. State whether S3D was pretrained on the 148 training volumes or loaded from released weights.

Reviewer_LNdHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper re-renders 3D MRI volumes in Unity into controllable 2D slice sequences, where each frame is paired with an 8-D “action” (slice position, orientation, scale). Pretraining has two stages: an MAE tokenizer, followed by an action-conditioned causal latent-denoising dynamics model. Both are frozen downstream, and lightweight heads are trained for brain-region segmentation, GM/WM/CSF segmentation, and dense 3D coordinate prediction on HCP T1 (184 subjects). The method is compared with UNETR and S3D. Additional experiments cover rotation robustness, action-use diagnostics, and cross-trajectory and cross-dataset transfer.

Critical strengths
  1. Using controlled viewing transitions as a self-supervision signal is a novel angle. The authors correctly note that the actions change the observation, not the anatomy.
  2. The action-use diagnostic (Fig. 5) is well designed and shows that the dynamics model does exploit actions during pretraining.
  3. The appendix is candid about several limitations, e.g., that S3D has higher absolute Dice, that the heart test set has only 3 volumes, and that the pose-encoding comparison is not a matched control.
Critical weaknesses
  1. The core claim lacks in-domain evidence. The introduction promises a comparison of tokenizer-only, zero-action, shuffled-action, and action-conditioned variants, but Table 1 reports none of them on downstream accuracy. The text (L371–372) nevertheless concludes that correct actions provide supervision “beyond static tokenizer pretraining.” Appendix A.4.1 itself states that the rotation study supports an advantage under orientation shift, “not a uniform in-domain downstream gain.” This effectively concedes that action conditioning has not been shown to help in the standard setting.
  2. Actions at inference leak positional information. The frozen dynamics model receives ground-truth actions at downstream inference, and these directly specify where the slice lies in the volume. For coordinate-field prediction, the target is nearly a deterministic function of the action and the pixel location, so this probe does not test whether the representation preserves 3D structure. For brain-region segmentation, HCP data are highly standardized (single scanner, healthy young adults, ACPC space), so knowing the slice pose acts as an atlas prior.
  3. Scale and generality are limited. Pretraining uses only 148 homogeneous single-site, single-contrast subjects, which does not match the abstract’s “large unlabeled MRI collections.” The main results rest on 18 test subjects; the cross-dataset experiment has only 3 test volumes, and its result is below S3D.
  4. The comparisons are unfair. Fig. 4 contrasts 5 epochs of frozen-head training with hundreds to thousands of baseline epochs. It ignores the method’s own 300k-step tokenizer and 200k-step dynamics pretraining, and an epoch over 2D slice videos is not comparable to an epoch over 3D volumes.
  5. Missing standard baselines, e.g., nnU-Net (2D/3D), FastSurfer/SynthSeg, and action-free video SSL (VideoMAE, V-JEPA) trained on the same slice videos.
Questions for authors
  1. What is the downstream accuracy of the four dynamics variants (tokenizer-only, zero, shuffled, action-conditioned) under the Table 1 protocol?
  2. What is Pack(·) in Eq.(6)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.