acceptodds
Under review as a conference paper at ICLR 2027

MedGen3D: Unified 3D Medical Prediction with a Pretrained Video Model

Abstract

Three-dimensional medical imaging is typically approached as a collection of specialized problems, with separate architectures, output representations, and training objectives for segmentation, restoration, reconstruction, synthesis, and generation. We ask whether these heterogeneous tasks can instead be addressed through a shared prediction interface built upon a pretrained video model. We introduce MedGen3D, a generalist framework that unifies five supervised 3D medical imaging task families within a single latent prediction formulation. To accommodate segmentation, we represent segmentation masks as signed distance fields, converting discrete targets into continuous volumetric representations that can be processed by the same latent prediction pipeline as image-valued targets. A frozen video VAE encodes all inputs and targets, while a single LoRA-adapted video Transformer predicts target latents using the same masked regression objective, without task-specific prediction heads or losses. Across AbdomenAtlas, BraTS 2021, and CT-RATE v2, a single MedGen3D checkpoint performs all five task families within the same framework, reducing restoration and sparse-view reconstruction MAE by 9.7% and 38.6% and improving T1-to-T2 synthesis PSNR by 2.76 dB over the strongest evaluated baselines, while exhibiting task-dependent trade-offs on segmentation and report-conditioned generation. Under matched per-task update budgets, unified training achieves better reported metrics than independently trained single-task counterparts on four of the five task families. These results demonstrate that a pretrained video model can provide a shared representation and prediction interface across substantially different 3D medical imaging tasks, suggesting a path toward generalist volumetric medical models without task-specific prediction architectures.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unified Model, Generative Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p1U4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper introduces MedGen3D, which aims to unify 3D medical imaging tasks, including segmentation, restoration, and so on. The motivation is worthwhile, but several issues concerning the method’s positioning, efficiency, key ablations, and dataset selection need more clarification.

Critical strengths

There is growing trend in using generative models or generative pretraining to unify visual representation learning and downstream tasks. Exploring it in 3D medical imaging is also meanful.

Critical weaknesses
  1. The figures, title, and main claims do not clearly match the actual method.
  • According to Sections 3.3 and 3.4, MedGen3D simply uses a pretrained DiT for a single forward pass, without multi-step denoising or flow integration. This differs substantially from the generative models. In some sense, MedGen3D can be seen as a TransUNet without skip connections, but supervised by a latent-to-latent loss. The authors should clearly distinguish between using a pretrained DiT backbone and adopting its generative framework, so that the figures, title, and terms such as “Video DiT” do not cause confusion.
  1. Some ablations on are missing.
  • The current experiments do not establish whether latent supervision is necessary or how much it contributes to performance. I suggest keeping the architecture unchanged, using the same frozen VAE encoder and decoder, and fine-tuning the intermediate network through LoRA. Three supervision settings should be compared: latent supervision only, original task supervision on the decoded outputs only, and both together. These experiments would help assess whether the proposed “generative-style supervision” is useful and whether the added cost of introducing SDFs is justified.
  • The efficiency of multi-organ segmentation is unclear. Does training require computing a SDF for each organ and encoding it through the VAE into the corresponding target tokens? If inference processes organs separately, abdominal CT segmentation with more than ten organs could take substantially longer than methods such as U-Net, which predict all organ classes in one pass. The authors should clarify how multiple organs are handled during training and inference, and provide comparisons of computational cost and inference time.
Questions for authors

I hope the authors can address the concerns raised in the Weaknesses section above and answer the following questions about the experiments and evaluation:

  • Why is only a small subset of AbdomenAtlas 1.0 Mini used instead of the full dataset?
  • Does BraTS 2021 contain 1,251 labeled cases? If so, why are only 1,250 used here?
3 comments
AuthorsOpenReview ↗

Response to Reviewer p1U4: Method Clarification and Dataset Selection

We thank the reviewer for the constructive feedback. We first clarify the positioning of MedGen3D and our dataset selection protocol.

Q1. Clarification of the Generative Framework

MedGen3D uses a pretrained video DiT for single-step conditional latent prediction, without iterative denoising or flow integration. The pretrained video generative model provides the backbone initialization and latent representation interface, while downstream adaptation is performed through direct latent regression.

Our contribution is not a new diffusion sampling algorithm, but a unified framework that adapts a pretrained video generative architecture to five heterogeneous 3D medical imaging tasks. All five tasks share the same latent prediction objective and a single checkpoint, without task-specific prediction heads.

This distinction is central to our method: we leverage the representation and backbone of a pretrained video generative model rather than its iterative sampling procedure.

Q4. AbdomenAtlas 1.0 Mini Dataset Selection

We adopt a fixed training data budget of 1,000 cases per task across all five task families to control differences in training dataset size.

For AbdomenAtlas 1.0 Mini, we use 1,300 of the 5,195 available CT volumes, with 1,000/100/200 cases for training, validation, and testing, respectively. Segmentation, restoration, and reconstruction share the same patient-disjoint split.

This controlled setting allows us to evaluate the unified framework without introducing large differences in training dataset sizes across tasks. Investigating scalability with larger datasets is an important direction beyond the current experimental scope.

Q5. BraTS 2021 Dataset Size

The original submission used 1,250 complete T1–T2 pairs from BraTS 2021, split into 1,000/50/200 training, validation, and test cases.

This selection maintained the same 1,000-case training budget across all five task families. We have now incorporated the remaining case into the validation set, updating the split to 1,000/51/200 while keeping the original training and test cohorts unchanged.

We appreciate the reviewer's feedback and will address the remaining concerns regarding latent supervision and multi-organ segmentation efficiency in separate responses.

AuthorsOpenReview ↗

Response to Reviewer p1U4: Single-Step Prediction vs. Flow Matching Ablation

Q1. Clarification of the Generative Framework

Thank you for highlighting this distinction. We agree that using a pretrained generative backbone is different from performing iterative generative sampling. MedGen3D repurposes the pretrained video DiT for direct conditional latent prediction and does not perform multi-step denoising or flow integration. Its contribution is the unified formulation of five heterogeneous tasks with a shared latent-space prediction objective, rather than a new iterative sampling procedure.

This single-step design is intentional. For tasks with paired inputs and targets, directly predicting the target latent provides a natural alternative to iterative sampling. However, paired supervision does not imply that the conditional distribution is deterministic, particularly for report-conditioned generation. We therefore do not claim that iterative generation is universally unnecessary.

Comparison with Flow Matching.

To examine this design choice, we conducted a separate, single-seed ablation comparing direct MSE prediction (NFE=1) with Flow Matching (FM) at NFE=1 and NFE=4. Both training runs used the same pretrained initialization source, training data, task-sampling ratio, and checkpoints at 148k updates. FM1 and FM4 use the same FM checkpoint.

These are separately retrained, matched-update ablation checkpoints, not the final MedGen3D model reported in the main table. The comparison is not convergence-matched. The ablation also uses a different training schedule: a global batch size of 1 with a fixed 6:1:1:1:1 task-sampling ratio, compared with a global batch size of 4 and staged task sampling in the original training.

On the same 200 in-distribution (ID) test cases per task, the results for MSE1, FM1, and FM4 are as follows:

Table R1. Direct latent prediction vs. Flow Matching.

MetricMSE1FM1FM4
Segmentation Dice (%) ↑60.9050.2254.25
Restoration PSNR (dB) ↑31.6630.4230.51
Reconstruction PSNR (dB) ↑28.5528.7928.38
Synthesis SSIM ↑0.90840.89940.9059
Generation FID ↓20.6019.7117.76
CT-CLIP Fréchet ↓0.7460.8400.684
Generation I2I ↑42.6538.7441.81

Importantly, the 60.90% segmentation Dice corresponds to the separately retrained ablation checkpoint and does not replace the 84.6% Dice reported for the final MedGen3D model in the main table.

Within this matched-update comparison, direct latent prediction performs better on segmentation, restoration, and synthesis, whereas Flow Matching offers advantages on certain reconstruction and generation metrics. These results indicate a task-dependent trade-off rather than universal superiority of either approach.

Overall, our findings support direct latent prediction as an effective single-step design for a unified framework across heterogeneous medical imaging tasks, while recognizing that iterative Flow Matching can benefit specific generation-related metrics. Accordingly, we characterize MedGen3D as a unified conditional latent prediction framework built on a pretrained generative backbone, rather than an iterative generative framework.

Reviewer_p1U4OpenReview ↗

Thank you for your response so far

I appreciate the authors’ response and their answers to my questions. Although their responses are not yet complete, the reviewer–author discussion period ends in about two days, so I would like to briefly comment on the responses provided so far.

I understand the authors’ motivation to unify 3D medical imaging tasks. However, “Gen” in MedGen3D suggests generative modeling, and the method uses a corresponding pretrained checkpoint. My main concern is that the experiments have not yet clearly demonstrated the benefits of this component (using the pretrained DiT checkpoint). Its role in the method could also be explained more clearly.

Some questions remain open, including whether introducing latent supervision provides meaningful benefits. I hope these can be discussed in the remaining discussion, as they are important for assessing the value of the specific design choices for a unified model. I have made some adjustments to my scores and look forward to the authors’ further responses.

Reviewer_XQPxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduce MedGen3D, a unified framework for five 3D medical imaging tasks. It converts segmentation masks into signed distance fields and uses a frozen VAE with a LoRA-adapted DiT to predict target latents. The method performs well on four of five task families.

Critical strengths
  1. The proposed framework accommodates five different task within a shared latent prediction interface.
  2. The proposed signed distance fields effectively convert discrete segmentation masks into continuous representations.
  3. The experiments show strong performance of the proposed method compared with the other methods.
Critical weaknesses
  1. The novelty is limited, as the main contribution lies in combining existing techniques and extending them to multiple medical imaging tasks, which mainly build upon a video DiT instead of proposing new approach.
  2. The ablation studies are insufficient to prove some key design choices like signed distance fields.
  3. The paper refers to several appendices, but they are missing, making it difficult to assess reproducibility.
  4. The paper lacks qualitative visualizations and comparisons with other methods.
  5. It is unclear whether the baseline methods in Table 1 were trained or fine-tuned on the same datasets as MedGen3D.
Questions for authors
  1. Can the authors provide the missing appendix, particularly the implementation details and additional experimental results?
  2. Were the baseline methods in Table 1 trained or fine-tuned on the same training datasets as MedGen3D?
  3. Can the authors clarify the limitations of MedGen3D and the scenarios where it might underperform?
2 comments
AuthorsOpenReview ↗

Response to Reviewer XQPx: Method Clarification and Experimental Details

We thank the reviewer for the constructive feedback and for recognizing the value of a shared latent prediction framework across five 3D medical imaging tasks. We address the concerns regarding methodological novelty, baseline comparisons, supplementary material, qualitative results, and limitations below.

1. Methodological contribution.

We agree that MedGen3D builds upon existing components, including a pretrained video DiT, a frozen VAE, and LoRA. Our contribution is not a new DiT architecture, but a unified formulation that enables five heterogeneous 3D medical imaging tasks to share the same latent prediction objective and a single model checkpoint.

Specifically, MedGen3D converts heterogeneous task outputs into a common continuous latent space, allowing segmentation, restoration, reconstruction, synthesis, and report-conditioned generation to be handled without task-specific prediction heads. Unlike conventional diffusion-based generation, our framework performs single-step conditional latent prediction without iterative denoising.

2. Baseline training and evaluation.

All learning-based baselines in Table 1 were trained or fine-tuned on the corresponding task-specific training datasets. Within each task, all methods were evaluated on the same held-out test cases using consistent evaluation metrics.

Following the reviewers' feedback, we re-examined baseline training convergence and extended training where necessary. For example, the updated SynDiff model achieves 23.30 dB PSNR and 0.890 SSIM, compared with 16.61 dB and 0.717 in the original submission. We will include the updated baseline results and training details in the final version.

3. Supplementary material and qualitative results.

We would like to clarify that the appendices were already submitted as a separate Supplementary Material PDF alongside the main manuscript.

In particular, Appendix B provides dataset and preprocessing details, Appendix C describes model implementation and training, and Appendix E reports zero-shot generalization results.

Furthermore, Appendix H.1 (Table 11) provides per-organ quantitative comparisons with TotalSegmentator, while Appendix H.2 (Figure 5) presents representative segmentation visualizations and Appendix H.3 (Figure 6) analyzes boundary errors and failure cases.

We apologize for not making the location of these materials sufficiently clear in the main manuscript.

4. Limitations.

We acknowledge that segmentation remains the most challenging task for MedGen3D. Although the SDF representation enables segmentation to share the continuous latent prediction interface, the model still underperforms specialized segmentation methods, particularly on challenging structures such as the pancreas and gallbladder.

Our qualitative analysis also identifies occasional false-positive regions and localization errors, indicating that precise boundary modeling remains a limitation of the current framework. Additionally, report-conditioned generation could benefit from stronger alignment between textual descriptions and generated anatomy.

These limitations are discussed in Appendix A (Discussion and Limitations) and will be made more explicit in the final version of the paper.

Thanks for the authors' clarification. However, my concerns about methodological novelty and experimental validation remain. The authors claim the main contribution is a unified formulation but do not sufficiently clarify its methodology compared with existing unified frameworks or methods. In addition, SDF is a main component of the proposed method, yet it lacks sufficient ablation. Furthermore, the rebuttal reports that extending the training of SynDiff improves its PSNR from 16.6 to 23.3 dB, indicating that the baseline training is insufficient in the original comparison. These concerns about the method's novelty and experiments remain unsolved. Therefore, I will keep the rating.

Reviewer_1fTJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies unified 3D medical imaging. MedGen3D uses a frozen video VAE to encode inputs and targets, and a LoRA video DiT to predict target latent in 1 step with masked MSE loss. Segmentation is formulated as signed distance field. Evaluated on 5 task families, vs. specialist models, and single-task models matched for task updates.

Critical strengths

(1) Single checkpoint trained on 5 tasks without task-specific heads/losses. Honest about shortcomings

(2) Includes matched-exposure single-task control (tab.2) and zero-shot external test (tab.4).

Critical weaknesses

(1) Training of baselines in tab.1 is not described, so their credibility is unclear. The claimed improvement of 9.7%, 38.6%, +2.76 dB is against baselines of unknown quality. Some published baselines are far below an uncited Residual 2D U-Net.

(2) The claims on unified training and on video pretraining are not isolated. Because the gain over single-task models may reflect more total training, and the benefit of video pretraining is untested. Tab.2 has 2 metrics per task, but no variance. Unified model is trained for 86k steps, but each single-task model is only trained for 15.46-24.17k steps. There is no control for random or image pretraining in the main text.

(3) Tab.2 omits metrics reported in tab.1, including ones where the proposed model trails the specialists (e.g., CT-CLIP Fréchet, ASSD), and fig.1 omits ASSD and Empty. Appendices B,C,E,G are not included in the pdf. Please include all metrics from tab.1 in tab.2 and fig.1, and include appendices.

Questions for authors

(1) How were baselines in tab.1 trained (data, hyperparams, tuning)?

(2) Do conclusions from tab.2 hold if single-task models are trained to convergence and results are reported as mean +/- std over 3 seeds?

(3) What happens if video pretraining is replaced with random or image pretraining?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1fTJ: Baseline Comparisons and Supplementary Material

We thank the reviewer for the constructive comments regarding baseline comparisons, evaluation metrics, and supplementary material.

1. Baseline implementation and training convergence.

Our Residual 2D U-Net is an in-house baseline based on the classical U-Net architecture (Ronneberger et al., 2015), incorporating residual convolutional blocks. We implemented it to provide a consistent CNN reference for restoration, reconstruction, and synthesis, rather than to reproduce a specific published method. Separate models were trained for each task.

All learning-based baselines were trained or fine-tuned on the corresponding task-specific training datasets and evaluated on the same held-out test cases within each task.

Following the reviewer's feedback, we re-examined the baseline training protocols and convergence, extending training where necessary. For example, the updated SynDiff model achieves 23.30 dB PSNR and 0.890 SSIM, compared with 16.61 dB and 0.717 in the original submission. We will include the updated baseline results, training configurations, and convergence details in the final version.

2. Supplementary material and evaluation metrics.

We would like to clarify that Appendices B, C, E, and G were already included in the separately submitted Supplementary Material PDF:

  • Appendix B: Dataset and Preprocessing Details
  • Appendix C: Training and Implementation Details
  • Appendix E: Zero-Shot Generalization to External Datasets
  • Appendix G: Layer-wise LoRA Sensitivity Details

We apologize for not making their location sufficiently clear in the main manuscript.

We also acknowledge that Table 2 and Figure 1 omit some applicable evaluation metrics reported in Table 1, including ASSD, Empty, and CT-CLIP Fréchet. We will include these missing metrics in the final version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.