- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces MedGen3D, which aims to unify 3D medical imaging tasks, including segmentation, restoration, and so on. The motivation is worthwhile, but several issues concerning the method’s positioning, efficiency, key ablations, and dataset selection need more clarification.
Critical strengths
There is growing trend in using generative models or generative pretraining to unify visual representation learning and downstream tasks. Exploring it in 3D medical imaging is also meanful.
Critical weaknesses
- The figures, title, and main claims do not clearly match the actual method.
- According to Sections 3.3 and 3.4, MedGen3D simply uses a pretrained DiT for a single forward pass, without multi-step denoising or flow integration. This differs substantially from the generative models. In some sense, MedGen3D can be seen as a TransUNet without skip connections, but supervised by a latent-to-latent loss. The authors should clearly distinguish between using a pretrained DiT backbone and adopting its generative framework, so that the figures, title, and terms such as “Video DiT” do not cause confusion.
- Some ablations on are missing.
- The current experiments do not establish whether latent supervision is necessary or how much it contributes to performance. I suggest keeping the architecture unchanged, using the same frozen VAE encoder and decoder, and fine-tuning the intermediate network through LoRA. Three supervision settings should be compared: latent supervision only, original task supervision on the decoded outputs only, and both together. These experiments would help assess whether the proposed “generative-style supervision” is useful and whether the added cost of introducing SDFs is justified.
- The efficiency of multi-organ segmentation is unclear. Does training require computing a SDF for each organ and encoding it through the VAE into the corresponding target tokens? If inference processes organs separately, abdominal CT segmentation with more than ten organs could take substantially longer than methods such as U-Net, which predict all organ classes in one pass. The authors should clarify how multiple organs are handled during training and inference, and provide comparisons of computational cost and inference time.
Questions for authors
I hope the authors can address the concerns raised in the Weaknesses section above and answer the following questions about the experiments and evaluation:
- Why is only a small subset of AbdomenAtlas 1.0 Mini used instead of the full dataset?
- Does BraTS 2021 contain 1,251 labeled cases? If so, why are only 1,250 used here?
3 comments
Response to Reviewer p1U4: Single-Step Prediction vs. Flow Matching Ablation
Q1. Clarification of the Generative Framework
Thank you for highlighting this distinction. We agree that using a pretrained generative backbone is different from performing iterative generative sampling. MedGen3D repurposes the pretrained video DiT for direct conditional latent prediction and does not perform multi-step denoising or flow integration. Its contribution is the unified formulation of five heterogeneous tasks with a shared latent-space prediction objective, rather than a new iterative sampling procedure.
This single-step design is intentional. For tasks with paired inputs and targets, directly predicting the target latent provides a natural alternative to iterative sampling. However, paired supervision does not imply that the conditional distribution is deterministic, particularly for report-conditioned generation. We therefore do not claim that iterative generation is universally unnecessary.
Comparison with Flow Matching.
To examine this design choice, we conducted a separate, single-seed ablation comparing direct MSE prediction (NFE=1) with Flow Matching (FM) at NFE=1 and NFE=4. Both training runs used the same pretrained initialization source, training data, task-sampling ratio, and checkpoints at 148k updates. FM1 and FM4 use the same FM checkpoint.
These are separately retrained, matched-update ablation checkpoints, not the final MedGen3D model reported in the main table. The comparison is not convergence-matched. The ablation also uses a different training schedule: a global batch size of 1 with a fixed 6:1:1:1:1 task-sampling ratio, compared with a global batch size of 4 and staged task sampling in the original training.
On the same 200 in-distribution (ID) test cases per task, the results for MSE1, FM1, and FM4 are as follows:
Table R1. Direct latent prediction vs. Flow Matching.
| Metric | MSE1 | FM1 | FM4 |
|---|---|---|---|
| Segmentation Dice (%) ↑ | 60.90 | 50.22 | 54.25 |
| Restoration PSNR (dB) ↑ | 31.66 | 30.42 | 30.51 |
| Reconstruction PSNR (dB) ↑ | 28.55 | 28.79 | 28.38 |
| Synthesis SSIM ↑ | 0.9084 | 0.8994 | 0.9059 |
| Generation FID ↓ | 20.60 | 19.71 | 17.76 |
| CT-CLIP Fréchet ↓ | 0.746 | 0.840 | 0.684 |
| Generation I2I ↑ | 42.65 | 38.74 | 41.81 |
Importantly, the 60.90% segmentation Dice corresponds to the separately retrained ablation checkpoint and does not replace the 84.6% Dice reported for the final MedGen3D model in the main table.
Within this matched-update comparison, direct latent prediction performs better on segmentation, restoration, and synthesis, whereas Flow Matching offers advantages on certain reconstruction and generation metrics. These results indicate a task-dependent trade-off rather than universal superiority of either approach.
Overall, our findings support direct latent prediction as an effective single-step design for a unified framework across heterogeneous medical imaging tasks, while recognizing that iterative Flow Matching can benefit specific generation-related metrics. Accordingly, we characterize MedGen3D as a unified conditional latent prediction framework built on a pretrained generative backbone, rather than an iterative generative framework.
Thank you for your response so far
I appreciate the authors’ response and their answers to my questions. Although their responses are not yet complete, the reviewer–author discussion period ends in about two days, so I would like to briefly comment on the responses provided so far.
I understand the authors’ motivation to unify 3D medical imaging tasks. However, “Gen” in MedGen3D suggests generative modeling, and the method uses a corresponding pretrained checkpoint. My main concern is that the experiments have not yet clearly demonstrated the benefits of this component (using the pretrained DiT checkpoint). Its role in the method could also be explained more clearly.
Some questions remain open, including whether introducing latent supervision provides meaningful benefits. I hope these can be discussed in the remaining discussion, as they are important for assessing the value of the specific design choices for a unified model. I have made some adjustments to my scores and look forward to the authors’ further responses.
Response to Reviewer p1U4: Method Clarification and Dataset Selection
We thank the reviewer for the constructive feedback. We first clarify the positioning of MedGen3D and our dataset selection protocol.
Q1. Clarification of the Generative Framework
MedGen3D uses a pretrained video DiT for single-step conditional latent prediction, without iterative denoising or flow integration. The pretrained video generative model provides the backbone initialization and latent representation interface, while downstream adaptation is performed through direct latent regression.
Our contribution is not a new diffusion sampling algorithm, but a unified framework that adapts a pretrained video generative architecture to five heterogeneous 3D medical imaging tasks. All five tasks share the same latent prediction objective and a single checkpoint, without task-specific prediction heads.
This distinction is central to our method: we leverage the representation and backbone of a pretrained video generative model rather than its iterative sampling procedure.
Q4. AbdomenAtlas 1.0 Mini Dataset Selection
We adopt a fixed training data budget of 1,000 cases per task across all five task families to control differences in training dataset size.
For AbdomenAtlas 1.0 Mini, we use 1,300 of the 5,195 available CT volumes, with 1,000/100/200 cases for training, validation, and testing, respectively. Segmentation, restoration, and reconstruction share the same patient-disjoint split.
This controlled setting allows us to evaluate the unified framework without introducing large differences in training dataset sizes across tasks. Investigating scalability with larger datasets is an important direction beyond the current experimental scope.
Q5. BraTS 2021 Dataset Size
The original submission used 1,250 complete T1–T2 pairs from BraTS 2021, split into 1,000/50/200 training, validation, and test cases.
This selection maintained the same 1,000-case training budget across all five task families. We have now incorporated the remaining case into the validation set, updating the split to 1,000/51/200 while keeping the original training and test cohorts unchanged.
We appreciate the reviewer's feedback and will address the remaining concerns regarding latent supervision and multi-organ segmentation efficiency in separate responses.