MedGen3D: Unified 3D Medical Prediction with a Pretrained Video Model

Three-dimensional medical imaging is typically approached as a collection of specialized problems, with separate architectures, output representations, and training objectives for segmentation, restoration, reconstruction, synthesis, and generation. We ask whether these heterogeneous tasks can instead be addressed through a shared prediction interface built upon a pretrained video model. We introduce MedGen3D, a generalist framework that unifies five supervised 3D medical imaging task families within a single latent prediction formulation. To accommodate segmentation, we represent segmentation masks as signed distance fields, converting discrete targets into continuous volumetric representations that can be processed by the same latent prediction pipeline as image-valued targets. A frozen video VAE encodes all inputs and targets, while a single LoRA-adapted video Transformer predicts target latents using the same masked regression objective, without task-specific prediction heads or losses. Across AbdomenAtlas, BraTS 2021, and CT-RATE v2, a single MedGen3D checkpoint performs all five task families within the same framework, reducing restoration and sparse-view reconstruction MAE by 9.7% and 38.6% and improving T1-to-T2 synthesis PSNR by 2.76 dB over the strongest evaluated baselines, while exhibiting task-dependent trade-offs on segmentation and report-conditioned generation. Under matched per-task update budgets, unified training achieves better reported metrics than independently trained single-task counterparts on four of the five task families. These results demonstrate that a pretrained video model can provide a shared representation and prediction interface across substantially different 3D medical imaging tasks, suggesting a path toward generalist volumetric medical models without task-specific prediction architectures.