A High-Compression Transformer VAE for Efficient 3D Medical Image Generation

Diffusion Transformers have enabled large-scale image and video generation, but their extension to three-dimensional medical images is constrained by long spatial token sequences and the resulting attention cost. High spatial compression offers a practical route to volumetric generation, yet preserving fine anatomical and pathological detail at such compression remains challenging. We introduce a high-capacity Transformer variational autoencoder trained on BrainTumor-256K, a large multi-sequence brain tumor MRI collection. Our model compresses each spatial dimension by a factor of $8$, yielding a Gaussian latent representation with $512\times$ fewer spatial locations. The model combines high reconstruction fidelity with efficient computation on this compact grid. Across four public brain MRI datasets, our model achieves the highest mean PSNR among the evaluated autoencoders, with consistent advantages in structural and perceptual metrics. Despite its large parameter count, it delivers the lowest encoding-decoding latency at $256^3$ among the evaluated autoencoders and requires substantially less training memory than comparably sized convolutional backbones. Trained exclusively on brain MRI, the model also reconstructs non-brain MRI and CT without target-domain fine-tuning, exceeding all evaluated medical autoencoders in PSNR on every non-brain dataset, and accommodates full volumes of varying sizes and shapes. Modality-controlled and tumor-mask-guided diffusion generation further demonstrate the utility of its compressed representation. These results establish high-capacity Transformer autoencoding as an effective approach to high-compression medical image representation and a practical foundation for volumetric generation. Code is available at https://anonymous.4open.science/r/transformer-vae3d-2D31