- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a high-capacity transformer VAE for volumetric medical images. The model applies isotropic spatial compression, reducing the number of spatial locations by , and produces a 16-channel Gaussian latent representation. In addition, it's trained on BrainTumour-256K, which is a quite large number and private multi-sequence brain MRI collection. The paper evaluates reconstruction fidelity, encoding-decoding latency, training memory, cross-domain reconstruction, tumor-region preservation, and the utility of the learned representation for modality-controlled and tumor-mask-guided generation with a conditioned DiT.
The proposed model achieves strong reconstruction results across multiple public brain MRI datasets and favorable latency and memory measurements despite its large parameter count.
Critical strengths
-
This paper addresses an important practical bottleneck in volumetric generation. Reducing each spatial dimension by a factor of 8 produces a () reduction in spatial locations, which can substantially reduce the token count and computational cost of the generative models. Sepcially, at higher spatial compression than most baselines, the C16 has the highest PSNR on all four public brain datasets, at Table 1. The distinction between spatial compression and total element compression is clearly formulated, and the proposed representation achieves strong reconstruction fidelity despite the compact spatial grid.
-
The empirial evalution is broad and carefully designed. The paper evaluates reconstruction on multiple public brain MRI datasets, cross-domain reconstruction on non-brain MRI and CT data, tumor-region presevation, encoding-decoding latency, training memory and downstream DiT resource usage. The proposed model achieves the highest mean PSNR across four public datasets and favorable encoding-decoding latency at resolution, while the controlled backbone comparison provides useful evidence that the Transformer design can reduce training memory at comparable parameter counts.
Critical weaknesses
-
It's unclear how much of the gain comes from the architecture. The paper's main takeaway is that high-capacity Transformer autoencoding works well, but the comparisons can't really show this. The proposed model has 526M parameters and is trained on 256K brain volumes, while the baselines differ in size, training data, objectives and target domains. So the results tell us the full system is strong, but not whether the Transformer backbone is what makes it strong, as opposed to model capacity or the training data. The efficiency claim has the same gap, the source of the reported efficiency advantage is unclear beacuse the comparison does not isolate the effects of the backbone architecture and early spatial compression, and no FLOPs are reported. Although Table 3 matches model capacity and training objective, it reports only memory consumption and therefore does not establish whether the Transformer architecture provides a better fidelity efficiency trade-off than a matched CNN VAE.
-
The generation results not show this latent space is a good basis for generation. This paper shows the f8/C16 latent maskes DiT training cheaper, but never compares sample quality against other latent spaces under the same DiT and training budget. This matters because Section 5.6 shows f4/C4 with a larger DiT patch size costs about the same, so the question is whether this latent actually gives better samples. The existing evidence is also limited to answer this.
Questions for authors
-
Could the authors include a small-scale controlled comparison between a Transformer VAE and a CNN VAE of similar size, trained on the same data subset with the same objective? Reporting both reconstruction quality and latency, along with FLOPs, would show whether the gains come from the architecture or from model and data scale.
-
Section 5.6 shows that an f4/C4 latent with a larger DiT patch size has a similar training cost. Given this, why should we expect the f8/C16 latent to give better generation quality, rather than just a different way of reaching the same token count? Is there evidence or reasoning that higher reconstruction fidelity in this latent translates into better samples?
-
How was patient-level overlap between BrainTumor-256K and the public evaluation sets (especially BraTS) ruled out?
2 comments
Thank you for your detailed rebuttal. And the additional experiments to compare the transformer VAE and CNN VAE under similar scale and the explination of compression and patient overlap address my previous concerns. Overall, I am inclined to increase my score from weak reject to weak accept.
Thank you for asking for a same-data, same-size comparison with latency and FLOPs. It turned out to be the most informative experiment in the revision, and the paper's main argument now rests on it.
1. Comparable-capacity fidelity and computational cost
Question: can similarly sized models be trained on the same data and objective, with quality, latency and FLOPs reported?
Yes. We trained a 249.2M Transformer VAE and a 246.5M CNN VAE from scratch on the same collection, both with f8/C16 Gaussian latents, 128³ crops and the same loss terms. The CNN uses the CONFLUX architecture because it already has this interface; the released CONFLUX checkpoint is evaluated separately. Batch sizes differ because of the CNN's memory footprint.
At equal size the Transformer leads on all four sets, by 0.09 dB on BraTS to 1.07 dB on OASIS. The larger difference is cost, which shapes the reconstruction–cost tradeoff:
Latency is whole-volume, batch one, on the same GPU type. The training step uses a shared MSE-only objective and measures per-step cost, not time to convergence. FLOPs count executed operations, including fused attention, with one MAC as two FLOPs (Table 2, Appendix D).
2. Compression, DiT patch size and generation quality
Question: why prefer f8/C16 over f4/C4 with a larger DiT patch, and does better reconstruction yield better samples?
Our resource benchmark shows that a larger patch makes f4/C4 about as cheap as f8/C16 at the same token count, so cost alone does not favor either. The case for f8/C16 is where spatial reduction is learned. DC-AE (Table 1) fixes the generator's token count and finds better FID when the autoencoder, trained for reconstruction, does more of the compression. These results motivate our design choice, but do not establish the corresponding generation-quality advantage for 3D MRI. Our study does not demonstrate superior generation quality at a matched DiT token or training budget, and the revised manuscript states this limitation explicitly.
Compression matters for conditioning too. Spatial references are concatenated with the target tokens, so compression shortens both streams, and every image condition must be encoded. Random holes and degradations are constructed in image space and encoded online during training, allowing one DiT checkpoint to support inpainting, deblurring, de-aliasing and motion correction alongside synthesis (Figure 5, Appendix J). The frozen VAE used for these tasks is the benchmarked 499.3M model, whose encoder takes 56.7 ms at 256³ compared with 690.7 ms for the CNN.
3. Patient overlap with public benchmarks
Question: how was overlap between BrainTumor-256K and public datasets, especially BraTS, ruled out?
We agree this matters, and we changed the evaluation accordingly. Within our collection, the training, validation and test splits are patient-disjoint. OASIS and IXI are non-tumor cohorts and do not overlap with our tumor collection.
Because BraTS pools cases from many institutions, the BraTS column of Table 1 now uses 200 BraTS-GLI subjects (800 volumes) sampled after excluding cases that may overlap with our training collection (Section 4, Appendix D). We also removed MSD Brain, which is a subset of earlier BraTS releases.
On the filtered BraTS set the ordering is unchanged: 33.53 dB for the 500M Transformer, 32.55 dB for the 250M Transformer and 32.46 dB for the CNN. The Transformer–CNN comparison also uses the same training data for both models, and the context experiment uses held-out internal patients, so neither depends on public-set overlap.