- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SynOphsis, a unified generative framework for ophthalmic imaging across multiple modalities and dimensionalities. The proposed method combines a frozen VAE, shape-specific patch embeddings, and a shared stochastic-interpolant transformer to model 2D images, 3D OCT volumes, and paired 3D–2D acquisitions within a common token sequence. A unified masking mechanism enables different conditional and unconditional generation tasks without task-specific fine-tuning.
Critical strengths
- The paper formulates heterogeneous ophthalmic generation as masked completion over a shared token sequence. This allows one model to accommodate multiple imaging modalities, dimensionalities, and conditioning configurations without introducing task-specific architectures (Secs. 3.1–3.2; Table 1). The joint representation of 3D volumes and paired 2D views is particularly interesting because the two outputs differ in both dimensionality and acquisition geometry. This unified interface could substantially simplify multimodal generative modeling and provide a useful foundation for subsequent research.
- The experiments span multiple imaging modalities, volumetric acquisition protocols, and conditional generation tasks. On internal OCT volumes, SynOphsis-XL achieves an FVD of 32.9 compared with 56.2 for the retrained 3D LDM baseline (Table 2). More importantly, the B-scale unified model maintains comparable conditional reconstruction accuracy to the task-specific SiT Inpaint variant, with an average PSNR difference of only +0.03 dB across fourteen shared conditions (Sec. 5.2; Table 3). These results suggest that the broader generation interface can be obtained without a substantial reconstruction penalty on the evaluated tasks.
Critical weaknesses
- One of the main contributions is the joint generation of 3D OCT volumes and their corresponding 2D en-face images. However, generating both modalities through the same transformer does not necessarily mean that their anatomical structures are consistent. Appendix C.3 provides a correspondence evaluation, but the authors acknowledge that the results demonstrate agreement at the aggregate level rather than for individual eyes. Since anatomical consistency is important for the practical use of generated medical images, I would like to see stronger evidence that the generated 2D and 3D pairs represent consistent anatomical structures.
- While SynOphsis presents a useful unified framework, its methodological novelty could be further clarified, as it largely builds upon established techniques, including VAE, SiT, masked conditioning, and REPA-style representation alignment. Although the paper includes comparisons with similarly sized SiT variants, the improvements over the 3D LDM baseline involve simultaneous changes in architecture, training objectives, latent representations, and model capacity. Therefore, it remains unclear to what extent these improvements can be attributed to the proposed methodological designs rather than the stronger backbone or increased model capacity.
Questions for authors
- The paper reports reconstruction results for 3D volumes (Table 3) and paired 2D images (Table 4), but these metrics evaluate the outputs separately. Could the authors provide more direct evidence that the jointly generated 2D and 3D images preserve consistent anatomical structures? In particular, can the authors evaluate structural correspondence beyond the projection-based retrieval results in Table 16?
- Could the authors report an ablation that removes the cross-dimensional alignment loss while keeping the joint tokenization and other training settings unchanged? It would be useful to see how this affects generation quality and the 3D–2D correspondence results in Table 16.
- Table 7 reports the reconstruction quality of the frozen VAE on OCT volumes. Could the authors further investigate whether independently encoding groups of three B-scans introduces structural discontinuities at group boundaries? For example, an analysis of adjacent-slice differences or anatomical boundary continuity before and after VAE reconstruction would be informative.
6 comments
Response to Reviewer dVPe (2/2): Novelty, Source of the Gains and Ablation of the Cross-Dimensional Loss
W2. Methodological novelty and the source of the gains over the 3D LDM
Thank you for this comment. We appreciate the chance to clarify both the novelty and the source of the gains. We see the main novelty of SynOphsis in a new formulation and in a pipeline designed for consistent joint generation across dimensionalities and modalities. The formulation answers a challenging clinical need and grows out of the idea of improving pre-training and its generalization by unifying as many modalities as possible. This formulation provides an exciting opportunity to train one model that flexibly generates the images of an eye examination together across dimensionalities (Contribution 1). Unlike a slice of a CT or MRI volume, an en-face retinal image lies in the plane orthogonal to every B-scan and on a different grid. The en-face image is therefore not one more frame of the volume.
SynOphsis writes the volume and its en-face image into one self-attention sequence on a 3D-aware SiT (Contribution 2) and conditions by masking that sequence. To tie the volume to its en-face image across their orthogonal planes, a cross-dimensional term that REPA lacks aligns the volume's depth-averaged features with the features of the paired image (Section 3.3). With this reformulation, it is STRAIGHTFORWARD and actually NATURAL to combine the VAE, SiT, masked conditioning and REPA-style alignment named in the review. As a working example, SynOphsis shows the benefit of embracing a more unified pre-training approach. In the words of the review, the interface could "provide a useful foundation for subsequent research", and we regard the formulation as a necessary stepping stone toward unified clinical foundation models.
For the source of the gains, most of the accuracy gain over the 3D LDM comes from our volumetric SiT design. Joint tokenization adds the en-face output at no cost in reconstruction accuracy and brings the generated volume closer to a given en-face image. No existing baseline performs all the tasks of SynOphsis, so each task is compared against its closest baseline. The need for a separate baseline per task also reflects how new the unified setting is. Internal SiT-B variants with our backbone, data and budget separate these sources (Section 5.2).
- Volumetric SiT design (Contribution 2). Under the same ControlNet-style conditioning, SiT-B Ctrl exceeds 3D LDM Ctrl by 3.28 dB on average, and the two models differ in transport, backbone, autoencoder and objective (Appendix H.4).
- Model size. The 3D LDM's denoiser has 82M parameters and SiT-B has 197M. Growing our backbone to 918M adds only 0.58 dB on average, so size likely explains a small part of the 3.28 dB.
- Conditioning mechanism. On the eight slice-conditioned settings, SiT-B Ctrl and SiT-B Inpaint differ by just −0.18 to +0.26 dB.
- Training objective. Representation alignment substantially cuts the GPU hours needed to reach similar quality (Appendix H.4), and Q2 below covers the cross-dimensional term.
- Joint tokenization. At equal backbone, data and budget, SynOphsis-B matches the task-specific SiT-B Inpaint (+0.03 dB over 14 conditions) and also generates the en-face image. Given an en-face image, SynOphsis-B generates a volume that agrees with the image at AUROC 0.729, against 0.657 for SiT-B Inpaint (Table 26).
The volumetric SiT design therefore brings the accuracy, and the joint tokens bring one model for every task and closer agreement with a given image at no cost in reconstruction accuracy. As the review notes, "the broader generation interface can be obtained without a substantial reconstruction penalty on the evaluated tasks".
Revision: Sections 1 to 3 and 5.2, Table 1, Appendix H.4 and Appendix K.
Q2. Ablation of the cross-dimensional alignment loss
Thank you for this suggestion. We appreciate the chance to measure the contribution of each component, and we are happy to provide more evidence. In our early experiments the term quickly improved the agreement between a generated volume and its en-face image (Appendix A), so we made the term part of the default objective. The ablation sets and keeps the joint tokens and the other loss terms unchanged. Given the reviewer's interest, we have continued training this ablation and will do our best to report the result on generation quality and 3D–2D correspondence during the discussion.
Revision: Appendix A describes the early experiments. If the ablation finishes in time, we will post it in this thread.
We thank the reviewer again for these constructive questions and would be glad to discuss any point further.
Thank you for the responses and additional experiments.
I noticed that adjacent-slice differences are nearly identical within and across groups in the original volumes, but a noticeable gap appears after VAE reconstruction. This suggests that independent group-wise encoding affects continuity differently within and across groups, which is an interesting finding.
For methodological novelty, it would be helpful to briefly distinguish existing techniques adapted in this work from those newly proposed.
Since the authors highlight the role of cross-dimensional alignment in 3D–2D consistency, I would also appreciate any available ablation results supporting its contribution.
Overall, the responses have addressed some of my concerns. I would therefore maintain my original score.
Response to Reviewer dVPe (1/2): Anatomical Consistency and VAE Group Boundaries
Thank you for the detailed review and the concrete questions. We are glad that the reviewer found the joint representation of 3D volumes and paired 2D views "particularly interesting because the two outputs differ in both dimensionality and acquisition geometry". We also appreciate the view that the unified interface "could substantially simplify multimodal generative modeling and provide a useful foundation for subsequent research". This response covers the anatomical consistency of the generated pairs (W1, Q1) and the continuity at VAE group boundaries (Q3). Our second response covers the novelty and the source of the gains (W2) and the ablation of the cross-dimensional loss (Q2).
W1, Q1. Anatomical consistency beyond the projection retrieval of Table 16 (now Table 25)
Thank you for this question. We agree that the anatomical consistency of the two outputs should be shown directly and for each eye, because practical use depends on this consistency. The revision therefore adds a direct, eye-by-eye test of structural correspondence (Appendix E.2).
Table 16 of the submission (now Table 25) compares image intensities. The new test compares anatomy directly and uses the retinal vessels, because both views show them. For each eye, the test places the volume's en-face projection on the en-face image at the scan position taken from the DICOM headers and compares the two vessel maps. AUROC then scores how reliably each en-face image agrees more closely with its own volume than with the volumes of other eyes from the same device and scan region. On acquired pairs the AUROC is 0.973, and the eye's own volume ranks first for 79.2% of en-face images. The test therefore tells individual eyes apart.
On 178 held-out eyes, pairs that SynOphsis-XL generates together reach AUROC 0.896 (chance 0.5; Table 26). Given a volume, SynOphsis-XL generates an en-face image that agrees with the volume at AUROC 0.955. Given an en-face image, SynOphsis-B generates a volume that agrees with the image at AUROC 0.729. SiT-B Inpaint receives the en-face image through cross-attention and reaches 0.657 with the same backbone, data and budget. The two models differ mainly in how the given image enters, so the joint sequence and the loss terms it allows make the generated volume agree more closely with the given en-face image.
Figures 16 to 18 show four eyes per device for joint generation and for each conditional direction. In jointly generated pairs, the optic disc, the fovea and the vessel arcades lie at the same positions in both outputs, as in acquired pairs. Given an en-face image, the generated volume places these structures where the image has them. Given a volume, the generated en-face image follows the vessels and lesions of the given volume. For pathology, Figure 19 shows four eyes per direction. In each eye, one lesion is marked in the en-face image, in the volume's projection and in the B-scan through the lesion. These measurements and cases show that a pair generated by SynOphsis depicts one eye.
Revision: new Section 5.3, Appendix E.2, Table 26 and Figures 16 to 19.
Q3. Structural continuity at VAE group boundaries
Thank you for this insightful question. We ran both proposed analyses on 100 held-out volumes per protocol before and after the VAE round trip.
Group boundaries are therefore not a problem. The unexpected smoothing within each group may point to a limitation of the VAE, and we thank the reviewer for bringing it to light.
Revision: new paragraph in Appendix H.2, Figures 30 and 31.