acceptodds
Desk rejected at ICLR 2027

SynOphsis: Unified Ophthalmic Image and Volume Generation Across Modalities and Dimensions

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

An ophthalmic examination records several 2D modalities, a volumetric OCT scan at one of two slab thicknesses, the en-face image registered to it, and device and free-text clinical metadata. Assessment rests on more than one of them. Generative models of this evidence are built one modality or one dimensionality at a time, so covering an examination takes a collection of models with no shared representation, and not even the volume and the en-face image acquired with it are generated together. We present SynOphsis, a unified generative model for ophthalmic imaging. A frozen autoencoder maps every input into one latent space and per-shape patch embedders place images and volumes on a common token grid, so a volume and the en-face view acquired with it occupy one self-attention sequence, which a single stochastic-interpolant transformer denoises. Conditioning is a mask over that sequence, so any subset of the evidence in either direction is the same operation on the same weights, and one set of weights covers eleven 2D modalities, all three volume protocols, sparse-observation completion, and generation conditioned on a paired image, modality and device, with text conditioning shown qualitatively. The pair is sampled along a single trajectory and either half may be generated from the other, a mode no prior ophthalmic generator provides. One checkpoint serves every mode we evaluate, with no task-specific head and no fine-tuning, while each conditioning baseline carries a mechanism built for one task family and several modes are outputs those architectures cannot emit. On the tasks those specialists do support, the broader interface keeps their volumetric reconstruction accuracy, and joint tokenization costs under 0.01% of the parameters. Separately, our SiT-based 3D stack improves over the retrained volumetric latent-diffusion stack by 3.28 dB PSNR under matched observation conditioning, and on held-out OCT volumes SynOphsis-XL takes FVD from that baseline's 56.2 to 32.9, against a floor of 29.5 measured between two disjoint halves of the real data.

Primary area:
generative models
Keywords:
ophthalmic imaging, medical image generation, diffusion models, stochastic interpolants, optical coherence tomography, multimodal generation, 3D generation, cross-dimensional generation, conditional generation, representation alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dVPeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SynOphsis, a unified generative framework for ophthalmic imaging across multiple modalities and dimensionalities. The proposed method combines a frozen VAE, shape-specific patch embeddings, and a shared stochastic-interpolant transformer to model 2D images, 3D OCT volumes, and paired 3D–2D acquisitions within a common token sequence. A unified masking mechanism enables different conditional and unconditional generation tasks without task-specific fine-tuning.

Critical strengths
  1. The paper formulates heterogeneous ophthalmic generation as masked completion over a shared token sequence. This allows one model to accommodate multiple imaging modalities, dimensionalities, and conditioning configurations without introducing task-specific architectures (Secs. 3.1–3.2; Table 1). The joint representation of 3D volumes and paired 2D views is particularly interesting because the two outputs differ in both dimensionality and acquisition geometry. This unified interface could substantially simplify multimodal generative modeling and provide a useful foundation for subsequent research.
  2. The experiments span multiple imaging modalities, volumetric acquisition protocols, and conditional generation tasks. On internal OCT volumes, SynOphsis-XL achieves an FVD of 32.9 compared with 56.2 for the retrained 3D LDM baseline (Table 2). More importantly, the B-scale unified model maintains comparable conditional reconstruction accuracy to the task-specific SiT Inpaint variant, with an average PSNR difference of only +0.03 dB across fourteen shared conditions (Sec. 5.2; Table 3). These results suggest that the broader generation interface can be obtained without a substantial reconstruction penalty on the evaluated tasks.
Critical weaknesses
  1. One of the main contributions is the joint generation of 3D OCT volumes and their corresponding 2D en-face images. However, generating both modalities through the same transformer does not necessarily mean that their anatomical structures are consistent. Appendix C.3 provides a correspondence evaluation, but the authors acknowledge that the results demonstrate agreement at the aggregate level rather than for individual eyes. Since anatomical consistency is important for the practical use of generated medical images, I would like to see stronger evidence that the generated 2D and 3D pairs represent consistent anatomical structures.
  2. While SynOphsis presents a useful unified framework, its methodological novelty could be further clarified, as it largely builds upon established techniques, including VAE, SiT, masked conditioning, and REPA-style representation alignment. Although the paper includes comparisons with similarly sized SiT variants, the improvements over the 3D LDM baseline involve simultaneous changes in architecture, training objectives, latent representations, and model capacity. Therefore, it remains unclear to what extent these improvements can be attributed to the proposed methodological designs rather than the stronger backbone or increased model capacity.
Questions for authors
  1. The paper reports reconstruction results for 3D volumes (Table 3) and paired 2D images (Table 4), but these metrics evaluate the outputs separately. Could the authors provide more direct evidence that the jointly generated 2D and 3D images preserve consistent anatomical structures? In particular, can the authors evaluate structural correspondence beyond the projection-based retrieval results in Table 16?
  2. Could the authors report an ablation that removes the cross-dimensional alignment loss while keeping the joint tokenization and other training settings unchanged? It would be useful to see how this affects generation quality and the 3D–2D correspondence results in Table 16.
  3. Table 7 reports the reconstruction quality of the frozen VAE on OCT volumes. Could the authors further investigate whether independently encoding groups of three B-scans introduces structural discontinuities at group boundaries? For example, an analysis of adjacent-slice differences or anatomical boundary continuity before and after VAE reconstruction would be informative.
6 comments
AuthorsOpenReview ↗

Response to Reviewer dVPe (1/2): Anatomical Consistency and VAE Group Boundaries

Thank you for the detailed review and the concrete questions. We are glad that the reviewer found the joint representation of 3D volumes and paired 2D views "particularly interesting because the two outputs differ in both dimensionality and acquisition geometry". We also appreciate the view that the unified interface "could substantially simplify multimodal generative modeling and provide a useful foundation for subsequent research". This response covers the anatomical consistency of the generated pairs (W1, Q1) and the continuity at VAE group boundaries (Q3). Our second response covers the novelty and the source of the gains (W2) and the ablation of the cross-dimensional loss (Q2).

W1, Q1. Anatomical consistency beyond the projection retrieval of Table 16 (now Table 25)

Thank you for this question. We agree that the anatomical consistency of the two outputs should be shown directly and for each eye, because practical use depends on this consistency. The revision therefore adds a direct, eye-by-eye test of structural correspondence (Appendix E.2).

Table 16 of the submission (now Table 25) compares image intensities. The new test compares anatomy directly and uses the retinal vessels, because both views show them. For each eye, the test places the volume's en-face projection on the en-face image at the scan position taken from the DICOM headers and compares the two vessel maps. AUROC then scores how reliably each en-face image agrees more closely with its own volume than with the volumes of other eyes from the same device and scan region. On acquired pairs the AUROC is 0.973, and the eye's own volume ranks first for 79.2% of en-face images. The test therefore tells individual eyes apart.

On 178 held-out eyes, pairs that SynOphsis-XL generates together reach AUROC 0.896 (chance 0.5; Table 26). Given a volume, SynOphsis-XL generates an en-face image that agrees with the volume at AUROC 0.955. Given an en-face image, SynOphsis-B generates a volume that agrees with the image at AUROC 0.729. SiT-B Inpaint receives the en-face image through cross-attention and reaches 0.657 with the same backbone, data and budget. The two models differ mainly in how the given image enters, so the joint sequence and the loss terms it allows make the generated volume agree more closely with the given en-face image.

Figures 16 to 18 show four eyes per device for joint generation and for each conditional direction. In jointly generated pairs, the optic disc, the fovea and the vessel arcades lie at the same positions in both outputs, as in acquired pairs. Given an en-face image, the generated volume places these structures where the image has them. Given a volume, the generated en-face image follows the vessels and lesions of the given volume. For pathology, Figure 19 shows four eyes per direction. In each eye, one lesion is marked in the en-face image, in the volume's projection and in the B-scan through the lesion. These measurements and cases show that a pair generated by SynOphsis depicts one eye.

Revision: new Section 5.3, Appendix E.2, Table 26 and Figures 16 to 19.

Q3. Structural continuity at VAE group boundaries

Thank you for this insightful question. We ran both proposed analyses on 100 held-out volumes per protocol before and after the VAE round trip.

  • Adjacent-slice differences. Across a group boundary, reconstructed neighbors differ by no more than acquired neighbors do (mean absolute difference 0.033 against 0.034 on thin volumes and 0.018 against 0.022 on thick). Interestingly, reconstructed neighbors inside a group differ less than acquired neighbors do (0.022 against 0.034, and 0.010 against 0.021).
  • Anatomical continuity. We aligned the depth profiles of adjacent B-scans column by column to measure how far the retina shifts between them. After the round trip, the retina shifts 0.12 pixels more between B-scans across a boundary than inside a group on thin volumes, and 0.07 pixels more on thick (about 0.9 and 0.7 µm). For the inner limiting membrane and the retinal pigment epithelium, the added shift is 2 to 4 µm. All of these added shifts lie below one pixel of the depth grid and below the 5 to 8 µm axial resolution of the scanners.

Group boundaries are therefore not a problem. The unexpected smoothing within each group may point to a limitation of the VAE, and we thank the reviewer for bringing it to light.

Revision: new paragraph in Appendix H.2, Figures 30 and 31.

AuthorsOpenReview ↗

Response to Reviewer dVPe (2/2): Novelty, Source of the Gains and Ablation of the Cross-Dimensional Loss

W2. Methodological novelty and the source of the gains over the 3D LDM

Thank you for this comment. We appreciate the chance to clarify both the novelty and the source of the gains. We see the main novelty of SynOphsis in a new formulation and in a pipeline designed for consistent joint generation across dimensionalities and modalities. The formulation answers a challenging clinical need and grows out of the idea of improving pre-training and its generalization by unifying as many modalities as possible. This formulation provides an exciting opportunity to train one model that flexibly generates the images of an eye examination together across dimensionalities (Contribution 1). Unlike a slice of a CT or MRI volume, an en-face retinal image lies in the plane orthogonal to every B-scan and on a different grid. The en-face image is therefore not one more frame of the volume.

SynOphsis writes the volume and its en-face image into one self-attention sequence on a 3D-aware SiT (Contribution 2) and conditions by masking that sequence. To tie the volume to its en-face image across their orthogonal planes, a cross-dimensional term that REPA lacks aligns the volume's depth-averaged features with the features of the paired image (Section 3.3). With this reformulation, it is STRAIGHTFORWARD and actually NATURAL to combine the VAE, SiT, masked conditioning and REPA-style alignment named in the review. As a working example, SynOphsis shows the benefit of embracing a more unified pre-training approach. In the words of the review, the interface could "provide a useful foundation for subsequent research", and we regard the formulation as a necessary stepping stone toward unified clinical foundation models.

For the source of the gains, most of the accuracy gain over the 3D LDM comes from our volumetric SiT design. Joint tokenization adds the en-face output at no cost in reconstruction accuracy and brings the generated volume closer to a given en-face image. No existing baseline performs all the tasks of SynOphsis, so each task is compared against its closest baseline. The need for a separate baseline per task also reflects how new the unified setting is. Internal SiT-B variants with our backbone, data and budget separate these sources (Section 5.2).

  • Volumetric SiT design (Contribution 2). Under the same ControlNet-style conditioning, SiT-B Ctrl exceeds 3D LDM Ctrl by 3.28 dB on average, and the two models differ in transport, backbone, autoencoder and objective (Appendix H.4).
  • Model size. The 3D LDM's denoiser has 82M parameters and SiT-B has 197M. Growing our backbone to 918M adds only 0.58 dB on average, so size likely explains a small part of the 3.28 dB.
  • Conditioning mechanism. On the eight slice-conditioned settings, SiT-B Ctrl and SiT-B Inpaint differ by just −0.18 to +0.26 dB.
  • Training objective. Representation alignment substantially cuts the GPU hours needed to reach similar quality (Appendix H.4), and Q2 below covers the cross-dimensional term.
  • Joint tokenization. At equal backbone, data and budget, SynOphsis-B matches the task-specific SiT-B Inpaint (+0.03 dB over 14 conditions) and also generates the en-face image. Given an en-face image, SynOphsis-B generates a volume that agrees with the image at AUROC 0.729, against 0.657 for SiT-B Inpaint (Table 26).

The volumetric SiT design therefore brings the accuracy, and the joint tokens bring one model for every task and closer agreement with a given image at no cost in reconstruction accuracy. As the review notes, "the broader generation interface can be obtained without a substantial reconstruction penalty on the evaluated tasks".

Revision: Sections 1 to 3 and 5.2, Table 1, Appendix H.4 and Appendix K.

Q2. Ablation of the cross-dimensional alignment loss

Thank you for this suggestion. We appreciate the chance to measure the contribution of each component, and we are happy to provide more evidence. In our early experiments the term quickly improved the agreement between a generated volume and its en-face image (Appendix A), so we made the term part of the default objective. The ablation sets λcross=0\lambda_{cross}=0 and keeps the joint tokens and the other loss terms unchanged. Given the reviewer's interest, we have continued training this ablation and will do our best to report the result on generation quality and 3D–2D correspondence during the discussion.

Revision: Appendix A describes the early experiments. If the ablation finishes in time, we will post it in this thread.

We thank the reviewer again for these constructive questions and would be glad to discuss any point further.

Thank you for the responses and additional experiments.

I noticed that adjacent-slice differences are nearly identical within and across groups in the original volumes, but a noticeable gap appears after VAE reconstruction. This suggests that independent group-wise encoding affects continuity differently within and across groups, which is an interesting finding.

For methodological novelty, it would be helpful to briefly distinguish existing techniques adapted in this work from those newly proposed.

Since the authors highlight the role of cross-dimensional alignment in 3D–2D consistency, I would also appreciate any available ablation results supporting its contribution.

Overall, the responses have addressed some of my concerns. I would therefore maintain my original score.

Reviewer_AHoBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on ophthalmic imaging and presents a unified generative model. To achieve this, the authors consider a stochastic interpolant transformer over modality-aware tokens, and a masking interface for finding evidence. Experiments show the advantages of the proposed method.

Critical strengths
  1. The topic is interesting, and a unified generative model for ophthalmology sounds interesting for real-world.
  2. The composition is extensive, including UNCONDITIONAL GENERATION and CONDITIONAL GENERATION.
Critical weaknesses
  1. Hard to understand. The writing is unacceptable, as it is hard for readers to follow this work.
  2. Limited novelty, as the different of the proposed method over existing methods is marginal.
Questions for authors
  1. Hard to understand. The writing is unacceptable, as it is hard for readers to follow this work.
  2. Limited novelty, as the different of the proposed method over existing methods is marginal.
4 comments
AuthorsOpenReview ↗

Response to Reviewer AHoB (1/2): Novelty Relative to Existing Methods

Thank you for the review. We are glad that the reviewer found the topic interesting and that "a unified generative model for ophthalmology sounds interesting for real-world" use. We are also grateful for the note that the experiments are extensive, from unconditional to conditional generation. The review is concerned with the novelty relative to existing methods (W2) and with the clarity of the writing (W1). This comment answers the novelty, and the next comment answers the clarity.

W2. Novelty relative to existing methods

Thank you for this comment. We appreciate the chance to set out the novelty more fully. We see the main novelty of SynOphsis in a new formulation and in a pipeline designed for consistent joint generation across dimensionalities and modalities. The formulation answers a challenging clinical need and grows out of the idea of improving pre-training and its generalization by unifying as many modalities as possible. This formulation provides an exciting opportunity to train one model that flexibly generates the images of an eye examination together across dimensionalities (Contribution 1).

The closest existing methods come from several lines of work, and to our knowledge none of them samples an OCT volume together with its en-face image (Section 2 and Appendix I).

  • General domain. The text-to-RGBD model LDM3D (Ben Melech Stan et al., 2023) and MVDream (Shi et al., 2024) generate several outputs together, and these outputs share one geometry. A depth map is registered to its image pixel for pixel, and the views of an object render one scene at known poses. FLUX3D (Ji et al., 2026) mixes 2D and 3D tokens in one sequence, but its 2D side serves only as the condition. Omni123 (Ye et al., 2026) generates image and 3D tokens one after another in one autoregressive sequence, so a shape that follows an image is conditioned on that image. An OCT volume and its en-face image describe the same eye on different grids, and SynOphsis can sample both together along one trajectory (Appendix I).
  • Joint generation in medical imaging. MedSyn (Xu et al., 2024) and DualDiT (García-Torres et al., 2026) sample several outputs at once, but each output lies on the grid of one scan, as an annotation or another contrast.
  • Ophthalmic and cross-dimensional generation. Ophthalmic models such as EyeDiff (Chen et al., 2026) and CA-LDM (Huang et al., 2024) cover 2D modalities or OCT volumes separately. De Vente et al. (2024) condition on the en-face image to upsample a volume, and other models generate a volume from a fundus photograph. In both cases the 2D image is given.

As a result, each generation mode needs its own baseline (Table 1). No conditioning module exists for a volumetric SiT, so we built two conditional baselines ourselves (SiT-B Ctrl and SiT-B Inpaint). An eye examination routinely acquires anatomically consistent images of several modalities, and handling them one at a time wastes this natural strength.

SynOphsis instead writes the volume and its en-face image into one self-attention sequence on a 3D-aware SiT and conditions by masking that sequence (Contribution 2). With this reformulation, it is STRAIGHTFORWARD and actually NATURAL to build the model from established techniques proven effective in the general machine learning domain. As a working example, SynOphsis shows the benefit of embracing a more unified pre-training approach. We regard this formulation as a necessary stepping stone toward the large-scale training of unified clinical foundation models. The measured gains show a much stronger pre-trained model for ophthalmology (Contribution 3).

  • On held-out internal OCT volumes, SynOphsis-XL lowers FVD from 56.2 for the retrained 3D LDM to 32.9 (Table 2).
  • In conditional generation, SynOphsis-B exceeds the 3D LDM by 3.39 dB PSNR on average and matches the task-specific SiT-B Inpaint with the same backbone, data and budget (+0.03 dB; Table 3).
  • A pair that SynOphsis-XL generates together shows the same eye, with AUROC 0.896 on 178 held-out eyes (acquired pairs 0.973, chance 0.5; Table 26).
  • Classifiers trained only on SynOphsis-XL volumes match classifiers trained on as many acquired volumes in five of six evaluations (Section 5.3).
  • Fine-tuned with the same tokens and masks, SynOphsis-B has a significantly higher PSNR than a 2D LDM in all 61 one-to-one directions between 2D modalities (Appendix F).

Together these results support the novelty of the formulation and its value for ophthalmic generation. If the reviewer has a specific method in mind, we would be very happy to compare SynOphsis with that method during the discussion.

Revision: Section 1 (contributions), Section 2, Table 1, Section 5 and Appendices E to G and I.

AuthorsOpenReview ↗

Response to Reviewer AHoB (2/2): Clarity of the Writing

W1. Clarity of the writing

We greatly appreciate this candid feedback. We substantially revised the writing and kept the main content and every table. The main text now fills the ten pages allowed during the discussion and carries substantial new results. Every section, figure and table of the submission remains, and the numbers in Tables 2 to 4 are unchanged. The introduction now lists the contributions clearly, and Table 1 gives each generation mode's input and output. Each results subsection now opens by stating the hypothesis under test. We hope these changes make the manuscript easier and more rewarding to read.

As a starting point we would suggest the introduction, Section 2 and the new Section 5.3. If any passage remains hard to follow, we would be glad to revise that passage before the discussion closes. We are also happy to provide a list of the main additions and their locations if the reviewer or the AC finds such a list useful.

Revision: throughout the manuscript. The main additions are Section 5.3 and Appendices E.2, F and G.

We thank the reviewer again and would be glad to discuss any point further.

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear reviewer AHoB:

Thank you again for the review. As the discussion period closes on November 18, we wanted to check whether our responses address the two concerns.

For the novelty, our first response sets out the formulation, compares it with the closest lines of work and lists the measured gains. No published method covers more than three of the seven generation modes of SynOphsis (Table 1), and on held-out internal OCT volumes SynOphsis-XL lowers FVD from 56.2 for the retrained 3D LDM to 32.9. For the writing, we revised the main text throughout and kept every section, figure and table, and our responses give the location of each change in the revised PDF. If a specific method would help the comparison, or a specific passage remains hard to follow, we would be glad to address that point before the discussion closes. If instead our responses address both concerns, we would kindly ask the reviewer to consider updating the score.

Best regards,

The Authors of Submission 58787

Dear reviewer AHoB:

Thank you again for the careful review of our work.

As the discussion period closes in a few hours, at the end of November 18 (AoE), we would be grateful if the reviewer could briefly revisit our responses and the revised PDF. Our responses and the revision answer both concerns, on the novelty and on the clarity of the writing.

If a specific passage or a specific existing method still raises a concern, we would be glad to address it before the discussion closes. If the concerns are addressed, it would be much appreciated if the reviewer could consider updating the score.

Thank you for your time and consideration.

Best regards,

The Authors of Submission 58787

Reviewer_dkcDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SynOphsis, a unified generative framework for ophthalmic imaging. The main idea is to use a single model to generate different types of eye images, including both 2D images and 3D OCT volumes. The framework also supports conditional generation, such as completing an OCT volume from a few observed slices or generating paired 2D and 3D images together.

Critical strengths

Unifying different ophthalmic imaging modalities within a single generative framework is valuable, as existing methods often focus on individual modalities or tasks. The authors evaluate the method across different ophthalmic modalities and generation settings. The results show promising generation quality, especially for OCT volumes. The detailed comparisons and additional analyses are also appreciated.

Critical weaknesses
  1. Most 2D modalities are mainly evaluated through unconditional generation, while joint generation focuses on OCT volumes and their paired 2D images. I would like to see more experiments on conditional generation between different 2D modalities. This would better demonstrate the generality of the proposed framework.
  2. The consistency between generated 2D and 3D images needs stronger validation. This is one of the key contributions of the paper. Although Appendix C.3 provides a paired-view consistency analysis, the results mainly show agreement at the overall level. I am not fully convinced that the generated images preserve the same anatomical structures and pathological findings.
  3. The paper reports notable improvements over the LDM baselines. However, as the authors also discuss, some of these improvements come from changes to the overall generation architecture rather than joint tokenization itself.
  4. The paper mainly relies on image quality metrics such as FID, FVD, and PSNR. While these metrics are useful, they do not necessarily reflect whether the generated images are medically meaningful. I appreciate that the authors acknowledge this limitation, but a clinical reader study or a downstream experiment, such as disease classification or segmentation using synthetic data, would make the work more convincing.
Questions for authors

No other questions.

7 comments
AuthorsOpenReview ↗

Response to Reviewer dkcD (1/2): Generation between 2D Modalities and Same-Eye Consistency

Thank you for the review and the concrete suggestions. We are glad that the reviewer found that "Unifying different ophthalmic imaging modalities within a single generative framework is valuable" and that the results "show promising generation quality, especially for OCT volumes". We are also grateful for the note on the detailed comparisons and additional analyses. We see four points in the review that would make the work more convincing. This response covers conditional generation between 2D modalities (W1) and the consistency of the generated 2D and 3D images (W2), and our second response covers the source of the gains (W3) and medical meaningfulness (W4). The revision answers each point with new results.

W1. Conditional generation between 2D modalities

Thank you for this insightful suggestion. We agree, and generation between 2D modalities is a use that SynOphsis naturally enables. The revision adds this experiment, and the same tokens and masks carry over to it with no change to the architecture. We fine-tuned SynOphsis-B on AI-READI. There an eye has up to seven kinds of 2D image at the macula and six around the optic disc, and the images come from four fundus cameras and three infrared devices. Given any subset of an eye's 2D images, the fine-tuned model generates the rest. As a baseline, we fine-tuned the conditional 2D LDM of the main comparisons on the same data.

SynOphsis-B has a significantly higher PSNR than the 2D LDM in all 61 one-to-one directions, and every 95% interval of the difference lies above zero (Table 27, Figure 21). In the examples of Figure 20, SynOphsis-B also keeps the vessels and the optic disc of the given eye more faithfully than the 2D LDM. Generating four images at once changes PSNR by at most 0.14 dB against generating each alone (Table 28), whereas the 2D LDM generates one target per run. The unified interface therefore carries over to new combinations of 2D modalities, and this result is an early sign that SynOphsis can be easily fine-tuned for new tasks.

Revision: new Appendix F, referenced from Section 5.2. Tables 27 and 28 and Figure 21 give the quantitative results, and Figure 20 gives qualitative examples.

W2. Anatomical and pathological consistency between the generated 2D and 3D images

Thank you for raising this important point. We agree that this consistency is central to the paper and should be shown for each eye. The revision therefore adds an eye-by-eye test beyond the aggregate agreement of Appendix C.3 of the submission (now Appendix E.1).

The new test (Appendix E.2) places the volume's en-face projection on the en-face image at the scan position taken from the DICOM headers and compares their vessel maps. Each en-face image then ranks its own volume among the other eyes' volumes from the same device and scan region. On acquired pairs, the test reaches AUROC 0.973 and ranks the eye's own volume first for 79.2% of en-face images, so it identifies single eyes. On 178 held-out eyes, pairs that SynOphsis-XL generates together reach AUROC 0.896 (chance 0.5; Table 26, also in our General Response).

New case figures show four eyes per device for joint generation and for each conditional direction (Figures 16 to 18). In joint generation the optic disc, the fovea and the vessel arcades fall at the same place in both outputs, as they do in acquired pairs. A volume generated from a given en-face image places these structures where the image has them, and an en-face image generated from a given volume follows the vessels and lesions of that volume. For pathology, Figure 19 covers joint generation and both conditional directions. In each eye shown, one lesion is marked in the en-face image, in the volume's projection and in the B-scan through the lesion. W4 in our second response adds quantitative evidence for pathology, since diagnostic models trained on acquired volumes recognize the prompted disease in generated volumes. Together these results show that a pair generated by SynOphsis describes one eye.

Revision: new Section 5.3, Appendix E.2, Table 26 and Figures 16 to 19.

AuthorsOpenReview ↗

Response to Reviewer dkcD (2/2): Source of the Gains and Clinical Utility

W3. Contribution of joint tokenization to the gains over the LDM baselines

Thank you for raising this. In short, most of the accuracy gain over the 3D LDM comes from our volumetric SiT design, and joint tokenization adds the en-face output at no cost in reconstruction accuracy and brings the generated volume closer to a given en-face image. Both are designs of this paper (Contribution 2). No existing baseline performs all the tasks of SynOphsis, so each task is compared against its closest baseline. The need for a separate baseline per task also reflects how new the unified setting is. Internal SiT-B variants with our backbone, data and training budget separate these sources (Section 5.2).

  • Volumetric SiT design. Under the same ControlNet-style conditioning, SiT-B Ctrl exceeds 3D LDM Ctrl by 3.28 dB on average. SiT-B Ctrl and the 3D LDM differ in transport, backbone, autoencoder and objective, and Appendix H.4 lists each component.
  • Model size. The 3D LDM's denoiser has 82M parameters and SiT-B has 197M. Growing our backbone to 918M adds only 0.58 dB on average, so size likely explains a small part of the 3.28 dB.
  • Conditioning mechanism. On the eight slice-conditioned settings, SiT-B Ctrl and SiT-B Inpaint differ by just −0.18 to +0.26 dB.
  • Joint tokenization. At equal backbone, data and budget, SynOphsis-B matches the task-specific SiT-B Inpaint (+0.03 dB over 14 conditions) and also generates the paired en-face image. Given an en-face image, SynOphsis-B generates a volume that agrees with the image at AUROC 0.729. SiT-B Inpaint receives the image through cross-attention and reaches 0.657 (Table 26).

The volumetric SiT design therefore brings the reconstruction accuracy, and the joint tokens bring one model for these tasks and closer agreement with a given image at no cost in reconstruction accuracy. One set of weights holds parity on average over the 14 conditions without task-specific training, and the 2D results in W1 show the same interface extending to new tasks. We expect later work to build on this base, for example segmentation or classification from joint multimodal inputs and more complex cross-modal transfer.

Revision: Section 5.2, Appendix H.4 and Appendix K.

W4. Medical meaningfulness beyond FID, FVD and PSNR

We greatly appreciate this suggestion, since a downstream evaluation was also the next step in our own plan for SynOphsis. The revision adds the proposed experiment. The generated volumes prove useful as training data for disease classification. We train four-class classifiers (normal, drusen, epiretinal membrane, macular edema) by fine-tuning each of three pretrained OCT encoders. We test them on 725 held-out internal volumes from patients in no training set and on the external Duke14 set.

  • Trained only on 100 SynOphsis-XL volumes per class, the classifier reaches the macro AUROC of a classifier trained on 100 acquired volumes per class in five of six evaluations (within −0.021 to +0.018).
  • With five acquired volumes per class, adding SynOphsis-XL volumes raises internal macro AUROC from about 0.59 to 0.76–0.83.
  • Central serous chorioretinopathy and Stargardt disease each served as a fifth class with no acquired training volume. For each, 100 SynOphsis-XL volumes raise its internal AUROC from 0.40–0.74 to 0.89–0.97. The result lies within 0.042 of the AUROC reached with all acquired volumes of that disease.
  • Diagnostic models trained on acquired volumes recognize the prompted disease in SynOphsis-XL volumes at least as often as the recorded diagnosis in held-out acquired volumes (Appendix G.3). One of these models is an OCT vision-language model trained by another group.
  • A glaucoma prompt thins the ganglion cell and inner plexiform layers by 1.95 and 1.70 healthy standard deviations in SynOphsis-XL volumes and by 0.30 and 0.19 in 3D LDM volumes. Acquired glaucoma thins the same two layers by 1.12 and 0.92 (Appendix G.2).
  • The classifier gains also hold on scanners from two other vendors (Appendix G.4).

The generated volumes therefore carry the features of the prompted disease, and in these evaluations they can stand in for acquired training volumes or add to them. A reader study and segmentation are the natural next steps, and Appendix K lists both as future work.

Revision: new Section 5.3 and Appendix G (Figures 22 to 29).

We fully agree that every suggestion has made the manuscript stronger, and we would be glad to discuss any point further.

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear reviewer dkcD:

Thank you again for the review.

As the discussion period closes on November 18, we wanted to check whether our responses address the four points raised.

In brief, the revision adds generation between 2D modalities on AI-READI (W1). There SynOphsis-B has a higher PSNR than a 2D LDM in all 61 one-to-one directions. It also adds an eye-by-eye consistency test and case figures for anatomy and lesions (W2). In the test, jointly generated pairs reach AUROC 0.896 against 0.973 for acquired pairs. Section 5.2 now separates the gains over the 3D LDM (W3), and classifiers trained only on generated volumes match classifiers trained on as many acquired volumes in five of six evaluations (W4). Our responses give the location of each change in the revised PDF. If they address the four points, we would kindly ask the reviewer to consider updating the score. We would also be glad to clarify any remaining point before the discussion closes.

Best regards,

The Authors of Submission 58787

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (3/3): Generation between 2D Modalities, Cross-Dimensional Loss Ablation, VAE Group Boundaries, Clarity and Summary Tables

Summary of the concerns and our responses (continued)

5. Conditional generation between 2D modalities (dkcD-W1). Reviewer dkcD asked for conditional generation between 2D modalities to show the generality of the framework. We agree and appreciate this suggestion. Generation between 2D modalities is a use that SynOphsis naturally enables.

Specifically, we fine-tuned SynOphsis-B on AI-READI with the same tokens and masks to generate any subset of an eye's 2D images from the others. The fine-tuned model has a significantly higher PSNR than a 2D LDM fine-tuned on the same data in all 61 one-to-one directions. Every 95% interval of the difference lies above zero. In the examples of Figure 20, SynOphsis-B also keeps the vessels and the optic disc of the given eye more faithfully than the 2D LDM (Appendix F).

6. Ablation of the cross-dimensional alignment loss (dVPe-Q2). Reviewer dVPe asked for an ablation that removes this loss and keeps everything else. We appreciate this suggestion to measure the contribution of each component, and we are happy to provide more evidence. In our early experiments the term quickly improved the agreement between a generated volume and its en-face image (Appendix A), so we made the term part of the default objective. The ablation keeps the joint tokens and the other loss terms unchanged. Given the reviewer's interest, we have continued training this ablation and will do our best to report the result during the discussion.

7. Continuity at VAE group boundaries (dVPe-Q3). Reviewer dVPe asked whether encoding three B-scans together leaves discontinuities where two groups meet. To answer the question, we compared 100 held-out volumes per protocol before and after the VAE round trip. Across a boundary, reconstructed neighbors differ by no more than acquired neighbors do (mean absolute difference 0.033 against 0.034 on thin volumes and 0.018 against 0.022 on thick), so group boundaries are not a problem. Interestingly, reconstructed neighbors inside a group differ less than acquired neighbors do (Appendix H.2, Figures 30 and 31). This unexpected smoothing within each group may point to a limitation of the VAE. We thank the reviewer for this insightful question.

8. Clarity of the writing (AHoB-W1). Reviewer AHoB found the paper hard to follow. We greatly appreciate this candid feedback. We substantially revised the writing and kept the main content and every table. The main text now fills the ten pages allowed during the discussion and carries substantial new results. Every section, figure and table of the submission remains, and the numbers in Tables 2 to 4 are unchanged. The introduction now lists the contributions clearly, and Table 1 gives each generation mode's input and output. Each results subsection now opens by stating the hypothesis under test. We hope these changes make the manuscript easier and more rewarding to read. We are also happy to provide a list of the main additions and their locations if the reviewers or the AC finds such a list useful.

Same-eye consistency on 178 held-out eyes (Table 26)

SettingModelAUROC
Acquired pair0.973
Joint, no image givenSynOphsis-XL0.896
2D→3DSynOphsis-B0.729
2D→3DSiT-B Inpaint0.657
2D→3D3D LDM Cross0.643
Chance0.500

Clinical utility of SynOphsis-XL volumes (Section 5.3, Appendix G)

TestSynOphsis-XLComparisonFigure
Generated volumes alone (100 per class) against 100 acquired per class, macro AUROC difference−0.021 to +0.018 in 5 of 6 evaluations3D LDM volumes reach the level of 5 to 25 acquired per class5
Gain in macro AUROC from adding generated volumes to 5 acquired per class+0.18 to +0.25 internal, +0.30 to +0.39 Duke140.08 to 0.13 and 0.11 to 0.21 above the gain from 3D LDM volumes5, 24
Fifth disease (CSC or Stargardt) with no acquired volume, internal AUROC0.40–0.74 → 0.89–0.97within 0.042 of training on all its acquired volumes27
Thinning of the ganglion cell and inner plexiform layers under a glaucoma prompt, in healthy SDs1.95 and 1.701.12 and 0.92 in acquired glaucoma, 0.30 and 0.19 for the 3D LDM22

Numbering changes from the submission. The submission's Appendix C.3 is now Appendix E.1, and its Tables 7 and 16 are now Tables 8 and 25.

The revision addresses every concern raised, and we welcome further discussion during the rest of the period.

AuthorsOpenReview ↗

General Response (2/3): Source of the Gains and Novelty

Summary of the concerns and our responses (continued)

3. Source of the gains over the 3D LDM (dkcD-W3; dVPe-W2). Reviewers dkcD and dVPe asked how much of the gain comes from joint tokenization and how much from the architecture, objective and capacity. We thank both reviewers for this question. In short, most of the accuracy gain comes from our volumetric SiT design, and joint tokenization adds the en-face output at no cost in reconstruction accuracy and brings the generated volume closer to a given en-face image. No existing baseline performs all the tasks of SynOphsis, so each task is compared against its closest baseline. The need for a separate baseline per task also reflects how new the unified setting is. Internal SiT-B variants with our backbone, data and budget separate these sources (Section 5.2).

  • Volumetric SiT design (Contribution 2). Under the same ControlNet-style conditioning, SiT-B Ctrl exceeds 3D LDM Ctrl by 3.28 dB on average, and the two models differ in transport, backbone, autoencoder and objective (Appendix H.4).
  • Model size. The 3D LDM's denoiser has 82M parameters and SiT-B has 197M. Growing our backbone to 918M adds only 0.58 dB on average, so size likely explains a small part of the 3.28 dB.
  • Conditioning mechanism. On the eight slice-conditioned settings, SiT-B Ctrl and SiT-B Inpaint differ by just −0.18 to +0.26 dB.
  • Training objective. Representation alignment substantially cuts the GPU hours needed to reach similar quality (Appendix H.4), and concern 6 covers the cross-dimensional term.
  • Joint tokenization. At equal backbone, data and budget, SynOphsis-B matches the task-specific SiT-B Inpaint (+0.03 dB over 14 conditions) and also generates the en-face image. Given an en-face image, SynOphsis-B generates a volume that agrees with the image at AUROC 0.729, against 0.657 for SiT-B Inpaint (Table 26).

4. Novelty relative to existing methods (AHoB-W2; dVPe-W2). Reviewer AHoB found the difference from existing methods marginal, and Reviewer dVPe asked us to clarify the methodological novelty of a model built on established components. We appreciate the reviewers' feedback and would like to bring a deeper discussion of the novelty to the table.

We see the main novelty of SynOphsis in a new formulation and in a pipeline designed for consistent joint generation across dimensionalities and modalities. This reformulation answers a challenging clinical need and grows out of the idea of improving pre-training and its generalization by unifying as many modalities as possible. This innovation in formulation provides an exciting opportunity to train one model that flexibly generates the images of an eye examination together across dimensionalities (Contribution 1). SynOphsis realizes this model through a straightforward and natural composition of recent advances in generative modeling. As a working example, SynOphsis shows the benefit of embracing a more unified pre-training approach. Such an approach trains one model on the data of every modality with the computation of modern hardware. We regard this formulation and its simple, flexible design as a necessary stepping stone toward the large-scale training of unified clinical foundation models.

Moreover, SynOphsis addresses key challenges of translating emerging generative modeling techniques to a clinical domain with its own complex structure. Unlike a slice of a CT or MRI volume, an en-face retinal image lies in the plane orthogonal to every B-scan and on a different grid. The en-face image is therefore not one more frame of the volume. Existing generative models generate either the volume or the en-face image alone, and conditional models add a module for each kind of input. No published method covers more than three of the seven generation modes of SynOphsis (Table 1). Handling the images separately also wastes a natural strength of the eye examination. The examination routinely acquires anatomically consistent images of several modalities. SynOphsis instead writes the volume and its en-face image into one self-attention sequence on a 3D-aware SiT (Contribution 2) and conditions by masking that sequence. To tie the volume to its en-face image across their orthogonal planes, a cross-dimensional term that REPA lacks aligns the volume's depth-averaged features with the features of the paired image (Section 3.3). With this reformulation, it is STRAIGHTFORWARD and actually NATURAL to combine established techniques proven effective in the general machine learning domain. The result is a much stronger pre-trained model for an important clinical domain. The model generates more consistent pairs (Contribution 2) and has more generalizable clinical capabilities (Contribution 3).

Concerns 5 to 8 and the summary tables follow in the third part of this response.

AuthorsOpenReview ↗

General Response (1/3): Summary of the Revision, Same-Eye Consistency and Clinical Utility

We thank the three reviewers for their careful reading and constructive comments. The reviewers valued the unified framework (Contribution 1). Reviewer dkcD found that "Unifying different ophthalmic imaging modalities within a single generative framework is valuable", and Reviewer AHoB found a unified generative model for ophthalmology interesting for real-world use. Reviewer dVPe noted that this unified interface "could substantially simplify multimodal generative modeling and provide a useful foundation for subsequent research". Reviewer dVPe also found the joint representation of 3D volumes and paired 2D views "particularly interesting because the two outputs differ in both dimensionality and acquisition geometry" (Contribution 2). Reviewer dkcD highlighted the "promising generation quality, especially for OCT volumes", and this quality comes from the 3D-aware SiT of Contribution 2. All three reviewers noted the breadth of the evaluation (Contribution 3).

We took every comment on board. We revised the writing thoroughly. The revision also adds new results on eye-by-eye consistency, on the clinical utility of generated volumes, on conditional generation between 2D modalities and on continuity at VAE group boundaries. Section 5.2 now separates the sources of the gains over the 3D LDM. Below we summarize each concern, what we did and where the revision shows it. Each reviewer's thread answers every point in full.

Summary of the concerns and our responses

1. Consistency between a jointly generated volume and its en-face image (dkcD-W2; dVPe-W1, Q1). Reviewers dkcD and dVPe asked for direct evidence that the two outputs show the same anatomy and pathology, beyond the aggregate agreement of Appendix C.3 of the submission. We agree that consistency should be shown for each eye. We therefore added an eye-by-eye test that compares vessel maps at the scan position taken from the DICOM headers. On 178 held-out eyes, pairs that SynOphsis-XL generates together reach AUROC 0.896 (acquired pairs 0.973, chance 0.5; Table 26). New case figures show four eyes per device for joint generation and for each conditional direction (Figures 16 to 18). In joint generation the optic disc, the fovea and the vessel arcades fall at the same place in both outputs, as they do in acquired pairs. A volume generated from a given en-face image places these structures where the image has them, and an en-face image generated from a given volume follows the vessels and lesions of that volume. To cover pathology as well, Figure 19 also marks lesions in each direction. Each marked lesion lies at the same position in the en-face image and in the volume (Section 5.3, Appendix E.2).

2. Medical value beyond image metrics (dkcD-W4). Reviewer dkcD noted that FID, FVD and PSNR may not reflect whether generated images are medically meaningful and suggested a reader study or a downstream task. We greatly appreciate this comment, since a downstream evaluation was also the next step in our own plan for SynOphsis. We therefore trained disease classifiers on generated volumes and tested them on 725 held-out internal volumes and the external Duke14 set. In five of six evaluations, classifiers trained only on SynOphsis-XL volumes match classifiers trained on as many acquired volumes. With five acquired volumes per class, adding SynOphsis-XL volumes raises internal macro AUROC from about 0.59 to 0.76–0.83. For a disease left out of the acquired training data, 100 SynOphsis-XL volumes raise its AUROC to within 0.042 of the AUROC reached with all acquired volumes of that disease (Section 5.3, Appendix G.4). Appendix G also tests the generated volumes from other angles. Diagnostic models trained on acquired volumes recognize the prompted disease in SynOphsis-XL volumes at least as often as the recorded diagnosis in held-out acquired volumes (Appendix G.3). One of these models is an OCT vision-language model trained by another group. A glaucoma prompt thins the ganglion cell and inner plexiform layers of the generated volumes, as glaucoma does in acquired volumes (Appendix G.2). The classifier gains also hold on scanners from two other vendors (Appendix G.4). The third part of this response tabulates the main numbers.

Concerns 3 to 8 and the summary tables follow in the second and third parts of this response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.