- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses the ill-posed problem of estimating a dense 3D medical image volume from a single 2D image, with application on color fundus photography to optical coherence tomography (OCT) estimation, and synthetic X-ray to computed tomography (CT) estimation. The proposed framework has two stages: 1) 2D-to-3D lifting stage which maps the input image into a 3D volume using TransUpNet; 2) Slice-by-slice perceptual refinement using a 2D flow matching model. Experiment has been performed on two benchmarks including color fundus to OCT benchmark and X-ray to CT benchmark, and results show that the proposed method achieves superior PSNR/SSIM fidelity and significantly lower FID compared to state-of-the-art baselines.
Critical strengths
- The separation of volumetric structural prediction from slice-wise appearance refinement is a natual and reasonable engineering choice.
- The ablation study is clear which investigates refinement steps, Noise Level, noise interpolation, choice of flow-matching network, and refinement-network architecture, and helps identifying which components contribute to the results.
Critical weaknesses
- The main concern is on whether the proposed problem setting (Color Fundus Photography to OCT, X-ray to CT) is suitable for medical imaging applications. The generated volumes' PSNR is low (18.0 to 19.4 as in Table 1), and improved perceptual realism DONOT corresponds to improved recovery of the patient's true unobserved anatomy.
- The performance improvement related with image quality is not very significant as stated in table 1 (i.e. Wasserstein vs. ours @ PSNR: 19.4 vs. 19.4).
Questions for authors
- Is there any planned downstream processing tasks for the generated OCT/CT volumes?
- Similarity transformation is used for alignment between fundus image and OCT, as stated in section 4, how about the sensitivity of generation result to such alignment accuracy.
- The fundus-OCT dataset has 403 pairs for training/test, how about the generalization capability of trained models on this dataset?
3 comments
Thanks the authors for detailed explanation. I still have some concerns including W1 (realism vs. true anatomy): the 'Table 7' in appendix is just a comparision between "TransUpNet + JiT" & "TransUpNet + Song", and the following text of "On the heart, whose size..." is only case study and not in original paper
Thank you for the follow-up. To avoid any confusion about the table references, we reproduce the relevant tables of the revised PDF below. The JiT vs. Song comparison is a backbone ablation (Appendix A.9, Table 12) and is not our evidence for anatomical fidelity. That evidence is Table 7 (Appendix A.4) for CT and Table 1 for OCT. The heart result is not a case study: it is the heart column of Table 7, computed over all 100 CT test cases.
Table 7 (A.4): patient-specific organ volumes recovered from the generated CT (100 test cases). Lungs are segmented with lungmask and the heart with TotalSegmentator, identically on real and generated volumes. MAE: mean absolute error of the per-case volume (mL, mean ± SD). Bias: mean signed error (mL). r: Pearson correlation with the volume on the real CT. p: paired Wilcoxon signed-rank test of the per-case absolute errors against the mean-volume baseline, which predicts the leave-one-out mean real volume for every case (mean real volume: lungs 4633 mL, heart 524 mL). †: significantly worse than the baseline.
| Method | Lung MAE ↓ | Lung bias | Lung r ↑ | Lung p | Heart MAE ↓ | Heart bias | Heart r ↑ | Heart p |
|---|---|---|---|---|---|---|---|---|
| Flow (normal noise) | 162 ± 158 | −69 | 0.98 | 6e-17 | 67 ± 66 | −23 | 0.75 | 6e-5 |
| Flow (slerp noise) | 163 ± 159 | −67 | 0.98 | 8e-17 | 67 ± 67 | −19 | 0.73 | 5e-5 |
| Flow, σ=0 | 169 ± 160 | −79 | 0.98 | 8e-17 | 146 ± 91 | +127 | 0.69 | 2e-4 † |
| TransUpNet (MMSE) | 206 ± 149 | +150 | 0.99 | 3e-15 | 107 ± 84 | +75 | 0.74 | 0.87 |
| Wasserstein | 189 ± 187 | −76 | 0.98 | 3e-16 | 145 ± 94 | +125 | 0.70 | 3e-4 † |
| Pix-2-Pix | 301 ± 228 | +128 | 0.95 | 8e-13 | 112 ± 109 | −28 | 0.58 | 0.84 |
| ED | 359 ± 305 | −211 | 0.93 | 2e-11 | 131 ± 127 | −111 | 0.39 | 0.14 |
| LDM | 1206 ± 881 | +183 | 0.15 | 9e-3 † | 175 ± 143 | −30 | 0.01 | 2e-6 † |
| LDM (WF-VAE) | 1289 ± 1080 | −302 | 0.05 | 9e-3 † | 163 ± 131 | −60 | −0.08 | 8e-6 † |
| PFM | 3947 ± 1228 | −3947 | −0.23 | 4e-18 † | 522 ± 118 | −522 | 0.18 | 4e-18 † |
| Mean-volume baseline | 972 ± 699 | 0 | – | – | 100 ± 69 | 0 | – | – |
The projection constrains the heart only weakly. Only our flow with noise significantly beats the mean-volume baseline there (67 vs. 100 mL). The MMSE estimate is no better than the baseline (p = 0.87), and the noise-free flow is significantly worse. The latent diffusion models produce realistic images, but their organ volumes are uncorrelated with the patient's (lung r ≤ 0.15, heart r ≤ 0.01).
Table 1 (main paper): OCT, 81 test eyes. Layer and vessel Dice are computed with independent segmentation networks, trained on disjoint cohorts, applied identically to real and generated volumes. They measure agreement with the specific patient's OCT, not realism. *: significant against the closest competitor.
| Method | PSNR ↑ | SSIM ↑ | FID ↓ | Dice layers ↑ | Dice vessels ↑ |
|---|---|---|---|---|---|
| ED (Liu et al., 2025b) | 19.4 ± 1.0 | 25.5 ± 1.9 | 448.3 | 23.0 ± 3.1 | 16.8 ± 9.3 |
| Pix-2-Pix (Isola et al., 2017) | 18.6 ± 1.0 | 22.3 ± 2.3 | 116.9 | 47.8 ± 5.3 | 44.4 ± 12.6 |
| Wasserstein (Pinetz et al., 2023) | 19.4 ± 1.2 | 26.1 ± 2.3 | 405.8 | 24.4 ± 2.6 | 28.6 ± 10.5 |
| LDM (Rombach et al., 2022) | 15.8 ± 0.7 | 15.5 ± 1.3 | 25.7 | 40.3 ± 5.5 | 8.9 ± 3.1 |
| LDM WF-VAE (Chen et al., 2026) | 14.1 ± 1.2 | 11.5 ± 1.7 | 118.4 | 22.8 ± 11.1 | 5.1 ± 3.8 |
| PFM (Li & He, 2026) | 17.9 ± 0.7 | 20.2 ± 1.4 | 236.0 | 28.7 ± 4.1 | 0.5 ± 1.2 |
| PRFM (Ohayon et al., 2025) | 17.7 ± 0.8 | 17.2 ± 1.8 | 29.7 | 47.5 ± 5.1 | 45.4 ± 12.5 |
| Ours (no flow) | 19.4 ± 1.2 | 26.6* ± 2.7 | 435.0 | 32.6 ± 4.9 | 45.9 ± 10.9 |
| Ours (σ=0 flow) | 19.1 ± 1.1 | 24.9 ± 2.5 | 354.9 | 51.3 ± 5.4 | 46.7 ± 11.3 |
| Ours (normal flow) | 18.0 ± 0.9 | 18.8 ± 2.0 | 17.8 | 48.3 ± 5.5 | 46.7 ± 11.9 |
| Ours (slerp flow) | 18.0 ± 0.9 | 19.1 ± 2.0 | 18.0 | 48.3 ± 5.5 | 48.0 ± 11.1 |
Methods with realistic outputs but no patient-specific anatomy score low here: LDM reaches FID 25.7 but vessel Dice 8.9. Our flow combines FID 18.0 with the highest vessel Dice and a layer Dice of 48.3. Without registration, from the fundus image alone (A.5, Table 8), layer Dice drops by less than one point (47.3 vs. 48.3) and the layer-volume error is unchanged (1.29 vs. 1.28).
Realism (FID) and patient-specific anatomy (organ volumes, layer and vessel Dice) are therefore measured separately, and our method is the only one that does well on both.
Response to Reviewer 3: W1–W2, Q1–Q3
We are grateful to the reviewer for their insightful suggestions, which have helped strengthen our manuscript.
W1 (low PSNR). On OCT, PSNR is limited by speckle, an interference pattern that changes with every acquisition and is not determined by the fundus image, so no method can predict it. We compared each real test volume with a speckle-denoised copy of itself (A.7, Table 10, 81 eyes): every denoiser that removes speckle scores only 22.7–25.3 dB against the original, which is what a perfect speckle-free reconstruction would score. Our MMSE estimate scores 19.1 dB against the real volume and 21.8 dB against its denoised copy (3D-SSIM 67.4 vs. 25.5). Even the pretrained WF-VAE of our latent baselines, which sees the real volume, reconstructs OCT at only 23.16 dB (Table 10). It keeps the retinal layers but regenerates speckle that does not match the input (Figure 9); on CT, which has no speckle, it reaches 31.53 dB (A.7). Low absolute OCT scores therefore reflect the modality.
W1 (realism vs. true anatomy). We agree that realism alone does not show this. We therefore compare per-case organ volumes on 100 CT test cases, segmenting real and generated volumes identically (lungmask, TotalSegmentator; A.4, Table 7), against a mean-volume baseline that predicts the population average for every patient. On the heart, whose size the projection only partly determines, only our flow with noise beats the baseline (67 vs. 100 mL, $p = 6\cdot10^{-5}$). The MMSE estimate overestimates the heart by 75 mL and is no better than the baseline (107 mL, $p = 0.87$). On the lungs our flow errs by 162 mL (about 3.5% of the mean real volume, $r = 0.98$) vs. 972 mL for the baseline. The latent diffusion baselines look realistic, but their organ volumes are essentially uncorrelated with the patient's (lungs $r \leq 0.15$, heart $r \leq 0.01$). Realism alone thus does not recover anatomy, and our method achieves both.
W2 (Wasserstein vs. ours, 19.4 vs. 19.4 dB). PSNR alone hides the distortion-perception tradeoff (Blau & Michaeli, 2018). The Wasserstein baseline uses the same TransUpNet generator without a pixel loss. At the same 19.4 dB it oversmooths (FID 405.8, layer Dice 24.4, vessel Dice 28.6), while our MMSE estimate reaches layer Dice 32.6 and vessel Dice 45.9. With the flow, FID drops to 18.0 and layer and vessel Dice rise to 48.3 and 48.0 (Table 1). Because the step count $S$ is chosen at test time, one model gives 19.4 dB at $S=1$, the best layer Dice at $S=2$ (49.1), the PSNR of Pix2Pix at $S=5$ with a lower FID (83.7 vs. 116.9), and FID 18.0 at $S=20$ (Table 3). The speckle ceiling (W1) also compresses all OCT PSNR differences into a few dB.
Q1 (downstream tasks). We evaluate the generated images with independent segmentation networks trained on disjoint cohorts (A.4). On OCT, the flow raises layer Dice from 32.6 to 48.3 and vessel Dice from 45.9 to 48.0, above every competing method (closest: Pix2Pix, layer Dice 47.8; Table 1). On CT we use lung and heart volumes (W1). We see two intended uses: a 3D layer view where only a fundus photograph is available, and data completion for multimodal foundation models.
Q2 (alignment sensitivity). Registration failures are removed by a quality threshold (A.1), so smaller misregistrations are already part of the training distribution. As a direct test, we ran inference with no registration, locating the fovea on the fundus image alone (A.5, Table 8). PSNR changes by 0.2 dB, FID is unchanged (18.0) and the layer-volume error is 1.29 vs. 1.28 registered (3.38 for the MMSE estimate). Vessel Dice falls (0.145 vs. 0.480) because the output is no longer pixel-aligned with the OCT. We also visualized a good case and the median case for this setup in Figure 7.
Q3 (generalization from 403 eyes). The 403 eyes come from 6 hospitals with a patient-wise 80/20 split (Section 5), and we also evaluate on a public CT benchmark (A.6, Table 9). To test generalization across sites, we retrained Stage 1 with one hospital held out (353 training and 50 held-out eyes; A.11, Table 13). The held-out model scores 19.1 dB, 3D-SSIM 25.2 and FID 442.2, vs. 19.4 dB, 26.6 and 435.0 (Table 8); This experiment shows that the results are similar and do not completely collapse given a new site with the same device. The cohort comes from one device type and early or intermediate AMD only, which limits broader claims.