acceptodds
Under review as a conference paper at ICLR 2027

Posterior-Mean Rectified Flow to estimate 3D structures from 2D medical images

Abstract

This study addresses the highly ill-posed challenge of 3D volume reconstruction from 2D medical images even in scenarios lacking a known physical forward model. We propose a two-stage framework that first employs a specialized “lifting” network to generate a structurally consistent Minimum Mean Squared Error (MMSE) estimate. This smooth baseline then serves as the foundation for a 2D paired flow-matching process, designed to recover high-frequency details across large 3D volumes efficiently. By decoupling structural translation from perceptual refinement, we leverage the generative power of flow matching in 2D while retaining structural consistency through spherical linear interpolation (slerp) without the need for explicit data-fidelity terms. Our approach operates directly in image space, bypassing the ambiguity of latent space correspondences and enabling test-time control over the distortion-perception tradeoff via adjustable refinement steps. When evaluated on the translation of synthetic X-ray to Computed Tomography (CT) and real Color Fundus Photography (CFP) to Optical Coherence Tomography (OCT), our method demonstrates superior structural fidelity and a significantly lower FID compared to state-of-the-art baselines. Code and example GIFs are available here: https://anonymous.4open.science/r/pmrf3dsynthesis-CEC5.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Imaging, Flow matching, Image Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cbvzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the ill-posed problem of estimating a dense 3D medical image volume from a single 2D image, with application on color fundus photography to optical coherence tomography (OCT) estimation, and synthetic X-ray to computed tomography (CT) estimation. The proposed framework has two stages: 1) 2D-to-3D lifting stage which maps the input image into a 3D volume using TransUpNet; 2) Slice-by-slice perceptual refinement using a 2D flow matching model. Experiment has been performed on two benchmarks including color fundus to OCT benchmark and X-ray to CT benchmark, and results show that the proposed method achieves superior PSNR/SSIM fidelity and significantly lower FID compared to state-of-the-art baselines.

Critical strengths
  1. The separation of volumetric structural prediction from slice-wise appearance refinement is a natual and reasonable engineering choice.
  2. The ablation study is clear which investigates refinement steps, Noise Level, noise interpolation, choice of flow-matching network, and refinement-network architecture, and helps identifying which components contribute to the results.
Critical weaknesses
  1. The main concern is on whether the proposed problem setting (Color Fundus Photography to OCT, X-ray to CT) is suitable for medical imaging applications. The generated volumes' PSNR is low (18.0 to 19.4 as in Table 1), and improved perceptual realism DONOT corresponds to improved recovery of the patient's true unobserved anatomy.
  2. The performance improvement related with image quality is not very significant as stated in table 1 (i.e. Wasserstein vs. ours @ PSNR: 19.4 vs. 19.4).
Questions for authors
  1. Is there any planned downstream processing tasks for the generated OCT/CT volumes?
  2. Similarity transformation is used for alignment between fundus image and OCT, as stated in section 4, how about the sensitivity of generation result to such alignment accuracy.
  3. The fundus-OCT dataset has 403 pairs for training/test, how about the generalization capability of trained models on this dataset?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3: W1–W2, Q1–Q3

We are grateful to the reviewer for their insightful suggestions, which have helped strengthen our manuscript.

W1 (low PSNR). On OCT, PSNR is limited by speckle, an interference pattern that changes with every acquisition and is not determined by the fundus image, so no method can predict it. We compared each real test volume with a speckle-denoised copy of itself (A.7, Table 10, 81 eyes): every denoiser that removes speckle scores only 22.7–25.3 dB against the original, which is what a perfect speckle-free reconstruction would score. Our MMSE estimate scores 19.1 dB against the real volume and 21.8 dB against its denoised copy (3D-SSIM 67.4 vs. 25.5). Even the pretrained WF-VAE of our latent baselines, which sees the real volume, reconstructs OCT at only 23.16 dB (Table 10). It keeps the retinal layers but regenerates speckle that does not match the input (Figure 9); on CT, which has no speckle, it reaches 31.53 dB (A.7). Low absolute OCT scores therefore reflect the modality.

W1 (realism vs. true anatomy). We agree that realism alone does not show this. We therefore compare per-case organ volumes on 100 CT test cases, segmenting real and generated volumes identically (lungmask, TotalSegmentator; A.4, Table 7), against a mean-volume baseline that predicts the population average for every patient. On the heart, whose size the projection only partly determines, only our flow with noise beats the baseline (67 vs. 100 mL, $p = 6\cdot10^{-5}$). The MMSE estimate overestimates the heart by 75 mL and is no better than the baseline (107 mL, $p = 0.87$). On the lungs our flow errs by 162 mL (about 3.5% of the mean real volume, $r = 0.98$) vs. 972 mL for the baseline. The latent diffusion baselines look realistic, but their organ volumes are essentially uncorrelated with the patient's (lungs $r \leq 0.15$, heart $r \leq 0.01$). Realism alone thus does not recover anatomy, and our method achieves both.

W2 (Wasserstein vs. ours, 19.4 vs. 19.4 dB). PSNR alone hides the distortion-perception tradeoff (Blau & Michaeli, 2018). The Wasserstein baseline uses the same TransUpNet generator without a pixel loss. At the same 19.4 dB it oversmooths (FID 405.8, layer Dice 24.4, vessel Dice 28.6), while our MMSE estimate reaches layer Dice 32.6 and vessel Dice 45.9. With the flow, FID drops to 18.0 and layer and vessel Dice rise to 48.3 and 48.0 (Table 1). Because the step count $S$ is chosen at test time, one model gives 19.4 dB at $S=1$, the best layer Dice at $S=2$ (49.1), the PSNR of Pix2Pix at $S=5$ with a lower FID (83.7 vs. 116.9), and FID 18.0 at $S=20$ (Table 3). The speckle ceiling (W1) also compresses all OCT PSNR differences into a few dB.

Q1 (downstream tasks). We evaluate the generated images with independent segmentation networks trained on disjoint cohorts (A.4). On OCT, the flow raises layer Dice from 32.6 to 48.3 and vessel Dice from 45.9 to 48.0, above every competing method (closest: Pix2Pix, layer Dice 47.8; Table 1). On CT we use lung and heart volumes (W1). We see two intended uses: a 3D layer view where only a fundus photograph is available, and data completion for multimodal foundation models.

Q2 (alignment sensitivity). Registration failures are removed by a quality threshold (A.1), so smaller misregistrations are already part of the training distribution. As a direct test, we ran inference with no registration, locating the fovea on the fundus image alone (A.5, Table 8). PSNR changes by 0.2 dB, FID is unchanged (18.0) and the layer-volume error is 1.29 vs. 1.28 registered (3.38 for the MMSE estimate). Vessel Dice falls (0.145 vs. 0.480) because the output is no longer pixel-aligned with the OCT. We also visualized a good case and the median case for this setup in Figure 7.

Q3 (generalization from 403 eyes). The 403 eyes come from 6 hospitals with a patient-wise 80/20 split (Section 5), and we also evaluate on a public CT benchmark (A.6, Table 9). To test generalization across sites, we retrained Stage 1 with one hospital held out (353 training and 50 held-out eyes; A.11, Table 13). The held-out model scores 19.1 dB, 3D-SSIM 25.2 and FID 442.2, vs. 19.4 dB, 26.6 and 435.0 (Table 8); This experiment shows that the results are similar and do not completely collapse given a new site with the same device. The cohort comes from one device type and early or intermediate AMD only, which limits broader claims.

Thanks the authors for detailed explanation. I still have some concerns including W1 (realism vs. true anatomy): the 'Table 7' in appendix is just a comparision between "TransUpNet + JiT" & "TransUpNet + Song", and the following text of "On the heart, whose size..." is only case study and not in original paper

Thank you for the follow-up. To avoid any confusion about the table references, we reproduce the relevant tables of the revised PDF below. The JiT vs. Song comparison is a backbone ablation (Appendix A.9, Table 12) and is not our evidence for anatomical fidelity. That evidence is Table 7 (Appendix A.4) for CT and Table 1 for OCT. The heart result is not a case study: it is the heart column of Table 7, computed over all 100 CT test cases.

Table 7 (A.4): patient-specific organ volumes recovered from the generated CT (100 test cases). Lungs are segmented with lungmask and the heart with TotalSegmentator, identically on real and generated volumes. MAE: mean absolute error of the per-case volume (mL, mean ± SD). Bias: mean signed error (mL). r: Pearson correlation with the volume on the real CT. p: paired Wilcoxon signed-rank test of the per-case absolute errors against the mean-volume baseline, which predicts the leave-one-out mean real volume for every case (mean real volume: lungs 4633 mL, heart 524 mL). †: significantly worse than the baseline.

MethodLung MAE ↓Lung biasLung r ↑Lung pHeart MAE ↓Heart biasHeart r ↑Heart p
Flow (normal noise)162 ± 158−690.986e-1767 ± 66−230.756e-5
Flow (slerp noise)163 ± 159−670.988e-1767 ± 67−190.735e-5
Flow, σ=0169 ± 160−790.988e-17146 ± 91+1270.692e-4 †
TransUpNet (MMSE)206 ± 149+1500.993e-15107 ± 84+750.740.87
Wasserstein189 ± 187−760.983e-16145 ± 94+1250.703e-4 †
Pix-2-Pix301 ± 228+1280.958e-13112 ± 109−280.580.84
ED359 ± 305−2110.932e-11131 ± 127−1110.390.14
LDM1206 ± 881+1830.159e-3 †175 ± 143−300.012e-6 †
LDM (WF-VAE)1289 ± 1080−3020.059e-3 †163 ± 131−60−0.088e-6 †
PFM3947 ± 1228−3947−0.234e-18 †522 ± 118−5220.184e-18 †
Mean-volume baseline972 ± 6990––100 ± 690––

The projection constrains the heart only weakly. Only our flow with noise significantly beats the mean-volume baseline there (67 vs. 100 mL). The MMSE estimate is no better than the baseline (p = 0.87), and the noise-free flow is significantly worse. The latent diffusion models produce realistic images, but their organ volumes are uncorrelated with the patient's (lung r ≤ 0.15, heart r ≤ 0.01).

Table 1 (main paper): OCT, 81 test eyes. Layer and vessel Dice are computed with independent segmentation networks, trained on disjoint cohorts, applied identically to real and generated volumes. They measure agreement with the specific patient's OCT, not realism. *: significant against the closest competitor.

MethodPSNR ↑SSIM ↑FID ↓Dice layers ↑Dice vessels ↑
ED (Liu et al., 2025b)19.4 ± 1.025.5 ± 1.9448.323.0 ± 3.116.8 ± 9.3
Pix-2-Pix (Isola et al., 2017)18.6 ± 1.022.3 ± 2.3116.947.8 ± 5.344.4 ± 12.6
Wasserstein (Pinetz et al., 2023)19.4 ± 1.226.1 ± 2.3405.824.4 ± 2.628.6 ± 10.5
LDM (Rombach et al., 2022)15.8 ± 0.715.5 ± 1.325.740.3 ± 5.58.9 ± 3.1
LDM WF-VAE (Chen et al., 2026)14.1 ± 1.211.5 ± 1.7118.422.8 ± 11.15.1 ± 3.8
PFM (Li & He, 2026)17.9 ± 0.720.2 ± 1.4236.028.7 ± 4.10.5 ± 1.2
PRFM (Ohayon et al., 2025)17.7 ± 0.817.2 ± 1.829.747.5 ± 5.145.4 ± 12.5
Ours (no flow)19.4 ± 1.226.6* ± 2.7435.032.6 ± 4.945.9 ± 10.9
Ours (σ=0 flow)19.1 ± 1.124.9 ± 2.5354.951.3 ± 5.446.7 ± 11.3
Ours (normal flow)18.0 ± 0.918.8 ± 2.017.848.3 ± 5.546.7 ± 11.9
Ours (slerp flow)18.0 ± 0.919.1 ± 2.018.048.3 ± 5.548.0 ± 11.1

Methods with realistic outputs but no patient-specific anatomy score low here: LDM reaches FID 25.7 but vessel Dice 8.9. Our flow combines FID 18.0 with the highest vessel Dice and a layer Dice of 48.3. Without registration, from the fundus image alone (A.5, Table 8), layer Dice drops by less than one point (47.3 vs. 48.3) and the layer-volume error is unchanged (1.29 vs. 1.28).

Realism (FID) and patient-specific anatomy (organ volumes, layer and vessel Dice) are therefore measured separately, and our method is the only one that does well on both.

Reviewer_DZCZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper estimates three-dimensional medical structures from two-dimensional images. It first produces a 3D initial estimate, then refines slices using a 2D posterior-mean rectified flow model. Correlated stochasticity and SLERP are used to improve inter-slice consistency. The experiments cover fundus-to-OCT and X-ray-to-CT tasks. The work combines paired-data preparation, initial volume estimation, and generative refinement, while the core flow framework and inter-slice consistency ideas have direct precedents.

Critical strengths
  1. The data preparation and two-task evaluation are substantive. The authors process paired medical data and compare multiple baselines and design choices. The retained OCT dataset contains 403 eyes, and the paper explicitly reports patient-level splitting. Tables 1–2 on page 7 show improvements in selected perceptual and structural metrics after generative refinement, together with lower PSNR. For example, OCT layer Dice increases from 32.6 to 48.3 relative to no-flow, while PSNR decreases from 19.4 to 18.0. These results present a meaningful quality trade-off.
  2. Refining a 3D initial estimate with a 2D generative model has a practical motivation. The design can use 2D models while attempting to control variation between slices. The paper also reports 3D-SSIM and segmentation-related metrics. The relevant question is whether this evidence validates the added mechanism. It would be inaccurate to say that the paper never evaluates 3D results.
Critical weaknesses
  1. The added 3D capability is not yet clearly separated from the existing frameworks. PMRF and inter-slice correlated stochasticity have direct precedents. SLERP does not consistently outperform the linear alternative: OCT FID is 18 versus 17.8; CT LPIPS is 0.19 for both, and SSIM is identical at the reported precision. These close point estimates do not establish statistical equivalence, but they also do not clearly demonstrate an independent SLERP benefit. Other 3D or diffusion controls simultaneously change the initial estimator, backbone, or training conditions. The experiments therefore establish that the combined system works more clearly than they establish a new solution to a previously unresolved 3D problem. This directly affects the depth of the methodological contribution.
  2. A key validation gap remains between the prepared paired data and a practical 2D-input setting. The pipeline on pages 5–6 and in the appendix uses paired OCT for registration, flattening, and selection. The authors disclose this limitation, and the network itself does take only the fundus image at inference. This is not evidence of hidden leakage. The unresolved question is how much structural performance remains when only information available in the intended 2D-input setting can be used. Predicting an output in flattened coordinates is possible in principle, but the availability of the required input localization, registration, and selection conditions needs separate validation. Together with the limited representation of details such as drusen, the current evidence more strongly supports a proof of concept under a particular data-preparation pipeline. 3.Page 2 inaccurately classifies DiffuX2CT as latent diffusion. Its diffusion target is in CT image space, while tri-plane features provide conditioning. This is not diffusion in a VAE latent space. Please correct the classification and revise the distinction from the present method. DiffuX2CT also uses two views, so it should not be treated as a numerically matched single-view baseline.
Questions for authors

1.Are direct inter-slice structural comparisons available for independent noise, shared noise, variance-matched linear mixing, and SLERP with the same initial estimator, 2D flow, and training conditions? 2.Has the fovea-only workflow that does not require paired OCT been evaluated separately? How does its performance compare with the current prepared and spatially registered pipeline? 3.What is the central methodological contribution beyond existing PMRF and ISCS? Which controlled experiment directly validates that contribution?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2: W1–W3, Q1–Q3

We sincerely thank the reviewer for their thorough evaluation and constructive feedback.

W1 (separation from PMRF and ISCS, slerp vs. linear, controls). Beyond PMRF, our contribution is Proposition 1 and the resulting volumetric guarantee (General response). ISCS (Du et al., 2026) correlates the stochastic noise across slices when 2D diffusion models are used for 3D imaging; we adopt this for our source noise (l. 211) and do not claim it. What differs is the source: our flow starts at a learned 3D MMSE estimate, needs no forward operator and keeps the PMRF guarantee slice by slice.

We agree that slice-wise metrics do not separate slerp from independent noise, and we no longer claim that slerp is better than other correlated couplings. A new ablation (Table 5) keeps the estimator, the 2D flow and its training fixed and varies only the coupling: independent, shared, linear, variance-matched linear and slerp. As a 3D metric we use the one-slice boundary pop rate of CT lung masks (Eq. 5), from a 2D segmenter that cannot smooth over flicker, reported as each case's deviation from the real CT. At 20 steps, independent noise flickers (19.99), while all four correlated couplings stay at the level of the MMSE estimate (0.49–0.54 vs. 0.48). The noise must also keep unit variance: plain linear interpolation reaches OCT FID 122.3 and CT FID 0.59, vs. 18.0–18.2 and 0.22–0.29 for slerp, shared and variance-matched noise. Lung volume barely changes (MAE 162–165 mL).

On the controls: Ours (no flow) and Ours (flow) share the estimator and the data, so Table 1 isolates the flow. Since PFM also changes the estimator, we added PRFM (Table 1), a 3D flow from the same TransUpNet estimate that changes only the dimension of the flow. It reaches 17.7 dB, 3D-SSIM 17.2 and FID 29.7, vs. 18.0, 19.1 and 18.0 for our 2D flow, and is also lower in both Dice scores. At batch size 1, only 16 base channels fit on a 48 GB GPU, which is the practical reason to refine in 2D.

W2 (fundus-only setting). We agree that this needs separate validation. Paired OCT is used only to curate the data: vessel-based registration with a quality threshold (vessel Dice 0.1, A.1) and image-quality checks excluded 124 of 527 eyes (Section 5). The network predicts in flattened coordinates, so inference does not need to register towards any OCT. In Appendix A.5 (Table 8, the same 81 test eyes as Table 1), we locate the fovea on the fundus image alone and crop the region the OCT device scans, with no OCT used for registration, scaling or selection. The Maestro2 centers acquisitions on the fovea with eye-tracking, so this is inline with the clinical field of view. Against the registered pipeline, PSNR is 17.8 vs. 18.0 dB, 3D-SSIM 17.9 vs. 19.1 and FID 18.0 for both, and the layer-volume error is 1.29 vs. 1.28 (3.38 for the MMSE estimate). Vessel Dice falls (0.145 vs. 0.480) because the output is no longer pixel-aligned with the OCT; this is the main cost of the fundus-only setting, and we will state it. Figure 7 shows a well-aligned and a median case.

W3 (DiffuX2CT). Thank you for pointing out the error. DiffuX2CT diffuses in CT image space, uses tri-plane features as conditioning and takes two orthogonal views. Section 2 will list it with image-space conditional diffusion models rather than VAE-based latent diffusion, and state that bi-planar methods use two views, unlike our single-view setting. It is not one of our numerical baselines; we will mark the single-view methods in Table 9.

Q1: see W1 (Table 5). Q2: see W2 (A.5, Table 8). Q3: see W1 and the General response. The controlled experiments are Ours (no flow) vs. Ours (flow), and PRFM vs. our 2D flow (Table 1).

Dear authors, sorry for the delay. Thank you for the detailed response and additional experiments.

The new noise-coupling ablation answers my Q1, and I appreciate that you no longer claim slerp is better than other couplings.The DiffuX2CT point is also resolved.

My main concern remains the methodological contribution. Once slerp is dropped, what is left beyond PMRF is Proposition 1, which, as you acknowledge, holds for any MMSE estimator, and the guarantee is stated for the noiseless case, while the method only works well with noise. So I am not sure the theory supports the setting that is actually used. The 3D flow comparison is useful, but I cannot tell whether the gap comes from 2D vs. 3D or from the smaller network.

On the fundus-only experiment, thank you for running it. However, the large drop in vessel Dice suggests that much of the patient-specific structure depends on the OCT-based registration, and layer Dice is not reported for this setting. Could you also check the appendix tables for consistency? Some numbers for the registered and fundus-only settings seem to be mixed.

Thank you for the follow-up, and for confirming that Q1 and the DiffuX2CT point are resolved.

Contribution beyond PMRF. That Proposition 1 holds for any MMSE estimator is intended. It is what allows the refinement to be carried out with a 2D flow at all. Without it, a 2D PMRF on slices of a 3D estimate would have no justification. It also identifies where this fails: a nonlinear latent encoder does not commute with the conditional expectation, so latent-space refinement loses the guarantee. Test-time control of the distortion-perception tradeoff follows from this construction (Table 3). Additionally, correlated noise has been used across video frames (Zhang et al., 2024), but PMRF was formulated for single 2D images, and it is not clear from the original work how its noise should be coupled when it is applied slice by slice to a volume. We show that this choice determines volumetric consistency, where independent noise produces strong slice-to-slice flicker (pop-rate deviation 20.0), while any correlated, variance-preserving coupling removes it (0.5, equal to the MMSE input). Furthermore, Slerp and variance-matched linear interpolation trace nearly the same path, since independent high-dimensional Gaussians are almost orthogonal. Plain linear interpolation also removes flicker but shrinks the noise variance between anchor slices, so the flow receives noise of the wrong scale and OCT FID degrades to 122.3, compared with about 18 for the variance-preserving couplings.

Noiseless guarantee vs. noisy practice. This gap is not specific to our setting. It exists in PMRF itself: Ohayon et al. (2025) state their result for a flow from the posterior mean and use a small source noise in practice for the same reason we do. What our analysis adds is that the property the practical method relies on carries over. Each slice’s source is its own posterior mean plus noise with the training variance (exact for the variance-matched coupling, approximate for slerp), so the 2D flow sees its training distribution at every slice. The zero-noise run stays close to the MMSE estimate (FID 354.9 vs. 18.0) and performs worse in 3 out of 4 downstream tasks. We agree that for σ > 0 the guarantee holds only approximately, but the data fit improves upon decreasing the noise. We state this explicitly in Section 3.

2D vs. 3D, or network size. We agree that the two are confounded. On a 48 GB GPU at batch size 1, a 3D flow on 128×256×256 volumes fits only with 16 base channels; 24 and 32 channels ran out of memory. Our claim is therefore the practical one: at equal hardware, a 2D flow can be far larger than any 3D flow that fits, and it performs better.

Vessel Dice is a pixel-wise overlap with the OCT-derived vessel map. Without registration, the prediction is no longer pixel-aligned with that map, so Dice falls even when the vessels are present but shifted, even though this would not affect downstream clinical assessment of these images. We report it to show that the fundus-only predictions do differ from those made with paired OCT. Metrics that do not depend on en-face alignment are unchanged: the layer-volume error is 1.29 vs. 1.28 registered (3.38 for MMSE; Table 8), and the newly added layer Dice is 47.8 vs. 48.3 registered (32.5 vs. 32.6 for MMSE). This is mostly because retinal layers vary little across neighbouring B-scans in our cohort, so a small transverse shift barely changes the overlap. FID is also unchanged 18 vs 18.

Thank you for checking. We went through all tables again, including an automated consistency check. We found one standard deviation in Table 2 (lung volume, LDM) that was copied incorrectly, and one heart-volume value for σ = 0 that was rounded incorrectly. Both are fixed.

Reviewer_AB8QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new method to do 3D reconstruction from 2D images without having access to a physical forward model. The method is flow matching-based and doesn't use a latent space compared to existing methods. It consists of two stages: an estimation of the MMSE using a new multiscale architecture, and then a refinement where one learns a flow between the previous MMSE distribution and the distribution of the clean 3D volumes. This last strategy called PMRF is based on the paper (Ohayon et al., 2025). The goal is to find a good perception-distortion tradeoff (the MMSE is known to produce over-smoothed reconstructions).

Critical strengths
  • To my knowledge, this type of PMRF approaches have not been used for 3D medical imaging applications yes, and it's a promising technology
  • The numerical experiments are convincing and seem to contain the necessary ablations, although the numerical part could be made clearer (see comments below).
Critical weaknesses
  • There is a small lack of novelty in the methodology since the PMRF approach already exists. This work appears mainly as applicative work to medical imaging, which nevertheless I guess could have some value.
Questions for authors

Method part

  1. How would incorporate a data-fidelity information in your approach if you had one?

  2. What does Proposition 1 teach us? I sounds quite obvious but how is it specific to the proposed architecture/ to equation (1)? Does it not work for any 3D MMSE?

  3. l191: why do you noise zjz_j? Is it necessary?

  4. About the spherical interpolation: Is it really direct that the Proposition 1 of Ohayon et al. (2025) applies for the spherical interpolation proposed? Please provide the full argument.

Experimental part I struggled understanding the different baselines. I think most ablations are there but they should be made clearer.

  1. I am not sure what is meant by “Paired Flow Matching (PFM)” here, and I think this should be defined more explicitly. First, if PFM simply means learning a flow between the paired coarse 3D estimate (x_0) and the corresponding clean 3D OCT volume (x_1), then this appears to be standard flow matching with a paired, data-dependent source, possibly using an (x_1)-prediction parameterization. In that case, I do not think Li and He should be the only reference. The paper of Li and He also uses a new JiT architecture which is not being used here if I am not mistaken. Second, could you specify the difference with your approach? My understanding is it that you use a neural 2D to 3D lifting first whereas "PFM" trains the flow directly from a simple lifting?

  2. Could you provide evidence that your proposed multi-scale MMSE improves reconstruction performance (without PMRF) over a basic MMSE estimation with a network having the same number of parameters (approximately) and with the same number of optimization steps? Maybe it's already what was intended through the baseline Encoder/Decoder. If so, it would be helpful to explain this more explicitly.

  3. Why is the FID for your method without flow (thus if I understood well, juste the 3D MMSE estimation step) so bad when the PSNR is good? It is known MMSE tends to smooth images but still this discrepancy is surprising. What do the output images look like?

Minor remarks:

  • It would be good to explicitely say what "no flow" and "linear flow" mean even if i guess the first one corresponds to just the MMSE, and the second to the PMRF with interpolation tx1+(1−t)x0t x_1 + (1-t)x_0.

  • l199 : Li and He did not propose this loss. The choice of parametrization for flow matching and diffusion has been studied in prior work, see for example Salimans and Ho (2022), Zhang et al. (2024). That part is not very clear. What you want to say is that you use the x1x_1-prediction and the vv-loss (i.e. a weighting 1/(1−t)1/(1-t) in a denoising loss) For a UNet there is no clear evidence (and rather opposite arguments) that x1x_1-pred is a good choice, see (Gagneux et al, 2026)

Zhang, X., Pu, Y., Kawamura, Y., Loza, A., Bengio, Y., Shung, D. L., & Tong, A. (2024). Trajectory flow matching with applications to clinical time series modelling. Advances in Neural Information Processing Systems, 37, 107198-107224.

Salimans, T., Ho, J., Progressive Distillation for Fast Sampling of Diffusion Models (2022), International Conference on Learning Representations (ICLR)

Gagneux, A., Martin, S., Gribonval, R., & Massias, M. (2026). Training flow matching: The role of weighting and parameterization. ICLR workshop 2026

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (1/2): W, Q1–Q4

We thank the reviewer for all the constructive comments.

W (novelty). Please see the General response. Proposition 1 makes a 3D MMSE estimate a valid PMRF source, so the guarantee holds for whole volumes in pixel space without a 3D generative model or a latent space, and Table 1 isolates the effect of the 2D flow.

Q1 (data fidelity). It depends on the forward operator. An operator that couples the whole volume, such as the X-ray projection (here the DRR from DiffDRR), enters Stage 1 as a differentiable re-projection loss or an unrolled projection layer in TransUpNet; as long as Stage 1 remains an MMSE estimator, Proposition 1 holds and Stage 2 is unchanged. A slice-local operator enters Stage 2 as a guidance term at each Euler step, analogous to diffusion posterior sampling, or, under a hard linear constraint, as a projection onto the measurement-consistent set, as in FLAIR (Erbach et al., NeurIPS 2025). Fundus-to-OCT has no physical forward operator, and this is the setting we target.

Q2 (Proposition 1). Thanks for correctly pointing out that it holds for any 3D MMSE estimator, not only TransUpNet or Eq. (1). Its role is to license training a 2D PMRF on slices of a 3D estimate, and to separate linear slicing, where the PMRF guarantee holds, from a learned latent encoder, where it does not. A better Stage 1 therefore directly gives Stage 2 a better source. It assumes that Stage 1 converges to the conditional mean, and the guarantee holds for slices pooled over position.

Q3 (l.191, why noise). In theory it is not needed. Proposition 1 of Ohayon et al. (2025) is stated for $\sigma_\epsilon = 0$. In practice it produces better results. A flow trained and sampled with $\sigma_\epsilon = 0$ (rows "$\sigma_\epsilon=0$ Flow", Tables 1 and 2) trains stably but stays close to the MMSE estimate. On OCT its FID is 354.9 vs. 18.0 with $\sigma_\epsilon = 0.1$. On CT it overestimates the heart by 127 mL, so its heart-volume error is 146 vs. 67 mL, worse than predicting the mean volume (Table 7). Small source noise is therefore what lets the flow add texture and remove the MMSE bias, in line with the observation of Ohayon et al. that a small non-zero noise is needed in practice (ll. 248–249). The noise-level ablation is in Tables 3 and 4.

Q4 (slerp and Proposition 1 of Ohayon et al.). The argument has four steps. Appendix A.10 gives steps 1 and 4; we will add steps 2 and 3.

  1. The proposition concerns the noiseless case, a flow started at the posterior mean. By our Proposition 1, each slice of the 3D estimate is the posterior mean of that slice, so the condition holds slice by slice. No property of slerp is used.
  2. For $\sigma_\epsilon > 0$ the guarantee holds approximately, as in Ohayon et al., provided that at inference each slice sees the source it was trained on: its MMSE slice plus $\mathcal{N}(0, \sigma_\epsilon^2 I)$ noise.
  3. Slerp sets $\epsilon_k = \frac{\sin((1-t_k)\Omega)}{\sin\Omega}\epsilon_0 + \frac{\sin(t_k\Omega)}{\sin\Omega}\epsilon_1$, with $\epsilon_0, \epsilon_1$ independent and $\Omega$ the angle between them. In dimension $d = 256^2$, $\Omega$ concentrates at $\pi/2$ with spread $\approx 1/\sqrt{d}$, so the weights tend to the cosine and sine of one angle and their squares sum to 1. Each slice's noise is therefore Gaussian with the training variance up to $O(1/\sqrt{d})$. Numerically (200 draws, 128 slices), $\Omega = 1.5709 \pm 0.0039$ rad, the per-slice variance stays within $[0.985, 1.008]$ and the kurtosis is 3.00. Fixing $\Omega = \pi/2$ makes the marginal exactly Gaussian; this is our "variance-matched linear" coupling, which performs like slerp (Table 5). Plain linear interpolation instead shrinks the variance to $0.5,I$ at the midpoint, and its OCT FID rises to 122.3 vs. 18.0 for slerp (Table 5, 20 steps).
  4. Since $\mathbb{E}|Y - \hat Y|_2^2 = \sum_k \mathbb{E}|Y_k - \hat Y_k|_2^2$, the volumetric bound follows from the slice-wise one whatever the inter-slice covariance of the $\epsilon_k$. The argument says nothing about the joint distribution across slices, which only the coupling affects (ll. 246–248).
AuthorsOpenReview ↗

Response to Reviewer 1 (2/2): Q5–Q7, minor remarks

Q5 (PFM). We thank the reviewer for on-point analysis. While PFM is a single-stage baseline, a 3D Song U-Net (3D convolutions, no attention, for memory) learns an x-prediction flow from a simple learned $1\times1$-convolution lifting of the fundus image, plus noise, directly to the 3D volume (A.2). Our method first trains TransUpNet as a 3D MMSE estimator and then learns a 2D flow on its slices. In our implementation the PFM network does not receive the time input; we will state this in A.2. Because PFM also changes the source, we added PRFM, a 3D flow started from the same TransUpNet estimate (Table 1). It reaches 17.7 dB PSNR, 3D-SSIM 17.2 and FID 29.7, vs. 18.0 dB, 19.1 and 18.0 for our 2D flow, and is also lower in both Dice scores. A 3D flow is thus feasible but worse; at batch size 1, only 16 base channels fit on a 48 GB GPU. On the references: we cite Li & He (2026) to justify our choice for the flow loss parameterization, and use the other citations to link the reader to additional formulations. PFM does not use the JiT architecture. Evaluated separately as a 2D backbone, JiT reaches FID 47.3 vs. 18.0 for the U-Net (A.9, Table 12). For the other references, see the minor remark on l.199.

Q6 (multi-scale MMSE vs. basic MMSE). Yes, the Encoder-Decoder (ED) baseline is meant for this explicitly. ED and Ours (no flow) are both deterministic 3D MMSE estimators, trained on the same data with the same budget (3 days on one A6000, A.2). They are not matched in parameters (TransUpNet 86.2 M, mostly its pretrained ViT-B/16 encoder; ED 2.50 M; we will add both counts to A.2), but on OCT ED costs 3.4× more per forward pass (5.46 vs. 1.60 TFLOPs; on CT 5.46 vs. 6.19, A.2). At equal OCT PSNR (19.4 dB), Ours (no flow) raises 3D-SSIM from 25.5 to 26.6, layer Dice from 23.0 to 32.6 and vessel Dice from 16.8 to 45.9 (Table 1). On CT it raises PSNR from 16.2 to 18.0 dB and 3D-SSIM from 40.9 to 58.3 (Table 2).

Q7 (poor FID at good PSNR). FID uses deep features (RETFound for OCT) that respond strongly to high-frequency texture, here OCT speckle. The MMSE estimator averages over all plausible speckle realizations and outputs a smooth anatomical envelope (Figures 3 and 4; slice fly-through GIFs at https://anonymous.4open.science/r/pmrf3dsynthesis-CEC5). Appendix A.8 quantifies this with degraded copies of the real test volumes (Table 11, Figure 10). Real against real gives FID 3.3. A Gaussian blur of only 1 px, which leaves the anatomy unchanged, gives 331.3, and removing speckle with non-local means gives 321.5–399.4. The MMSE estimate (435.5) lies just above real OCT blurred by 4 px (420.7). FID in the hundreds therefore means that speckle statistics are missing, not that the anatomy is wrong; the flow restores them (FID 18.0).

Minor remarks. "No flow" is the TransUpNet MMSE estimate (Stage 1 only). The former "linear flow" is now named "normal flow" (Tables 1 and 2, Section 5): the Stage 2 flow with a new, independent Gaussian noise draw for every slice. "Slerp flow" couples the noise across slices by spherical interpolation (Eq. 3). On l.199, we agree that Li & He did not propose this loss. The revised text now cites Gagneux et al. (2026). We use x-prediction as a design choice, not as a claimed advantage.

Dear authors, Sorry for the delay. Thanks for your careful answers, they helped me understand better the paper.

About Q5. I did not find in A2 the reason why you use a network independent on time. It could significantly weaken the baseline. Could you explain the motivation for this choice and clarify whether PRFM is also trained without time conditioning? Regarding the additional comparison with the 3D flow, how can we determine whether the performance gap is due to the proposed method or to differences in network architecture and capacity?

Overall assessment on the paper: I like the 3D-MMSE estimation strategy proposed in the paper. I find the ablations on the method quite complete. I unfortunately don't have a very good idea of how impactful this new approach can be on the medical imaging community because I don't know very well the domain.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal General Comments

General response to all reviewers: central contribution and summary of new results

We thank all reviewers for their time and constructive feedback. Changes in the manuscript are highlighted in blue. Summarizing the critique, two themes recur. Namely, our methodological contribution relative to PMRF and inter-slice noise coupling. We address both here; the per-reviewer comments give the details.

Central contribution. We demonstrate that a dense 3D medical volume can be perceptually refined with a 2D flow-matching model in pixel space. This keeps the PMRF guarantee, while controlling the refinement at test time.

  1. Theoretical grounding (Proposition 1). Slicing is linear, so $\mathbb{E}
Yk∣XY_k \mid X

= (\mathbb{E}

Y∣XY \mid X

)_k$. If Stage 1 converges to the conditional mean, each slice of the 3D MMSE estimate is the exact 2D MMSE estimate of that slice. A 2D PMRF trained on slices therefore starts from the source that PMRF theory (Ohayon et al., 2025) requires. MSE is additive over slices, $|Y-\hat Y|_2^2=\sum_k|Y_k-\hat Y_k|_2^2$, so the PMRF distortion bound carries over to the volume. No 3D generative model and no latent space are needed. A VAE encoder $E$ is nonlinear, so $E(\mathbb{E}

Y∥XY\|X

) \neq \mathbb{E}

E(Y)∥XE(Y)\|X

$, and latent MSE does not bound pixel MSE. Because the 2D flow is not given the slice index, the guarantee holds for slices pooled over position.
2. Controlled validation. We keep the Stage 1 estimator (TransUpNet) and the data fixed while introducing only the Stage 2 flow. On OCT, this lowers FID from 435.0 to 18.0 and raises retinal-layer Dice from 32.6 to 48.3 (Table 1). On CT, it lowers FID from 1.4 to 0.22 and heart-volume error from 107 to 67 mL (Table 2). Notably, a 3D flow started from the same estimate (PRFM, Table 1) is worse than our 2D flow on every OCT metric.
3. Test-time control. The number of Euler steps $S$ moves a single trained model along the distortion-perception frontier (Table 3). At $S=5$ the model matches the PSNR of Pix2Pix (18.6 dB) at a much lower FID (83.7 vs. 116.9, Tables 1 and 3).

Noise coupling. The new ablation (Table 5, Figure 11) shows that the source noise must be correlated across slices to avoid flicker, and retain the trained variance across slices to stay realistic. Slerp follows the inter-slice consistent stochasticity of Du et al. (2026) and presents a valid choice. Shared noise and variance-matched linear noise work equally well, and thus we drop the claim that slerp is better.

New results in the revised paper.

  • Speckle ceiling and VAE reconstruction on OCT (Appendix A.7, Table 10, Figure 9): for Reviewer 1 Q7 and Reviewer 3 W1.
  • FID of degraded real OCT (Appendix A.8, Table 10, Figure 10): for Reviewer 1 Q7.
  • Zero-noise flow (rows $\sigma_\epsilon=0$ in Tables 1 and 2): for Reviewer 1 Q3.
  • Five noise couplings, with a 3D flicker metric and FID (Table 5): for Reviewer 2 W1/Q1.
  • 3D flow from the same MMSE estimate (PRFM, Table 1): for Reviewer 1 Q5 and Reviewer 2 W1/Q3.
  • Fundus-only inference without any OCT, including layer-volume error (Appendix A.5, Table 8): for Reviewer 2 Q2 and Reviewer 3 Q2.
  • CT lung and heart volumes (Table 2): for Reviewer 3 W1/Q1.
  • Held-out-hospital evaluation (Appendix A.11, Table 13): for Reviewer 3 Q3.
AuthorsOpenReview ↗

Deadline Reminder

Dear Reviewers, With the discussion deadline approaching, we wanted to kindly follow up on our rebuttal. We kept our replies intentionally brief so they can be read quickly. Please let us know if our responses adequately address your questions or if any open issues remain that we can clarify now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.