acceptodds
Under review as a conference paper at ICLR 2027

A Domain-Aligned Encoder Evaluation Framework for FID-like Quality Metrics on Synthetic 3D Medical Imaging

Abstract

Fréchet Inception Distance (FID) is the de facto standard for evaluating synthetic image data. For 2D natural images, FID's validity has been shown to depend strongly on the features deemed salient by the used image encoder. For 3D medical imaging, FID is either omitted due to 2D-3D incompatibility or is implemented with the Med3D image encoder as feature extractor, although this substitution has not been scrutinized. We present a framework that evaluates Med3D and other encoders along multiple criteria with controlled signals, including domain-relevant distortions and synthesis-derived fidelity and empirical-coverage signals, which ranks encoders without requiring expert annotation. Additionally, we conduct an exploratory analysis of alignment with expert visual perception, covering overall, anatomical and vessel quality as well as anatomical and pathological variation. In a use-case on Time-of-Flight Magnetic Resonance Angiography with six candidate encoders, Med3D showed no detectable response to vessel masking or to controlled changes in empirical coverage. Self-supervised ResEncL encoders pre-trained on OpenMind responded to most controlled signals, with ResEncL-OM-S3D stronger than Med3D in five of six and ranking first overall. Expert-rating associations were weak for all encoders and did not separate them, reinforcing the need for systematic encoder evaluation. The framework facilitates reliable evaluation of synthetic medical imaging and is designed to transfer to other medical imaging modalities.

Primary area:
datasets and benchmarks
Keywords:
generative model evaluation, Fréchet Inception Distance, image synthesis, 3D medical imaging, self-supervised encoders, TOF-MRA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Mo5UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for evaluating encoders used in FD-based metrics for synthetic 3D medical images. It compares six encoders based on feature relevance, sensitivity to image fidelity, and empirical coverage, together with expert annotations. On TOF-MRA dataset, Med3D shows limited sensitivity to vessel removal and coverage changes, while RedEncL-OM-S3D performs best across the tests. However, the correlation between encoders with annotated quality remain limited.

I overall like this paper's motivation, the three aspect evaluation, but also feel this current experiments need to be expanded (both breadth and depth) to give us more general insights for medical image synthesis.

Critical strengths
  1. The motivation is important, an encoder pertained on medical images may still miss clinically relevant differences when used for image evaluation.
  2. The experimental design evaluates feature relevance, fidelity, and coverage separately, providing a more comprehensive aspects than just FD comparison.
  3. Vessel masking is a useful clinical orientated test, and they also provide expert ratings for controlled experiments.
Critical weaknesses
  1. the evaluation is limited to one modality, one anatomical region. It is hardly to show a broad conclusion for 3D medical image generation tasks.
  2. The study identifies encoder limitations, but does not test some practical ways to improve them, limiting their actionable insights to this field.
  3. The fidelity ranking use mixed real-synthetic datasets, it remains unclear whether the encoder ranking for real-versus-synthetic FD.
Questions for authors
  1. Does the framework give consistent findings on another modality or anatomical region?
  2. What is the more general insights the authors would bring to a broader audience who works on medical image synthesis? Don't use current encoder based FDs?
Reviewer_dUpvOpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a framework for evaluating 3D medical image encoders used in Fréchet Distance–based quality metrics, testing their sensitivity to image distortions, fidelity, and empirical coverage. It also introduces Branching Diffusion, which generates related images by sharing an initial denoising trajectory and then branching into independent stochastic continuations, controlling dataset redundancy while preserving individual sample marginals.

Critical strengths

The lack of well-validated metrics for evaluating synthetic 3D medical images is an important problem, and the authors make a valuable attempt to address it through a systematic, domain-specific evaluation of the encoders underlying these metrics.

Critical weaknesses
  1. Although the evaluation includes seven datasets, all focus on brain TOF-MRA. This demonstrates performance across datasets within one imaging setting, but does not establish generalization to other domains.
  2. C1 relies on small differences between leave-one-out FD values computed from heavily overlapping data subsets. Consequently, d_j (in Eq. 2) may be sensitive to the distortion type, severity, and random realization.
  3. C2 assumes that longer generative-model training produces higher-fidelity images. However, fidelity may plateau or fluctuate during training, so training duration alone may not provide a reliable quality ordering.
  4. The weak correlation between the final ranking and expert-rated image quality raises concerns about whether the framework captures clinically meaningful quality differences.
Questions for authors
  1. Could the authors apply multiple severity levels of each distortion type to the same fixed set of images and measure FD at each level? Repeating this across distortion seeds would help verify that C1 reflects distortion sensitivity rather than image-specific leave-one-out effects.
  2. Could the authors independently verify that image fidelity improves monotonically across the evaluated checkpoints, including intermediate and late epochs?
  3. Could the authors further investigate the weak associations with expert ratings? In particular, how much might they reflect low inter-rater reliability, limited sample size, or the mismatch between per-image FD influence and perceived quality, rather than limitations of the encoders or controlled criteria?
Reviewer_3FGDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an annotation-free framework to evaluate 3D feature encoders used inside Fréchet Distance (FD) metrics for synthetic 3D medical imaging, auditing the field-standard supervised Med3D backbone against five self-supervised OpenMind 3D encoders (ResEncL and PrimusM pretrained with MAE, VoCo, and S3D). Encoders are scored and ranked across three controlled-signal criteria -- sensitivity to image/domain distortions including vessel masking (C1), sensitivity to generative training epochs (C2), and sensitivity to sample-diversity loss via a proposed Branching Diffusion procedure that varies within-sample correlation while preserving the generator's marginal distribution (C3), alongside an exploratory 4-clinician visual rating study on 100 synthetic scans.

Evaluated on 3,304 Time-of-Flight MR Angiography (TOF-MRA) Circle-of-Willis scans with 3D-MedDiffusion, Med3D responds to global noise and training epochs but shows no detectable response to vessel masking (ρ=−0.007\rho = -0.007) or coverage loss (ρˉ=0.027\bar{\rho} = 0.027), whereas ResEncL-OM-S3D responds to all six probes and ranks first; associations with clinician quality ratings are weak (∣τb∣≤0.26|\tau_b| \le 0.26) and fail to separate any encoders.

Critical strengths
  1. Demonstrating that the standard Med3D FD backbone is completely insensitive to removing up to 5,000 vessel voxels (ρ=−0.007\rho = -0.007) and to empirical coverage collapse (ρˉ=0.027\bar{\rho} = 0.027) is a concrete, high-value warning for 3D medical synthesis, and Branching Diffusion cleanly isolates finite-sample coverage/clustering from marginal fidelity (pθ,b(x0)=pθ(x0)p_{\theta,b}(x_0) = p_\theta(x_0)).

  2. Comparing Med3D against five modern 3D self-supervised backbones (OpenMind) identifies ResEncL-OM-S3D as responsive across all controlled signals, while transparently reporting low inter-rater agreement (αord≤0.01\alpha_{\text{ord}} \le 0.01) and null encoder separation in the 4-clinician reader study.

Critical weaknesses
  1. In C2 and C3, FD compares 3,304 real scans against 3,204 real + 100 synthetic scans (only a 3% perturbation) across vastly different embedding dimensions (d = 320 for ResEncL, 864 for PrimusM, 2048 for Med3D). In C3 (where means match and only sample covariance shifts), finite-sample Gaussian covariance noise scales heavily with dd and is never disentangled from representation quality via dimension-matched projections or kernel metrics (CMMD). Moreover, declaring C2/C3 ties via the raw min–max range over 10 seeds rewards noisy encoders: ResEncL-OM-VoCo ranks #2 overall despite having no significant sensitivity to training fidelity (C2: ρˉ=−0.092,p=0.109\bar{\rho} = -0.092, p = 0.109), outranking ResEncL-OM-MAE which passes all 6/6 tests.

  2. Despite pitching a general 3D medical evaluation framework for ICLR, experiments cover only one cropped sub-volume (TOF-MRA Circle of Willis) and one generator (3D-MedDiffusion, with C2 fidelity monotonicity verified only visually on a few slices). Because the clinician study fails to separate any encoders and no multi-generator comparison (e.g., GAN vs. LDM vs. WDM) or downstream task utility (e.g., synthetic-trained vessel segmentation) is tested, it remains unproven whether ranking encoders by C1–C3 actually improves generative model selection.

Questions for authors
  1. Because ResEncL (d=320d=320), PrimusM (d=864d=864), and Med3D (d=2048d=2048) differ by over 6×6\times in dimension while C2/C3 evaluate a 3% synthetic mixture (3,204 real + 100 synthetic) using full-covariance Gaussian FD, does Med3D and PrimusM's insensitivity on C3 persist under (a) dimension-matched PCA projections (d=320d=320), (b) non-Gaussian kernel metrics like CMMD (which works directly on M=100M=100 pure synthetic samples without the 97%-real mixture), or (c) larger pure synthetic sets (M≥1,000M \ge 1,000)?

  2. Section 4 states that responding to C1–C3 is a necessary condition, yet ResEncL-OM-VoCo fails C2 completely (ρˉ=−0.092,p=0.109\bar{\rho} = -0.092, p = 0.109) and still ranks #2 overall because its high seed variance ([-0.597, +0.531]) triggers ties under the 10-seed min–max rule (even tying ResEncL-OM-S3D on C2 in Table 8). How does the ranking change if you use proper paired 95% CIs / Wilcoxon tests across seeds and gate encoders on passing all three necessary criteria?

  3. Can you validate on existing embeddings or a second generator/modality whether ResEncL-OM-S3D FD better ranks distinct generative models (e.g., 3D GAN vs. LDM checkpoints, or against quantitative vesselFM morphological statistics rather than visual-only inspection in C2), or generalizes to a second 3D modality (e.g., brain T1 MRI or chest CT)?

Comments on the paper

1 comment

We sincerely thank all three reviewers for their time and for their careful, constructive reviews. We are glad that all reviewers found the problem important, and we appreciate the specific, actionable suggestions.

Unfortunately, due to unforeseen circumstances we were not able to carry out the additional experiments within the discussion period, and we prefer not to respond with arguments alone where new evidence is needed. We would nevertheless like to confirm that we found the feedback very valuable, and we will incorporate it into a revised version of this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.