- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for evaluating encoders used in FD-based metrics for synthetic 3D medical images. It compares six encoders based on feature relevance, sensitivity to image fidelity, and empirical coverage, together with expert annotations. On TOF-MRA dataset, Med3D shows limited sensitivity to vessel removal and coverage changes, while RedEncL-OM-S3D performs best across the tests. However, the correlation between encoders with annotated quality remain limited.
I overall like this paper's motivation, the three aspect evaluation, but also feel this current experiments need to be expanded (both breadth and depth) to give us more general insights for medical image synthesis.
Critical strengths
- The motivation is important, an encoder pertained on medical images may still miss clinically relevant differences when used for image evaluation.
- The experimental design evaluates feature relevance, fidelity, and coverage separately, providing a more comprehensive aspects than just FD comparison.
- Vessel masking is a useful clinical orientated test, and they also provide expert ratings for controlled experiments.
Critical weaknesses
- the evaluation is limited to one modality, one anatomical region. It is hardly to show a broad conclusion for 3D medical image generation tasks.
- The study identifies encoder limitations, but does not test some practical ways to improve them, limiting their actionable insights to this field.
- The fidelity ranking use mixed real-synthetic datasets, it remains unclear whether the encoder ranking for real-versus-synthetic FD.
Questions for authors
- Does the framework give consistent findings on another modality or anatomical region?
- What is the more general insights the authors would bring to a broader audience who works on medical image synthesis? Don't use current encoder based FDs?
We sincerely thank all three reviewers for their time and for their careful, constructive reviews. We are glad that all reviewers found the problem important, and we appreciate the specific, actionable suggestions.
Unfortunately, due to unforeseen circumstances we were not able to carry out the additional experiments within the discussion period, and we prefer not to respond with arguments alone where new evidence is needed. We would nevertheless like to confirm that we found the feedback very valuable, and we will incorporate it into a revised version of this work.