- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
CRAFT adapts a medical image generator by combining reconstruction with three feedback terms on sampled images: Visual Description Consistency, Clinical Criteria Satisfaction, and Diagnostic Discriminability. It is evaluated on four datasets using DINOv2 target-label recognition, MetaCLIP2 similarity, KID, LPIPS, and a two-physician preference study. The main claim is that sampled-image feedback makes requested conditions more recognizable than reconstruction alone.
Critical strengths
The direction is reasonable: adding feedback on sampled images, rather than only fitting real images, is a sensible way to push medical generators toward clinically specified content.
The experimental effort is substantial: matched continuation on CheXpert, component removals, four domains, extra training seeds for the DD removal, and a blinded reader study.
Critical weaknesses
The central claim is not independently validated. DD directly optimizes a target-class classifier trained on real-image features, and the primary endpoint is target-label recognition by another feature-based classifier trained on real images. Using MedSigLIP for training and DINOv2 for evaluation reduces direct metric reuse, but the study still mainly shows that reward optimization transfers across pretrained representation spaces. It does not show that generated images contain clinically valid, diagnostically sufficient findings.
The physician study does not close this gap. Readers pick a preferred image among four candidates using a mix of label consistency, plausibility, realism, and image quality. They do not judge whether the requested finding is actually present, whether disease-specific criteria are satisfied, or whether each image is diagnostically correct. The paper also notes that the reader panel and the automatic benchmark use different generated images, so the two evaluations provide no per-image joint validation. The human study therefore supports relative preference, not the paper's main clinical claim.
Selection and evidence scope weaken the four-domain results. Checkpoints for CRAFT and TI+LoRA are selected with a SigLIP-family validation score that includes target-class accuracy and semantic similarities close to CRAFT's own objectives, which the authors acknowledge may favor CRAFT. The matched CheXpert continuation partially addresses this, but no equivalent control is given for the other three domains. Reported uncertainty also conditions on evaluated checkpoints, and the five seeds are generation seeds, not independent training runs, so optimization variability is not captured.
Questions for authors
Can you quantify agreement between the DINOv2 endpoint and physician judgments of disease presence on the same generated images?