acceptodds
Withdrawn from ICLR 2027

CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating an image for a requested medical condition requires more than plausible appearance: the condition must be visible in the image. Medical-domain adaptation, richer descriptions, and reward or preference alignment improve different aspects of generation, but agreement with text or visual plausibility alone does not necessarily make the requested condition recognizable in a new sample. We ask whether direct feedback on generated images improves expression of the requested condition beyond equal additional text-guided reconstruction. CRAFT (Clinical Reward-Aligned Finetuning for Medical Image Synthesis) tests this question by retaining reconstruction and scoring sampled images against an instance description, class-shared visual criteria, and the target label using a classifier trained on real images. In a matched CheXpert continuation from the same TI+LoRA initialization, CRAFT exceeds reconstruction-only training in DINOv2 balanced accuracy by 10.70 percentage points after five additional epochs [paired source-bootstrap 95% CI 7.28, 14.12]. Across three CheXpert training runs, removing diagnostic feedback lowers balanced accuracy for label recognition, while MetaCLIP2 description and checklist similarities barely change. Across four domains, CRAFT has the highest synthetic-method point estimates for DINOv2 balanced accuracy and both MetaCLIP2 similarities. Two physicians selected CRAFT in 55.3% versus TI+LoRA in 26.3% of 800 method-anonymous four-way judgments. Feedback on sampled images thus makes requested conditions more recognizable to a held-out classifier trained on real images while improving alignment with instance descriptions and shared clinical criteria.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
medical image synthesis, diffusion models, reward fine-tuning, differentiable rewards, text-to-image generation, multimodal alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1UyrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CRAFT adapts a medical image generator by combining reconstruction with three feedback terms on sampled images: Visual Description Consistency, Clinical Criteria Satisfaction, and Diagnostic Discriminability. It is evaluated on four datasets using DINOv2 target-label recognition, MetaCLIP2 similarity, KID, LPIPS, and a two-physician preference study. The main claim is that sampled-image feedback makes requested conditions more recognizable than reconstruction alone.

Critical strengths

The direction is reasonable: adding feedback on sampled images, rather than only fitting real images, is a sensible way to push medical generators toward clinically specified content.

The experimental effort is substantial: matched continuation on CheXpert, component removals, four domains, extra training seeds for the DD removal, and a blinded reader study.

Critical weaknesses

The central claim is not independently validated. DD directly optimizes a target-class classifier trained on real-image features, and the primary endpoint is target-label recognition by another feature-based classifier trained on real images. Using MedSigLIP for training and DINOv2 for evaluation reduces direct metric reuse, but the study still mainly shows that reward optimization transfers across pretrained representation spaces. It does not show that generated images contain clinically valid, diagnostically sufficient findings.

The physician study does not close this gap. Readers pick a preferred image among four candidates using a mix of label consistency, plausibility, realism, and image quality. They do not judge whether the requested finding is actually present, whether disease-specific criteria are satisfied, or whether each image is diagnostically correct. The paper also notes that the reader panel and the automatic benchmark use different generated images, so the two evaluations provide no per-image joint validation. The human study therefore supports relative preference, not the paper's main clinical claim.

Selection and evidence scope weaken the four-domain results. Checkpoints for CRAFT and TI+LoRA are selected with a SigLIP-family validation score that includes target-class accuracy and semantic similarities close to CRAFT's own objectives, which the authors acknowledge may favor CRAFT. The matched CheXpert continuation partially addresses this, but no equivalent control is given for the other three domains. Reported uncertainty also conditions on evaluated checkpoints, and the five seeds are generation seeds, not independent training runs, so optimization variability is not captured.

Questions for authors

Can you quantify agreement between the DINOv2 endpoint and physician judgments of disease presence on the same generated images?

Reviewer_ZsS1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CRAFT adds reward-based fine-tuning to a textual-inversion plus LoRA diffusion model for medical images. Alongside reconstruction, it uses three forms of feedback. Two measure agreement with an instance description and a class-level clinical checklist, while the third comes from a diagnostic classifier trained on real images. Experiments cover dermatology, chest X-ray, breast histopathology, and retinal imaging. The main result is that generated images from CRAFT are more often recognized as the requested condition by a separate classifier, including in a matched CheXpert continuation against the same amount of additional reconstruction training.

Critical strengths

The matched-continuation experiment is well designed. Both CheXpert arms start from the same selected TI+LoRA checkpoint and share the prompts, generation seeds, reconstruction optimizer and learning rate, batch size, and update count; the generated-image rewards are the main difference. That is a much cleaner control than simply comparing separately trained methods. The component-removal results are also useful because diagnostic feedback primarily affects label recognition while the text feedback terms affect semantic alignment.

Critical weaknesses

My main worry is what "recognizable" means in this setup. CRAFT is trained using one diagnostic representation and evaluated with a different classifier, which is better than evaluating on the training critic itself, but correlated visual shortcuts could still transfer between them. Figure A2 makes this concern concrete: several Fitzpatrick17k source and generated images contain visible "SFS"-like marks. A classifier-based reward could potentially exploit artifacts of this kind rather than only disease morphology. I would therefore like an explicit artifact analysis before interpreting the classification gain as improved clinical expression of the requested condition.

The physician study helps, but it does not fully resolve this issue. There are only two readers. CheXpert images are displayed at 512 pixels, while the other three domains are displayed at 128 pixels, and inter-reader Cohen's kappa ranges from 0.29 to 0.56. The endpoint is also the reader's preferred image among four candidates, not a direct assessment of whether the requested finding is present. I therefore see the reader study as preference evidence rather than a definitive validation of diagnostic correctness.

The methodological novelty also seems incremental. Reward-based diffusion alignment with text- and classifier-based signals is already established, including in medical imaging. The DPO comparator is difficult to interpret because the paper adapts a method originally used in a source-conditioned image-to-image setting to from-noise generation, and I could not determine clearly what model state the DPO training starts from. Since DPO performs close to plain TI in several settings, this initialization and adaptation detail matters. I would also like to see simple inference-time alternatives such as best-of-N selection or classifier guidance using the same diagnostic signal.

Questions for authors
  1. How often do generated images contain watermarks, text, or similar acquisition artifacts, and do the diagnostic-recognition gains remain after masking or otherwise controlling for them?

  2. How do best-of-N selection and classifier guidance compare with CRAFT at a similar generation or compute budget? Also, what exact checkpoint is used to initialize the DPO baseline?

  3. What are the two physicians' clinical specialties, and how closely are those specialties matched to the four imaging domains being evaluated?

Reviewer_fZCKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a method for improving the clinical accuracy of synthetic medical images generated by diffusion models by ensuring they are aligned with desirable properties of these images. The proposed method combines conventional diffusion reconstruction training with three differentiable rewards for visual description accuracy, clinical criteria, and diagnostic accuracy. To generate these, the proposed method uses multimodal LLMs to generate image descriptions and disease-specific clinical checklists and uses a frozen medical VLM and a classifier trained on real images provide feedback during optimization.

The authors conducted experiments across four imaging datasets in dermatology, chest X-rays, histopathology, and retinal fundus images and demonstrate improvements in diagnostic accuracy and alignment compared with existing adaptation methods. They also conduct a controlled experiment on CheXpert which shows significant improvement in balanced accuracy over reconstruction-only training, and they conducted a blinded physician study across four domains which shows their method is favored by the evaluators.

Critical strengths

This paper addresses an important problem of aligning generative models in non-verifiable domains and presents a technically sound approach for this problem in a medical application. Its experiments demonstrate reasonably consistent improvements in automated and physician evaluation metrics.

Critical weaknesses

(1) My main concern regarding this submission is that the proposed approach is that its main technical contribution has already been established in many other publications from 2-3 years ago. For example, Sun et al., NeurIPS 2023 (which the authors cite) also conduct alignment of diffusion models using an almost identical method where a reward model is trained to predict human adjudications of rubric criteria that evaluates the validity of synthetic medical images. There are many similar methods that follow the same idea, e.g. "Doctor Approved: Generating Medically Accurate Skin Disease Images through AI-Expert Feedback" published in NeurIPS 2025. I understand that CRAFT differs mechanically in that it uses differentiable vision-language similarity and diagnosis classifier rewards for alignment/fine-tuning, but these are incremental modifications of established approaches. I think that the authors should more clearly articulate their technical novelty relative to these works, and ideally they should provide stronger empirical comparisons demonstrating the advantages of their specific reward formulation. Otherwise, it won't be clear if this contribution would be informative to the broader NeurIPS readership.

(2) There are also possible issues with the clinical validity of the evaluation metrics. If I understand correctly, the diagnostic accuracy is measured using a DINOv2-based classifier rather than expert diagnostic annotations of generated images. Also, this classifier achieves only 47.06% balanced accuracy on real CheXpert images, so I am not sure about its reliability as a proxy for accurate diagnosis.

Questions for authors

Please respond to the weaknesses I raised above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.