ValidCF: Evaluating Synthetic Counterfactual Evidence in Medical Vision-Language Models

Synthetic counterfactuals are increasingly used to evaluate medical vision-language models (VLMs). They test whether a model responds to clinically meaningful visual changes. However, the editing process can also introduce unintended changes. This creates a fundamental question: Did the model respond to the intended clinical change or to the editing process itself? We introduce ValidCF, a framework for evaluating synthetic counterfactual evidence before using it to interpret medical VLM behavior. ValidCF pairs each clinical counterfactual with a matched sham intervention. The sham follows the corresponding editing pipeline while preserving the target clinical state. This provides a control for measuring editing-associated response instability. ValidCF also examines intervention validity evidence and consistency across independent counterfactual generators. Uncertain interventions can then be accepted, deferred, or rejected rather than treated as equally reliable evidence. We evaluate ValidCF on 500 patient-distinctchest radiographs using two independent counterfactual generators and a medical VLM. We find a striking disagreement between conventional counterfactual evaluation and sham-controlled analysis. The generator with higher clinical transition accuracy also produces substantially greater response changes under sham interventions. Once this instability is considered, the apparent comparison between the generators reverses. Our findings show that an expected response to a counterfactual edit is not, by itself, sufficient evidence of clinical sensitivity. The intervention used to produce that evidence must also be evaluated. ValidCF provides a controlled approach for distinguishing intended clinical responsiveness from sensitivity to the editing process.