- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper asks whether a medical VLM's response to a synthetic counterfactual edit reflects the intended clinical change or the editing process itself. The authors propose ValidCF, which pairs each clinical counterfactual with a "sham" edit from the same generator that is supposed to keep the target finding unchanged. They define Sham Flip Rate (SFR, how often the VLM's answer changes under the sham), Clinical Transition Accuracy (CTA, how often the VLM gives the expected answer on the clinical edit), and Matched Response Contrast (MRC = CTA − SFR). They also build an automated "intervention evidence" score (SIV) from six proxy terms (classifier outputs, SSIM, MAE, cross-generator agreement) and map it to Accept / Defer / Reject with two hand-set thresholds. Experiments use 500 NIH ChestX-ray14 images (pleural effusion, edited in both directions), two editors (RadEdit, InstructX2X), and one VLM (CheXagent-2-3b). InstructX2X has higher CTA than RadEdit (73.4% vs 67.0%) but a much higher SFR (44.8% vs 11.6%), so RadEdit gets a higher MRC (55.4% vs 28.6%). The authors conclude that CTA alone is not enough to judge counterfactual evidence, and that the editing pipeline itself should be evaluated.
Critical strengths
-
The question is a good one. Synthetic counterfactual edits are increasingly used to test whether medical VLMs look at the right evidence, and it is true that a response change after an edit does not by itself show the model reacted to the clinical content. Using a negative-control (sham) condition is a simple and sensible idea that could be applied to existing editors without retraining anything. This is why I think the paper is worth discussing, even though I do not think it delivers on its claims.
-
The paper is fairly honest about what it does not show. The authors state that the sham is not the true patient counterfactual, that MRC is descriptive and not causal, and that SIV weights and thresholds are design choices rather than clinically calibrated. The observation that the two editors behave very differently under the sham (44.8% vs 11.6% flips) is a useful warning for people who rely on edit-based evaluation, even if I think the evidence is too thin to support the stronger conclusions.
Critical weaknesses
-
The experiments do not support the main claim. The motivation (a model reacting to a synthetic edit is not the same as the model understanding real clinical change) is reasonable, and I agree with it. But the paper never tests anything real. All the evidence comes from two off-the-shelf editors applied to a single X-ray dataset. There is no real longitudinal data (NIH ChestX-ray14 has follow-up studies, MIMIC-CXR has sequential studies, and MS-CXR-T has temporal pleural effusion labels), and no radiologist review of the edits. The sham is just another edit from the same generator, so what it measures is how unstable one generator is under one kind of "do-nothing" edit. It says nothing about real clinical change, and not much about "the editing process" in general. The authors themselves admit (Sec. 4.2) that sham and clinical edits may have different nuisance distributions, which undercuts the abstract's claim that ValidCF separates "intended clinical responsiveness from sensitivity to the editing process." The headline "reversal" is also partly an artifact of the authors' own metric: MRC = CTA − SFR subtracts two different quantities (accuracy against an expected answer vs. any change from the original answer), and the 33-point SFR gap trivially outweighs the 6-point CTA gap. No confidence intervals or significance tests are given for CTA, SFR or MRC, with only 500 cases and a single run.
-
The sham and the measurements are under-specified, and the scope is too narrow for the general claims. The most important object in the paper, the sham edit z^sham (Eq. 2), is never defined. I could not find the prompts or instructions, masks, edit strength, sampling steps or seeds used for each generator. It is also unclear what "matched" means (same region? same edit size?), and how "target-preserving" was verified (only by a classifier and SSIM, not by a human). The 44.8% flip rate for InstructX2X may simply mean that a "no change" instruction is out-of-distribution for that model, and not that it is generally unstable. CTA is the VLM's accuracy on the edited image, so calling it the generator's "clinical transition accuracy" mixes up editor failure and VLM failure. The SIV score uses hand-picked weights and proxy metrics (SSIM, MAE), contains a sham term that partly repeats the sham finding, and is never checked against clinician judgement, so it is unclear what Accept/Defer/Reject actually mean (only 0.6% are rejected). The study also covers one modality (chest X-ray), one dataset with noisy text-mined labels, one finding (pleural effusion), 500 images, one 3B VLM, and two editors. I do not think any general conclusion about "medical VLMs" or "counterfactual evidence" can be drawn from this.
-
Presentation and missing statements. Figure 1, the main figure, is a plain left-to-right box diagram with tiny text; the first and last boxes are cut off at the edges, and it does not show the core idea (the clinical/sham pair and what is compared). It also uses names that do not match the text (v_anat, v_art, p_valid vs. v_struct, v_img; SIV described as "not a probability" in the text), and it needs a lot of redrawing. Key experimental details are missing (VLM prompt and answer parsing, which classifier is used, per-direction results). The paper says there are 3,000 VLM queries but only 2,500 images are described. I also could not find an AI/LLM usage statement, which is required this year, nor an ethics or reproducibility statement or code link.
Questions for authors
-
How exactly is the sham built for each generator (instruction/prompt, mask, strength, steps, seeds)? Is it matched to the clinical edit in region and magnitude? Has a radiologist checked, on at least a subset, that the sham really leaves the effusion status unchanged? Would a trivial control (re-encoding the image, small noise, or an auto-encoder round trip) give similar SFR numbers? Please also report results separately for Present→Absent and Absent→Present.
-
Can you show that SFR, MRC or SIV tracks anything real? For example, on real follow-up pairs (e.g., MS-CXR-T, MIMIC-CXR sequential studies), or by showing that SIV agrees with expert ratings of edit validity. Also, does filtering to "Accepted" interventions change any conclusion about the VLM? Without this, I cannot tell whether ValidCF is useful or just an extra score.
-
Please clarify the metrics. (a) Could the "reversal" survive other reasonable ways of combining CTA and SFR, such as computing both only on cases where the original answer was correct, or using direction-specific flips, and what are the confidence intervals? (b) Why are there 3,000 VLM queries when only 2,500 images are described? (c) Was CheXagent trained or tuned on NIH ChestX-ray14, and could that affect the results?