acceptodds
Withdrawn from ICLR 2027

CARAE: Matched Counterfactual Transition Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Counterfactual supervision can make a model respond to an edited input, but the response may reflect nuisance variation introduced by the edit rather than the intended task change. A targeted counterfactual alone provides no reference for this ambiguity. Matched controls turn absolute response fitting into a relative comparison of target and non-target transitions within the same input. CARAE instantiates this idea with a question-conditioned visual state, a relative transition objective, and question-conditioned token-weight-modulated residual coupling to a vision–language model’s generation pathway. Expert localization in chest radiographs permits controlled target and non-target edits; the masks construct interventions offline and are never model inputs. On the canonical CheXlocalize protocol, CARAE raises target-versus-control answer AUROC from 0.5133 to 0.8038 at 3B scale and from 0.7053 to 0.8527 at 7B scale. The 7B paired improvement is 0.1474 (95% patient-bootstrap interval [0.0737, 0.2266]). Frozen checkpoints retain an advantage under held-out control constructions and replacement operators; dataset transfer within chest radiography is moderate on MS-CXR and weak in absolute terms on VinDr-CXR. These results support intervention-specific response under the evaluated construction family, without treating synthetic removal as a clinically adjudicated diagnosis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
counterfactual learning, vision-language models, matched controls, multimodal learning, medical vision-language models, chest radiography, counterfactual reasoning, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_p25wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a counterfactual-intervention-based learning framework called CARAE for improving reliability of multimodal large language models (MLLMs). Using chest X-ray images as a testbed, the authors first investigate the behaviors of models in identifying targeted and matched control edits. Then, they propose to optimize models’ relative behavior under these constructed interventions by learning more robust question-conditioned visual representations and via a gated residual injection module. Finally, they evaluate the performance of the counterfactual-intervention optimized models through cross-dataset experiments on two benchmark datasets. Experiments demonstrate the effectiveness of CARAE against several baselines.

Critical strengths

S1. The paper presents a new counterfactual-intervention-based learning framework called CARAE to improve model reliability. Employing targeted and matched control interventions simultaneously to investigate and optimize model behavior is an interesting idea.

S2. The way the authors propose optimize models’ relative behavior under these interventions and to learn more robust question-conditioned visual representations through this seem novel. Moreover, the authors demonstrate the effectiveness of using these constructed interventions to optimize models’ relative behavior against the competing approaches.

S3. The authors present detailed ablation studies to assess the contribution of each component of the suggested framework.

Critical weaknesses

W1. Most of the critical technical details are given in the Appendix, which makes the paper a hard-read.

W2. I found the paper highly related to the hallucination detection/migitation techniques suggested for MLLMs. However, no specific discussions and/or comparisons are provided in the paper, which weakens the paper’s quality.

W3. The following recent work is highly relevant to the submitted work, but is missing in the paper.

  • Lu et al., Counterfactual Anatomy-Guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs, MICCAI 2026

W4. Experiments are carried out by using Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct as the main backbones. No other generic MLLMs or medical MLLMs have been evaluated.

W5. The counterfactual interventions only include masking (filling the regions with mean color).

Questions for authors

Q1. Are these findings still be valid for MLLM families other than Qwen2.5-VL-3B-Instruct, especially for medical MLLMs?

Q2. Could interventions other than masking (blurring, adding noise, mix-up, etc.) could be used within the suggested framework? If so, how do they affect the learning?

Reviewer_4ZroOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors introduce CARAE, a method to help VLMs learn to respond correctly when a part of an image is edited or cropped’ through counterfactual learning. They use and compare two types of edits: one with an important part removed and one with an unimportant part. In this way, they train the model to be more precise and help it respond to the correct part of the image.

Critical strengths

For me, the critical strength is the overall idea. The setup was generally OK, tried different things and the matched control idea seems simple and shows some effectiveness.

Critical weaknesses

My critical concerns are:

  1. My most critical concern is that CARAE looks a bit unstable, in some methodology and results. Firstly, this method needs precise localization masks to create those targeted and controlled edits; and now, it’s unclear how it’ll work if something like this is not available. Then, the counterfactuals are not real clinical changes; rather made with a fake, blurry X-ray; which may induce some shortcut/bias. And, also correlation is not co-causation; the paper shows that the model's internal evidence state changes, but again, it doesn’t prove that these changes caused the correct answer. Instability issues are seen in results too, in Table 2, CIs are very very very wide; and unstable; due to very low sample size (20) used. And, the transfer is weak. When trying a new dataset (VinDr-CXR) the performance dropped a lot; so, it's not clear how it’ll be useful in different hospitals using different machines.
  2. Another critical problem is that, in the whole introduction paragraphs, there are no single references used. All these claims, backgrounds, research gaps and problem statements are given without any single reference of prior work; and makes it harder for reviewers to check and verify the claims made. It also shows how immaturely and carelessly this work is done; without maintaining proper scientific standards. Also, the references are not formatted properly, \citep vs \citet clash is visible in Related Works; strengthening the concern even more.
Questions for authors
  1. Following Critical Weakness 1, please resolve these concerns. Can you please report results with a larger set for Table 2? Instead of reporting AUC only in major tables, report accuracy, F!, Recall, AUPRC and other metrics too, to show the full picture. Can you please also explore how CARAE will work without an expert localization mask? And, if possible, can you show a strong transfer example and explore why VinDr-CXR failed?
  2. Also, can you provide any proper causal relationship,as you claimed? And, can you explain and prove that it can be used in diverse real-world scenarios with minimum overhead? Maybe, some human-doctor evaluation can be done in future works.
  3. Also, can you please answer how CARAE compares to some simpler methods; like, attention-based regularization, contrastive learning and so on?
  4. I’m also interested in exploring what the token weights actually represent? And, how sensitive CARAE is to the control matching quality? Can we also transfer this method to other domains like MRI, CT scans and so on? (given limited utility in X-ray in current stage)
  5. The training pipeline is very complex and long. How would you justify the gains over it?
  6. Can you please check and confirm if all references are done properly? Several works (for example, Müller et al. (ChEX) and Zhang et al. (CF-VLM)) refer to arxiv, despite they are old and published versions with proper DOIs are available.
Reviewer_LLU7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CARAE, a framework for improving the evidence-specific counterfactual responses of vision-language models. The authors argue that conventional counterfactual supervision may encourage models to respond to artifacts introduced by image editing rather than changes in task-relevant visual evidence. To address this issue, CARAE constructs factual, targeted, and matched-control image triplets, and introduces Evidence-Specific Transition Learning (EST) to encourage different responses to targeted and non-target interventions in a question-conditioned representation space. The learned representation is further integrated into the VLM's generation pathway through a token-weight-modulated gated residual mechanism. Experiments on chest radiographs demonstrate improved target-versus-control discrimination.

Critical strengths

None

Critical weaknesses

The paper suffers from substantial presentation issues that make it difficult to follow despite careful reading. My primary concern is that the paper does not clearly articulate the specific research problem it aims to address or sufficiently explain its motivation and significance. Furthermore, several key concepts and terms, such as transition learning, matched controls, and target-versus-control behavior, are introduced without adequate explanations or intuitive interpretations. As a result, it is difficult to establish a clear understanding of the research objective, the rationale behind the proposed methodology, and how the individual components contribute to solving the intended problem.

Questions for authors
  1. Could the authors clarify the specific research problem CARAE aims to address, including the limitations of existing approaches and the practical significance of solving this problem?
  2. Could the authors provide more intuitive explanations of key concepts, particularly matched counterfactual transition learning and target-versus-control behavior, and clarify how these concepts relate to the proposed methodology and its intended objectives?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.