acceptodds
Under review as a conference paper at ICLR 2027

Hidden Harm in Non-Atomic Demonstrations: Selective Counterfactual Component Intervention for Multimodal In-Context Learning

Abstract

Multimodal in-context learning (MICL) commonly constructs and evaluates in-context sequences at the level of complete image-text demonstrations, although prior studies show that visual and textual information can contribute differently across tasks. We ask a finer-grained question: for a query, do the image and text within the same demonstration exert the same effect? They need not. Their query-conditioned utilities can differ, and sometimes disagree in sign, such that a demonstration appearing useful at the whole-demonstration level may still contain a component that harms the prediction. We refer to this failure mode as hidden component harm, arising from demonstration non-atomicity. To mitigate it, we propose Selective Counterfactual Component Intervention (SCCI), a training-free framework that decomposes each demonstration into visual and textual components and derives query-specific counterfactual signals through leave-one-component-out inference. A reliability-aware decision rule intervenes only when the evidence is sufficiently informative, while otherwise preserving the context. Across six vision-language benchmarks and two primary multimodal language models, Qwen2.5-VL-7B and InternVL2.5-8B, our method consistently improves 4-shot MICL over the unmodified context and outperforms both random component removal and matched whole-demonstration removal. Further analyses of image-text utility disagreement, hidden component harm cases, and whole-demonstration attribution support the value of semantic component granularity, with gains remaining robust across demonstration sampling seeds and order permutations. Anonymous code is available at https://anonymous.4open.science/r/SCCI-3F45/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal In-Context Learning, Large Vision-Language Models, Counterfactual Explanation, Model Attribution, Context Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_97KGOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates hidden component harm in multimodal in-context learning, where a demonstration is beneficial overall but contains an image or text component that harms prediction. It proposes Selective Counterfactual Component Intervention (SCCI), which evaluates the original context and individual component removals using the model’s confidence in its generated answers. A reliability-aware router combines evidence concentration, calibration priors and abstention to remove at most one component or preserve the original context without updating model parameters. Experiments cover six benchmarks with two primary LVLMs and additional Idefics2-8B validation, supported by routing ablations, controlled removal comparisons and robustness analyses

Critical strengths
  • A precisely defined failure mode. The paper distinguishes harmful components from harmful demonstrations and formalizes cases where their removal has opposite effects. This provides a concrete reason to evaluate images and text separately rather than treating each demonstration as an indivisible unit. The analysis offers a useful perspective on preserving helpful context while removing interference.

  • A complete framework with substantive empirical support. SCCI integrates component probing, calibrated routing and abstention into a coherent inference procedure rather than applying deletion indiscriminately. Improvements across six benchmarks, particularly on InternVL2.5-8B, together with routing ablations, provide evidence that selective component intervention can meaningfully improve prediction accuracy.

Critical weaknesses
  • High inference cost with limited gains. The default setting requires nine inference branches per query, costing 8.43× and 8.11× as much as the original inference on the two primary models. Yet on Qwen2.5-VL, SCCI improves over TACO by only 0.10–0.59 percentage points on five benchmarks and falls behind by 0.02 points on GQA. These marginal gains do not convincingly justify the additional computation.

  • Insufficient validation of confidence as a reliability signal. SCCI compares confidence in answers generated under different contexts, while its entropy criterion only measures the concentration of candidate scores. Both can favor a confident but incorrect answer.

  • Potential amplification of textual priors. Removing demonstration images while retaining their text may strengthen reliance on demonstration answers and language patterns rather than the query’s visual evidence. Selecting outputs by confidence may further reinforce this bias. The paper lacks targeted analysis to distinguish genuine improvements in visual grounding from greater confidence driven by textual priors.

  • Unclear connection between hidden component harm and performance gains. The paper does not report what fraction of selected interventions satisfy Eq. (4), where removing a component helps but removing its entire demonstration hurts. It also does not quantify how much of the improvement comes from these cases. Consequently, it remains unclear whether SCCI’s gains actually arise from resolving the specific failure mode that motivates the method.

Questions for authors
  1. Requested comparison under matched computation. How does SCCI compare with trying multiple complete demonstration sets and selecting or voting on their answers under the same inference budget?

  2. Requested validation of routing reliability. How reliably do confidence gains and the entropy criterion identify beneficial interventions when the original answer is incorrect or the intervention changes the prediction?

Other questions please refer to weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 97KG

Dear Reviewer 97KG,

Thank you for recognizing the clear definition of hidden component harm and SCCI's empirical contributions. We address the four concerns using existing results and additional targeted analyses.

W1. Inference cost and matched-budget comparisons

A1: We acknowledge the substantial cost and modest gains over TACO on Qwen2.5-VL-7B (mean +0.26 pp; Table 1). The gain on InternVL2.5-8B is +1.86 pp. The original Fig. 7 already reports 8.43×/8.11× probing overhead; we do not claim inference acceleration.

(1) Matched candidates. We added nine-generation controls using resampled demonstration sets (Demo-set), reordered demonstrations (Demo-order), and self-consistency (Sec. 5.6; App. E.3). Under majority voting, they reach 60.02%, 60.21%, and 59.09%, versus 60.98% for SCCI on Qwen2.5-VL-7B (Table 14). Additional selection-rule comparisons test Max-logP and voting on candidate pools (Table 15).

(2) Isolating routing and cost. PCP and SCCI evaluate the identical nine component-removal branches, obtaining 59.68% and 60.98% (Table 14). Thus, the +1.30 pp gain isolates reliability-aware routing, not more generations. Other controls match generation counts, not necessarily FLOPs or latency; one-inference baselines are cheaper. SCCI is an accuracy–computation trade-off whose utility depends on deployment constraints.

W2. Reliability of confidence and entropy

A2: We agree that higher self-log-likelihood and concentrated evidence do not guarantee correctness. We added calibration and prediction-change analyses (Sec. 5.4; App. C).

(1) Confidence reliability. Mean ECE changes from 0.210 to 0.212 after intervention. For answer-changing cases, target-free evidence Δ\Delta achieves 0.668 mean AUC, exceeding first-token log-probability (0.549), entropy gain (0.561), and candidate consistency (0.553) (Table 4; App. C.2). Moreover, 9.5%–31.5% of high-confidence original predictions are wrong (App. C.4). We therefore treat Δ\Delta as an informative but imperfect signal.

(2) Conditional outcomes. When the original answer is wrong, the raw Δ\Delta-gated selector gains +13.72/+5.27 pp in low-/higher-entropy groups. Among answer-changing cases, gains are +24.53/−2.14 pp, with harmful-selection rates of 8.6%/35.0% (App. C.4, Table 8). Full SCCI also applies conflict calibration and abstention (Table 2; Fig. 5). These results show useful discrimination, not guaranteed reliability.

W3. Visual grounding versus textual priors

A3: We agree that removing an image might increase reliance on textual priors. The original component-utility analysis (Fig. 2) and random-component/whole-demo controls (Table 3) support selective intervention, but do not prove improved visual grounding. We added two diagnostics (App. K).

(1) Removal modality. Among executed interventions on Qwen2.5-VL-7B, 65.9% remove text while retaining images and 34.1% remove images while retaining text (Table 30). Thus, SCCI does not preferentially delete images.

(2) Harmful copying. We measure incorrect predictions that copy a demonstration answer. The overall rate increases by +1.2 pp versus the unchanged context; changes are +0.85 pp after image removal and +1.80 pp after text removal, including +3.7 pp for image removal on TextOCR (Table 31). This reveals a real copying risk, not a systematic image-removal effect. We do not claim these diagnostics conclusively prove stronger query-image grounding; direct query-image counterfactual tests remain necessary.

W4. Hidden component harm and observed gains

A4: The original paper defines hidden component harm (Eq. 4), analyzes component-utility disagreement (Fig. 2), and compares component-level with whole-demonstration removal (Table 3). To directly quantify the contribution of the motivating failure mode, we added selected hidden-harm attribution (App. G.2, Table 19).

Among executed interventions, 28.4%/31.6% satisfy the hidden-harm condition on Qwen2.5-VL-7B/InternVL2.5-8B, respectively. These cases contribute +0.96/+3.41 pp to SCCI's overall accuracy gains of +2.20/+8.67 pp, accounting for approximately 43.6%/39.3% of the net improvements. The remaining cases contribute +1.24/+5.26 pp. All utility-based labels are computed post-hoc and never used for routing.

These results directly connect hidden component harm with SCCI's performance gains while showing that this failure mode explains only part of the improvements.

We appreciate the reviewer's feedback; our revisions sharpen both the empirical evidence and the limits of our claims.

Reviewer_dqGPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies multimodal in-context learning (MICL), where a vision-language model sees a few image–text examples before answering a query. It argues that a demonstration’s image, question, and answer can affect the prediction in opposite directions. So a demonstration that helps overall can still contain a harmful part. The authors propose SCCI, a training-free method that scores the model’s confidence in its answer and suggests a protocol to return the most confident run, blend the scores with a prior learned on a labeled calibration set, or keep the original answer.

Critical strengths

Discovering the hidden component harm phenomenon in MICL. Treating a demonstration’s image and text as separate units fills a gap between choosing whole demonstrations and pruning tokens.

Critical weaknesses

W1. It seems the gains come from picking the most confident of nine answers at about 8× the compute, not from removing components.

  • SCCI produces 9 answers from 9 prompt variants and returns one the model is most confident in. This is essentially best-of-9 answer selection.
  • The baselines are not matched in compute cost. RICES, MMICES, TACO and CATP each use about one inference.
  • The controls don’t isolate the mechanism. No control applies the same rule to nine answers produced in other ways (resampled or reordered demonstrations, sampled decoding), and none takes a majority vote. A fair ablation test would keep the same ‘nine answers, pick one’ procedure but create the nine candidates differently.

W2. The evidence for generality is narrow, and the gains are largest where the baseline is weakest.

  • Only InternVL2.5 (2024) and Qwen2.5-VL (2025) are tested. Newer models with stronger multi-image training (e.g., InternVL3.5, Qwen3.5-VL) and other model sizes are not evaluated on the proposed method.
  • Zero-shot results are not reported.
  • All benchmarks are short-answer recognition or VQA. They include no math or multi-step reasoning datasets (e.g., MathVista), no tasks where the demonstrations define the task (e.g., VL-ICL operator induction), etc.
  • Main comparisons are 4-shot only.
  • The sampling protocol for demonstrations is not described.
  • Other shot counts report only gains over vanilla, with no baselines or cost.
  • In-distribution only. Demonstrations, calibration queries and test queries all come from the same benchmark, and routing thresholds and priors appear to be calibrated per model and benchmark using labeled data. Transfer of calibration across tasks and robustness to demonstrations from a different dataset are not tested, so each new task needs its own labeled calibration set.
Questions for authors

Please see weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer dqGP

Dear Reviewer dqGP,

Thank you for recognizing the importance of hidden component harm and for your detailed concerns. We address W1 and W2 through new controlled experiments, generalization tests, and clarifications.

W1. Candidate selection, cost, and mechanism isolation

A1: The original paper already reports router ablations (Table 2), removal controls (Table 3), and inference cost (Fig. 7). We added generation-matched candidate comparisons (Sec. 5.6, Table 6; App. E.3, Tables 14–15).

(1) Best-of-nine. SCCI is not unconditional confidence maximization: its router uses evidence concentration, a calibration prior, and abstention (Sec. 4.2). On identical nine component-removal candidates, PCP (Max-logP), majority vote, and SCCI achieve 59.68%, 59.18%, and 60.98%, respectively. Thus, SCCI gains 1.30 pp over PCP through routing rather than extra candidate generation (App. E.3, Tables 14–15). Original random-component and whole-demo removal controls provide complementary granularity evidence (Table 3).

(2) Cost fairness. We agree that RICES, MMICES, TACO, and CATP are not compute-matched. SCCI's measured 4-shot probing overhead is 8.43×/8.11× on Qwen2.5-VL-7B/InternVL2.5-8B (Fig. 7). New controls use nine generations; PCP and SCCI use exactly the same cached branches, matching their generation cost (Table 6; App. E.3). We do not equate matching generation counts with matching FLOPs or latency across other methods. SCCI trades computation for accuracy, not speed.

(3) Candidate-source and voting controls. With nine generations, self-consistency, Demo-set, and Demo-order achieve 59.09%, 60.02%, and 60.21%, versus 60.98% for SCCI (Table 6; App. E.3, Table 14). We also apply the same Max-logP and majority-vote rules to each candidate source. For component removal, temperature sampling, Demo-set, and Demo-order, Max-logP yields 59.68/59.56/60.10/60.28%, versus 59.18/59.09/60.02/60.21% under voting (App. E.3, Table 15). SCCI exceeds the strongest simple-selection alternative by 0.70 pp. Component removal alone is not always best; the evidence supports its combination with reliability-aware routing, not a universal advantage of candidate construction.

W2. Generalization and calibration

A2: We expanded the originally 4-shot, mainly in-domain evaluation across backbones, task types, shot counts, distribution shifts, and calibration regimes.

(1) Models and baseline strength. On new InternVL3.5-8B and Qwen3.5-4B, SCCI improves Vanilla from 69.40% to 70.53% (+1.13 pp) and 67.25% to 68.08% (+0.83 pp), respectively, also exceeding PCP (App. J, Table 27). We acknowledge larger gains on weaker benchmarks: lower-/higher-baseline gains are 3.12/1.29 pp (Qwen2.5-VL-7B) and 11.02/6.32 pp (InternVL2.5-8B), but gains remain positive on stronger tasks (Sec. 5.2).

(2) Task diversity and zero-shot. On MathVista, gains are +0.84/+0.73 pp (Qwen2.5/InternVL2.5); on VL-ICL Operator Induction they are +0.31/−0.16 pp, indicating limited transfer when demonstrations define the task (App. J, Table 28). At zero-shot, no demonstration component exists to remove; SCCI therefore reduces to Vanilla by construction (App. B.3).

(3) Shots and cost. We added PCP and TACO comparisons at 2/4/6/8 shots (App. E.4, Table 16), beyond the original Fig. 6. SCCI outperforms both at each shot budget on both models. Overhead grows 4.68×→15.86× (Qwen2.5) and 4.55×→15.41× (InternVL2.5) from 2 to 8 shots.

(4) Sampling and shifted demonstrations. App. B.3 specifies disjoint benchmark-specific support pools, matched contexts, and fixed queries; App. D tests sampling and order. With cross-dataset demonstrations on VQAv2/OK-VQA/VizWiz, SCCI retains +1.12/+0.57/+0.23 pp under target-task/LODO/source-only calibration, respectively (App. J, Table 29), though gains shrink under shift.

(5) Calibration transfer. LODO transfers label-selected routing hyperparameters from the other five benchmarks while using unlabeled target statistics, obtaining 60.19% versus 58.77% Vanilla and 60.98% target-calibrated (Sec. 5.4, Table 5; App. F.4). Fully source-calibrated transfer needs no target calibration queries or labels and achieves 59.62% (+0.85 pp) (App. F.5, Table 18). Labeled target calibration is therefore helpful, but not required to retain average gains.

We thank the reviewer for prompting these more rigorous controls and clearer limitations.

Reviewer_wN55OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies a failure mode in multimodal in-context learning (MICL), where the image and text components of the same demonstration can have different or even opposing effects on model predictions. The authors term this phenomenon hidden component harm, where a demonstration that appears beneficial overall may contain an individually harmful component. To address this, the paper proposes Selective Counterfactual Component Intervention (SCCI), a training-free framework that evaluates the effects of removing individual image or text components through counterfactual inference. A reliability-aware routing mechanism uses model confidence, entropy-based gating, and calibration statistics to selectively remove harmful components while preserving useful context. Experiments across six vision-language benchmarks and three LVLMs demonstrate improvements over standard MICL and several context-optimization baselines, with additional analyses of component utility and intervention reliability.

Critical strengths

S1. The paper identifies an interesting and relevant limitation of multimodal in-context learning by investigating the potentially conflicting contributions of individual demonstration components.

S2. The experimental evaluation is reasonably comprehensive, covering six benchmarks, multiple LVLM architectures, and comparisons with existing context-optimization approaches.

S3. The paper is generally well written and includes useful ablation studies, controlled removal comparisons, and robustness analyses supporting the proposed approach.

Critical weaknesses

W1. Reliability of Counterfactual Evidence Is Not Sufficiently Established: The proposed framework relies on changes in the model's self-log-likelihood across counterfactual contexts to identify harmful components. However, higher confidence in a generated answer does not necessarily imply improved correctness, particularly when different interventions produce different answers. The reported top-action agreement between the target-free evidence and ground-truth-based utility is only 28.6%–44.6%, indicating substantial disagreement. More rigorous evaluations of confidence calibration, incorrect high-confidence interventions, and alternative utility estimators are necessary to establish the reliability of this signal.

W2. High Computational Overhead and Lack of Compute-Matched Comparisons: SCCI requires evaluating the original context and every single-component counterfactual, resulting in nine inference branches for the default 4-shot setting. The reported computational overhead is approximately 8.1–8.4× compared to vanilla MICL, which is substantial for a method intended to improve inference-time performance. Moreover, some improvements over the strongest baselines are relatively marginal, particularly on Qwen2.5-VL-7B. The paper should include compute-matched comparisons against approaches such as self-consistency, multiple candidate generation, or confidence-based answer selection to establish whether the improvements justify the additional computation.

W3. Limited Methodological Novelty and Unclear Generalization of the Routing Mechanism: The proposed method largely combines leave-one-component-out counterfactual evaluation with heuristic confidence-based selection, entropy thresholds, and calibration priors. While the component-level perspective is interesting, the individual methodological ingredients are relatively straightforward, and the paper does not establish why the particular routing formulation is preferable to simpler alternatives. Furthermore, the routing parameters are selected using labeled calibration data, raising questions about sensitivity to task-specific calibration and generalization to unseen tasks. Cross-task calibration experiments and stronger comparisons with simpler confidence-based routing strategies would make the methodological contribution more convincing.

Questions for authors

See Weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer wN55

Dear Reviewer wN55,

Thank you for your constructive feedback and for recognizing the relevance of our research problem, experimental breadth, and presentation quality. We have addressed all three weaknesses through additional experiments, analyses, and clarifications in the revised manuscript. Our point-by-point responses are provided below.

W1. Reliability of counterfactual evidence

A1: We agree that higher self-log-likelihood does not necessarily imply improved correctness. We added calibration and prediction-change analyses (Sec. 5.4; App. C.2–C.4). On Qwen2.5-VL-7B, mean ECE changes from 0.210 to 0.212 after intervention, indicating no consistent calibration improvement. For answer-changing cases, the counterfactual evidence Δ achieves a mean AUC of 0.668, outperforming first-token log-probability (0.549), entropy gain (0.561), and candidate-answer consistency (0.553). High-confidence original predictions (confidence ≥ 0.8) remain incorrect in 9.5%–31.5% of cases. Among answer-changing cases, the raw Δ-gated selector exhibits lower harmful-selection rates in low-entropy than high-entropy groups (8.6% vs. 35.0%). We acknowledge the limited 28.6%–44.6% top-action agreement across model–benchmark settings and emphasize that Δ is an informative but imperfect routing signal, not a correctness guarantee.

W2. Computational overhead and compute-matched comparisons

A2: We agree that SCCI incurs substantial inference overhead and is not an acceleration method. We added nine-generation comparisons against self-consistency, resampled demonstration sets, reordered demonstrations, and raw PCP (Sec. 5.6; App. E.3). On Qwen2.5-VL-7B, SCCI achieves 60.98% mean accuracy across six benchmarks, outperforming Demo-order (60.21%), Demo-set (60.02%), self-consistency (59.09%), and PCP (59.68%). Crucially, PCP and SCCI evaluate exactly the same original and eight counterfactual branches, providing matched LVLM generation cost and isolating the contribution of reliability-aware routing. SCCI improves over PCP by 1.30 percentage points, demonstrating that its gains cannot be attributed solely to additional candidate generation. The new comparisons, together with Fig. 7, provide complementary evidence on routing effectiveness under matched branch-generation cost and computational overhead. We acknowledge that equal generation counts across other baselines do not guarantee identical FLOPs or latency, and that the reported 8.43×/8.11× probing overheads remain substantial. Thus, SCCI improves candidate utilization at additional computational cost rather than accelerating inference.

W3. Methodological novelty and routing generalization

A3: We clarify that SCCI's primary contribution lies in identifying hidden component harm and enabling query-conditioned component-level intervention, rather than introducing individually novel routing heuristics. We added the routing rationale (Sec. 4.2) and comparisons with simpler strategies (Sec. 5.3; App. F.3). All routing variants reuse the same nine cached branches. SCCI achieves 60.98% mean accuracy, outperforming Gated PCP (60.30%), peak-evidence gating (60.01%), base-confidence gating (59.86%), and candidate-confidence gating (58.87%). We further added leave-one-dataset-out (LODO) calibration (Sec. 5.4; App. F.4), selecting routing hyperparameters from the other five benchmarks without target-task label-based tuning. LODO achieves 60.19% mean accuracy, compared with 60.98% under target-task calibration and 58.77% for Vanilla MICL, retaining improvements across all six benchmarks. Together with the existing ablations and calibration sensitivity analyses (App. F.1–F.2), these results support the effectiveness of SCCI's routing design and the cross-task transferability of its routing hyperparameters, while acknowledging task-dependent degradation.

We appreciate the reviewer's suggestions, which have helped us strengthen the empirical validation, clarify the methodological contributions, and present the limitations of SCCI more explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.