- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates hidden component harm in multimodal in-context learning, where a demonstration is beneficial overall but contains an image or text component that harms prediction. It proposes Selective Counterfactual Component Intervention (SCCI), which evaluates the original context and individual component removals using the model’s confidence in its generated answers. A reliability-aware router combines evidence concentration, calibration priors and abstention to remove at most one component or preserve the original context without updating model parameters. Experiments cover six benchmarks with two primary LVLMs and additional Idefics2-8B validation, supported by routing ablations, controlled removal comparisons and robustness analyses
Critical strengths
-
A precisely defined failure mode. The paper distinguishes harmful components from harmful demonstrations and formalizes cases where their removal has opposite effects. This provides a concrete reason to evaluate images and text separately rather than treating each demonstration as an indivisible unit. The analysis offers a useful perspective on preserving helpful context while removing interference.
-
A complete framework with substantive empirical support. SCCI integrates component probing, calibrated routing and abstention into a coherent inference procedure rather than applying deletion indiscriminately. Improvements across six benchmarks, particularly on InternVL2.5-8B, together with routing ablations, provide evidence that selective component intervention can meaningfully improve prediction accuracy.
Critical weaknesses
-
High inference cost with limited gains. The default setting requires nine inference branches per query, costing 8.43× and 8.11× as much as the original inference on the two primary models. Yet on Qwen2.5-VL, SCCI improves over TACO by only 0.10–0.59 percentage points on five benchmarks and falls behind by 0.02 points on GQA. These marginal gains do not convincingly justify the additional computation.
-
Insufficient validation of confidence as a reliability signal. SCCI compares confidence in answers generated under different contexts, while its entropy criterion only measures the concentration of candidate scores. Both can favor a confident but incorrect answer.
-
Potential amplification of textual priors. Removing demonstration images while retaining their text may strengthen reliance on demonstration answers and language patterns rather than the query’s visual evidence. Selecting outputs by confidence may further reinforce this bias. The paper lacks targeted analysis to distinguish genuine improvements in visual grounding from greater confidence driven by textual priors.
-
Unclear connection between hidden component harm and performance gains. The paper does not report what fraction of selected interventions satisfy Eq. (4), where removing a component helps but removing its entire demonstration hurts. It also does not quantify how much of the improvement comes from these cases. Consequently, it remains unclear whether SCCI’s gains actually arise from resolving the specific failure mode that motivates the method.
Questions for authors
-
Requested comparison under matched computation. How does SCCI compare with trying multiple complete demonstration sets and selecting or voting on their answers under the same inference budget?
-
Requested validation of routing reliability. How reliably do confidence gains and the entropy criterion identify beneficial interventions when the original answer is incorrect or the intervention changes the prediction?
Other questions please refer to weakness.
Response to Reviewer 97KG
Dear Reviewer 97KG,
Thank you for recognizing the clear definition of hidden component harm and SCCI's empirical contributions. We address the four concerns using existing results and additional targeted analyses.
W1. Inference cost and matched-budget comparisons
A1: We acknowledge the substantial cost and modest gains over TACO on Qwen2.5-VL-7B (mean +0.26 pp; Table 1). The gain on InternVL2.5-8B is +1.86 pp. The original Fig. 7 already reports 8.43×/8.11× probing overhead; we do not claim inference acceleration.
(1) Matched candidates. We added nine-generation controls using resampled demonstration sets (Demo-set), reordered demonstrations (Demo-order), and self-consistency (Sec. 5.6; App. E.3). Under majority voting, they reach 60.02%, 60.21%, and 59.09%, versus 60.98% for SCCI on Qwen2.5-VL-7B (Table 14). Additional selection-rule comparisons test Max-logP and voting on candidate pools (Table 15).
(2) Isolating routing and cost. PCP and SCCI evaluate the identical nine component-removal branches, obtaining 59.68% and 60.98% (Table 14). Thus, the +1.30 pp gain isolates reliability-aware routing, not more generations. Other controls match generation counts, not necessarily FLOPs or latency; one-inference baselines are cheaper. SCCI is an accuracy–computation trade-off whose utility depends on deployment constraints.
W2. Reliability of confidence and entropy
A2: We agree that higher self-log-likelihood and concentrated evidence do not guarantee correctness. We added calibration and prediction-change analyses (Sec. 5.4; App. C).
(1) Confidence reliability. Mean ECE changes from 0.210 to 0.212 after intervention. For answer-changing cases, target-free evidence Δ achieves 0.668 mean AUC, exceeding first-token log-probability (0.549), entropy gain (0.561), and candidate consistency (0.553) (Table 4; App. C.2). Moreover, 9.5%–31.5% of high-confidence original predictions are wrong (App. C.4). We therefore treat Δ as an informative but imperfect signal.
(2) Conditional outcomes. When the original answer is wrong, the raw Δ-gated selector gains +13.72/+5.27 pp in low-/higher-entropy groups. Among answer-changing cases, gains are +24.53/−2.14 pp, with harmful-selection rates of 8.6%/35.0% (App. C.4, Table 8). Full SCCI also applies conflict calibration and abstention (Table 2; Fig. 5). These results show useful discrimination, not guaranteed reliability.
W3. Visual grounding versus textual priors
A3: We agree that removing an image might increase reliance on textual priors. The original component-utility analysis (Fig. 2) and random-component/whole-demo controls (Table 3) support selective intervention, but do not prove improved visual grounding. We added two diagnostics (App. K).
(1) Removal modality. Among executed interventions on Qwen2.5-VL-7B, 65.9% remove text while retaining images and 34.1% remove images while retaining text (Table 30). Thus, SCCI does not preferentially delete images.
(2) Harmful copying. We measure incorrect predictions that copy a demonstration answer. The overall rate increases by +1.2 pp versus the unchanged context; changes are +0.85 pp after image removal and +1.80 pp after text removal, including +3.7 pp for image removal on TextOCR (Table 31). This reveals a real copying risk, not a systematic image-removal effect. We do not claim these diagnostics conclusively prove stronger query-image grounding; direct query-image counterfactual tests remain necessary.
W4. Hidden component harm and observed gains
A4: The original paper defines hidden component harm (Eq. 4), analyzes component-utility disagreement (Fig. 2), and compares component-level with whole-demonstration removal (Table 3). To directly quantify the contribution of the motivating failure mode, we added selected hidden-harm attribution (App. G.2, Table 19).
Among executed interventions, 28.4%/31.6% satisfy the hidden-harm condition on Qwen2.5-VL-7B/InternVL2.5-8B, respectively. These cases contribute +0.96/+3.41 pp to SCCI's overall accuracy gains of +2.20/+8.67 pp, accounting for approximately 43.6%/39.3% of the net improvements. The remaining cases contribute +1.24/+5.26 pp. All utility-based labels are computed post-hoc and never used for routing.
These results directly connect hidden component harm with SCCI's performance gains while showing that this failure mode explains only part of the improvements.
We appreciate the reviewer's feedback; our revisions sharpen both the empirical evidence and the limits of our claims.