acceptodds
Withdrawn from ICLR 2027

When Examples Fail to Teach: The Demonstration Gap in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal in-context learning (ICL) can help or harm vision-language model predictions, complicating demonstration selection. To understand this difficulty, we quantify the structure of demonstration utility across 13 models, three multiple-choice visual question answering datasets, and 27 conditions through exhaustive one-shot query–demonstration evaluations, measuring changes in correct-answer support and prediction correctness. Models that solve many of the same questions at zero-shot often benefit from different demonstrations on questions they both get wrong, despite partial utility transfer. On zero-shot errors, demonstration main effects are small, while query and pairing effects dominate utility variation. Selecting demonstrations by their average rescue rates rarely yields consistent gains across held-out query splits. Measured content similarities show weak linear alignment with pair-specific rescue patterns. In two end-to-end settings, the evaluated retrievers do not consistently improve accuracy. As a simple example of using query characteristics, we combine a simple trained gate for likely errors with voting over separate one-shot predictions using random demonstrations. This procedure improves accuracy over zero-shot by 1.22 percentage points on A-OKVQA and 0.53 on AI2D, while breaking fewer correct answers than the retrievers. These findings suggest the potential value of studying multimodal ICL through the characteristics of queries themselves.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal In-Context Learning, Vision-Language Models, Demonstration Utility, Demonstration Selection, Multimodal Reasoning, Visual Question Answering, Retrieval-Augmented Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WG6gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why a demonstration could be useful or harmful in the multimodal in context learning setting. It quantifies the structure of demonstration utility across 13 models, three multiple-choice visual question answering datasets, and 27 conditions through one-shots evaluations, measuring changes in correctness.

It gets several findings through experiments:

  1. which demonstrations can help rarely generalize across the models and queries
  2. the query matters more than the demonstration itself on the final performance
  3. the content similarity between query and demonstration cannot used as a signal to predict where that demonstration will be useful or not.

It also develops a gate-vote method which can moderately improve the final answer accuracy in the multimodal ICL setting.

Critical strengths

The paper has the below strengths:

  1. Large scale of experiments. It did large scale of the matrix evaluations (over 250,000 runs per condition) which provides robust statistical foundation
  2. Metrics decouple. It decouple the performance metrics into rescue and breaks, which clearly explain why the standard retrievers often fail
  3. Clear control. The Option -Text Control successfully isolates and exposed superficial answer-latter copying biases.
Critical weaknesses

I have a few concerns about the experiments design and their results.

  1. Data leak.
    App. A says that A-OKVQA and AI2D are from their training splits. Among the evaluated models, some of them may have seen these dataset during the tuning (e.g LLaVA-1.5's training datasets include A-OKVQA). Could the authors state which models were exposed to which training splits? It raises a concern about the data contamination. Memorized queries would have high zero shot confidence and leave less room for any demonstration to help. Also the contamination would differ across models which pollutes the cross model comparison in Sec.3. It may also explain in sec 5 why the retrievers failing below the zero-shot.

  2. Unbalanced query letters may turn letter copying into apparent demonstration main effects.
    App. A says demonstration candidates are balanced by the gold letter but the queries are not. Assuming letter A is the most common correct answers for the queries, based on the section 4.2's conclusion which a model tends to copy the demonstration's answer letter, then demonstration ending with B will rescue more queries then other letters. It will look like a good demonstration but the rescue has nothing to do with its actual content.

  3. Real content vs. Prompt noise.
    While the paper says the aggregate variance share of pairing residuals is consistent across baseline and explicit wordings, this does not show the pairing effect is from prompt sensitivity or actual content. If the pairing residual is largely from the prompt sensitivity, then it won't be replicated across wordings. Since the paper has evaluated both instruction settings, it should also report the correlation between R_baseline and R_explicit to demonstrate if the pairing effect is from stable semantic interaction or unreplicable prompt sensitivity

  4. Multiple variables in the E2E evaluation The proposed Gate + Arbitration pipeline use 20 random demonstrations with a selective gate to outperform retrievers. However, this comparison changes three variables at a time including 1. when to use the demonstration (selective gating) 2. ensemble size (20-draw voting vs 1 draw) 3. selection strategy (random vs retrieved). To improve the retrieved is actually weaker, the authors need evaluate the Gate + Retriever Arbitration baseline (using the retriever's top-20 candidates for the vote)

Questions for authors

No specific questions apart from the above concerns. And for the above concerns, can the authors answer / address them?

Reviewer_bKJEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies demonstration utility in multimodal in-context learning for vision-language models. The authors perform large-scale exhaustive one-shot query, demonstration evaluations across 13 models, 3 multiple-choice VQA datasets, 27 conditions, and analyze the relative contributions of query effects, demonstration effects, and pairing effects. The paper finds that query and pairing effects dominate, while demonstration main effects are relatively small. Based on these findings, the paper further proposes a simple query-centric strategy, gate + arbitration, which uses a trained gate to identify likely zero-shot errors and then aggregates multiple random one-shot predictions.

Critical strengths
  1. The objective of the paper is clear and the structure is well organized, and the empirical analyses are presented in a reasonably systematic way.

  2. The paper conducts research on an interesting diagnostic question about multimodal ICL. Understanding whether utility mainly comes from the query, the demonstration itself, or their pairing is a useful perspective for studying multimodal ICL.

Critical weaknesses
  1. Experiments are only in the 1-shot setting, while shot number is an important factor in ICL and 1-shot alone cannot fully reflect demonstration and pairing effects;
  2. The scope is narrow because all datasets are multiple-choice VQA datasets, so conclusions may not generalize to broader multimodal reasoning tasks; In fact, the true significance of ICL lies in generalizing the model's ability to new tasks, and the VQA task has already learned the form of this task during the training of VLMs.
  3. Too few ICL baselines are compared, mostly relatively basic similarity-based approaches. Better ICL retrieval methods do not have any comparisons. For example, the Lever LM (2024 NIPS) retrieves demonstrations based on pairing effects.
  4. gate + arbitration requires 20 separate one-shot runs and may benefits from self-consistency, so comparison to standard retrieval methods is not fair. Experiments such as RICES + arbitration or gate + RICES are missing.
  5. the introduction asks "how much do query, demonstration, and pairing effects contribute to variation in demonstration utility", but the paper does not provide a sufficiently clear answer to that question.
Questions for authors

see Weaknesses above.

Reviewer_KWwWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents an extensive empirical analysis of demonstration utility in vision-language models, covering 13 models and 27 experimental conditions. While the scale of the experiments and the query-demonstration variance decomposition are valuable, I have significant concerns about the paper's novelty, methodological contribution, and practical significance. The limitations of multimodal in-context learning have already been extensively investigated in prior work (See weaknesses). These studies have documented unreliable demonstration benefits, shallow pattern matching, inadequate utilization of visual context, and failures in transferring information from demonstrations to queries. The present paper offers additional quantitative characterization of these phenomena, but does not clearly establish a sufficiently distinct conceptual contribution. Furthermore, the proposed gate-and-arbitration approach does not directly address the underlying failure of multimodal demonstration learning. Instead, it uses an error detector and multiple randomly sampled one-shot predictions to selectively revise model outputs. The resulting accuracy improvements are modest, and the evaluation does not control for the additional inference computation relative to retrieval baselines.

Critical strengths
  1. The paper studies 13 VLMs across three datasets and 27 conditions, with exhaustive query–demonstration pair evaluations. This gives a much richer picture of multimodal ICL behavior than reporting only average few-shot accuracy.
  2. Separating utility into query, demonstration, and query–demonstration pairing effects is a useful empirical tool. The rescue/break analysis is also helpful for understanding when demonstrations help versus harm.
  3. The paper considers answer-letter effects, content similarity, multiple retrieval baselines, and several model families. This makes the empirical analysis relatively thorough.
  4. The probelm of in-context learning in multimodal setting is very practically relevant and solving it would have huge impact on the field overall.
Critical weaknesses
  1. The unreliability of multimodal ICL, shallow use of demonstrations, and failures to exploit visual context are already well studied in recent work such as Huang et al. (2025), Wang & Li (2026), and dos Santos et al. (2025). The paper adds quantitative characterization, but the conceptual advance over this literature is not fully clear. 2/ The proposed gate-and-arbitration approach is mainly an inference-time workaround rather than a method that improves demonstration understanding or selection. The reported gains are also modest, especially on AI2D.
  2. The proposed method uses many more forward passes than the retrieval baselines, making it difficult to attribute the gains specifically to the query-centric strategy. A fairer comparison would use the same gating, arbitration, and inference budget for random versus retrieved demonstrations.

Huang et al. (2025), Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models

Wang & Li (2026), Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

dos Santos et al. (2025), What Do Vision-Language Models See in the Context? Investigating Multimodal In-Context Learning

Questions for authors
  1. How do the authors distinguish the paper’s main contribution from prior work showing that multimodal ICL is unreliable and often driven by shallow or incomplete use of demonstrations, e.g., Huang et al. (2025), Wang & Li (2026), and dos Santos et al. (2025)?
  2. Since the proposed gate + arbitration method uses substantially more forward passes than the retrieval baselines, can the authors provide a compute-matched comparison under the same gating and arbitration budget?
  3. Can the authors provide stronger evidence that the proposed method addresses the underlying demonstration-learning problem, rather than simply avoiding harmful demonstrations through selective prediction revision?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.