acceptodds
Under review as a conference paper at ICLR 2027

Definitely Maybe: Can Multimodal In-Context Learning Induce Visual Concepts?

Abstract

Multimodal large language models (MLLMs) can adapt their predictions to multimodal demonstrations in context, but whether they can induce visual concepts from a few demonstrations during multimodal in-context learning (MICL) remains unclear. Existing evaluations often confound concept induction with recognition of familiar semantic associations. It is therefore difficult to determine whether demonstrations cause MLLMs to acquire new information or only change how existing information is accessed. We study this question in a controlled setting and disentangle these two possibilities. Specifically, we decompose MICL performance into signal power, measuring how separable the target concept is inside the low-dimensional subspace the frozen language head can see, and readability, measuring how effectively the head aligns with that separation. Across our analysis, we find that MLLMs can induce visual concepts from context. However, demonstrations primarily improve the readability of pre-existing visual signals by steering them toward the frozen readout, rather than substantially changing what the query representation encodes. Additionally, we find that this steering is not selective: demonstrations can also align irrelevant or competing concepts with the readout, creating interference that limits concept induction. Our findings suggest that current MLLMs primarily learn to read rather than write during MICL: demonstrations largely teach the model how to access information already encoded in its representations, rather than providing new task-relevant information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Multimodal In-Context Learning, Few-Shot Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_E2itOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether multimodal models can infer a classification rule from image examples paired with meaningless labels. In clevr-4, the same image can be classified by color, shape, count, or texture. The authors define a readout frame using the unembedding vectors of the candidate labels. They split pairwise discrimination in this frame into signal power and readout loss. Examples usually reduce readout loss. Shape also interferes with classification by other concepts, and removing shape information improves accuracy on some tasks. The paper finds that multimodal in-context learning (MICL) mostly helps the model read existing visual information, with less change to the query representation.

Critical strengths
  1. The experiment setting is interesting where the same images can be classified by four independent attributes. Meaningless labels help test whether the model learns the rule and the label mapping from examples.
  2. The readout frame gives a clear way to study the final-layer state and the fixed output head. The interference analysis and LEACE intervention also provide useful evidence for shape bias.
Critical weaknesses
  1. Under the paper's first-token label setup, the final decision uses only the readout frame, but this identity says nothing about other representations or intermediate computation. Stable probe accuracy across shots can hide changes in geometry and margins, therefore I'm concerned that the “read rather than write” claim may be a bit too strong.
  2. The squared-angle readability measure ignores direction, and distances between class means ignore the fixed decision boundary. Lower readout loss may therefore fail to improve label mapping or ten-class accuracy. The paper may need controls for class-to-word permutations, nonsense-word sets, and prior label logits, especially given the collapse of Count predictions onto one word at k=1.
  3. All experiments use Clevr-4 classification, with no tests of visual reasoning or natural scenes. Claims that MLLMs in MICL usually “learn to read rather than write” should be limited to this setting to avoid overclaiming.
Questions for authors
  1. Could you compare representation geometry, classification margins, and probe transfer across shot counts? A probe trained under one condition and tested under another would test the claim that representations stay largely unchanged more directly.
  2. Do the main trends in Figures 2–4 hold when class-to-word assignments are randomly permuted, nonsense-word sets are changed, and zero-shot label logits are controlled?
  3. How were the intermediate layers for LEACE chosen, and does erasing random or irrelevant attributes produce similar gains? Can these effects be reproduced on another dataset with several valid classification rules?
Reviewer_fcwWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether demonstrations with nonsense labels induce different visual partitions of the same images. It compares semantic labels and linear probes, projects representations into a label-difference readout frame, and factorizes pairwise deflection into signal power and readability. LEACE interventions examine competing-attribute interference. The authors then conclude that demonstrations mainly improve access to existing visual information.

Critical strengths
  • The writing is rather clear. The setting separates abilities that few-shot evaluations usually conflate.
  • I like the idea of intervention beyond correlational probing. The results show the role for competing-attribute information is consequential.
Critical weaknesses
  • I appreciate your abundant experiment results, but I was expecting a figure illustrating your method or main idea.
  • Your main "read, not write" conclusion seems not fully established by the evidence? Correct me if I'm wrong. The idea's main support is that separately fitted linear probes keep similar accuracy as k increases (Fig. 2). On Clevr-4 every attribute of the query is present in the image regardless of the demonstrations, so I guess this test can hardly fail. Can you clarify whether an in-frame signal increase is redirected or newly formed information?
Questions for authors
  • What's your opinion on the paper by Yang et al., Unifying Attention Heads and Task Vectors (NeurIPS 2025), which already distinguishes separability from label unembedding alignment?
  • How do signed pairwise margins and per-label offsets relative to the zero threshold behave across conditions? After offset calibration, do Count and Texture remain limited to signals?
  • Please specify query counts and splits, label-mapping and demonstration-order randomization, episode variability, and how the layer 23 was selected.
Reviewer_acTQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether multimodal in-context learning enables models to induce visual concepts from demonstrations. It introduces a readout frame to distinguish concept separability from alignment with the frozen output head. Experiments suggest that demonstrations primarily improve access to existing visual information, while competing attributes, particularly shape, can interfere with the target task.

Critical strengths
  • The readout frame provides an interesting perspective on the gap between encoded information and its use by the output head.

  • The analysis forms a coherent evidence chain, combining behavioral evaluation, linear probing, geometric decomposition, and intervention.

  • The identification of shape bias as interference in other visual tasks is insightful.

Critical weaknesses
  • The practical implications remain limited: the LEACE intervention requires labeled confounders, and its applicability beyond the controlled setting is unclear.

  • Retraining probes independently for each condition allows them to adapt to representation changes, weakening the conclusion that demonstrations leave the encoded information largely unchanged.

  • Colour, shape, count, and texture are likely familiar attributes. Tasks involving unseen attributes, attribute combinations, or relational rules are needed to support broader claims about concept induction.

  • Experiments cover only one synthetic dataset and three episodes per setting, with insufficient analysis of sensitivity to label words, class-to-word mappings, and demonstration order.

Questions for authors
  • How well do probes trained under one demonstration condition transfer to other conditions?

  • Do the findings extend to novel attribute combinations or relational concepts?

  • How robust are the results across more episodes, different label sets, randomized class-to-word mappings, and demonstration orders?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.