- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether multimodal models can infer a classification rule from image examples paired with meaningless labels. In clevr-4, the same image can be classified by color, shape, count, or texture. The authors define a readout frame using the unembedding vectors of the candidate labels. They split pairwise discrimination in this frame into signal power and readout loss. Examples usually reduce readout loss. Shape also interferes with classification by other concepts, and removing shape information improves accuracy on some tasks. The paper finds that multimodal in-context learning (MICL) mostly helps the model read existing visual information, with less change to the query representation.
Critical strengths
- The experiment setting is interesting where the same images can be classified by four independent attributes. Meaningless labels help test whether the model learns the rule and the label mapping from examples.
- The readout frame gives a clear way to study the final-layer state and the fixed output head. The interference analysis and LEACE intervention also provide useful evidence for shape bias.
Critical weaknesses
- Under the paper's first-token label setup, the final decision uses only the readout frame, but this identity says nothing about other representations or intermediate computation. Stable probe accuracy across shots can hide changes in geometry and margins, therefore I'm concerned that the “read rather than write” claim may be a bit too strong.
- The squared-angle readability measure ignores direction, and distances between class means ignore the fixed decision boundary. Lower readout loss may therefore fail to improve label mapping or ten-class accuracy. The paper may need controls for class-to-word permutations, nonsense-word sets, and prior label logits, especially given the collapse of Count predictions onto one word at k=1.
- All experiments use Clevr-4 classification, with no tests of visual reasoning or natural scenes. Claims that MLLMs in MICL usually “learn to read rather than write” should be limited to this setting to avoid overclaiming.
Questions for authors
- Could you compare representation geometry, classification margins, and probe transfer across shot counts? A probe trained under one condition and tested under another would test the claim that representations stay largely unchanged more directly.
- Do the main trends in Figures 2–4 hold when class-to-word assignments are randomly permuted, nonsense-word sets are changed, and zero-shot label logits are controlled?
- How were the intermediate layers for LEACE chosen, and does erasing random or irrelevant attributes produce similar gains? Can these effects be reproduced on another dataset with several valid classification rules?