- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Universal multimodal embedders are often evaluated on MMEB, which pairs each task with one fixed instruction string. The paper argues that this convention hides a blind spot. Because the string never varies within a task, a model can learn to use it as a task cue as well as a description, and fixed-wording accuracy cannot tell the two apart. The authors vary only the query instruction while holding the image, query content and candidate index fixed. Each of 36 MMEB tasks is scored under six conditions, namely its own instruction, three author-curated paraphrases, another task's instruction and no instruction. The audit is run on five checkpoints, and on the MMEB-V2 visual-document split with 27 datasets and 12,917 queries. Training controls include a randomized wording lottery and a nonce control. The paper also tests paraphrase sampling during training and releases an audit protocol.
Critical strengths
-
The audit is a clean, direct measurement of a real blind spot. Six conditions across 36 tasks and five checkpoints isolate the instruction effect while holding everything else fixed. On VLM2Vec-Qwen2VL-2B, a meaning-preserving paraphrase costs 3.32 accuracy points and a wrong-task instruction costs 9.97, with the ordering clear on 31 of 36 tasks and on four of five checkpoints. This matters because it changes what a fixed-wording score can be taken to mean.
-
The training controls and the release make the claim credible and usable. The randomized wording lottery reverses the model's preference, the nonce control shows that a meaningless task cue nearly matches fixed-instruction training, and paraphrase sampling cuts the held-out gap from 8.18 to 3.55 while fixed-wording accuracy stays flat at 62.3 versus 61.8. The audit scripts and instruction strings are released, and the reproduction scores 59.59 against a reported 60.1.
Critical weaknesses
1.The empirical scope is currently MMEB-specific. The paper argues that the protocol convention generalizes and releases a benchmark-agnostic audit, but the experiments cover one benchmark family. Other UME benchmarks, late-interaction models and reasoning-driven embedders are not audited. This limits the generality of the recommendation.
- The task-ID token reading is not yet established. The lottery shows that training exposure creates a preference for a wording, not that the model then uses the string as a task identifier. The nonce control is consistent with the reading, and the paper says so, but the title and framing go a little beyond the evidence. It would help to add an intervention that separates the identifier reading from instruction semantics.
Questions for authors
-
How does the audit transfer to other UME benchmarks, such as ViDoRe or VisIT-Bench, and to late-interaction or reasoning-driven embedders? Even a small transfer study would clarify generality.
-
Can you design a stronger intervention to separate the task-ID token reading from instruction semantics? For example, a cue that is present but never paired with a task, or a within-task wording swap at training.
-
The lottery is not significant in classification alone. What would a larger draw or a different stratification show?
-
The augmentation is tested on eight tasks and one checkpoint in the main text. How does it scale to 36 tasks and to the other checkpoints, especially Qwen3-VL-Embedding-8B?
-
A short reporting recipe for practitioners, including how to build paraphrases that are not overfitted to the audit, would make the tool easier to adopt.