acceptodds
Under review as a conference paper at ICLR 2027

One String Per Task: How a Benchmark's Fixed Instructions Can Become Task-ID Tokens

Abstract

Universal multimodal embedding benchmarks pair each task with one fixed instruction string. This makes a familiar wording useful as a task cue, while leaving its contribution to the score unmeasured. We expose this blind spot by changing only the query instruction and holding the image, query content, and candidate index fixed. Across 36 MMEB tasks, a meaning-preserving rewrite costs VLM2Vec-Qwen2VL-2B 3.3 accuracy points on average, compared with 10.0 for a wrong-task instruction. The ordering replicates on three further checkpoints, with sensitivity varying by training recipe. Training controls explain why the distinction matters: a model given meaningless task tokens nearly matches fixed-instruction training, and randomly assigning which wording is trained reverses the model's wording preference. These results support learned reliance on the exact string alongside instruction meaning. Sampling paraphrases during training more than halves the gap on held-out wordings. We release a benchmark-agnostic audit and recommend reporting paraphrase and wrong-instruction gaps beside every fixed-wording score, so that evaluation measures how much performance survives an equivalent instruction.

Primary area:
datasets and benchmarks
Keywords:
universal multimodal embedding, MMEB, benchmark auditing, instruction sensitivity, paraphrase robustness, shortcut learning, evaluation protocol, multimodal retrieval, contrastive learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xACeOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Universal multimodal embedders are often evaluated on MMEB, which pairs each task with one fixed instruction string. The paper argues that this convention hides a blind spot. Because the string never varies within a task, a model can learn to use it as a task cue as well as a description, and fixed-wording accuracy cannot tell the two apart. The authors vary only the query instruction while holding the image, query content and candidate index fixed. Each of 36 MMEB tasks is scored under six conditions, namely its own instruction, three author-curated paraphrases, another task's instruction and no instruction. The audit is run on five checkpoints, and on the MMEB-V2 visual-document split with 27 datasets and 12,917 queries. Training controls include a randomized wording lottery and a nonce control. The paper also tests paraphrase sampling during training and releases an audit protocol.

Critical strengths
  1. The audit is a clean, direct measurement of a real blind spot. Six conditions across 36 tasks and five checkpoints isolate the instruction effect while holding everything else fixed. On VLM2Vec-Qwen2VL-2B, a meaning-preserving paraphrase costs 3.32 accuracy points and a wrong-task instruction costs 9.97, with the ordering clear on 31 of 36 tasks and on four of five checkpoints. This matters because it changes what a fixed-wording score can be taken to mean.

  2. The training controls and the release make the claim credible and usable. The randomized wording lottery reverses the model's preference, the nonce control shows that a meaningless task cue nearly matches fixed-instruction training, and paraphrase sampling cuts the held-out gap from 8.18 to 3.55 while fixed-wording accuracy stays flat at 62.3 versus 61.8. The audit scripts and instruction strings are released, and the reproduction scores 59.59 against a reported 60.1.

Critical weaknesses

1.The empirical scope is currently MMEB-specific. The paper argues that the protocol convention generalizes and releases a benchmark-agnostic audit, but the experiments cover one benchmark family. Other UME benchmarks, late-interaction models and reasoning-driven embedders are not audited. This limits the generality of the recommendation.

  1. The task-ID token reading is not yet established. The lottery shows that training exposure creates a preference for a wording, not that the model then uses the string as a task identifier. The nonce control is consistent with the reading, and the paper says so, but the title and framing go a little beyond the evidence. It would help to add an intervention that separates the identifier reading from instruction semantics.
Questions for authors
  1. How does the audit transfer to other UME benchmarks, such as ViDoRe or VisIT-Bench, and to late-interaction or reasoning-driven embedders? Even a small transfer study would clarify generality.

  2. Can you design a stronger intervention to separate the task-ID token reading from instruction semantics? For example, a cue that is present but never paired with a task, or a within-task wording swap at training.

  3. The lottery is not significant in classification alone. What would a larger draw or a different stratification show?

  4. The augmentation is tested on eight tasks and one checkpoint in the main text. How does it scale to 36 tasks and to the other checkpoints, especially Qwen3-VL-Embedding-8B?

  5. A short reporting recipe for practitioners, including how to build paraphrases that are not overfitted to the audit, would make the tool easier to adopt.

Reviewer_zNKzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that benchmarks for universal multimodal embedders cannot tell whether a model reads its instruction or keys on the exact wording, because every query of a task gets the same instruction string. It audits MMEB by changing only the query instruction under six conditions: the original, three meaning-preserving paraphrases, another task's instruction, and no instruction. On VLM2Vec-Qwen2VL-2B over 36 tasks, a paraphrase costs 3.3 accuracy points on average and a wrong-task instruction costs 10.0. The wrong instruction costs more than a paraphrase on four of five checkpoints and on the MMEB-V2 visual-document split.

Training experiments link the cost to exposure. When a lottery decides which of two wordings is used in training, the trained wording wins. A model trained with a meaningless token in place of each instruction nearly matches one trained with the real instructions. Rewriting the instruction on both the query and the candidate side leaves most of the cost, which argues against pure misalignment with the candidate index. Sampling paraphrases during training reduces the gap on held-out paraphrases from 8.18 to 3.55 points on eight tasks with three seeds. On all 36 tasks with one seed, it falls from 3.33 to 1.56.

Critical strengths

The experimental design isolates the variable. Only the query instruction changes, while the image, query content and candidate index stay fixed. Blind raters confirm that the paraphrases keep the meaning (κ = 0.82). The randomized lottery and the nonce control test the cause from the training side instead of relying on correlation. This is stronger evidence than a typical prompt-sensitivity study provides.

The paper calibrates its claims. It separates what is directly measured from what the lottery supports and from the task-identifier interpretation, and Section 7 lists specific limits. The released audit and the recommendation to report paraphrase and wrong-instruction gaps next to accuracy are practical for benchmark designers.

Critical weaknesses
  1. The novelty over closely related work is limited. Sensitivity to instruction wording is well documented, as the paper says. Kostiuk & Enevoldsen (2026) audit instruction sensitivity in text embedding evaluation with 38 to 45 prompts per task, against three here. Sánchez Aimar et al. (2025) show that training with several templates per task cuts the cost of unseen rewordings by a similar factor. The new elements are the application to MMEB's protocol, the lottery and the nonce control. These are useful, but the main finding is a modest step beyond that work.

  2. The causal evidence behind the title is narrow, and the title claims more than the text. The lottery uses one randomization over 13 groups, with one recipe and one replacement wording per group. The two seeds share that draw, so they are two measurements of one randomization. The result is not significant for classification alone, and Appendix B.29 notes that much of the +6.6 against −1.3 asymmetry comes from the quality of the replacement wording. The paper itself says the lottery "does not show that the model then uses that wording as a task identifier". It also says the probe cannot separate a template identifier from a representation of task type. Yet the title, "Fixed Instructions Can Become Task-ID Tokens", will be read as the main finding.

The presentation adds to this problem. Most main-text sentences carry several numbers, and the argument depends on about 40 pages of appendices (A, B.1 to B.30). A reader who stays in the main text has difficulty separating the central result from supporting checks.

Minor points (these do not drive my rating).

  • The abstract says the ordering "replicates on three further checkpoints". Four further checkpoints were audited, and the fourth (GME-Qwen2VL-2B) shows no clear ordering. The abstract should say so.
  • The average paraphrase cost of 3.3 points is concentrated in three tasks with small candidate pools. On the 26 tasks with 1,000 candidates it is 2.74 (Section 3). The abstract could note this.
  • Appendix B.20 states that the 8-task training subset was chosen after the audit and that the 36-task held-out panel is the unselected estimate of the fix. The main text should say this directly next to Table 6.
  • The paraphrases were drafted with an LLM and screened by the authors, and raters did not agree on their fluency (κ close to 0). Three paraphrases per task cover a small part of the possible wordings.
Questions for authors

Questions that could change my assessment:

  1. Beyond the change of benchmark, what does this audit show that Kostiuk & Enevoldsen (2026) and Sánchez Aimar et al. (2025) do not?
  2. Does the lottery result hold under a second, independent randomization, or with a different replacement wording per group?
  3. Why does GME-Qwen2VL-2B show no ordering? If its training used varied instructions, this would support the exposure explanation. If it did not, the explanation needs another factor.

Suggestion:

  1. Would the authors consider a title that matches the confidence of Section 7? For example, one that names the measured blind spot rather than the task-identifier interpretation.
Reviewer_uLRXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the impact of using fixed task prompts during the training and evaluation of multimodal embedding models. Through experiments on a single benchmark and a single multimodal embedding model, the authors observe that model performance is highly sensitive to prompt phrasing, and that training with multiple prompt variants helps mitigate this sensitivity.

Critical strengths

Auditing the sensitivity and robustness of benchmark evaluations to prompt variations is a practical and important problem for reliable multimodal representation evaluation.

Critical weaknesses
  • Unvetted AI-Generated Writing: The manuscript shows clear signs of heavy, unreviewed AI generation in its writing and literature review. For instance, the text includes automated search-log artifacts such as: "A search of 95 full-text 2025–26 UME papers finds ..." This raises concerns about the rigor of the scholarship and the care taken in preparing the manuscript.
  • Limited Novelty and Technical Insight: The phenomenon that foundation models are sensitive to prompt phrasing, and that multi-prompt data augmentation improves robustness, is already well established in the literature. Applying this known observation and standard mitigation to a multimodal embedding setting offers limited conceptual or technical novelty.
  • Limited Experimental Scope: The empirical evaluation relies on a single multimodal embedding model and a single benchmark. This narrow scope is insufficient to draw generalizable conclusions about multimodal embedding architectures, training paradigms, or downstream tasks more broadly.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.