acceptodds
Under review as a conference paper at ICLR 2027

Instruction Following in Multimodal Embeddings: Measurable, Diagnosable, Controllable

Abstract

Current multimodal embedding models map images and text into a unified vector space and achieve strong performance on general image-text retrieval and visual question answering (VQA). However, in practical applications such as e-commerce, agent-driven Q&A, and medical diagnosis, users perform cross-modal retrieval with diverse perspectives and intentions, requiring multimodal embedding models to dynamically adjust their representations according to natural language instructions. Existing benchmarks primarily focus on behavioral-level performance evaluation and fail to quantitatively measure the instruction-following capability of multimodal embedding models. To address this limitation, we propose IFME-Bench (Instruction-Following for Multimodal Embeddings Benchmark) and the 4STEP (4-Sequential Testing for Embedding Progression) evaluation pipeline. As the core component of 4STEP, IFME-Bench integrates eight measurable metrics and pioneers end-to-end instruction-following diagnosis from the behavioral layer to the embedding layer under multimodal settings. Crucially, our analysis identifies a verifiable mapping between IFME-Bench diagnostic metrics and training hyperparameters, demonstrating the feasibility of controlling the model's specific instruction-following sub-capabilities via hyperparameter optimization during training. To facilitate future research, we release two baseline models trained on Qwen3-VL-Embedding-2B: GF-Embedding (General instruction-following embedding), which achieves a favorable balance between general embedding quality and instruction-following performance, and SF-Embedding (Specialized instruction-following embedding), which outperforms all evaluated models in terms of nearly all metrics of IFME-Bench.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Embeddings, Instruction Following, Diagnostic Benchmark, Representation Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZpumOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes IFME-Bench and the 4STEP pipeline for measuring whether multimodal embedding models follow natural-language instructions. Step 1 checks base alignment on the VQA and classification tasks of MMEB-V2. Steps 2 to 4 use IFME-Bench, built from 1,000 ADE20K images with five GPT-generated (instruction, answer) pairs per image, each targeting a different object, and per instruction four correct variants (paraphrases), four implicit variants (object name removed) and four incorrect variants (the other objects' instructions). Eight metrics cover retrieval behavior (Hit@1 over the global pool of 5,000 answers, and ISA, KIA, IPR and IFA over each image's five-candidate pool) and instruction-induced embedding shifts (IMS, IDA, ICE). The authors fine-tune Qwen3-VL-Embedding-2B with LoRA on Visual Genome region descriptions paired with generated instructions, sweep temperature, LoRA α/rank and learning rate, and introduce a Relative Sensitivity Index (RSI) to map each metric to the hyperparameter that controls it.

Critical strengths

(+) The per-image candidate pool (used by ISA and KIA) and the noise-instruction baseline in IMS are sound design choices. All five candidates describe objects present in the image, so only the instruction identifies the target and a model that ignores it sits at 20%, and Table 3 shows that Qwen3-VL-Embedding-8B, the strongest model on MMEB-V2-Sub (81.1 VQA, 74.2 CLS), reaches only 43.97 ISA and 33.37 KIA. IMS subtracts the similarity obtained with generic prompts, which removes the shift that adding any text to an image query produces.

Critical weaknesses

(---) The eight metrics are proposed without validation, and most duplicate one another or measure something other than instruction following. The title's "measurable" and "diagnosable" are not supported. Apart from ISA, KIA and IMS, the metrics do not measure what they claim. For example, IPR gives a perfect score to a model that ignores the instruction (UME-R1-2B has the best IPR, 99.31, and the worst Hit@1, 0.68), and IDA measures the direction in which the instruction moves the embedding, which doesn't matter in practice, since ranking depends only on how close the final query is to each candidate. Neither reflects how the embedding ranks candidates, which is what most tasks depend on.

(---) The title's "controllable" means tuning training hyperparameters to move metric scores, a different sense from ABC and related work, where controllable means steering an embedding with instructions. Even in the sense of hyperparameter tuning, raising LoRA α/rank or learning rate raises all eight IFME-Bench metrics together, temperature is only effective on two metrics in Figure 4, and Tables 5-7 show its effects are marginal. Again, out of 7 new metrics, only ISA, KIA, IMS look valid to me. Thus the hyperparameter tuning discussion seems not meaningful to me.

(--) GF- and SF-Embedding are tested on the proposed evaluation only, but not tested in general benchmarks (full MMEB, CtrlBench) and tasks (retrieval is still the primary application for embeddings).

Questions for authors
  • Clarification: what is the unit of the 50% manual check, how many annotators took part, and were failing items removed or corrected?

  • Clarification: were the MMEB-V2-Sub scores of the base Qwen3-VL-Embedding-2B re-run in your pipeline, or taken from its paper? GF-Embedding's margin over it is 0.3 VQA and 0.2 CLS, which is likely within a margin of error (caused by the variance of the evaluation pipeline). Also, where does ABC's classification score of 56.1 come from? Was it reproduced by you? ABC paper reports 60.0 on 9 tasks.

  • How do GF- and SF-Embedding perform on CtrlBench, and on MMEB's image retrieval and visual grounding tasks? CtrlBench tests the same kind of instruction following on data built independently of your design. Both benchmarks would show whether the gains of the trained models carry beyond IFME-Bench.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Zpum (Part1)

Thank you for your constructive and technically detailed review. We have added experiments and uploaded a revised manuscript, with the additions marked in blue. We address your concerns below.

Response to Weakness 1: Validation and interpretation of the diagnostic metrics

IFME-Bench uses retrieval to evaluate embeddings produced when a model jointly encodes an image and a text instruction. Our goal is to provide an instruction-following profile that supports model analysis, comparison, and selection.

Hit@1 measures the final global retrieval outcome under the original instruction. The remaining metrics diagnose controlled responses to instructions and changes in the embedding, with potential relevance to model selection and validation in other tasks that use instruction-conditioned embeddings. Global retrieval rankings also depend on the model’s underlying semantic capabilities, the competing candidate answers, and the size of the candidate pool. These factors allow diagnostic scores to behave differently from Hit@1.

IPR measures the stability of target similarity when the wording changes and the intent is preserved. In the blank-instruction control in Appendix H, all four models obtain an IPR of 100 and low Hit@1. The five instruction variants for each image have identical inputs in this control, leading to identical target similarities. This is consistent with IPR’s intended role as a measure of similarity stability across variants.

Appendix F provides complementary evidence. Under instruction–target mismatch, SF-Embedding’s ISA drops from 92.25 to 2.00, while its IPR rises from 96.56 to 98.02. The target similarity can therefore remain stable even when the model fails to select the correct candidate answer.

This explains why UME-R1-2B can have a high IPR of 99.31 alongside a low Hit@1 of 0.68. The two values reflect stability of target similarity across paraphrases and correctness of global retrieval, respectively.

IFME-Bench-Robust and IFME-Bench-Force characterize different aspects of the model’s response to instructions. In applications such as threshold-based retrieval and clustering, the similarity values matter directly, making the stability measured by IPR relevant. Image generation or local editing can also use embeddings as continuous conditioning signals. In such settings, IDA and ICE characterize the direction of the shift toward the target and the change in target similarity. The contribution of 4STEP is to organize these measurements into a layered profile of instruction-following behavior.

Response to Weakness 2: Scope of controllability and interpretation of RSI

We appreciate the reviewer’s question about controllability. ABC controls embeddings at inference time by changing the input instruction, enabling the model to retrieve different candidate answers for the same image. We study the complementary question of how training hyperparameters shape the instruction-following profile measured by IFME-Bench. In this context, controllability refers to using reproducible hyperparameter–metric response patterns to select training configurations for a particular diagnostic objective.

Figure 4 highlights the hyperparameter with the highest RSI for each metric among the three tested hyperparameters. For the temperature sweep from 0.01 to 0.12, four metrics have an RSI greater than one, indicating that their relative gains exceed the average relative gain of that sweep. The highlighting identifies the most concentrated relative response for a metric across the tested sweeps. Several metrics can respond to temperature while only some attain their highest RSI in its sweep.

The hyperparameters operate on different scales and have different step sizes. RSI normalizes a metric’s relative gain by the mean relative gain of all metrics in the same sweep, allowing us to compare how prominently that metric responds within each sweep.

For example, in the temperature sweep, the average relative gain across the eight metrics is 5.73%, while IMS increases by 12.67%, yielding an RSI of 2.21. In the learning-rate sweep, IMS increases by 57.57% and the average relative gain is 46.73%, giving an RSI of 1.23. The learning rate produces a larger IMS gain in the reported sweep, while IMS responds more prominently relative to the other metrics in the temperature sweep.

RSI therefore indicates which hyperparameter produces the strongest relative response for a given diagnostic metric. The raw results quantify the magnitude of the change.

Appendix I adds a held-out evaluation. Each of the eight metrics has the same highest-RSI hyperparameter on the original IFME-Bench and on the held-out images. This supports the reproducibility of the original response patterns on unseen evaluation images.

AuthorsOpenReview ↗

Response to Reviewer Zpum (Part2)

Response to Weakness 3: Evaluation on general image tasks and external instruction-guided retrieval

Thank you for this suggestion. We agree that performance on external data is important when assessing the reference models. We therefore expanded the general-task evaluation and added an external instruction-guided retrieval task.

Table 3 now reports results across 36 static-image MMEB-V2 subsets covering visual question answering, classification, retrieval, and visual grounding. GF-Embedding obtains 74.4 on retrieval and 87.9 on grounding, close to the base model Qwen3-VL-Embedding-2B at 74.8 and 88.5, respectively, while improving IFME-Bench Hit@1 from 5.00 to 11.24. SF-Embedding reaches 32.84 on IFME-Bench Hit@1 and obtains 52.3 on retrieval and 70.0 on grounding. These configurations represent different trade-offs. GF-Embedding improves instruction-guided retrieval while retaining general image-task performance close to the base model. SF-Embedding obtains larger instruction-guided gains alongside lower scores on general tasks.

We obtained these Table 3 results by running the evaluations locally. Appendix K reports sample variances from three evaluation runs per model with fixed weights for all 11 models to assess repeatability.

Section 5 reports results on OVEN from M-BEIR. Given an image and a text question, OVEN requires retrieving the corresponding entity from a large pool of Wikipedia entity text. GF-Embedding and SF-Embedding improve on the base model by 1.93 and 8.63 percentage points, respectively, and SF-Embedding achieves the highest score among the evaluated 2B models. The gains also appear on external question-guided retrieval. We could not locate CtrlBench test data suitable for reproduction through the official ABC release channels, so we used the publicly available OVEN benchmark for the external evaluation.

Response to Question 1: Manual inspection protocol

We sampled at the image level. Two initial reviewers, two annotators responsible for corrections, and one final reviewer took part. For each sampled image, the initial reviewers checked the following criteria.

  1. All five candidate answers refer to five distinct objects that are present in the image.
  2. Each original instruction uniquely identifies its designated candidate answer.
  3. Each Correct variant preserves the original instruction’s intended meaning.
  4. Each Implicit variant can be answered by the same candidate answer.
  5. Each Implicit variant omits words that directly name the target object.

Each initial reviewer inspected 250 images and flagged problematic examples. The two annotators corrected the flagged items. The final reviewer rechecked the corrected data and recorded the final inspection decisions.

AuthorsOpenReview ↗

Response to Reviewer Zpum (Part3)

Response to Question 2: MMEB-V2-Sub score provenance and the ABC-7B classification result

Score provenance. We obtained the MMEB-V2-Sub scores in Table 3 from our own evaluation runs. Appendix K reports sample variances across repeated evaluations for all 11 models to assess the repeatability of the reported results.

GF-Embedding compared with the base model. We designed GF-Embedding to improve instruction following while retaining general embedding performance close to Qwen3-VL-Embedding-2B. The repeated runs in Appendix K show that the small VQA and CLS differences persist across the reported evaluations. Table 3 also includes the newly added MMEB-V2 retrieval and visual grounding results. GF-Embedding scores slightly below Qwen3-VL-Embedding-2B on both tasks. Taken together, these results support interpreting GF-Embedding’s general capability as comparable to that of the base model, with the main improvement in instruction following.

ABC-7B classification. The MMEB classification evaluation comprises 10 tasks. The ABC paper did not report N24News. For consistency with the other models, we included this dataset when reevaluating ABC-7B. Our results on the other nine datasets are broadly consistent with the published numbers. The reported 56.1 is the classification score from our evaluation including N24News.

Response to Question 3: MMEB-V2 retrieval and grounding, and external instruction-guided retrieval

We have added MMEB-V2 retrieval and visual grounding results to Table 3. GF-Embedding achieves 74.4 and 87.9, respectively, close to Qwen3-VL-Embedding-2B at 74.8 and 88.5. SF-Embedding achieves 52.3 and 70.0. Together with the IFME-Bench results, these scores show that GF-Embedding improves instruction-guided retrieval while retaining general image-task performance close to the base model. SF-Embedding yields a larger instruction-guided gain alongside lower scores on general retrieval and grounding.

We could not locate CtrlBench test data suitable for reproduction through the official ABC release channels. Section 5 therefore reports results on OVEN under Task 6 of M-BEIR. OVEN tests whether a model can combine an image and a text question to retrieve the relevant Wikipedia entity from a large pool of entity text. In Table 4, GF-Embedding and SF-Embedding attain R@5 scores of 20.87 and 27.57, respectively. These exceed the base model’s 18.94 by 1.93 and 8.63 percentage points. The two models’ gains relative to the base model thus also appear on an independent external question-guided retrieval task.

Reviewer_wDpHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper investigates whether multimodal embedding model possess instruction-following capabilities with protocol called 4STEP. The steps are: (1) measures base VL alignment on MMEB-V2, (2)-(4) utilizes IFME-Bench, their constructed benchmark from 1k ADE20k images paired with object-specific instruction-answer target sets, which are meant to measure the IF capability. The benchmark includes instruction-answer variants including paraphrases, rewriiting, and negative cross-object instructions to evaluate directional shifts. They also propose sensitivity index to attribute metric variance to fine-tuning hyperparameters. Finally, the authors also contribute 2 baselines of embedding model that achieves state-of-the-art results on IFME-Bench metrics.

Critical strengths
  1. Valid isolation of instruction-following over multimodal context design. By conditioning target answers on the exact same underlying source image (section 3.2), the model cannot rely just via visual features and must seek context by processing the text instruction to retrieve correctly. Furthermore, including negative and paraphrased instruction variants also allows extended tests on representation robustness if there exists instruction modifications (section 3.5)

  2. Comprehensive documentation and reproducible sweeps. Providing a comprehensive hyperparameter sweeping (App. E), prompt configurations (App. D), and noise templates (Table 4) is a really big contribution which strengthen direct verification of the calculations that support the RSI indexing claim.

Critical weaknesses
  1. Lack of text-only control baselines. While above experiment design is robust in multimodal scenario, instruction templates often contain target object name directly (Fig, 6), so it is unclear whether the retrieval success is driven by text-to-text keyword matching or image-text conditioning. Furthermore, several metrics is a bit irregular -> IPR achieves highest score on UME-R1-2B despite a near-zero Hit@1 score, and IFA does seem redundant with ISA.

  2. Flawed sensitivity metric and benchmark-bound fine-tuning. The proposed RSI metric, while being principled and well motivated on high-level, measures gain over parameter's relative proportion. This is undermining because, for example, IMS wins by only 12.7% absolute performance improvement compared to LoRA which is already a "lightweight" method that delviers +134% gain. Furthermore, the specialized SF-Embedding baseline is trained directly on the benchmark without any out-of-domain validation + rely on single experimental runs which I condemn suggest narrower gain claim of the effectiveness of the contributed model in a more general use cases.

Questions for authors
  1. What are the benchmark result if evaluating a text-only baseline (or blank image text pairing maybe)? Relating to weakness #1, if the text-only baselines achieve comparable Hit@1 and ISA scores, it suggests that the benchmark evaluates text matching rather instruction following in multimodal context

  2. How does the proposed metric correlate with each other? reason -> metrics like IFA and ISA appear near identical in table 3 + IPR reports a peak score of 99.31 for a model with Hit@1 of 0.68, so identifying redundancy would further contributes to cleaner metrics design and whether the 4STEP protocol is a sound framework overall rather than "spammy"

  3. Does SF-Embedding model gain generalized to external instruction benchmarks (like CtrlBench or GeneCIS)? Since it is trained directly on the benchmark, there could be benchmark-specific overfitting here, so demonstrating positive transfer to independent datasets would confirm generality.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wDpH (Part1)

Thank you for your constructive and technically detailed review. We have added experiments and uploaded a revised manuscript, with the additions marked in blue. We address your concerns below.

Response to Weakness 1: Visual–text controls, interpretation of IPR, and distinction between ISA and IFA (Part 1)

Text-only and visual controls

Appendix G introduces shuffled-image, text-only, and blank-image controls. It also reports a shuffled-image subset filtered by target-object annotation and a TF-IDF lexical baseline. These experiments test whether text information, annotation overlap, or simple lexical matching can account for the IFME-Bench results.

Correctly paired images contribute to global retrieval. With the instructions and candidate answer set fixed, SF-Embedding’s global Hit@1 drops from 32.72 under normal input to 6.70 with shuffled images, 11.60 with text-only input, and 10.14 with blank images. Most other models show a similar decline. Text information can support some retrieval, and correctly paired images contribute substantially to the performance achieved with normal inputs.

Some models gain little from visual input on local candidate answer selection. With text-only input, Qwen3-VL-Embedding-2B, GF-Embedding, and UME-R1-2B achieve higher ISA and KIA than with normal image–text input. Hit@1 also increases for UME-R1-2B. These models can use text semantics to distinguish candidate answers in the local pool, and their joint image–instruction representations provide uneven additional gains. The results show that models differ in how they combine visual information with instruction semantics.

The TF-IDF baseline provides a lexical comparison. The benchmark uses Implicit instructions that omit explicit target-object names. KIA assesses candidate answer selection under this condition and helps evaluate reliance on explicit keywords. Instruction-following models such as SF-Embedding and ABC-7B achieve relatively high scores. In Table 12, TF-IDF obtains a KIA of 16.21, below the 20% reference for the local candidate pool, while all four evaluated embedding models score higher. Thus, the embedding models achieve substantially more than simple TF-IDF matching on these instructions.

Taken together, the controls indicate that both visual content and instruction semantics contribute to the observed performance.

IPR interpretation

IPR measures how stable the similarity to the designated candidate answer remains when an instruction is rephrased without changing its meaning. Retrieval correctness and rank stability depend on how the model scores the entire candidate answer set.

More broadly, IFME-Bench uses retrieval to evaluate the embedding representations produced by jointly encoding images and text instructions. In the instruction-response layer, IPR captures target-similarity stability, a property that complements Hit@1.

Appendix H adds the blank-instruction evaluation. Under this condition, all four models show the same pattern of an IPR of 100 and low Hit@1. With blank instructions, all five instruction variants for each image have identical inputs, and the similarity to the target stays constant. This is consistent with the intended interpretation of IPR as a measure of target-similarity stability across phrasings.

Appendix F provides complementary evidence. Under instruction–target mismatch, SF-Embedding’s ISA drops from 92.25 to 2.00, while its IPR increases from 96.56 to 98.02. Similarity to the designated candidate answer can remain stable even when correct selection deteriorates.

For the same reason, UME-R1-2B’s high IPR of 99.31 is compatible with its low Hit@1 of 0.68. IPR reflects similarity stability across paraphrases, and Hit@1 reflects the correctness of global retrieval.

AuthorsOpenReview ↗

Response to Reviewer wDpH (Part2)

Response to Weakness 1: Visual–text controls, interpretation of IPR, and distinction between ISA and IFA (Part 2)

IFA and ISA relationship

The two metrics examine complementary responses to instruction variation. ISA measures correct selection under meaning-preserving changes, and IFA measures departure from the original target when the instruction intent changes.

ISA keeps the instruction’s intent and correct candidate answer fixed and varies its wording. For example, “Find the dog in the image” can be rephrased as “Which animal in the image is a canine?” The model should select the same candidate answer under both phrasings. ISA therefore tests whether models recognize the same intent across alternative expressions.

IFA keeps the image and candidate answer set fixed and replaces the instruction with one referring to a different object in the same image. For example, when “Find the dog in the image” is changed to “Find the car in the image,” IFA examines whether the model still selects the candidate answer referring to the dog. This control tests whether changing the instruction changes the model’s selection for an otherwise identical image.

ISA examines responses across equivalent instruction variants, while IFA examines changes prompted by a different intent. They provide complementary readings under different inputs. In the single-variant case, ISA and IFA would have more mathematical overlap. The benchmark uses multiple Correct variants, so the evaluation designs examine different sets of instruction responses.

Response to Weakness 2: Interpretation of RSI, external validation, and run-to-run variation

RSI design

We appreciate the reviewer’s analysis of relative sensitivity. Within a specified hyperparameter range, RSI compares a diagnostic metric’s relative gain with the mean relative gain across all eight metrics in the same sweep. The hyperparameters use different scales and step sizes, so this normalization makes it possible to compare how strongly a given metric responds relative to the other measured outcomes under each sweep.

We also added held-out RSI results in Appendix I. The held-out set contains 1,000 images disjoint from the original benchmark and 5,000 evaluation items. For all eight metrics, the hyperparameter yielding the highest RSI is the same on the original and held-out sets. The response patterns measured in the original sweeps therefore reproduce on unseen images.

External validation of GF-Embedding and SF-Embedding

To evaluate SF-Embedding on external data, we added Section 5, which reports results on OVEN under Task 6 of the open M-BEIR benchmark. OVEN requires a model to combine an image with a text question and retrieve text for the relevant Wikipedia entity. It evaluates instruction-conditioned retrieval through visual recognition, question understanding, and joint image–text representation.

Table 4 shows that GF-Embedding and SF-Embedding achieve Recall@5 scores of 20.87 and 27.57, respectively, improving on Qwen3-VL-Embedding-2B by 1.93 and 8.63 percentage points. SF-Embedding obtains the highest score among the evaluated 2B models. The improvements observed on IFME-Bench thus also appear on an independent external instruction-conditioned retrieval task.

Run-to-run variation

To assess variation across runs, we repeated the complete 4STEP evaluation three times for each of the 11 models, keeping the model weights, data, and evaluation settings fixed. Appendix K reports the sample variance of each metric.

The ISA score of Qwen3-VL-Embedding-8B has a sample variance of 0.192, and the remaining variances are below 0.1. The repeated runs show limited variation for most metrics under the current evaluation setup and support the stability of the main observations across runs.

AuthorsOpenReview ↗

Response to Reviewer wDpH (Part3)

Response to Question 1: Text-only and visual-control baseline results

Appendix G reports shuffled-image, text-only, and blank-image controls, as well as a TF-IDF lexical baseline. These controls change the IFME-Bench results and help identify the contribution of visual information.

For most models, global Hit@1 is higher with normal image–text input than under any of the three controls. For SF-Embedding, Hit@1 drops from 32.72 with normal input to 6.70 with shuffled images, 11.60 with text-only input, and 10.14 with blank images. Correctly paired visual information is an important contributor to the global retrieval performance observed with normal input.

On local candidate answer selection, removing the image improves ISA and KIA for Qwen3-VL-Embedding-2B, GF-Embedding, and UME-R1-2B. These models can use instruction semantics to distinguish local candidate answers, and the benefit of adding the image varies across conditions.

TF-IDF achieves a KIA of 16.21 on Implicit instructions, below the 20% reference for five candidate answers. All four evaluated embedding models outperform this baseline, indicating that simple lexical matching alone provides an incomplete account of their performance. Appendix G and our response to Weakness 1 provide the full results and analysis.

Response to Question 2: Metric relationships and rationale for layered evaluation

IFME-Bench evaluates instruction-guided multimodal retrieval through three levels covering task outcomes, responses to instructions, and changes in the embedding space. Together, these measurements form an instruction-following profile that supports model analysis, comparison, and selection. Correlated scores can arise from shared aspects of instruction understanding while reflecting different responses under controlled conditions.

Relationship between IFA and ISA

ISA examines responses across meaning-preserving instruction variants, and IFA tests how the model responds to a change in the intended target. The different inputs capture complementary aspects of instruction sensitivity, as illustrated in our response to Weakness 1. With a single ISA instruction variant, the two metrics would have more mathematical overlap. Our dataset uses five Correct variants, and the two metrics are designed around distinct instruction interventions.

Interpretation of high IPR

IPR measures target-similarity stability across meaning-preserving rephrasings. Hit@1 evaluates correct retrieval. As discussed in our response to Weakness 1, a model with Hit@1 of 0.68 can have high IPR because similarity to the designated answer can remain stable across paraphrases even when that answer is ranked incorrectly.

Rationale for layered evaluation

IFME-Bench uses retrieval to evaluate an image representation conditioned on a text instruction. IFME-Bench-Ctrl uses Hit@1 to report whether the model retrieves the correct candidate answer. IFME-Bench-Robust varies instruction wording or intent and examines the resulting responses. IFME-Bench-Force measures changes in the embedding after the instruction is introduced. Together, the three layers distinguish whether the model completes the task, how it responds to changes in the instruction, and how the instruction changes its representation. Each layer contributes to the instruction-following profile.

Please see our response to Weakness 1 for further discussion.

Response to Question 3: External transfer of GF-Embedding and SF-Embedding

We appreciate the reviewer’s question about external generalization. GF-Embedding and SF-Embedding were trained on data separate from the IFME-Bench evaluation set. To assess transfer to independent external data, Section 5 adds OVEN under Task 6 of the open M-BEIR benchmark released with UniIR. Given an image and a question, OVEN requires the model to retrieve the relevant Wikipedia entity from a large pool of entity titles and summaries. The task draws on visual recognition, question understanding, and joint image–text representation, making it an external test of instruction-conditioned multimodal retrieval.

GF-Embedding and SF-Embedding achieve R@5 scores of 20.87 and 27.57, respectively, versus 18.94 for Qwen3-VL-Embedding-2B. These correspond to improvements of 1.93 and 8.63 percentage points. SF-Embedding achieves the highest R@5 among the evaluated 2B models. The gains relative to the base model thus also appear on an independent external instruction-conditioned retrieval task.

We selected OVEN because it offers publicly available evaluation data. We were unable to locate publicly available CtrlBench evaluation data, and GeneCIS is constructed from Visual Genome, which is also the source of our fine-tuning data. We use OVEN as complementary evidence of transfer to an external task.

Please see our response to Weakness 2 for further details.

AuthorsOpenReview ↗

Response to Reviewer (Follow-up)

Thank you again for your review. We would appreciate it if you could take a look at our response and the additional results in the revised manuscript. If these additions address your main concerns, we hope you will consider raising your overall score. We would be happy to clarify any remaining questions.

Reviewer_qHf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces IFME-Bench and a four-step evaluation pipeline for instruction following in multimodal embeddings. The benchmark uses 1,000 images, multiple instructions and answers per image, and instruction variants. Eight metrics measure retrieval performance, sensitivity to instructions, and changes in embedding geometry. The paper also trains GF-Embedding and SF-Embedding and studies how training hyperparameters affect these metrics. The main intended contribution is a diagnostic benchmark that can guide training, rather than a new embedding architecture.

Critical strengths
  1. The evaluation targets a meaningful failure mode. Pairing the same image with several valid answers makes it harder to succeed by image–text matching alone. Paraphrases and implicit instructions provide useful tests beyond a single retrieval score. This is a worthwhile direction for multimodal embedding evaluation.

  2. The paper makes the general-quality tradeoff visible. Table 3 evaluates general capability alongside instruction following. For example, GF-Embedding improves IFME Hit@1 from 5.00 to 11.24 while keeping the reported MMEB scores close to the base model. This is more informative than reporting instruction-following gains alone.

Critical weaknesses
  1. The new diagnostic metrics need stronger validation. The main distinction from CtrlBench is the embedding-level diagnosis, but some interpretations are not justified. In Section 3.6, IDA compares two displacement vectors that share the same image embedding as their origin. These directions are not independent: for independent random unit embeddings in high dimension, this construction gives a cosine near 0.5, not zero. Also, stable target similarity in IPR does not ensure stable retrieval rankings. These issues matter because the paper uses the metrics to infer specific capabilities, rather than only report descriptive statistics. ABC/CtrlBench already evaluates instruction-conditioned visual retrieval, so the added diagnostic value needs to be convincing.

  2. The evidence supports local tuning more clearly than targeted control. Section 4.2 and Appendix B show that hyperparameter changes can improve several metrics together. However, RSI is calculated from sweep endpoints, and the paper does not describe a separate held-out test of predictions made from RSI. Nor does it compare diagnostic-guided tuning with a matched-budget search using retrieval accuracy alone. The central claim that the benchmark identifies and controls specific sub-capabilities therefore remains only partly supported.

Questions for authors
  1. How do IDA and the other geometric metrics behave under shuffled instructions, shuffled targets, and an instruction-ignoring model? Does correcting the IDA null baseline change the diagnostic conclusions?
  2. Were the hyperparameter sweeps, RSI estimates, and final reported IFME scores computed on separate development and test subsets? If so, please report the held-out results.
  3. Can RSI predict a useful intervention on a held-out setting, and improve the intended capability more reliably than equally budgeted tuning on Hit@1? A text-only or shuffled-image control would also help establish that the measured gains require visual grounding.
5 comments
AuthorsOpenReview ↗

Response to Reviewer qHf8 (Part1)

Thank you for your constructive and technically detailed review. We have added experiments and uploaded a revised manuscript, with the additions marked in blue. We address your concerns below.

Response to Weakness 1: Empirical calibration of IDA and interpretation of IPR

IDA null baseline

The reviewer correctly points out that IDA compares two displacement vectors that share the same image embedding as their origin. With independent, isotropically distributed unit embeddings in high dimensions, the expected cosine of the resulting displacement directions converges to 0.5. We have removed the previous near-zero baseline statement.

Equation (10) in Appendix J derives this theoretical limit. Because learned embeddings can exhibit anisotropy, we estimate a model-specific empirical reference in place of a universal baseline of 0.5. We sample with replacement from the five candidate answers associated with the same image, using 20 draws per evaluation item and 100,000 draws per model for the reference. The resulting random-reference values differ across models on IFME-Bench. We define excess IDA as the raw score minus the empirical reference.

All four models have positive excess IDA, and their rankings by raw and excess IDA are identical. The original comparative conclusion therefore holds after calibration. Raw IDA describes the extent to which the instruction-induced shift points toward the designated candidate answer. We interpret it relative to its theoretical maximum of 100 and use model-specific references when assessing alignment beyond chance.

IPR interpretation

We agree that stable target similarity alone provides insufficient evidence for stable retrieval rankings. IPR measures the stability of the similarity to the designated candidate answer across meaning-preserving paraphrases. IFME-Bench uses retrieval to evaluate the representations produced when a model jointly encodes an image and a text instruction.

Appendix H supports this interpretation. With blank instructions, all four models obtain an IPR of 100 despite low Hit@1. The same image is paired with identical inputs across the five instruction variants, so the target similarity stays constant. Appendix F provides a further example of this separation. Under instruction–target mismatch, SF-Embedding’s ISA drops from 92.25 to 2.00, while its IPR increases from 96.56 to 98.02. The high IPR and low Hit@1 of UME-R1-2B in Table 3, at 99.31 and 0.68, respectively, are consistent with the distinct properties measured by these metrics.

IFME-Bench-Ctrl follows the idea of CtrlBench by evaluating instruction-conditioned rankings. IFME-Bench-Robust and IFME-Bench-Force extend the analysis to observed representation behavior. Threshold-based retrieval and clustering can use similarity values directly, so stability across paraphrases matters alongside the ability to interpret those paraphrases correctly. IPR measures this stability. In applications such as image generation or local editing that use embeddings as continuous conditioning signals, the direction and magnitude of movement toward a target can also be informative. IDA and ICE provide measures of these properties. CtrlBench and IFME-Bench therefore provide complementary views of task outcomes and representation behavior.

AuthorsOpenReview ↗

Response to Reviewer qHf8 (Part 2)

Response to Weakness 2: Held-out validation and scope of RSI-guided diagnosis

In Appendix I, we report an evaluation on IFME-Bench Held-out Data. We sampled 1,000 ADE20K images disjoint from the original benchmark and followed the same task structure and templates to construct 5,000 evaluation items.

We reevaluated the checkpoints from the original hyperparameter sweeps with fixed weights. For all eight metrics, the hyperparameter yielding the highest RSI was the same on the original and held-out sets. The hyperparameter–metric response patterns observed in the original sweeps therefore reproduce on unseen images.

Regarding diagnostic-guided tuning and a matched-budget search based on retrieval accuracy, RSI identifies the hyperparameters to which a specific diagnostic metric responds most strongly relative to the other metrics in the same sweep. It summarizes the results of the sweep and provides guidance for selecting settings for a particular diagnostic objective. Hit@1 supplies a direct selection criterion when the objective is global retrieval accuracy.

For example, in the temperature sweep in Table 6, the average relative gain across the eight metrics is 5.73%, while IMS improves by 12.67%, yielding an RSI of 2.21. In the learning-rate sweep, the average relative gain is 46.73%, while IMS improves by 57.57%, yielding an RSI of 1.23. The learning rate produces a larger absolute gain in IMS, while IMS responds more prominently relative to the other metrics in the temperature sweep.

RSI describes relative response patterns within the reported ranges. Absolute optimization efficiency would require normalization by quantities such as the hyperparameter step size, sweep width, or computational budget. The raw gains report the size of the improvements, and RSI characterizes how pronounced each gain is relative to the other responses in its sweep.

Response to Question 1: Instruction–target mismatch and blank-instruction controls

Appendices F and H report the results for shuffled instructions, shuffled targets, and blank instructions.

Under instruction and target mismatch, most IFME-Bench scores decline across the four models. SF-Embedding’s IMS and ICE become negative, indicating lower target similarity relative to their respective references. IPR remains high under mismatch, consistent with its focus on similarity stability across phrasings.

Raw IDA also declines for all four models. Relative to the same-image random reference in Appendix J, excess IDA is negative under mismatch, indicating weaker directional alignment with the designated candidate answer than the reference. With a single space in place of each instruction, excess IDA is close to zero, indicating little additional directional alignment over the random reference.

After calibration, the pattern is consistent with the intended behavior of IDA. Excess IDA is positive under normal instructions, negative under mismatch, and close to zero under blank instructions. We retain raw IDA as the primary direction-alignment metric and interpret its magnitude relative to its theoretical maximum of 100. Excess IDA adds the comparison with the model-specific empirical reference.

AuthorsOpenReview ↗

Response to Reviewer qHf8 (Part 3)

Response to Question 2: RSI consistency on held-out images

Following the reviewer’s suggestion, we added the held-out evaluation in Appendix I. The set contains 1,000 ADE20K images disjoint from the original benchmark and 5,000 evaluation items constructed with the same task structure and templates. Figures 4 and 10 show that the hyperparameter yielding the highest RSI is the same across both sets for all eight metrics. This result supports the reproducibility of the original hyperparameter–metric response patterns on unseen images.

Response to Question 3: Held-out RSI consistency and visual–text controls

1. RSI consistency on unseen images

As discussed in our responses to Weakness 2 and Question 2, Appendix I shows agreement between the original and held-out RSI results. All eight metrics retain the same highest-RSI hyperparameter on both sets. This supports the generalization of the observed response patterns to unseen evaluation images.

2. Text-only and shuffled-image controls

In Appendix G, we report shuffled-image, text-only, and blank-image controls. The text-only condition removes the image, while the instruction and candidate answer set remain fixed in all three controls. We also report a shuffled-image subset filtered to exclude matching target annotations and a TF-IDF lexical baseline to assess the roles of annotation overlap and lexical matching.

Correctly paired images contribute to global retrieval. With instructions and candidate answers fixed, SF-Embedding’s global Hit@1 drops from 32.72 under normal input to 6.70 with shuffled images, 11.60 with text-only input, and 10.14 with blank images. Most other models also show declines. Text provides useful signals for global retrieval. Correctly paired visual information contributes to the higher performance observed with normal image–text input.

Some models improve on local selection with text-only input. Removing the image increases ISA and KIA for Qwen3-VL-Embedding-2B, GF-Embedding, and UME-R1-2B. For UME-R1-2B, Hit@1 also improves. These models can use text semantics to distinguish candidate answers in the local pool, and the benefit of joint image–instruction encoding varies across models. The different responses at the global and local levels illustrate the diagnostic value of reporting both.

The TF-IDF baseline provides a lexical comparison. The Implicit instructions omit explicit target-object names, and KIA measures how accurately a model selects the designated candidate answer under this condition. Models such as SF-Embedding and ABC-7B obtain higher KIA scores. In Table 12, TF-IDF obtains a KIA of 16.21, below the 20% reference associated with five candidate answers, while all four evaluated embedding models score higher. This result supports a contribution from semantic information beyond simple TF-IDF matching.

Together, these experiments show that visual content and instruction semantics contribute to IFME-Bench performance. The declines in global Hit@1 highlight the role of correctly paired images, while the local metrics reveal differences in how models combine visual information with instructions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.