- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes IFME-Bench and the 4STEP pipeline for measuring whether multimodal embedding models follow natural-language instructions. Step 1 checks base alignment on the VQA and classification tasks of MMEB-V2. Steps 2 to 4 use IFME-Bench, built from 1,000 ADE20K images with five GPT-generated (instruction, answer) pairs per image, each targeting a different object, and per instruction four correct variants (paraphrases), four implicit variants (object name removed) and four incorrect variants (the other objects' instructions). Eight metrics cover retrieval behavior (Hit@1 over the global pool of 5,000 answers, and ISA, KIA, IPR and IFA over each image's five-candidate pool) and instruction-induced embedding shifts (IMS, IDA, ICE). The authors fine-tune Qwen3-VL-Embedding-2B with LoRA on Visual Genome region descriptions paired with generated instructions, sweep temperature, LoRA α/rank and learning rate, and introduce a Relative Sensitivity Index (RSI) to map each metric to the hyperparameter that controls it.
Critical strengths
(+) The per-image candidate pool (used by ISA and KIA) and the noise-instruction baseline in IMS are sound design choices. All five candidates describe objects present in the image, so only the instruction identifies the target and a model that ignores it sits at 20%, and Table 3 shows that Qwen3-VL-Embedding-8B, the strongest model on MMEB-V2-Sub (81.1 VQA, 74.2 CLS), reaches only 43.97 ISA and 33.37 KIA. IMS subtracts the similarity obtained with generic prompts, which removes the shift that adding any text to an image query produces.
Critical weaknesses
(---) The eight metrics are proposed without validation, and most duplicate one another or measure something other than instruction following. The title's "measurable" and "diagnosable" are not supported. Apart from ISA, KIA and IMS, the metrics do not measure what they claim. For example, IPR gives a perfect score to a model that ignores the instruction (UME-R1-2B has the best IPR, 99.31, and the worst Hit@1, 0.68), and IDA measures the direction in which the instruction moves the embedding, which doesn't matter in practice, since ranking depends only on how close the final query is to each candidate. Neither reflects how the embedding ranks candidates, which is what most tasks depend on.
(---) The title's "controllable" means tuning training hyperparameters to move metric scores, a different sense from ABC and related work, where controllable means steering an embedding with instructions. Even in the sense of hyperparameter tuning, raising LoRA α/rank or learning rate raises all eight IFME-Bench metrics together, temperature is only effective on two metrics in Figure 4, and Tables 5-7 show its effects are marginal. Again, out of 7 new metrics, only ISA, KIA, IMS look valid to me. Thus the hyperparameter tuning discussion seems not meaningful to me.
(--) GF- and SF-Embedding are tested on the proposed evaluation only, but not tested in general benchmarks (full MMEB, CtrlBench) and tasks (retrieval is still the primary application for embeddings).
Questions for authors
-
Clarification: what is the unit of the 50% manual check, how many annotators took part, and were failing items removed or corrected?
-
Clarification: were the MMEB-V2-Sub scores of the base Qwen3-VL-Embedding-2B re-run in your pipeline, or taken from its paper? GF-Embedding's margin over it is 0.3 VQA and 0.2 CLS, which is likely within a margin of error (caused by the variance of the evaluation pipeline). Also, where does ABC's classification score of 56.1 come from? Was it reproduced by you? ABC paper reports 60.0 on 9 tasks.
-
How do GF- and SF-Embedding perform on CtrlBench, and on MMEB's image retrieval and visual grounding tasks? CtrlBench tests the same kind of instruction following on data built independently of your design. Both benchmarks would show whether the gains of the trained models carry beyond IFME-Bench.
5 comments
Response to Reviewer Zpum (Part2)
Response to Weakness 3: Evaluation on general image tasks and external instruction-guided retrieval
Thank you for this suggestion. We agree that performance on external data is important when assessing the reference models. We therefore expanded the general-task evaluation and added an external instruction-guided retrieval task.
Table 3 now reports results across 36 static-image MMEB-V2 subsets covering visual question answering, classification, retrieval, and visual grounding. GF-Embedding obtains 74.4 on retrieval and 87.9 on grounding, close to the base model Qwen3-VL-Embedding-2B at 74.8 and 88.5, respectively, while improving IFME-Bench Hit@1 from 5.00 to 11.24. SF-Embedding reaches 32.84 on IFME-Bench Hit@1 and obtains 52.3 on retrieval and 70.0 on grounding. These configurations represent different trade-offs. GF-Embedding improves instruction-guided retrieval while retaining general image-task performance close to the base model. SF-Embedding obtains larger instruction-guided gains alongside lower scores on general tasks.
We obtained these Table 3 results by running the evaluations locally. Appendix K reports sample variances from three evaluation runs per model with fixed weights for all 11 models to assess repeatability.
Section 5 reports results on OVEN from M-BEIR. Given an image and a text question, OVEN requires retrieving the corresponding entity from a large pool of Wikipedia entity text. GF-Embedding and SF-Embedding improve on the base model by 1.93 and 8.63 percentage points, respectively, and SF-Embedding achieves the highest score among the evaluated 2B models. The gains also appear on external question-guided retrieval. We could not locate CtrlBench test data suitable for reproduction through the official ABC release channels, so we used the publicly available OVEN benchmark for the external evaluation.
Response to Question 1: Manual inspection protocol
We sampled at the image level. Two initial reviewers, two annotators responsible for corrections, and one final reviewer took part. For each sampled image, the initial reviewers checked the following criteria.
- All five candidate answers refer to five distinct objects that are present in the image.
- Each original instruction uniquely identifies its designated candidate answer.
- Each Correct variant preserves the original instruction’s intended meaning.
- Each Implicit variant can be answered by the same candidate answer.
- Each Implicit variant omits words that directly name the target object.
Each initial reviewer inspected 250 images and flagged problematic examples. The two annotators corrected the flagged items. The final reviewer rechecked the corrected data and recorded the final inspection decisions.
Response to Reviewer Zpum (Part3)
Response to Question 2: MMEB-V2-Sub score provenance and the ABC-7B classification result
Score provenance. We obtained the MMEB-V2-Sub scores in Table 3 from our own evaluation runs. Appendix K reports sample variances across repeated evaluations for all 11 models to assess the repeatability of the reported results.
GF-Embedding compared with the base model. We designed GF-Embedding to improve instruction following while retaining general embedding performance close to Qwen3-VL-Embedding-2B. The repeated runs in Appendix K show that the small VQA and CLS differences persist across the reported evaluations. Table 3 also includes the newly added MMEB-V2 retrieval and visual grounding results. GF-Embedding scores slightly below Qwen3-VL-Embedding-2B on both tasks. Taken together, these results support interpreting GF-Embedding’s general capability as comparable to that of the base model, with the main improvement in instruction following.
ABC-7B classification. The MMEB classification evaluation comprises 10 tasks. The ABC paper did not report N24News. For consistency with the other models, we included this dataset when reevaluating ABC-7B. Our results on the other nine datasets are broadly consistent with the published numbers. The reported 56.1 is the classification score from our evaluation including N24News.
Response to Question 3: MMEB-V2 retrieval and grounding, and external instruction-guided retrieval
We have added MMEB-V2 retrieval and visual grounding results to Table 3. GF-Embedding achieves 74.4 and 87.9, respectively, close to Qwen3-VL-Embedding-2B at 74.8 and 88.5. SF-Embedding achieves 52.3 and 70.0. Together with the IFME-Bench results, these scores show that GF-Embedding improves instruction-guided retrieval while retaining general image-task performance close to the base model. SF-Embedding yields a larger instruction-guided gain alongside lower scores on general retrieval and grounding.
We could not locate CtrlBench test data suitable for reproduction through the official ABC release channels. Section 5 therefore reports results on OVEN under Task 6 of M-BEIR. OVEN tests whether a model can combine an image and a text question to retrieve the relevant Wikipedia entity from a large pool of entity text. In Table 4, GF-Embedding and SF-Embedding attain R@5 scores of 20.87 and 27.57, respectively. These exceed the base model’s 18.94 by 1.93 and 8.63 percentage points. The two models’ gains relative to the base model thus also appear on an independent external question-guided retrieval task.
Response to Reviewer Zpum (Part1)
Thank you for your constructive and technically detailed review. We have added experiments and uploaded a revised manuscript, with the additions marked in blue. We address your concerns below.
Response to Weakness 1: Validation and interpretation of the diagnostic metrics
IFME-Bench uses retrieval to evaluate embeddings produced when a model jointly encodes an image and a text instruction. Our goal is to provide an instruction-following profile that supports model analysis, comparison, and selection.
Hit@1 measures the final global retrieval outcome under the original instruction. The remaining metrics diagnose controlled responses to instructions and changes in the embedding, with potential relevance to model selection and validation in other tasks that use instruction-conditioned embeddings. Global retrieval rankings also depend on the model’s underlying semantic capabilities, the competing candidate answers, and the size of the candidate pool. These factors allow diagnostic scores to behave differently from Hit@1.
IPR measures the stability of target similarity when the wording changes and the intent is preserved. In the blank-instruction control in Appendix H, all four models obtain an IPR of 100 and low Hit@1. The five instruction variants for each image have identical inputs in this control, leading to identical target similarities. This is consistent with IPR’s intended role as a measure of similarity stability across variants.
Appendix F provides complementary evidence. Under instruction–target mismatch, SF-Embedding’s ISA drops from 92.25 to 2.00, while its IPR rises from 96.56 to 98.02. The target similarity can therefore remain stable even when the model fails to select the correct candidate answer.
This explains why UME-R1-2B can have a high IPR of 99.31 alongside a low Hit@1 of 0.68. The two values reflect stability of target similarity across paraphrases and correctness of global retrieval, respectively.
IFME-Bench-Robust and IFME-Bench-Force characterize different aspects of the model’s response to instructions. In applications such as threshold-based retrieval and clustering, the similarity values matter directly, making the stability measured by IPR relevant. Image generation or local editing can also use embeddings as continuous conditioning signals. In such settings, IDA and ICE characterize the direction of the shift toward the target and the change in target similarity. The contribution of 4STEP is to organize these measurements into a layered profile of instruction-following behavior.
Response to Weakness 2: Scope of controllability and interpretation of RSI
We appreciate the reviewer’s question about controllability. ABC controls embeddings at inference time by changing the input instruction, enabling the model to retrieve different candidate answers for the same image. We study the complementary question of how training hyperparameters shape the instruction-following profile measured by IFME-Bench. In this context, controllability refers to using reproducible hyperparameter–metric response patterns to select training configurations for a particular diagnostic objective.
Figure 4 highlights the hyperparameter with the highest RSI for each metric among the three tested hyperparameters. For the temperature sweep from 0.01 to 0.12, four metrics have an RSI greater than one, indicating that their relative gains exceed the average relative gain of that sweep. The highlighting identifies the most concentrated relative response for a metric across the tested sweeps. Several metrics can respond to temperature while only some attain their highest RSI in its sweep.
The hyperparameters operate on different scales and have different step sizes. RSI normalizes a metric’s relative gain by the mean relative gain of all metrics in the same sweep, allowing us to compare how prominently that metric responds within each sweep.
For example, in the temperature sweep, the average relative gain across the eight metrics is 5.73%, while IMS increases by 12.67%, yielding an RSI of 2.21. In the learning-rate sweep, IMS increases by 57.57% and the average relative gain is 46.73%, giving an RSI of 1.23. The learning rate produces a larger IMS gain in the reported sweep, while IMS responds more prominently relative to the other metrics in the temperature sweep.
RSI therefore indicates which hyperparameter produces the strongest relative response for a given diagnostic metric. The raw results quantify the magnitude of the change.
Appendix I adds a held-out evaluation. Each of the eight metrics has the same highest-RSI hyperparameter on the original IFME-Bench and on the held-out images. This supports the reproducibility of the original response patterns on unseen evaluation images.