- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines how natural-language task instructions affect query representations in retrieval models and questions whether current task-isolated evaluation overestimates embedding models’ instruction-following ability. The authors test different instructions on eight models using ARCChallenge and Tatoeba in eight languages. They find that, although task-relevant instructions have an advantage at the distributional level overall, irrelevant or nonsensical instructions often achieve similar scores. Meanwhile, after adding “query-side distractors” generated by GPT-5.4-mini—candidates semantically similar to the query but inconsistent with the target task relation—ARCChallenge Recall@1 nearly collapses, while the effect on Tatoeba varies across models. The authors therefore hypothesize that the dominance of target-document-side negatives and the lack of query-side negatives in commonly used training may cause models to rely excessively on semantic similarity. To mitigate this problem, the authors fine-tune Qwen3-Embedding-0.6B on SQuAD and multilingual Tatoeba. The results show that, with a single paraphrase distractor, SQuAD Recall@1 increases from 0.014 to 0.470 and mean Tatoeba Recall@1 increases from 0.102 to 0.569, while performance changes only slightly in the normal setting and the average change across 41 MTEB English v2 tasks is also small.
Critical strengths
-
The problem setting is valuable. It identifies the mismatch between training, evaluation, and real-world multi-task shared indexing: standard evaluation usually makes all candidates belong to the same task, so it cannot adequately test whether a model can use the instruction to exclude semantically similar content that does not satisfy the required task relation.
-
The diagnostic setup can directly expose failure modes that conventional evaluation may miss. Query paraphrases are used to construct candidates that are semantically similar but have the wrong task relation, making the distinction between “topically related” and “satisfying the instruction” explicit.
-
The instruction coverage is relatively extensive. The 47/51 instructions include task-specific instructions, MTEB instructions, instructions for unrelated tasks, vague instructions, and nonsensical instructions (pp. 3, 15–16). This reduces the risk that the conclusions depend on one or two carefully selected prompts.
Critical weaknesses
-
The “query-side distractors” may not test instruction understanding alone. The distractor is a paraphrase of the query. In ARC, the candidate forms are “question” versus “answer passage,” while in Tatoeba they are “English sentence” versus “target-language translation”. The model may identify candidates using text type or language features rather than actually interpreting the task instruction. The paper does not include same-type but relationally incorrect candidates, such as incorrect answer passages, incorrect translations, or incorrect entities.
-
There is still no decisive evidence separating distractor robustness from general instruction following. Although training includes both retrieval and STS, the main evaluation primarily measures whether the model can find A under a retrieval instruction. It does not directly report whether, for the same test query and the same candidate pool containing (A, D), switching the instruction to STS causes the target to switch accordingly from A to D. Consequently, explanations such as “the QA model prefers answer passages over questions” and “the translation-retrieval model prefers the target language over English” have not been directly ruled out.
-
The distractor generation is limited to a single source and lacks quality validation. All 185,254 paraphrases are generated by the same GPT-5.4-mini model using nearly identical prompts. The authors provide no manual quality audit, error rate, semantic-equivalence statistics, or ablation using different generation models.
Questions for authors
-
If the distractors were replaced with same-type but relationally incorrect candidates—incorrect answer passages, incorrect translations, or incorrect parallel sentences—would the performance collapse and the fine-tuning gains still hold?
-
Could you provide a manual quality audit of the distractors and quantify dependence on the generator by replacing the generation model, changing the prompt, or using human paraphrases?
-
How is N=200 in Table 3 defined? Does cross-validation split at the instruction, query, or language level? Why do 47/51 instructions correspond to N=200?
-
Why does Table 5 report maximum performance? Could you provide the distribution across instructions and languages, the worst-case values, and the complete performance of task-relevant instructions?
-
Do the authors consider the “lack of query-side negatives” a fundamental training deficiency, or merely a related factor that can be remedied in the current post-training stage? Is there an experiment that directly controls the source of negatives during pre-training or contrastive training to establish causality?
3 comments
Response to review
Thank you for your review and patience while we run new experiments. We are grateful that you found several strengths, and we believe that we have managed to respond to your main concerns. Here is our detailed response.
W1 and Q1 Text type difference: We agree that targets and distractors differ in text type, but see this as part of the task rather than a limitation: in multi-task retrieval, the instruction often determines which text type is relevant, so selecting the type according to instruction is desired behavior. We have now included the instruction-swap results in a separate message to directly evaluate this, showing that the fine-tuned model changes its target based on the instruction. Since the candidate pool also includes normal same type negatives (all other passages / translations), avoiding the wrong text type alone is not sufficient, the model still needs to identify the correct answer passage out of all passages.
W2 Instruction swap: See our new instruction-swap experiments in a separate message. To summarize, the results show that the STS instruction used during fine-tuning consistently (100% of the time) retrieves a paraphrase distractor, whereas QA and bitext instructions more often returns the correct target type (answer, translation). Empty and gibberish (keysmash) instructions also continue to return distractors.
W3 and Q2 Paraphrase validity: We have now manually annotated a sample of 200 SQuAD question–paraphrase pairs to evaluate their validity. This sample did not include any errors where the model would generate an answer instead of the question. All annotated examples retained the meaning of the original question. Two of them were verbatim copies of the original question (both were very short questions) and the remaining 198 were valid paraphrases.
Q3 N=300 in Table 3: We consider each model-dataset combination separately. For ARCChallenge this means we have 8 datasets (one for each model) which all have 47 data points (number of instructions). We calculate the ROC-AUC score on these 8 datasets separately by using 5-fold cross-validation and iterate over different splits 200 times. This means we essentially fit logistic regression 1000 times and calculate the ROC-AUC as the mean of these results. The same applies to Tatoeba with the difference of multiple languages and 51 instances in each: In this case, we normalize the scores and metrics by model to make the data comparable.
Q4 Reported max performance in Table 5: We report the maximum performance over all instructions (and all languages on Tatoeba) to show how dire the failure really is: if the maximum performance is 0.032, then all other instructions receive worse results. While we agree that seeing the distribution would in the general case be interesting, we would like to point out that we include irrelevant instructions on purpose, and they make up the majority of the instructions, and hence, looking at the distribution would not be informative in this case. As an example, the distribution of all relevant instructions for Tatoeba:fra can be seen in Fig. 7.
Q5 Pre/Post-training: We see the lack of query-side negatives as a gap in training data and objective rather than a fundamental limitation of embedding models. We were able to compare added query-side negatives to normal domain-specific post-training (Appendix H), with identical data and setup, only query-side negatives improving the distractor performance. Unfortunately our compute budget did not allow us to compare post-training to full pre-training setting, however, we see this as an interesting direction in the future.
We believe these additions make the paper stronger and that we managed to address the main concerns. We hope that you could reconsider your evaluation. Please let us know if anything remains unclear.
Response to Authors
I appreciate the authors' detailed response and the additional experiments. The new experiments are highly valuable; they clearly demonstrate the impact of retrieval quality on task performance and confirm that fine-tuned models follow instructions more reliably. Overall, the authors have adequately addressed my primary concerns, so I am raising my score to 3.
Additionally, I have a suggestion: the structure and presentation of the response could be made more reader-friendly. When addressing each point, providing a brief conclusion followed by specific explanations or experimental results—while highlighting key data and revisions—would help reviewers verify the authors' responses more efficiently.
Response to review
Thank you for your review and patience while we are working on our responses.
Instruction swap behavior: See our new instruction-swap experiment results in a separate message. To summarize, the results show that the STS instruction used during fine-tuning consistently (100% of the time) retrieves a paraphrase distractor, whereas QA and bitext instructions more often return the correct target type (answer, translation). Empty and gibberish (keysmash) instructions also continue to return distractors. We hope our new results addresses these concerns.
We are still working on other aspects, and we try to send rest of the responses, as well as update the pdf, as soon as possible.