acceptodds
Under review as a conference paper at ICLR 2027

Your Prompt Should Do More: Effects of Retrieval Instructions in Embedding Models

Abstract

Prompted embedding models have recently received increasing attention, particularly for retrieval, where detailed retrieval instructions are provided as part of the retrieval prompt. Several new datasets and studies have examined this setting, showing that the current embedding models often struggle to follow such instructions reliably. In this paper, we study the mechanism of how instructions actually affect the representations of retrieval queries in asymmetric retrieval tasks. We show that models can fail to follow even simple task instructions when query-side distractors are included in the evaluation. We hypothesize that this behavior is driven by the training setup of current embedding models and their evaluation, and show that fine-tuning with added query-side distractors leads to substantial improvements, with minimal effect on other tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
text embedding models, task instructions, question-answering, bitext-mining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2GG4OpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how natural-language task instructions affect query representations in retrieval models and questions whether current task-isolated evaluation overestimates embedding models’ instruction-following ability. The authors test different instructions on eight models using ARCChallenge and Tatoeba in eight languages. They find that, although task-relevant instructions have an advantage at the distributional level overall, irrelevant or nonsensical instructions often achieve similar scores. Meanwhile, after adding “query-side distractors” generated by GPT-5.4-mini—candidates semantically similar to the query but inconsistent with the target task relation—ARCChallenge Recall@1 nearly collapses, while the effect on Tatoeba varies across models. The authors therefore hypothesize that the dominance of target-document-side negatives and the lack of query-side negatives in commonly used training may cause models to rely excessively on semantic similarity. To mitigate this problem, the authors fine-tune Qwen3-Embedding-0.6B on SQuAD and multilingual Tatoeba. The results show that, with a single paraphrase distractor, SQuAD Recall@1 increases from 0.014 to 0.470 and mean Tatoeba Recall@1 increases from 0.102 to 0.569, while performance changes only slightly in the normal setting and the average change across 41 MTEB English v2 tasks is also small.

Critical strengths
  1. The problem setting is valuable. It identifies the mismatch between training, evaluation, and real-world multi-task shared indexing: standard evaluation usually makes all candidates belong to the same task, so it cannot adequately test whether a model can use the instruction to exclude semantically similar content that does not satisfy the required task relation.

  2. The diagnostic setup can directly expose failure modes that conventional evaluation may miss. Query paraphrases are used to construct candidates that are semantically similar but have the wrong task relation, making the distinction between “topically related” and “satisfying the instruction” explicit.

  3. The instruction coverage is relatively extensive. The 47/51 instructions include task-specific instructions, MTEB instructions, instructions for unrelated tasks, vague instructions, and nonsensical instructions (pp. 3, 15–16). This reduces the risk that the conclusions depend on one or two carefully selected prompts.

Critical weaknesses
  1. The “query-side distractors” may not test instruction understanding alone. The distractor is a paraphrase of the query. In ARC, the candidate forms are “question” versus “answer passage,” while in Tatoeba they are “English sentence” versus “target-language translation”. The model may identify candidates using text type or language features rather than actually interpreting the task instruction. The paper does not include same-type but relationally incorrect candidates, such as incorrect answer passages, incorrect translations, or incorrect entities.

  2. There is still no decisive evidence separating distractor robustness from general instruction following. Although training includes both retrieval and STS, the main evaluation primarily measures whether the model can find A under a retrieval instruction. It does not directly report whether, for the same test query and the same candidate pool containing (A, D), switching the instruction to STS causes the target to switch accordingly from A to D. Consequently, explanations such as “the QA model prefers answer passages over questions” and “the translation-retrieval model prefers the target language over English” have not been directly ruled out.

  3. The distractor generation is limited to a single source and lacks quality validation. All 185,254 paraphrases are generated by the same GPT-5.4-mini model using nearly identical prompts. The authors provide no manual quality audit, error rate, semantic-equivalence statistics, or ablation using different generation models.

Questions for authors
  1. If the distractors were replaced with same-type but relationally incorrect candidates—incorrect answer passages, incorrect translations, or incorrect parallel sentences—would the performance collapse and the fine-tuning gains still hold?

  2. Could you provide a manual quality audit of the distractors and quantify dependence on the generator by replacing the generation model, changing the prompt, or using human paraphrases?

  3. How is N=200 in Table 3 defined? Does cross-validation split at the instruction, query, or language level? Why do 47/51 instructions correspond to N=200?

  4. Why does Table 5 report maximum performance? Could you provide the distribution across instructions and languages, the worst-case values, and the complete performance of task-relevant instructions?

  5. Do the authors consider the “lack of query-side negatives” a fundamental training deficiency, or merely a related factor that can be remedied in the current post-training stage? Is there an experiment that directly controls the source of negatives during pre-training or contrastive training to establish causality?

3 comments
AuthorsOpenReview ↗

Response to review

Thank you for your review and patience while we are working on our responses.

Instruction swap behavior: See our new instruction-swap experiment results in a separate message. To summarize, the results show that the STS instruction used during fine-tuning consistently (100% of the time) retrieves a paraphrase distractor, whereas QA and bitext instructions more often return the correct target type (answer, translation). Empty and gibberish (keysmash) instructions also continue to return distractors. We hope our new results addresses these concerns.

We are still working on other aspects, and we try to send rest of the responses, as well as update the pdf, as soon as possible.

AuthorsOpenReview ↗

Response to review

Thank you for your review and patience while we run new experiments. We are grateful that you found several strengths, and we believe that we have managed to respond to your main concerns. Here is our detailed response.

W1 and Q1 Text type difference: We agree that targets and distractors differ in text type, but see this as part of the task rather than a limitation: in multi-task retrieval, the instruction often determines which text type is relevant, so selecting the type according to instruction is desired behavior. We have now included the instruction-swap results in a separate message to directly evaluate this, showing that the fine-tuned model changes its target based on the instruction. Since the candidate pool also includes normal same type negatives (all other passages / translations), avoiding the wrong text type alone is not sufficient, the model still needs to identify the correct answer passage out of all passages.

W2 Instruction swap: See our new instruction-swap experiments in a separate message. To summarize, the results show that the STS instruction used during fine-tuning consistently (100% of the time) retrieves a paraphrase distractor, whereas QA and bitext instructions more often returns the correct target type (answer, translation). Empty and gibberish (keysmash) instructions also continue to return distractors.

W3 and Q2 Paraphrase validity: We have now manually annotated a sample of 200 SQuAD question–paraphrase pairs to evaluate their validity. This sample did not include any errors where the model would generate an answer instead of the question. All annotated examples retained the meaning of the original question. Two of them were verbatim copies of the original question (both were very short questions) and the remaining 198 were valid paraphrases.

Q3 N=300 in Table 3: We consider each model-dataset combination separately. For ARCChallenge this means we have 8 datasets (one for each model) which all have 47 data points (number of instructions). We calculate the ROC-AUC score on these 8 datasets separately by using 5-fold cross-validation and iterate over different splits 200 times. This means we essentially fit logistic regression 1000 times and calculate the ROC-AUC as the mean of these results. The same applies to Tatoeba with the difference of multiple languages and 51 instances in each: In this case, we normalize the scores and metrics by model to make the data comparable.

Q4 Reported max performance in Table 5: We report the maximum performance over all instructions (and all languages on Tatoeba) to show how dire the failure really is: if the maximum performance is 0.032, then all other instructions receive worse results. While we agree that seeing the distribution would in the general case be interesting, we would like to point out that we include irrelevant instructions on purpose, and they make up the majority of the instructions, and hence, looking at the distribution would not be informative in this case. As an example, the distribution of all relevant instructions for Tatoeba:fra can be seen in Fig. 7.

Q5 Pre/Post-training: We see the lack of query-side negatives as a gap in training data and objective rather than a fundamental limitation of embedding models. We were able to compare added query-side negatives to normal domain-specific post-training (Appendix H), with identical data and setup, only query-side negatives improving the distractor performance. Unfortunately our compute budget did not allow us to compare post-training to full pre-training setting, however, we see this as an interesting direction in the future.

We believe these additions make the paper stronger and that we managed to address the main concerns. We hope that you could reconsider your evaluation. Please let us know if anything remains unclear.

Reviewer_2GG4OpenReview ↗

Response to Authors

I appreciate the authors' detailed response and the additional experiments. The new experiments are highly valuable; they clearly demonstrate the impact of retrieval quality on task performance and confirm that fine-tuned models follow instructions more reliably. Overall, the authors have adequately addressed my primary concerns, so I am raising my score to 3.

Additionally, I have a suggestion: the structure and presentation of the response could be made more reader-friendly. When addressing each point, providing a brief conclusion followed by specific explanations or experimental results—while highlighting key data and revisions—would help reviewers verify the authors' responses more efficiently.

Reviewer_VvnaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how natural-language task instructions affect query embeddings in asymmetric retrieval, arguing that the standard "tasks-in-isolation" evaluation (e.g., MTEB) overestimates genuine instruction-following because candidate pools rarely contain semantically-similar-but-task-irrelevant distractors. The authors (i) evaluate 47-51 instructions (including deliberately irrelevant and keysmash prompts) across 8 embedding models on ARCChallenge (retrieval) and Tatoeba (bitext mining), introducing three geometric measures (similarity improvement, displacement, angulation); (ii) show that irrelevant instructions often reach competitive scores and that low displacement/small similarity gain correlate with high standard-eval scores, especially on ARCChallenge; (iii) introduce query-side "distractors" (GPT-generated paraphrases) that destroy ARCChallenge performance for nearly all models; and (iv) fine-tune Qwen3-Embedding-0.6B with query-side negatives on SQuAD and Tatoeba, showing substantial robustness gains to paraphrase distractors (e.g., SQuAD Recall@1 0.014→0.470) with little change on Normal retrieval and MTEB English v2. An ablation (Appendix H) attributes the gains specifically to query-side negatives rather than task fine-tuning alone.

Critical strengths

The three structural metrics (similarity improvement, displacement, angulation; are simple, defined formally, and used consistently throughout to connect behavior to geometry.

Reasonable model diversity in evaluation: Eight models spanning different sizes (308M to 7.6B) and embedding dimensions (768-4096) are used, with a stated effort to avoid only selecting top scorers.

Attention to confounds: The authors explicitly omit NQ/FEVER to mitigate contamination, use normalized scores and a linear mixed model to control for per-language variation, and filter OPUS training data against the test sets by both exact match and TF-IDF similarity. The key causal ablation is present. Appendix H (fine-tuning on the same data without query-side negatives) directly supports the central claim that query-side negatives, not task-specific fine-tuning, drive the robustness gains. This is an important control. Fine-tuning gains come with minimal collateral damage. MTEB English v2 differences are reported as small, supporting the "minimal effect on other tasks" claim.

Cross-validated, significance-aware statistics. ROC-AUC with stratified folds and Spearman correlations with significance markers (Tables 3-4) go beyond anecdotal plots.

Critical weaknesses

Very narrow task coverage undermines the central generalization: The entire analysis rests on exactly two task types - ARCChallenge (retrieval) and Tatoeba (bitext mining). The abstract/intro make broad claims about "the standard evaluation paradigm," but the authors themselves acknowledge (Section 6) that clustering and other categories are omitted. With only one retrieval dataset, it is impossible to know whether the dramatic ARCChallenge distractor collapse is a property of instruction-following in general or an artifact of that specific, multiple-choice-style, short-answer dataset.

Fine-tuning evidence rests on a single model: Only Qwen3-Embedding-0.6B is fine-tuned. The headline claim that fine-tuning with query-side negatives "substantially improves robustness" is thus demonstrated on one small model, with no evidence it transfers to larger or architecturally different models. The ARCChallenge transfer result (Appendix E, Table 11) is weak: baseline Recall@1 0.059→0.051 (Normal) and paraphrase 0.000→0.003 - an improvement so small as to be near-noise, and the authors note NDCG@10 actually decreased.

Synthetic distractors generated by a single model with no validation of quality.: Distractors are GPT-generated paraphrases , and the authors generate exactly one per query. There is no human or automatic check that these paraphrases preserve meaning (are true query-side negatives) rather than occasionally changing meaning (becoming false negatives). The claim that "the choice of generation model [has] only a limited effect" (Section 6) is asserted, not tested - no ablation over generators is performed, as the authors admit.

The two tasks are not clean analogs, confounding the ARCChallenge vs. Tatoeba contrast: Much of the analysis hinges on bitext mining behaving "more desirably" than retrieval. But bitext mining is near-saturated (Normal Tatoeba max recall often >0.98, Table 5) and is a fundamentally different relation (translation/near-paraphrase) than QA retrieval. The explanation offered ("mixing tasks during fine-tuning exposes the model to slightly more varied negatives," Section 4.3) is speculative and unsupported - the authors do not have access to the training data of the evaluated models, so this is conjecture.

"Distractors completely destroy performance" may partly reflect metric choice. Table 5 uses Recall@1 for the distractor setting while the main ARCChallenge metric is NDCG@10. A single distractor collapsing Recall@1 is less surprising when the distractor is a near-paraphrase of the query and the metric is top-1; some discussion of how much of the effect is metric-driven versus genuinely about instruction-following would strengthen the claim.

Potentially circular fine-tuning/evaluation setup: The fine-tuning uses the same paraphrase-distractor construction (GPT paraphrases as query-side negatives) that is then used at evaluation ("Paraphrase distractors"). While Random distractors provide one control and Appendix H provides another, the model is trained on exactly the type of negative it is tested on, which risks overstating robustness; the residual gap under paraphrase distractors (SQuAD Recall@1 fine-tuned 0.470 still well below Normal 0.706) hints the model has not fully solved the task.

Questions for authors

Does the ARCChallenge distractor collapse replicate on at least one additional, more conventional retrieval dataset (document-length targets, not multiple-choice answers)? Without this, how can the claim be generalized beyond ARCChallenge?

Did you validate that the GPT-generated paraphrases are genuine meaning-preserving query-side negatives (not accidental false negatives)? What fraction, if any, change the answer? Can the fine-tuning robustness gains be reproduced on a second, larger, or architecturally different model, or is Qwen3-Embedding-0.6B the only fine-tuned model?

In Table 5, why is Recall@1 used for the distractor setting while NDCG@10 is the primary ARCChallenge metric? How much of the "complete destruction" is attributable to the top-1 metric versus genuine instruction-following failure?

The "MV" column in Table 2 is described as Mann-Whitney effect size — how exactly is it computed and normalized, and what does a value like 0.821 mean in interpretable terms?

2 comments
AuthorsOpenReview ↗

Response to review

Thank you for your review and patience while we are working on our responses. We are still working on the full response, but see our new results on instruction swap behavior to see how the fine-tuned model uses instructions.

To summarize, the results show that the STS instruction used during fine-tuning consistently (100% of the time) retrieves a paraphrase distractor, whereas QA and bitext instructions more often return the correct target type (answer, translation). Empty and gibberish (keysmash) instructions also continue to return distractors. We hope our new results addresses these concerns.

We are still working on other aspects, and we try to send rest of the responses, as well as update the pdf, as soon as possible.

AuthorsOpenReview ↗

Response to review

Thank you for your review and patience while we run new experiments. We are grateful that you found several strengths, and we believe that we have managed to address your main concerns. Here is our detailed response.

W1 Narrow task coverage: We have now increased the number of tasks used in distractor evaluation from two to four (adding SQuAD and TextbookQA). See our new results in a separate message (Promptriever and TextbookQA).

W2 and Q2 Single model fine-tuning: We have now extended the fine-tuning experiments to also include a larger model. See our new results in a separate message (additional fine-tuned models). We agree that the transfer to ARCChallenge is limited, we will add a more explicit discussion of out-of-domain transfer in the results and limitations, moving the discussion of negative NDCG@10 from appendix to main text.

W3 and Q2 Paraphrase validity: We have now manually annotated a sample on 200 SQuAD question–paraphrase pairs to evaluate their validity. This sample did not include any false positives. All annotated examples retained the meaning of the original question, two of them being verbatim copies of the original (both were very short questions) while the remaining were valid paraphrases.

W4 Speculative language: We agree that the explanation in Section 4.3 is speculative. We will rephrase it and remove claims about the models’ training data. Regarding saturation, the Tatoeba performance varies a lot between languages and the performance is near-saturated only in the normal retrieval setting. Our new SQuAD and TextbookQA results also show the same collapse on standard retrieval tasks, so our conclusions do not depend on the ARCChallenge–Tatoeba contrast. We will revise the discussion accordingly.

W5 Metric choice: Our choice to use Recall@1 as the primary metric in distractor evaluation comes from the design setting where we add one paraphrase distractor per question to the pool. Therefore, R@1 directly measures what our probe tests: a model following a retrieval instruction should not rank a question above its answer. We will replace the framing “completely destroy” to be more specific what we measure.

W6 Potentially circular setup: We have now conducted an experiment where we explicitly break the potential circularity in fine-tuning/evaluation. We fine-tuned the same Qwen3-Embedding-0.6B by using the original queries themselves as negatives, preserving the generated paraphrases for evaluation only. This model performs on par with the previous fine-tuned model. Detailed results are provided in a separate message (additional fine-tuned models).

Q1 Distractor collapse replicate on one additional dataset: We have expanded the evaluation to use ARCChallenge, Tatoeba, SQuAD and TextbookQA. See our new results in a separate message (Promptriever and TextbookQA).

Q3 Metric choice: See our metric response above. We now also provide new instruction-swap results that show that retrieval behavior depends on instruction after fine-tuning. See our new results in a separate message (instruction swap experiment).

Q4 Mann-Whitney effect size: The MV column contains the rank-biserial correlation coefficient which is an effect size measure for the Mann-Whitney U test. It is calculated as (r_rb):

stat, p = stats.mannwhitneyu(relevant, not_relevant, alternative='greater')  # one tailed, hypothesis: relevant receives greater values
r_rb =  (2 * stat) / (len(relevant) * len(not_relevant)) -1 #formula flipped from standard presentation due to mannwhitneyu() returning the U for the greater group

Values close to +1 mean the relevant (task related) instructions receive higher scores, while values close to -1 would mean that the unrelated instructions (like keysmashes, wrong task instructions) would receive higher scores. The star (*) is the statistical significance of the Mann-Whitney U test (p<0.05 in the above code).

We believe these additions make the paper stronger and that we managed to address the main concerns. We hope that you could reconsider your evaluation. Please let us know if anything remains unclear.

Reviewer_UMtcOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how instructions change query embeddings in asymmetric retrieval across 8 embedding models. It shows that instructions barely move queries toward their targets, and that adding a paraphrase of each query to the corpus makes retrieval collapse for most models. The authors attribute this to the lack of query-side negatives in training, and show that fine-tuning with such negatives substantially improves robustness with little effect on other tasks.

Critical strengths
  1. The paper identifies a real evaluation gap. Standard benchmarks evaluate each task in isolation, so they never test whether an instruction can override plain semantic similarity. This matters for shared multi-task indices.
  2. The query-side distractor probe is simple and cheap and exposes the gap clearly. A single paraphrase per query is enough to make retrieval fail almost completely, and the failure is consistent across model families.
  3. The ablation without query-side negatives shows that the gain comes from the negatives, not from extra in-domain fine-tuning. The MTEB evaluation checks for side effects on other tasks.
Critical weaknesses
  1. GTE and Qwen3-Embedding already use InfoNCE variants with in-batch query–query negatives, and Promptriever trains with instruction-specific negatives. The paper neither discusses nor compares against these approaches. A simple alternative, filtering out retrieved candidates that are near-duplicates of the query, is also not considered.
  2. The gains may reflect a task-type shortcut rather than instruction following. Distractors differ from targets in surface form (questions vs. paragraphs in SQuAD, English vs. other languages in Tatoeba), so the model may simply learn to avoid such texts regardless of the instruction. Consistent with this, vague instructions also improve substantially, and the STS instruction is never tested on the distractor pool.
  3. The improvement is limited and largely in-domain. After fine-tuning, distractor performance stays below the normal setting. Transfer to the unseen ARCChallenge is negligible. Only one small model is fine-tuned, with a single seed, and the checkpoint is selected on the paraphrase-distractor dev set.
Questions for authors
  1. Does the fine-tuned model actually use the instruction? With the paraphrase distractor in the pool, please report what it retrieves under the STS instruction, under irrelevant or gibberish instructions, and with no instruction.
2 comments
AuthorsOpenReview ↗

Response to review

Thank you for your review and patience while we are working on our responses.

Instruction swap behavior: See our new instruction-swap experiment results in a separate message. To summarize, the results show that the STS instruction used during fine-tuning consistently (100% of the time) retrieves a paraphrase distractor, whereas QA and bitext instructions more often return the correct target type (answer, translation). Empty and gibberish (keysmash) instructions also continue to return distractors. We hope our new results addresses these concerns.

We are still working on other aspects, and we try to send rest of the responses, as well as update the pdf, as soon as possible.

AuthorsOpenReview ↗

Response to review

Thank you for your review and patience while we run new experiments. We are grateful that you found several strengths, and we believe that addressing your concerns makes the paper stronger. Here is our detailed response.

W1 InfoNCE variants: To our knowledge, the Qwen3-Embedding and GTE’s query–query negatives are other, unrelated queries, which matches our Random distractor evaluation setting. We show that Qwen3 still collapses under the paraphrase distractor setting, indicating that random query–query negatives are not sufficient. We believe one reason may be that training uses a negative mask to remove in-batch negatives that have high similarity (Qwen3 Embedding technical report, Section 3.1, https://arxiv.org/pdf/2506.05176). Such a setting explicitly removes negatives like our paraphrase distractors. We will add a discussion about this to the related work section.

W1 Promptriever: Promptriever’s instruction negatives are corpus-side (passages relevant to the query but not the instruction), complementary to our query-side negatives. We have added it to the pool of distractor evaluated models, see our new results in a separate message.

W1 Near-duplicate filtering: In a shared multi-task index, near-duplicates of the query are correct answers for tasks such as STS, so a filter would need to know the task in advance. This is exactly what the instruction should communicate to the model. Since our focus is on model behaviour that generalizes to other task pairs in the future, we did not consider task-specific filtering or reranking.

W2 and Q1 Instruction swap: See our new instruction-swap experiments in a separate message. To summarize, the results show that the STS instruction used during fine-tuning consistently (100% of the time) retrieves an added distractor, whereas QA and bitext instructions more often returns the correct target type (answer, translation). Empty and gibberish (keysmash) instructions also continue to return paraphrase distractors.

W3 Limited and in-domain improvement: We agree that the fine-tuning does not fully solve the problem, it is intended as a case study of the model behavior rather than a full solution. Our new results (additional fine-tuned models) addresses several of the other concerns. Different model size is tested, on-par performance is observed without checkpoint selection, and a fresh training with different negative source (original questions instead of paraphrases) gives comparable result. We agree that transfer to ARCChallenge is negligible and the instruction-swap analysis shows that transfer to unseen instructions is also limited, we will add an explicit discussion of out-of-domain transfer in the results and limitations.

We believe these additions make the paper stronger and that we managed to address your main concerns. We hope that you could reconsider your evaluation. Please let us know if anything remains unclear.

Comments on the paper

4 comments
AuthorsOpenReview ↗

New experimental results (instruction swap experiment)

To address the reviewers' concerns about how the fine-tuned model uses instructions, we have run a new experiment, and provide its results here.

To test whether the fine-tuned model actually uses the instruction, we made an additional experiment, where we keep the candidate pool fixed (containing all target answers (A) and paraphrase distractors (D)) and vary only the instruction. We report if the top-1 hit is the query’s own answer (A), its own distractor (D), another answer (OA), or another distractor (OD). A model that follows instructions should switch between them. Values are % of test queries.

SQuAD

instruction typeQwen baseline (A/D/OA/OD)Qwen finetuned
QA / finetune1.4 / 95.5 / 0.2 / 2.947.0 / 34.5 / 17.3 / 1.3
QA / ARCC0.2 / 97.6 / 0.0 / 2.212.2 / 82.9 / 2.9 / 2.0
STS / finetune0.0 / 98.2 / 0.0 / 1.80.0 / 98.1 / 0.0 / 1.9
STS / other0.1 / 94.5 / 0.0 / 5.59.1 / 69.9 / 3.8 / 17.2
empty instr.0.0 / 98.5 / 0.0 / 1.50.0 / 98.2 / 0.0 / 1.8
keysmash0.1 / 97.7 / 0.0 / 2.210.3 / 84.9 / 2.6 / 2.2

General results: The baseline retrieves the distractor regardless of instruction. After fine-tuning, the QA fine-tuning instruction (QA / finetune) returns an answer passage in 64.3% of queries, the correctness (A vs. OA) reflects model’s normal ~70% R@1 in normal setting (fine-tuned, Normal in Table 6). The STS fine-tuning instruction and the empty instruction never return an answer passage, and a gibberish keysmash instruction returns an answer passages in 12.9% of cases.

Transfer to unseen: Transfer to unseen instructions is limited but shows a compatible pattern to our general ARCChallenge results. ARCC instruction (QA / ARCC) retrieves a passage in 15.1% cases (baseline 0.2%). Generic STS instruction (STS / other, “Retrieve semantically similar text.”) mostly retrieves questions, as intended, with passages in 12.9% cases.

Tatoeba (French)

instructionQwen baseline (A/D/OA/OD)Qwen finetuned
bitext / finetune13.7 / 82.6 / 0.4 / 3.383.3 / 10.3 / 6.0 / 0.4
bitext / other26.5 / 69.3 / 1.6 / 2.679.9 / 14.0 / 5.2 / 0.9
STS / finetune0.0 / 92.8 / 0.0 / 7.20.0 / 93.3 / 0.0 / 6.7
STS / other6.9 / 88.7 / 0.2 / 4.210.0 / 85.3 / 0.3 / 4.4
empty instr.1.2 / 94.6 / 0.0 / 4.20.5 / 95.2 / 0.0 / 4.3
keysmash1.2 / 93.4 / 0.1 / 5.31.6 / 92.8 / 0.1 / 5.5

French: The pattern is the same. The fine-tuned bitext instruction returns a French sentence in 89.3% of queries, while the STS never does so. Also empty and keysmash instructions rarely return French top-1 hits (0.5% and 1.7%, respectively). Unlike Qa, transfer to unseen bitext instruction is strong (85.1%), however, the name of the language may be a strong cue in the instruction. The generic STS instruction (STS / other, “Find a sentence that has similar meaning.”) retrieves French sentences in 10.3% of cases.

Other languages: All eight languages show the same general pattern, most variation being in baseline bitext performance and generic STS instruction results. Due to length limitations, we are not able to include all languages here, but we will update the paper as soon as possible.

AuthorsOpenReview ↗

New experimental results (Promptriever + TextbookQA)

To test whether the distractor effect generalizes beyond ARCChallenge, we extended the distractor analysis (Table 5) with two retrieval datasets with paragraph-length targets, SQuAD and TextbookQA (subset of MRQA including science-textbook passages) as suggested by reviewer Vvna, and one additional model, Promptriever, which is trained with instruction-specific negatives (suggested by reviewer UMtc). As before, we test 47 instructions (51 for Tatoeba) per model and report the maximum Recall@1 over instructions.

Updated Table 5

ARCChallengeTatoeba (max)SQuADTextBookQA
Modelnormaldistr.normaldistr.normaldistr.normaldistr.
BAAI/bge-m30.0320.0010.9870.4710.7180.0050.4940.005
codefuse-ai/F2LLM-v2-4B0.2140.0620.9980.6260.8070.0610.5970.021
google/embeddinggemma-300m0.0190.0050.7250.2500.7340.0000.5610.001
multilingual-e5-large-instruct0.0740.0070.9940.9540.7860.0240.5020.037
microsoft/harrier-oss-v1-0.6b0.1060.0070.9880.9560.8060.0120.5990.083
Octen/Octen-Embedding-8B0.1490.0050.9920.6710.7650.0300.6110.024
Qwen/Qwen3-Embedding-0.6B0.0590.0070.9770.5000.7310.0270.5300.015
Qwen/Qwen3-Embedding-4B0.1120.0380.9940.5290.7890.0220.5970.037
promptriever-llama3.1-8b*0.0590.0020.9710.9220.8100.3840.5920.130

* Full model name: samaya-ai/promptriever-llama3.1-8b-instruct-v1

The results are consistent with our earlier findings. Unlike ARCChallenge, SQuAD and TextbookQA have high performance in the normal setting, but adding distractors again reduces Recall@1 to near zero for all models except Promptriever. Promptriever is clearly more robust, consistent with our hypothesis that the choice of negatives during training matters, but it also loses a large part of its performance. Interestingly, the Promptrievers best SQuAD distractor performance (0.384) comes from a clustering instruction (“Identify the main and secondary category of Arxiv papers based on the titles.”), while the best relevant instruction (“Retrieve'”) has much lower score (0.297), compatible to our earlier findings related to task relevance (Section 4.1).

AuthorsOpenReview ↗

New experimental results (additional fine-tuned models)

To test fine-tuning with a larger model (reviewers UMtc and Vvna) and remove the potential circularity of fine-tuning and evaluating on generated paraphrases (reviewer Vvna), we fine-tuned two additional models on SQuAD: (1) Qwen3-Embedding-4B, and (2) Qwen3-Embedding-0.6B using the original question itself as the query-side negative, so that generated paraphrases are used only in evaluation. All models use same hyperparameters as the original experiment and the final checkpoint, without any selection on development data. For the original fine-tuned model, the final checkpoint gives practically identical results to the selected one (reported in Table 6), so checkpoint selection does not affect our conclusions.

SQuAD results:

modelR@1 NormalR@1 Paraphr.R@5 NormalR@5 Paraphr.
Qwen3 0.6B baseline0.7240.0140.9180.778
Qwen3 0.6B finetune0.7060.4700.9110.889
Qwen3 0.6B question neg0.7020.4890.9090.889
Qwen 4B baseline0.7740.0160.9490.838
Qwen 4B finetune0.6870.6430.9140.909

Instruction-swap results: We report if the top-1 hit is the query’s own answer (A), its own distractor (D), another answer (OA), or another distractor (OD).

promptQwen-baseline-4B (A/D/OA/OD)Qwen-finetuned-4B
QA / finetune1.6 / 95.8 / 0.2 / 2.564.3 / 6.9 / 28.4 / 0.4
QA / ARCC0.3 / 97.7 / 0.0 / 2.130.8 / 61.5 / 6.4 / 1.3
STS / finetune0.0 / 98.3 / 0.0 / 1.70.0 / 98.3 / 0.0 / 1.7
STS / other0.0 / 96.3 / 0.0 / 3.60.1 / 94.1 / 0.0 / 5.8
empty instr.0.0 / 98.5 / 0.0 / 1.40.0 / 98.5 / 0.0 / 1.5
keysmash0.1 / 98.1 / 0.0 / 1.817.9 / 77.5 / 3.1 / 1.5

The results support our earlier findings. Training with the original questions as negatives (Qwen3 0.6B question neg) performs on par with training on paraphrases, showing that the gains do not depend on potential circularity. The larger model benefits more from fine-tuning, reaching 0.643 under paraphrase distractors and clearly outperforming its smaller variant. However, its normal performance decreases, likely because we reused the hyperparameters of the 0.6B model without further tuning. The instruction-swap analysis confirms that the larger model also selects the target type according to the instruction.

AuthorsOpenReview ↗

Discussion period ending soon

Dear AC,

As the discussion period has almost ended and we have not received engagement from two reviewers, we would like to summarize our new contributions and rebuttals:

TL;DR:

  1. We have run a substantial amount of new experiments, each showing positive results and negating the pointed-out weaknesses.
  2. Our instruction swap experiment shows that the models respond to instructions as designed.
  3. Our comparison to new baselines increases the validity of our claims, in both new datasets and model sizes, as well as comparison to Promptriever.
  4. We have now updated the pdf to include all new experiments (in Appendices K, L, and M) and other changes discussed in our responses.

Reviewer UMtc:

  • (InfoNCE variants): We clarify that InfoNCE variants match our random distractor evaluation, not the STS distractor evaluation, which is at the core of our claims.
  • (Promptriever): We clarify that Promptriever uses corpus-side negatives, which are complementary to our query-side negatives. We added Promptriever as a baseline and show that while it is more robust than other baselines, our fine-tuned models show better resilience to distractors.
  • (Applicability of filtering): We explain that since our focus is on the model behavior with instructions, filtering is not aligned with this goal.
  • (Following instructions): We presented the instruction swap experiment, which showed that instruction following works as intended.
  • (Out-of-domain performance): We conceded that fine-tuning does not solve the issue fully, but our study is intended to study model behavior and inform the community of shortcomings in evaluation, not to solve the problem conclusively.

Reviewer Vvna:

  • (Task coverage): We increased the number of tasks in the discussion period, with all results supporting our conclusions.
  • (Model size): We have now included experiments with multiple model sizes, with all results supporting our conclusions.
  • (Paraphrase validity): We manually annotated 200 question-paraphrase pairs and found no false positives.
  • (Speculative Language): We modified the language in Section 4.3.
  • (Metric Choice): We explained why we use Recall@1 and why it is central to our experiments.
  • (Circularity): We ran a new experiment with all possible sources of circularity removed, with all results supporting our conclusions.
  • Additionally, we answered all other questions.

Reviewer 2GG4:

  • We thank the reviewer for engagement!
  • The discussion shows that we have successfully responded to the brought-up weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.