acceptodds
Under review as a conference paper at ICLR 2027

Failure Is Not Absence: Selective Low Rank Supervision For Dense Retrieval

Abstract

Dense retrieval failures are often attributed to representations that do not sufficiently capture the discriminative structure required to distinguish relevant from non relevant documents. We challenge this view by showing that retrieval failure does not necessarily imply the absence of such structure. Using frozen single vector representations, we show that queries that fail under the original retrieval function are recovered through low rank readouts. We further find that the discriminative signals enabling this recovery are extracted even from very low dimensional subspaces. By decomposing the full space score into selected subspace and residual contributions, we show that the low rank contributions associated with recovery are counteracted by residual contributions, leading to dense retrieval failure. Building on these findings, we propose Selective Low Rank Supervision, which selects beneficial rank one signals and aligns the full embedding representation with the selected signals during training. The proposed method requires no additional subspace computation at inference, while outperforming standard contrastive fine tuning. These results indicate that the low rank structure revealed by our analysis is not only informative for understanding retrieval failure, but can also provide an effective training signal for improving the full embedding representation for retrieval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
dense retrieval, representation learning, low rank subspaces, embedding geometry, selective supervision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7aYqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a dense embedding can contain useful retrieval signals even when its full-space score fails. It first studies whether selected low-rank projections recover failed queries, then uses a direction constructed from each training query and its positive and negative examples to modify a distillation target. The training method improves over standard fine-tuning on most of the evaluated datasets.

Critical strengths

The distinction between information present in an embedding and information expressed by its full-space score is interesting. The score decomposition gives the diagnostic analysis a clear basis, and the paper is appropriately explicit that its subspace recovery results use an oracle. The training study covers several datasets and includes ablations that help assess the contribution of the added supervision.

Critical weaknesses
  1. The striking recovery results depend on selecting a successful subspace after seeing relevance labels. They show that a suitable direction can exist, but do not show that one can be found for an unseen query without that information. A null or label-permutation comparison, together with a feasible selection procedure, would make the diagnostic result easier to interpret. The analysis also uses a different data and input setup from the training experiments, so its connection to the proposed learning method remains indirect.

  2. The practical improvement is modest and uneven. Moreover, the proposed gate activates for almost every training query, and forcing it on gives similar results where that comparison is available. This makes the claimed selectivity less persuasive. A plain distillation baseline using the same candidates, along with paired uncertainty estimates for the retrieval results, would better isolate the value of the constructed target.

Questions for authors
  1. How much oracle recovery remains with permuted relevance labels, and what can a selector achieve without access to test labels?
  2. How does the method compare with ordinary score distillation under the same training setup?
1 comment

Sections 3 and 4 address distinct but related research questions. Section 3 investigates whether retrieval failures can be recovered through low dimensional subspaces of frozen embeddings, while Section 4 explores whether discriminative signals associated with such recovery can be constructed and utilized as training supervision. Thus, Section 3 examines recoverability rather than proposing a practical procedure for selecting subspaces at inference.

However, we acknowledge that the current manuscript does not sufficiently explain the transition from Section 3 to Section 4. In particular, the theoretical motivation for explicitly constructing the Rank 1 direction and the supporting experimental findings are not clearly presented or summarized. We will revise the manuscript to clarify this connection, provide a more explicit justification for the direction construction, and summarize the relevant diagnostic results while distinguishing their experimental settings from those used for training.


We acknowledge that the practical improvements are modest and vary across datasets. Nevertheless, this study aims to investigate whether discriminative signals associated with retrieval failure recovery can be utilized as supervision for full space representation learning. We believe that demonstrating this possibility remains a meaningful contribution.

Regarding the gate, its activation rate is close to 100%, so the similar results obtained with g=1g=1 are expected. The comparison with g=0g=0, which removes the additional supervision, provides evidence that incorporating the proposed target contributes to the observed improvements. As already discussed in the manuscript, Table 5 shows that the gate is activated for nearly all training queries, resulting in similar performance when g=1g=1. The primary purpose of the ablation is to examine the contribution of the additional supervision through g=0g=0 and the effect of selective gating through g=1g=1.


We thank the reviewer for this observation. We conducted an additional matched random document null comparison, preserving the original embeddings, candidate pool, failure cohort, and number of relevant documents per query. Control documents were selected outside the original top 10 and excluded from positive qrels.

DatasetRank 1 RelevantRank 1 ControlRank 32 RelevantRank 32 Control
FiQA31.4127.8096.1516.97
NFCorpus98.8698.0897.7392.90
SciFact97.9683.49100.0054.72
ArguAna66.0266.50100.0031.35
SCIDOCS81.7184.7999.1264.80

We acknowledge that high oracle recovery alone does not establish relevance specificity or provide a practical method for selecting suitable subspaces for unseen queries. These limitations, together with the absence of consistently shared recovery directions in our overlap analysis, motivate the explicit construction of a Rank 1 direction from the query and labeled relevant and nonrelevant documents in Section 4. Rather than relying on random subspace selection, the constructed direction is used to provide training supervision. At inference, the trained encoder requires neither relevance labels nor subspace selection.


In Section 4, the constructed Rank 1 direction explicitly depends on relevant and nonrelevant document embeddings, whose identities are available during supervised training but unknown at inference. Therefore, the same construction cannot be directly applied to unseen queries without relevance information.


The g=0g=0 setting in Table 5 corresponds to pretrained self distillation, where the teacher and student are initialized from the same pretrained encoder. Consequently, their initial score distributions are identical under deterministic evaluation, resulting in no initial KL gradient. Therefore, this setting is not intended to serve as a comprehensive baseline for conventional score distillation.

Instead, the g=0g=0 ablation is designed to examine the contribution of the additional supervision introduced by our Rank 1 correction. The observed performance differences indicate that incorporating this correction provides a learning signal beyond the original pretrained distribution.

Considering the related concerns raised by other reviewers, we will conduct additional comparisons with label smoothed InfoNCE and KL distillation using a mixture of the pretrained distribution and one hot labels under matched training conditions and temperatures. We will report the results and provide further discussion in a subsequent response.

Reviewer_QTxBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work analyzes whether dense retrieval failure implies the absence of information in frozen embeddings at a lower subspace. Using random low-rank subspaces, the authors demonstrate that most failed queries can be recovered by some certain low-rank subspaces. Also, they shows that residual components counteract this recovery signal. Based on this observation, Selective Low Rank Supervision is proposed. It constructs a rank-1 direction from the query and embeddings from positive and negative documents and adds its projected score to the full-space score to form a teacher distribution. The retrieval is trained with KL divergence loss. The method improves nDCG over InfoNCE FT on four of five BEIR datasets.

Critical strengths
  • The central question is interesting. The subspace and residual decomposition give an intuitive explanation of retrieval failure even though they have the useful signal.
Critical weaknesses
  • Gap between the analysis and the method.
    • The analysis studies random subspaces selected using relevance labels. However, the proposed method uses an analytic direction built from the query, positive, and negative embeddings. The findings from the analysis are not exploited beyond choosing rank 1.
    • To address this, the authors should show that the subspaces analyzed in Section 3 themselves provide a useful training signal.
  • The method is close to self-distillation with soft labels, but comparisons with such methods are missing.
      • Expanding Eq. 8 gives sP=12(sF+δ⊤d)s_P = \frac{1}{2}(s_F + \delta^\top d), so the teacher logit is 1.5⋅sF+0.5⋅δ⊤d1.5\cdot s_F + 0.5\cdot\delta^\top d. Since uu bisects qq and δ\delta, their alignment cancels exactly, so the projection adds no query information beyond sFs_F. The only new term, δ⊤d\delta^\top d, depends on the labeled pair (d+,d−)(d^+, d^-) but not on the query embedding. The method therefore reduces to self-distillation with a label-dependent term.
    • The g=0g=0 ablation does not isolate the recovery signal. Its target equals the student's initial distribution with no label information, so it only shows that labels are necessary, not that the recovery direction is.
    • To address this, the authors should compare with self-distillation baselines for retrieval (e.g., label-smoothed InfoNCE or KL to a mixture of the pretrained distribution and the one-hot label) under matched temperature.
Questions for authors
  • Given that the query term cancels, what does the rank-1 projection contribute beyond a label-dependent document-side bonus?
  • Can randomly chosen non-relevant documents be "recovered" into the top-10 at a similar rate under the same oracle selection?
2 comments

We sincerely thank the reviewer for the careful evaluation and valuable feedback. We greatly appreciate the constructive suggestions.

Our analysis first asks whether failed queries can be recovered using subspaces of frozen embeddings. The overlap analysis does not reveal consistently shared directions beyond the random baseline. This motivates constructing a recovery direction directly from the query and the relevant and nonrelevant documents. Additional diagnostics for Qwen3 Embedding 0.6B confirm that these constructed directions produce recovery:

DatasetRecovered / Full FailuresRecovery (%)
FiQA1741 / 174499.883
NFCorpus714 / 714100.000
SCIDOCS243 / 243100.000

These are diagnostics on labeled training queries using frozen embeddings with a maximum input length of 512 tokens, not held out performance. They support the recovery capability of the constructed direction, but do not directly establish the training utility of the random subspaces from Section 3.


Algebraic Interpretation and Motivation for Rank One Supervision

We thank the reviewer for the algebraic observation. Under Eq. (7), the projected score satisfies

sP(q,d)=12q⊤d+12δ⊤ds_P(q,d)=\tfrac12 q^\top d+\tfrac12\delta^\top d

when q+δ≠0q+\delta\neq0. With the gate active, sT(q,d)=32q⊤d+12δ⊤ds_T(q,d)=\tfrac32 q^\top d+\tfrac12\delta^\top d. We agree that the projection introduces no new query interaction: it rescales the original similarity and adds a label dependent document correction.

Our goal is not to use Rank 1 projection as a new retrieval function. Our analysis shows that low dimensional subspaces can recover retrieval failures, but selecting a suitable direction for an unseen query without relevance labels is difficult. Directly applying such directions was also sensitive to their selection in our exploration. This motivates using their discriminative information as training supervision instead. Labeled document pairs define the correction, which enters the teacher distribution only when it increases the positive document probability. KL training then transfers this signal to a student that uses the full embedding space at inference.

This correction differs algebraically from mixing the pretrained distribution with a one hot label: its geometry dependent term can also alter relative scores among negative candidates. We agree that this difference alone does not establish a performance advantage.

Table 5 is intended to evaluate the contribution of the proposed supervision and the effect of selective gating, rather than to isolate the recovery direction itself. Specifically, g=0g=0 removes the additional supervision, while g=1g=1 applies it to all queries. Comparing g=1g=1 with our selective gating strategy evaluates the benefit of selectively incorporating the Rank 1 signal. We acknowledge that these ablations do not establish whether the discriminative signal derived from the recovery direction provides benefits beyond conventional label based supervision.

To address this distinction, we will conduct direct comparisons with label smoothed InfoNCE, KL distillation using a mixture of the pretrained distribution and one hot labels, and a score scaling control under matched training conditions and temperatures. We will clarify the algebraic interpretation and the purpose of the ablations and provide the additional experimental results and discussion in a subsequent response.


We conducted an additional control experiment using the original embeddings and baseline failure queries, evaluating 1,024 candidate subspaces at each rank in 1,2,4,8,16,32{1,2,4,8,16,32}. For each query, we sampled control documents outside the original top 10 and positive qrels, matching the number of relevant documents.

DatasetRank 1 RelevantRank 1 ControlRank 32 RelevantRank 32 Control
FiQA31.4127.8096.1516.97
NFCorpus98.8698.0897.7392.90
SciFact97.9683.49100.0054.72
ArguAna66.0266.50100.0031.35
SCIDOCS81.7184.7999.1264.80

All values are percentages. At rank 1, control recovery is comparable to or higher than relevant recovery in several datasets. At rank 32, relevant recovery exceeds control recovery across all five datasets. However, these differences are descriptive and do not establish statistical significance.

We agree that high oracle recovery alone does not establish relevance specificity. Furthermore, our overlap analysis does not reveal consistently shared recovery directions beyond the random baseline. These observations highlight the limitations of relying on randomly selected subspaces and provide part of the motivation for Section 4, where we explicitly construct a Rank 1 direction from the query and labeled document pairs rather than relying on random subspace selection. We will clarify this connection and the limitations of oracle recovery in the revised manuscript.

Thanks for the response.

The additional experiments and explanation about algebric interpretation are useful for understanding the method. However, I think the comparison with label smoothing should be considered to strengthen the benefit of the proposed method. So, I will keep the current score.

Reviewer_8RmqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes frozen query and document embeddings from Qwen3-Embedding-0.6B. It shows that for many queries that fail at top 10, the relevant document can be brought into the top 10 by scoring in a low-rank linear subspace, chosen per query from 1024 random projections. Building on this, it constructs a rank-1 direction from the query and the positive–negative difference for each training query. Adding the projected score to the frozen teacher's scores moves the query towards the positive and away from the negatives. The resulting distribution is the soft target for fine-tuning the same model with KL. On five BEIR datasets (FiQA, NFCorpus, SciFact, ArguAna, SCIDOCS), it reports small, mixed gains over InfoNCE fine-tuning (+0.18 nDCG@10 on average).

Critical strengths
  • The paper asks an interesting research question: whether an embedding model fails because it fails to encode the relevant information in the embeddings or because the similarity function (e.g., cosine) can not to extract the information already encoded.
Critical weaknesses
  • Experiments need to be more rigor and transparent. (1) The source of training data and its distribution needed to be clearly stated in the paper. (2) For evaluation, please separate between in-distribution and out-of-distribution evaluation. A method that performs well on in-distribution data might not generalize to out-of-distribution data. In addition, please report the numbers on full BEIR benchmark, do not just select a few small subsets. (3) In analysis 3.2, please add a random baseline (the chance that the relevant document reaches the top 10 in at least one trial across 1024 trials or as many trials as the number of projections tested) and please use a larger corpus (ArguAna is very small for a retrieval dataset). Analysis 3.3 (Table 2) interpolates between a failing score and a projection already chosen to succeed, so flips are expected and the conclusion is biased.

  • Fine-tuning baselines needs to be more comprehensive. The method can be seen as teacher-guided label smoothing: about 81% of the target mass is on the positive, and the rest follows the frozen teacher. The paper should therefore compare against label smoothing and CE + KD baselines. The comparison is also confounded by temperature: InfoNCE uses t = 0.01 and the proposed KL loss uses t = 0.05. Please sweep the InfoNCE temperature, selecting on validation as for the proposed method.

Questions for authors
  1. What training data is used for each dataset, and how are the validation and custom ArguAna/SCIDOCS splits built?

  2. Does the method generalize out of distribution, e.g. trained on MS MARCO and evaluated zero-shot on the full BEIR benchmark?

  3. In 3.2, what is the recovery rate under a chance baseline and on larger corpora?

  4. How does the method compare to label smoothing, CE + KD, and InfoNCE with a temperature sweep? Why do the baseline (t = 0.01) and the proposed method (t = 0.05) use different temperatures?

  5. How does the method work on other embedding models that are not not Qwen?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.