- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a dense embedding can contain useful retrieval signals even when its full-space score fails. It first studies whether selected low-rank projections recover failed queries, then uses a direction constructed from each training query and its positive and negative examples to modify a distillation target. The training method improves over standard fine-tuning on most of the evaluated datasets.
Critical strengths
The distinction between information present in an embedding and information expressed by its full-space score is interesting. The score decomposition gives the diagnostic analysis a clear basis, and the paper is appropriately explicit that its subspace recovery results use an oracle. The training study covers several datasets and includes ablations that help assess the contribution of the added supervision.
Critical weaknesses
-
The striking recovery results depend on selecting a successful subspace after seeing relevance labels. They show that a suitable direction can exist, but do not show that one can be found for an unseen query without that information. A null or label-permutation comparison, together with a feasible selection procedure, would make the diagnostic result easier to interpret. The analysis also uses a different data and input setup from the training experiments, so its connection to the proposed learning method remains indirect.
-
The practical improvement is modest and uneven. Moreover, the proposed gate activates for almost every training query, and forcing it on gives similar results where that comparison is available. This makes the claimed selectivity less persuasive. A plain distillation baseline using the same candidates, along with paired uncertainty estimates for the retrieval results, would better isolate the value of the constructed target.
Questions for authors
- How much oracle recovery remains with permuted relevance labels, and what can a selector achieve without access to test labels?
- How does the method compare with ordinary score distillation under the same training setup?
Sections 3 and 4 address distinct but related research questions. Section 3 investigates whether retrieval failures can be recovered through low dimensional subspaces of frozen embeddings, while Section 4 explores whether discriminative signals associated with such recovery can be constructed and utilized as training supervision. Thus, Section 3 examines recoverability rather than proposing a practical procedure for selecting subspaces at inference.
However, we acknowledge that the current manuscript does not sufficiently explain the transition from Section 3 to Section 4. In particular, the theoretical motivation for explicitly constructing the Rank 1 direction and the supporting experimental findings are not clearly presented or summarized. We will revise the manuscript to clarify this connection, provide a more explicit justification for the direction construction, and summarize the relevant diagnostic results while distinguishing their experimental settings from those used for training.
We acknowledge that the practical improvements are modest and vary across datasets. Nevertheless, this study aims to investigate whether discriminative signals associated with retrieval failure recovery can be utilized as supervision for full space representation learning. We believe that demonstrating this possibility remains a meaningful contribution.
Regarding the gate, its activation rate is close to 100%, so the similar results obtained with g=1 are expected. The comparison with g=0, which removes the additional supervision, provides evidence that incorporating the proposed target contributes to the observed improvements. As already discussed in the manuscript, Table 5 shows that the gate is activated for nearly all training queries, resulting in similar performance when g=1. The primary purpose of the ablation is to examine the contribution of the additional supervision through g=0 and the effect of selective gating through g=1.
We thank the reviewer for this observation. We conducted an additional matched random document null comparison, preserving the original embeddings, candidate pool, failure cohort, and number of relevant documents per query. Control documents were selected outside the original top 10 and excluded from positive qrels.
We acknowledge that high oracle recovery alone does not establish relevance specificity or provide a practical method for selecting suitable subspaces for unseen queries. These limitations, together with the absence of consistently shared recovery directions in our overlap analysis, motivate the explicit construction of a Rank 1 direction from the query and labeled relevant and nonrelevant documents in Section 4. Rather than relying on random subspace selection, the constructed direction is used to provide training supervision. At inference, the trained encoder requires neither relevance labels nor subspace selection.
In Section 4, the constructed Rank 1 direction explicitly depends on relevant and nonrelevant document embeddings, whose identities are available during supervised training but unknown at inference. Therefore, the same construction cannot be directly applied to unseen queries without relevance information.
The g=0 setting in Table 5 corresponds to pretrained self distillation, where the teacher and student are initialized from the same pretrained encoder. Consequently, their initial score distributions are identical under deterministic evaluation, resulting in no initial KL gradient. Therefore, this setting is not intended to serve as a comprehensive baseline for conventional score distillation.
Instead, the g=0 ablation is designed to examine the contribution of the additional supervision introduced by our Rank 1 correction. The observed performance differences indicate that incorporating this correction provides a learning signal beyond the original pretrained distribution.
Considering the related concerns raised by other reviewers, we will conduct additional comparisons with label smoothed InfoNCE and KL distillation using a mixture of the pretrained distribution and one hot labels under matched training conditions and temperatures. We will report the results and provide further discussion in a subsequent response.