- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The work focuses on the domain transfer for dense retriever commonly used in RAG or recommendation system, where the conventional approach of training a lightweight adapters, although requires less labels, may still suffer from insufficient labels. To this end, the authors propose the Representation Correction Distillation (RCD), a semi-supervised framework that distills the task-specific representation more efficiently. Experimental results show effectiveness of the approach.
Critical strengths
-
The domain adaptation where relevance definition is beyong semantic similarity and different for domain is true, and important in many scenarios.
-
The proposed approach is reasonable to me, and the idea of "distill the change" is well-motivated and well-supported by ablations.
Critical weaknesses
-
The author claims that the main challenge of the adaptor is due to label scarcity. However, given the reasoning capability, can we directly use LLMs as teacher to generate retrieval labels and fine-tune the retrieval model? Or as judge to improve student model. If there are existing works, a comparison will be helpful. If not, it'd be valuable to explain why the proposed approach is the most promising.
-
The main end-to-end evaluation result looks problematic in the following aspects
- The test set is small (e.g., PrivacyQA has about 19 test queries, i.e., 10% of ~195).
- The improvement looks incremental with small margin (i.e., +3.7% over the teacher comes from 3 wins and 2 ties across five datasets).
- Some patterns/trends in the results weaken the major claim: (1) while main tables suggests that student beats the teacher only at nDCG@8, it loses at @16, @32 and @64; (2) when the labeled percentage is at 100% (i.e., no unlabeled), the student still beats teacher, which may suggesting the improvement does not come from the semi-supervised approach.
Questions for authors
I suggest the authors to include more discussion on the result corresponding to weakness 2 above, and strengthen the evaluation.
2 comments
Dear Reviewer XY46,
Thank you for your feedback and questions. In our responses, we have clarified the relationship between RCD and LLM-generated relevance supervision, provided additional statistical evidence for improvements over the teacher, and discussed performance across retrieval cutoffs and the contribution of unlabeled queries.
We hope these clarifications have addressed your concerns. We would appreciate it if you could consider revisiting your assessment, and we would be happy to answer any remaining questions.
Response to Reviewer XY46
We thank the reviewer for their thoughtful feedback and questions.
W1: LLM-Generated Relevance Supervision
LLM-generated relevance supervision is a valid approach to addressing label scarcity. Section 2 discusses existing methods, including GPL, InPars, and Promptagator, and Figure 2 already includes InPars as an LLM-based synthetic-supervision baseline. RCD addresses the complementary problem of transferring specialization learned by a retriever adapter into its underlying dense retriever using unlabeled queries. LLM-generated relevance judgments could be used to train the adapter teacher or provide additional supervision for the student retriever in an extended setting. Such approaches introduce additional generation or relevance-scoring costs, whereas RCD uses the specialization already learned by the adapter without requiring these stages during student training. We will clarify how these approaches differ and how they could be used together in the revised manuscript.
W2: Evaluation Results and Interpretation
RCD's central objective is to transfer adapter-learned specialization into a standalone dense retriever using unlabeled queries. Surpassing the adapter teacher is an additional outcome whose magnitude varies across datasets and retrieval settings.
1. Test-set size: We acknowledge that PrivacyQA has a relatively small test set (~19 queries), making its performance estimates more sensitive to individual queries. However, our evaluation spans five benchmarks, including ContractNLI, a related legal-domain benchmark with a substantially larger test set, where RCD outperforms its teacher by 18.3% (Table 3). Thus, our findings do not depend on the small PrivacyQA test set alone.
2. Improvement over the teacher: At 50% supervision, RCD outperforms the IMRNN teacher on ContractNLI (+18.3%), ComLQ (+5.0%), and SciFact (+3.5%), achieving a 3.7% relative macro-average improvement (Table 3). To assess statistical significance, we conducted a two-sided paired randomization test using per-query nDCG@8 scores averaged across three seeds, with equal weighting across all five datasets. The macro-average improvement is statistically significant (p = 0.046). While RCD slightly underperforms the teacher on PrivacyQA (-0.7%) and NFCorpus (-0.6%), the results support our claim that RCD can outperform its teacher in the evaluated setting.
3. Performance across retrieval cutoffs: RCD outperforms the IMRNN teacher in macro-average nDCG@2 (0.463 vs. 0.443), nDCG@4 (0.476 vs. 0.460), and nDCG@8 (0.505 vs. 0.487), as reported in Appendix B (Table 10). At larger cutoffs, performance varies across datasets. Notably, RCD consistently outperforms the teacher on ComLQ across all six evaluated cutoffs, including @16 (0.843 vs. 0.815), @32 (0.847 vs. 0.822), and @64 (0.847 vs. 0.823). RCD also remains competitive or better on ContractNLI at @16 and @32, and PrivacyQA at @64. While the macro-average falls below the teacher at larger cutoffs, these results show that the gains extend beyond our primary metric, nDCG@8, and can persist at greater retrieval depths.
4. Results at 100% supervision: RCD can outperform its teacher even without additional unlabeled queries, as demonstrated by the Search-Adaptor results at 100% supervision (Table 11). This shows that transferring adapter-learned specialization into the student encoder can itself improve retrieval performance. Additional unlabeled queries provide further gains, as demonstrated by the controlled experiment with IMRNNs at 50% supervision (Table 5). RCD Labeled Only already surpasses the teacher on ContractNLI and ComLQ, while RCD All Queries further improves performance across all five datasets using the same frozen teacher and optimization budget. The macro-average improvement is statistically significant (p = 0.021, two-sided paired randomization test). Thus, the 100% supervision result is consistent with the independent benefit of unlabeled queries established in Table 5.