acceptodds
Under review as a conference paper at ICLR 2027

Retriever Adapters as Teachers: Representation Correction Distillation for Dense Retrieval

Abstract

Adapting dense retrievers to new target domains is severely constrained by the high cost of manual relevance labeling. While lightweight adapters can learn domain specialization from limited labeled data, they leave abundant unlabeled queries unused. To address this, we introduce Representation Correction Distillation (RCD), a semi-supervised framework that transfers adapter-learned specialization into a standalone dense retriever using both labeled and additional unlabeled target queries, without using relevance labels during student training. Rather than matching full representation spaces, RCD explicitly isolates and distills the task-specific representation updates induced by the adapter, preserving the base model's pre-existing structure while integrating new domain preferences via ranking distillation. Consequently, the distillation phase requires zero additional relevance labels. Across five benchmark datasets, RCD improves macro-average nDCG over the base retriever by up to 21.7%. When the teacher is trained on only 50% of the labeled queries, RCD uses additional unlabeled target queries to train a student that outperforms the teacher by 3.7% in macro-average nDCG.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
dense-retrieval, semi-supervised, representation learning, information retrieval, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XY46OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work focuses on the domain transfer for dense retriever commonly used in RAG or recommendation system, where the conventional approach of training a lightweight adapters, although requires less labels, may still suffer from insufficient labels. To this end, the authors propose the Representation Correction Distillation (RCD), a semi-supervised framework that distills the task-specific representation more efficiently. Experimental results show effectiveness of the approach.

Critical strengths
  • The domain adaptation where relevance definition is beyong semantic similarity and different for domain is true, and important in many scenarios.

  • The proposed approach is reasonable to me, and the idea of "distill the change" is well-motivated and well-supported by ablations.

Critical weaknesses
  • The author claims that the main challenge of the adaptor is due to label scarcity. However, given the reasoning capability, can we directly use LLMs as teacher to generate retrieval labels and fine-tune the retrieval model? Or as judge to improve student model. If there are existing works, a comparison will be helpful. If not, it'd be valuable to explain why the proposed approach is the most promising.

  • The main end-to-end evaluation result looks problematic in the following aspects

    • The test set is small (e.g., PrivacyQA has about 19 test queries, i.e., 10% of ~195).
    • The improvement looks incremental with small margin (i.e., +3.7% over the teacher comes from 3 wins and 2 ties across five datasets).
    • Some patterns/trends in the results weaken the major claim: (1) while main tables suggests that student beats the teacher only at nDCG@8, it loses at @16, @32 and @64; (2) when the labeled percentage is at 100% (i.e., no unlabeled), the student still beats teacher, which may suggesting the improvement does not come from the semi-supervised approach.
Questions for authors

I suggest the authors to include more discussion on the result corresponding to weakness 2 above, and strengthen the evaluation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer XY46

We thank the reviewer for their thoughtful feedback and questions.

W1: LLM-Generated Relevance Supervision

LLM-generated relevance supervision is a valid approach to addressing label scarcity. Section 2 discusses existing methods, including GPL, InPars, and Promptagator, and Figure 2 already includes InPars as an LLM-based synthetic-supervision baseline. RCD addresses the complementary problem of transferring specialization learned by a retriever adapter into its underlying dense retriever using unlabeled queries. LLM-generated relevance judgments could be used to train the adapter teacher or provide additional supervision for the student retriever in an extended setting. Such approaches introduce additional generation or relevance-scoring costs, whereas RCD uses the specialization already learned by the adapter without requiring these stages during student training. We will clarify how these approaches differ and how they could be used together in the revised manuscript.

W2: Evaluation Results and Interpretation

RCD's central objective is to transfer adapter-learned specialization into a standalone dense retriever using unlabeled queries. Surpassing the adapter teacher is an additional outcome whose magnitude varies across datasets and retrieval settings.

1. Test-set size: We acknowledge that PrivacyQA has a relatively small test set (~19 queries), making its performance estimates more sensitive to individual queries. However, our evaluation spans five benchmarks, including ContractNLI, a related legal-domain benchmark with a substantially larger test set, where RCD outperforms its teacher by 18.3% (Table 3). Thus, our findings do not depend on the small PrivacyQA test set alone.

2. Improvement over the teacher: At 50% supervision, RCD outperforms the IMRNN teacher on ContractNLI (+18.3%), ComLQ (+5.0%), and SciFact (+3.5%), achieving a 3.7% relative macro-average improvement (Table 3). To assess statistical significance, we conducted a two-sided paired randomization test using per-query nDCG@8 scores averaged across three seeds, with equal weighting across all five datasets. The macro-average improvement is statistically significant (p = 0.046). While RCD slightly underperforms the teacher on PrivacyQA (-0.7%) and NFCorpus (-0.6%), the results support our claim that RCD can outperform its teacher in the evaluated setting.

3. Performance across retrieval cutoffs: RCD outperforms the IMRNN teacher in macro-average nDCG@2 (0.463 vs. 0.443), nDCG@4 (0.476 vs. 0.460), and nDCG@8 (0.505 vs. 0.487), as reported in Appendix B (Table 10). At larger cutoffs, performance varies across datasets. Notably, RCD consistently outperforms the teacher on ComLQ across all six evaluated cutoffs, including @16 (0.843 vs. 0.815), @32 (0.847 vs. 0.822), and @64 (0.847 vs. 0.823). RCD also remains competitive or better on ContractNLI at @16 and @32, and PrivacyQA at @64. While the macro-average falls below the teacher at larger cutoffs, these results show that the gains extend beyond our primary metric, nDCG@8, and can persist at greater retrieval depths.

4. Results at 100% supervision: RCD can outperform its teacher even without additional unlabeled queries, as demonstrated by the Search-Adaptor results at 100% supervision (Table 11). This shows that transferring adapter-learned specialization into the student encoder can itself improve retrieval performance. Additional unlabeled queries provide further gains, as demonstrated by the controlled experiment with IMRNNs at 50% supervision (Table 5). RCD Labeled Only already surpasses the teacher on ContractNLI and ComLQ, while RCD All Queries further improves performance across all five datasets using the same frozen teacher and optimization budget. The macro-average improvement is statistically significant (p = 0.021, two-sided paired randomization test). Thus, the 100% supervision result is consistent with the independent benefit of unlabeled queries established in Table 5.

Dear Reviewer XY46,

Thank you for your feedback and questions. In our responses, we have clarified the relationship between RCD and LLM-generated relevance supervision, provided additional statistical evidence for improvements over the teacher, and discussed performance across retrieval cutoffs and the contribution of unlabeled queries.

We hope these clarifications have addressed your concerns. We would appreciate it if you could consider revisiting your assessment, and we would be happy to answer any remaining questions.

Reviewer_Yh77OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work tackles the issue of adapting dense retrievers to new domains, where not a lot of relevance labeled data exists. It proposes Representation Correction Distillation (RCD), a semi-supervised approach for adapter-trained retriever built on top of a base retriever (teacher; trained on small labeled data) to distill task-specific representation updates, rather than its final embeddings, to the base retriever (student; trained on large unlabeled queries using RCD). Experiments on ContractNLI, PrivacyQA, ComLQ, SciFact, and NFCorpus show improvements over the base retriever and other distillation methods.

Critical strengths
  • This work tackles a practical problem of domain adaptation of dense retriever with limited relevance labeled data.
  • The idea to distill representation updates rather than full representations given an adapter-trained student as the teacher is intuitive.
Critical weaknesses
  • The technical novelty appears to be somewhat incremental relative to existing representation-level retrieval distillation methods. Rather than distilling a teacher’s final representation, RCD distills representation updates relative to a shared base retriever, which can be seen as a relatively incremental modification of the distillation objective within an existing distillation framework.
  • While the claim is that RCD can outperform adapter-trained teacher by leveraging additional unlabeled data, the performance improvements over the teacher are modest and inconsistent across datasets. And since the paper does not provide statistical significance testing results, it is unclear if the gains are robust.
  • Additional analysis of what information is learned by the student retriever along with why it matters more in particular datasets would be beneficial.
Questions for authors
  • The experiments are conducted with adapter-trained teachers. Is there evidence that the results are robust to other forms of teacher training, e.g., sft from the same base retriever?
  • Which type of queries benefit the most from the propagation of adapter's learned specialization by unlabeled data, and does this explain the varying magnitude of performance improvements across the datasets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Yh77

We thank the reviewer for their thoughtful feedback and questions.

W1: Methodological Novelty

We address this concern in our shared author comment, "RCD: Novel Training Setting and Methodological Contributions." The key distinction lies in RCD's adapter-to-backbone distillation setting and correction-direction supervision, supported by the controlled ablation in Table 6 (0.505 vs. 0.411 macro nDCG@8).

W2: Statistical Robustness and Teacher Improvements

To assess statistical robustness, we conducted two-sided paired randomization tests at 50% supervision, testing the null hypothesis of no performance difference between methods. We used per-query nDCG@8 scores averaged across three seeds, randomly swapping paired method scores within each dataset to construct the null distribution. The test statistic was the macro-average performance difference across five datasets, with each dataset weighted equally. RCD All Queries achieves statistically significant macro-average improvements over RCD Labeled Only (p = 0.021) and the IMRNN teacher (p = 0.046). These results provide additional evidence for the contribution of unlabeled queries and RCD's ability to outperform its teacher on average. While the magnitude of improvement varies across datasets (Table 3), our claim is that RCD can outperform its teacher, without assuming consistent improvements in every setting.

W3: Analysis of Transferred Specialization

The paper already provides task-level analysis of the transferred specialization (Section 5.3, Table 4). On ComLQ, RCD outperforms the IMRNN teacher across all evaluated logical query structures, with a particularly notable improvement on 3-hop queries (0.818 vs. 0.731 nDCG@8), providing evidence of improved retrieval on compositionally complex queries. Section 5.2 also discusses dataset-level differences: NFCorpus offers limited teacher improvement to transfer, while PrivacyQA has only 78 additional unlabeled queries at 50% supervision. These observations provide plausible explanations for the varying gains.

A1: Generalization to Fully Fine-Tuned Teachers

RCD's formulation is not inherently restricted to adapter-trained teachers. As defined in Eqs. (1)-(2), representation corrections are measured relative to a shared pretrained retriever. Therefore, a teacher obtained through supervised fine-tuning of the same base retriever could, in principle, provide the corresponding correction signals, as discussed in Appendix D. Our focus on adapter-trained teachers is motivated by their effectiveness under limited relevance supervision. Table 2 shows that IMRNNs improves over pretrained E5 by 4.3%, 10.6%, and 17.3% at 10%, 30%, and 50% supervision, compared with 0.8%, 0.8%, and 7.1% for supervised E5 fine-tuning using the same label budgets. We empirically validate RCD with two adapter architectures and two retriever backbones (Tables 7 and 12). We will clarify this broader applicability in the revised manuscript, while keeping the empirical focus on adapter-trained teachers.

A2: Query Types Benefiting from Unlabeled Data

To address this question, we conducted an additional analysis on ComLQ at 50% supervision, comparing RCD trained with all queries against RCD trained with labeled queries only. The results below report nDCG@8 averaged across three seeds.

Query TypeTest QueriesAll QueriesLabeled OnlyΔ nDCG@8
Overall2900.8290.810+0.019
Logical structure
Negation1040.8090.795+0.014
Conjunction870.8010.782+0.019
Union410.8410.818+0.023
Projection1790.8370.818+0.019
Number of hops
1-hop150.9750.9750.000
2-hop2110.8210.801+0.020
3-hop640.8190.795+0.024

The largest gains occur on union queries (+0.023) and multi-hop queries (+0.020 for 2-hop and +0.024 for 3-hop), indicating that additional unlabeled queries particularly benefit compositional retrieval. This helps explain the stronger performance on ComLQ, where most queries require multiple hops. As discussed in Section 5.2, dataset-level gains also depend on the specialization available from the teacher and the amount of unlabeled data: NFCorpus offers limited teacher improvement to transfer, while PrivacyQA has only 78 additional unlabeled queries. Appendix C (Table 14) further provides qualitative retrieval examples illustrating how RCD improves the selection of relevant passages across datasets.

Dear Reviewer Yh77,

We appreciate your thoughtful feedback and the opportunity to address your concerns. In particular, we have provided additional statistical significance tests, a detailed analysis of which query types benefit from unlabeled data, and clarifications on RCD's novelty and applicability to other teacher-training settings.

We hope these additions have addressed your main concerns and would be grateful if you could revisit your evaluation. We are happy to discuss any remaining questions.

Reviewer_U9E4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a semi-supervised retrieval optimization method called RCD. They first train an adapter as a teacher using a small amount of labeled data, and then distill the teacher's representation correction direction and ranking information into the student model through unlabeled queries. Meanwhile, the authors' experiments on five datasets show that this method outperforms the original retrieval model and various distillation baselines.

Critical strengths
  1. The author utilizes a small amount of labeled data and a large amount of unlabeled queries to enhance retrieval performance, and no adapter is required during inference.

  2. They achieved good results under different supervision ratios, and ablation experiments also verified the effectiveness of unlabeled data and representation correction distillation.

Critical weaknesses
  1. However, the method primarily combines cosine alignment representing the direction of correction with traditional ranking distillation, lacking deeper theoretical innovation and explanation.

  2. The performance on some datasets is inferior to that of the teacher model, and individual distillation modules outperform the complete method, indicating that the applicable conditions of the method are not yet clear.

Questions for authors
  1. Does the performance improvement mainly come from removing the shared base representation, or from the cosine direction alignment itself?

  2. Why is RCD inferior to some teacher models under high supervision ratio?

  3. Should the weights for representation correction and ranking distillation be adjusted according to different tasks?

  4. Can you provide a comparison of the training time, GPU memory, and computational cost between the proposed method and traditional distillation methods, including any potential supplements and fine-tuning?

  5. When there is a distribution discrepancy between unlabeled queries and teacher training data, does the method still remain effective?

5 comments
AuthorsOpenReview ↗

Response to Reviewer U9E4 (Part 1)

We thank the reviewer for their thoughtful feedback and questions.

W1: Methodological Novelty

We address this concern in our shared author comment, "RCD: Novel Training Setting and Methodological Contributions." RCD introduces an adapter-to-backbone distillation setting and aligns adapter-induced correction directions to transfer learned specialization (Section 3.2). Table 6 supports this distinction: correction alignment outperforms final-representation alignment across all five datasets (0.505 vs. 0.411 macro nDCG@8), with cosine alignment and ranking distillation held fixed. We will clarify these contributions in the revised manuscript.

W2: Consistency of Improvements

RCD's central contribution is transferring adapter-learned specialization into a standalone dense retriever using unlabeled queries. Across all supervision budgets, RCD improves macro-average nDCG@8 over the pretrained E5 retriever with both IMRNN (+6.9%, +13.3%, +21.7%; Table 2) and Search-Adaptor (+0.9%, +8.3%, +5.6%; Table 7) teachers. While teacher-relative improvements vary across datasets and architectures, RCD surpasses the IMRNN teacher on three of five datasets at 50% supervision (Table 3). Furthermore, although individual objectives perform better on certain datasets, their combination achieves the highest macro-average nDCG@8 (0.505 vs. 0.463 for ranking-only and 0.470 for correction-only; Table 6). These results support effective transfer despite the variability discussed in Sections 5.2 and 5.7. We will revise the introduction and conclusion to clarify that outperforming the adapter teacher is a setting-dependent outcome rather than a general guarantee.

A1: Correction Directions vs. Complete Representations

The primary benefit comes from isolating the adapter-induced corrections from the shared base representation (Section 3.2). When these corrections are small relative to the base embeddings, complete teacher and student representations can remain highly similar even when their correction directions differ. Cosine alignment plays a complementary role: it transfers the directions of these corrections without requiring the student to reproduce their exact magnitudes (Eq. 3).

Importantly, from Eqs. (1)–(2), Euclidean matching of the corrections satisfies

∥Δqθ−ΔqT∥22=∥fθ(q)−qT∥22\lVert \Delta q_\theta-\Delta q_T\rVert_2^2 = \lVert f_\theta(q)-q_T\rVert_2^2

making it mathematically equivalent to full-representation L2 matching. Thus, direction-based alignment is important to obtaining a distinct correction supervision signal.

Table 6 isolates the representation target: with cosine alignment and ranking distillation held fixed, correction alignment outperforms final-representation alignment on all five datasets (0.505 vs. 0.411 macro nDCG@8). We will expand Section 3.2 to clarify the complementary roles of isolating adapter-induced corrections and aligning their directions, including the limits of matching complete representations.

A2: Sensitivity to Teacher Architecture and Quality

RCD's sensitivity to teacher architecture can be understood through the different forms of specialization learned by IMRNNs and Search-Adaptor. IMRNNs produces query-conditioned document representations, whereas Search-Adaptor applies a shared embedding transformation independently to queries and documents (Section 2). Since the RCD student produces a query-independent representation for each document, distilling IMRNNs requires learning a common representation from corrections that vary across query contexts. This may allow the student to capture recurring adaptation patterns across unlabeled queries, providing a possible explanation for its generalization beyond the teacher. With Search-Adaptor, the corrections are already query-independent, leaving a different form of specialization to transfer.

Table 7 supports the sensitivity to architecture and supervision level: RCD surpasses the Search-Adaptor teacher at 10% and 30% supervision. At 50%, it still improves over pretrained E5 by 5.6%, compared with 9.6% for the teacher and 9.2% for Margin-MSE. These results demonstrate that RCD transfers useful specialization across architectures, while its teacher-relative gains depend on the teacher and supervision setting. We will expand Section 5.7 to clarify these architectural differences and discuss their possible implications for distillation.

A3: Weighting of Distillation Objectives

RCD uses equal weights for correction alignment and ranking distillation (Eq. 5), with the same configuration maintained across datasets to avoid task-specific tuning. While individual objectives perform better on certain datasets, their combination achieves the highest macro-average nDCG@8 (0.505 vs. 0.463 for ranking-only and 0.470 for correction-only, Table 6). Task-specific weighting may offer further improvements, but the current results demonstrate the effectiveness of combining both signals under a fixed configuration.

AuthorsOpenReview ↗

Response to Reviewer U9E4 (Part 2)

A4: Computational Cost

Appendix A.3 (Table 9) reports student training times for RCD and conventional distillation methods on NFCorpus at 50% supervision using a single NVIDIA L40S GPU. To further address the computational-cost concern, we examined the existing logs and report the additional recorded costs, including shared preparation, adapter teacher training, and student training where available.

MethodTraining TimeRecorded Pipeline Subtotal
RCD1:52:271:53:40
EmbedDistill1:52:191:53:31
Margin-MSE1:29:451:30:57
Supervised E51:01:011:02:56

The recorded pipeline subtotals include shared frozen-E5 preparation, teacher preparation/training, candidate/student setup, and student training where recorded. RCD incurs only 9 seconds of additional recorded runtime compared with EmbedDistill, while achieving substantially better retrieval performance (Table 2). The IMRNN teacher, a two-layer neural network, itself requires only 4.82 seconds of training.

A5: Distribution Shift in Unlabeled Queries

RCD considers an in-domain semi-supervised setting where labeled and unlabeled queries come from the same target-domain distribution (Sections 3.1 and 4). This reflects a practical deployment scenario in which a subset of queries from an existing query log is annotated, while the remaining queries are available without relevance labels.

Under distribution shift, RCD's effectiveness depends on whether the teacher's learned corrections remain applicable to the shifted queries. If the new queries require similar relevance distinctions or reasoning patterns, the teacher's corrections may still provide useful supervision. However, if they require specialization the adapter has not learned, its corrections may become unreliable and negatively affect student training. Thus, the relevant factor is how well the teacher's specialization generalizes to the unlabeled-query distribution. Our experiments establish RCD's effectiveness under in-domain adaptation across five benchmarks. We will clarify this assumption and discuss the expected behavior under distribution shift in the revised manuscript, while leaving its empirical evaluation to future work.

Dear Reviewer U9E4,

Thank you again for your careful assessment and constructive feedback. We hope our responses have clarified RCD's methodological contributions, the motivation for correction-direction alignment, and the differences observed across teacher architectures. We have also provided additional details on computational costs and discussed the assumptions regarding distribution shifts.

We would appreciate it if you could reconsider your assessment in light of these clarifications. Please let us know if any concerns remain, and we would be happy to address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

RCD: Novel Training Setting and Methodological Contributions

We thank the reviewers for their feedback. Below, we clarify RCD's training setting and methodological contributions relative to prior work.

Novelty relative to prior work: RCD introduces an adapter-to-backbone distillation setting for label-efficient dense retrieval, where a trained retriever adapter acts as a teacher to improve its own pretrained dense retriever using unlabeled target-domain queries. The distinction from existing adapter-based approaches is summarized below (Section 2).

ApproachRole of the AdapterOutcome
IMRNNs / Search-AdaptorLearns task-specific specialization while the pretrained retriever remains frozenAdapter retained at inference
AdapterDistillationTransfers learned specialization between adapters while keeping the pretrained retriever frozenStudent adapter retained
RCD (Ours)Acts as a teacher to train its underlying dense retriever using unlabeled queriesUpdated standalone retriever; adapter removed

To our knowledge, prior work has not investigated this adapter-to-backbone distillation setting. It is particularly relevant to practical retrieval systems where relevance judgments are scarce but unlabeled query logs are readily available. RCD leverages these queries to propagate adapter-learned specialization into a standalone dense retriever without requiring additional relevance labels or retaining the adapter at inference.

Methodological distinction and evidence: When teacher and student share a pretrained retriever, their final embeddings contain a common base representation. If the adapter-induced corrections are small relative to this shared component, the final embeddings can remain highly similar even when their correction directions differ. RCD isolates these changes (Eqs. 1–2) and directly aligns their directions (Eq. 3), together with the teacher's ranking behavior (Eqs. 4–5). Table 6 isolates the representation target by holding cosine alignment and ranking distillation fixed: correction alignment outperforms final-representation alignment on all five datasets (0.505 vs. 0.411 macro nDCG@8).

Qualitative evidence: Appendix C (Table 14) further illustrates the resulting retrieval behavior. On ComLQ, the IMRNN teacher retrieves both ground-truth relevant passages at ranks 4 and 7, whereas RCD ranks them 1 and 2. On ContractNLI, neither E5 nor the IMRNN teacher retrieves any of the two relevant passages within the top eight, whereas RCD retrieves one at rank 1. Similarly, on PrivacyQA, E5 and IMRNN each retrieve only one of four relevant passages within the top eight, while RCD retrieves three at ranks 1, 6, and 8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.