Retriever Adapters as Teachers: Representation Correction Distillation for Dense Retrieval

Adapting dense retrievers to new target domains is severely constrained by the high cost of manual relevance labeling. While lightweight adapters can learn domain specialization from limited labeled data, they leave abundant unlabeled queries unused. To address this, we introduce Representation Correction Distillation (RCD), a semi-supervised framework that transfers adapter-learned specialization into a standalone dense retriever using both labeled and additional unlabeled target queries, without using relevance labels during student training. Rather than matching full representation spaces, RCD explicitly isolates and distills the task-specific representation updates induced by the adapter, preserving the base model's pre-existing structure while integrating new domain preferences via ranking distillation. Consequently, the distillation phase requires zero additional relevance labels. Across five benchmark datasets, RCD improves macro-average nDCG over the base retriever by up to 21.7%. When the teacher is trained on only 50% of the labeled queries, RCD uses additional unlabeled target queries to train a student that outperforms the teacher by 3.7% in macro-average nDCG.