acceptodds
Under review as a conference paper at ICLR 2027

Distill What You Retrieve: Candidate-Set On-Policy Distillation with Reliable Preference Correction

Abstract

Dense retrieval is essential for LLM agents to identify relevant tools and reusable skills from large candidate collections. Knowledge distillation (KD) has become a widely adopted paradigm for transferring the capabilities of powerful teachers to compact student retrievers. However, conventional offline distillation relies on static candidate pools that cannot adapt to the student’s evolving retrieval behavior. On-policy distillation (OPD) addresses this mismatch by collecting teacher supervision on student-induced candidates, yet its application to dense retrieval remains limited by two intrinsic issues: query-candidate-level supervision fails to capture the setwise competition that governs retrieval, and soft distribution matching does not explicitly guarantee that annotated positives outrank negatives. To address these issues, we propose a two-stage framework that combines candidate-set on-policy distillation with targeted preference optimization. In Stage 1, Candidate-Set On-Policy Distillation (CS-OPD) constructs query-specific Top-K candidate sets from the training student and transfers the teacher’s listwise preferences over the same candidates via KL divergence. In Stage 2, Reliable-Disagreement Direct Preference Optimization (RD-DPO) identifies reliable residual ranking errors through teacher-validated margin disagreement and corrects them using a score-level DPO objective tailored to dense retrieval. Extensive experiments on ToolRet and SkillRet demonstrate consistent improvements over strong retrieval and distillation baselines. Further analyses confirm that CS-OPD improves candidate-set distribution alignment, whereas RD-DPO resolves residual ordering inconsistencies, highlighting their complementary contributions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Dense Retrieval, Tool Retrieval, Knowledge Distillation, On-Policy Distillation, Preference Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1XRAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a two-stage framework consisting of CS-OPD and RD-DPO to transfer ranking knowledge from teacher model to a compact dense retriever. CS-OPD aligns the student's retrieval score distribution with the frozen teacher over student-induced Top-K candidate sets via reliability-weighted KL divergence to capture listwise competition, while RD-DPO mines teacher-validated positive-negative ranking inversions and applies a score-level DPO objective for targeted ranking correction.

Critical strengths
  • CS-OPD avoids the limitation of static offline candidate pools by letting the student dynamically generate is own candidate support during distillation.
  • RD-DPO successfully targets and resolves residual local ranking errors near the top of the list.
  • Experiments on ToolRet and SkillRet benchmarks show that the framework significantly outperforms offline distillation and contrastive baselines across multiple student scales.
Critical weaknesses
  • The two-stage framework requires careful hyperparameter tuning, which makes complex to implement than standard contrastive fine-tuning.
  • Training relies heavily on a powerful teacher model to compute candidate score distributions and validate preference directions, while experiments on computational and training efficiency are missing.
Questions for authors
  • What is the theoretical justification for mapping sequence-level preference optimization directly to score margins, and why doesn't direct margin regression achieve the same targeted correction?
  • Could the authors provide an efficiency analysis detailing the training cost breakdown between Stage 1 and Stage 2?
  • Does the two-stage framework generalize zero-shot to broader, traditional passage retrieval domains, or are CS-OPD and RD-DPO structurally specialized for tasks where candidates are highly categorical and densely clustered?
4 comments

We thank the reviewer for the constructive feedback and important questions.

Regarding W1: hyperparameter tuning and implementation complexity

We thank the reviewer for this helpful comment. Our experiments use largely shared hyperparameters across ToolRet and SkillRet. In Stage I, the only dataset-specific setting is the candidate-set size: Top-10 for ToolRet and Top-30 for SkillRet. All other Stage-I hyperparameters are shared, and Stage II uses the same configuration on both benchmarks. Learning rates and other general training settings were kept fixed across the two datasets rather than extensively tuned for each benchmark. Section 4.5 and Figure 4 also evaluate Stage-I sensitivity to candidate-set size on ToolRet, with NDCG@10 ranging from 50.80 to 51.22 across the tested values of K.

The framework introduces additional retrieval and preference-pair construction steps compared with InfoNCE fine-tuning, while achieving better retrieval performance, particularly at the first rank. On ToolRet, our 4B student outperforms the InfoNCE-trained Tool-Embed (Tool-DE)-4B baseline by 5.26 percentage points in NDCG@1 and 3.41 points in Recall@1.

Regarding Q1: theoretical justification and comparison with margin regression

We thank the reviewer for this important question. Our score-level formulation follows from defining a local retrieval policy over a positive–negative pair:

πθ(c∣q,{p,n})=exp⁡(sθ(q,c)/τs)exp⁡(sθ(q,p)/τs)+exp⁡(sθ(q,n)/τs),τs>0.\pi_{\theta}(c\mid q,\{p,n\}) = \frac{\exp(s_{\theta}(q,c)/\tau_{s})} {\exp(s_{\theta}(q,p)/\tau_{s})+\exp(s_{\theta}(q,n)/\tau_{s})}, \qquad \tau_{s}>0.

Its log-odds satisfy the exact identity

log⁡πθ(p∣q)πθ(n∣q)=sθ(q,p)−sθ(q,n)τs=Δθ.\log\frac{\pi_{\theta}(p\mid q)}{\pi_{\theta}(n\mid q)} = \frac{s_{\theta}(q,p)-s_{\theta}(q,n)}{\tau_{s}} =\Delta_{\theta}.

Applying this identity to both the student and reference policies yields the per-pair DPO loss:

LDPO=−log⁡σ[β(Δθ−Δref)],L_{\mathrm{DPO}} = -\log \sigma\left[\beta(\Delta_{\theta}-\Delta_{\mathrm{ref}})\right],

where the reference score margin is defined analogously using the frozen Stage-I checkpoint. Thus, score-level DPO is standard DPO applied to the score-induced local retrieval policy. This derivation is presented in Section 3.3 (Eqs. 12–15) of the main text.

Direct margin regression can also correct the selected inversions. MarginMSE minimizes

(mθ−mT)2,(m_{\theta}-m_{T})^{2},

where the unscaled student and teacher margins are

mθ=sθ(q,p)−sθ(q,n),mT=sϕ(q,p)−sϕ(q,n).m_{\theta}=s_{\theta}(q,p)-s_{\theta}(q,n),\qquad m_{T}=s_{\phi}(q,p)-s_{\phi}(q,n).

It therefore requires reproducing the teacher’s numerical margin, introducing a calibration requirement beyond recovering the correct ordering. In RD-DPO, the teacher validates the preference direction, while the frozen Stage-I reference defines the starting preference. DPO encourages reference-relative preference improvement without requiring reconstruction of the teacher’s margin magnitude. The targeted supervision comes from reliable-disagreement pair selection; the objective choice is evaluated separately below.

For the 0.6B student, we compared both objectives from the same Stage-I checkpoint, using identical reliable-disagreement pairs and matched optimizer-update budgets:

DatasetMethodN@10 ↑R@10 ↑C@10 ↑
ToolRetCS-OPD (Stage-I checkpoint)51.0761.1750.26
ToolRetMarginMSE51.1161.5250.48
ToolRetScore-level DPO51.4261.9351.00
SkillRetCS-OPD (Stage-I checkpoint)80.6587.2378.19
SkillRetMarginMSE81.6487.7978.94
SkillRetScore-level DPO82.0889.0581.03

Both objectives improve all three @10 metrics over the Stage-I checkpoint. DPO exceeds MarginMSE by 0.31/0.41/0.52 percentage points on ToolRet and 0.44/1.26/2.09 points on SkillRet. These comparisons support DPO as the correction objective in our evaluated setting.

Regarding W2 and Q2: teacher dependence and training efficiency

We thank the reviewer for this important point. We agree that teacher quality affects the reliability of both relevance distributions and preference directions. This dependence is shared by existing retrieval-distillation methods; some approaches, including the original Rabtriever, additionally rely on computationally expensive generative rerankers. Our framework directly transfers knowledge from an embedding teacher to an embedding student, without requiring a generative reranker or an additional predictor. Under the same frozen 8B embedding teacher, our framework outperforms the evaluated distillation baselines on ToolRet and SkillRet, including MarginMSE, Score-Only KD, and the adapted Rabtriever baseline.

Our teacher-size comparison in Appendix Tables 5–6 also evaluates a smaller teacher. For the 0.6B student, a 4B teacher yields NDCG@10 scores of 50.50 on ToolRet and 81.68 on SkillRet, exceeding the corresponding InfoNCE-trained baselines by 2.86 and 2.00 percentage points, respectively. Using the 8B teacher further improves these scores to 51.42 and 82.08. These results show that the framework improves upon contrastive fine-tuning with a smaller teacher while benefiting further from the larger teacher.

We separate training cost into Stage-I distillation, Stage-II data preparation, and DPO optimization. All reported times are elapsed wall-clock times measured using the eight NVIDIA A800 GPUs. For the 0.6B student, CS-OPD takes approximately 90 minutes, followed by 10 minutes of data selection and 30 minutes of DPO training, totaling 130 minutes. For the 4B student, Stage-I distillation takes approximately 330 minutes, data preparation takes 20 minutes, and DPO optimization takes 60 minutes, totaling 410 minutes.

Regarding Q3: zero-shot generalization

We thank the reviewer for raising this concern. CS-OPD constructs candidate support using student retrieval scores, while RD-DPO selects residual errors using relevance annotations and teacher–student score margins. Neither stage requires tool-category labels or explicit cluster assignments. To assess generalization beyond tool and skill catalogs, we evaluated the 0.6B student on selected retrieval tasks from BEIR and MTEB: HotpotQA for multi-hop question answering, SciFact for scientific evidence retrieval, and ArguAna for counterargument retrieval. We compared the same student before and after our two-stage training, without additional training on these datasets.

DatasetModelNDCG@10 ↑Recall@10 ↑MRR@10 ↑
HotpotQAOriginal student66.2867.8383.44
HotpotQACS-OPD + RD-DPO66.8568.5383.92
SciFactOriginal student70.2684.1066.39
SciFactCS-OPD + RD-DPO70.8484.5266.95
ArguAnaOriginal student67.4792.8959.87
ArguAnaCS-OPD + RD-DPO67.4992.9659.91

Our method improves all three reported metrics across HotpotQA, SciFact, and ArguAna. These results provide evidence that the two-stage framework can transfer beyond tool and skill catalogs to different text-retrieval tasks.

AuthorsOpenReview ↗

A gentle discussion-phase reminder

Dear Reviewer 1XRA,

Thank you again for your constructive feedback. As the discussion phase draws to a close, we would appreciate any further thoughts on whether our responses and additional results address your concerns.

Best regards,

The Authors

Dear Reviewer 1XRA,

As the discussion period is coming to a close, we would appreciate your feedback on whether our responses have addressed your concerns. Please let us know if any further clarification is needed.

Best regards, The Authors

Reviewer_Qkv4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies distillation for dense tool and skill retrieval. CS-OPD retrieves a candidate set with the current student and matches the teacher's softmax score distribution on that set using a reliability-weighted forward KL loss. RD-DPO then mines positive-negative inversions from the Stage I student, keeps pairs supported by the teacher and a student-margin filter, and applies a reference-relative pairwise logistic objective with a frozen Stage I reference. Experiments on ToolRet and SkillRet use 0.6B and 4B students distilled from task-trained embedding teachers. The full pipeline improves ranking and coverage metrics, while the stage-wise analysis attributes most of the gain to CS-OPD and a smaller additional gain to preference correction.

Critical strengths
  1. The pipeline addresses a practical retrieval problem and preserves efficient inference: only the student embedding model is needed after training. The improvements extend across two student sizes and two benchmarks, including SkillRet's disjoint training and evaluation skill pools. For example, Table 1 reports average NDCG@10 gains over InfoNCE of 3.09 points for the 0.6B student and 2.59 points for the 4B student. Improvements in completeness are also useful for queries that require multiple tools or skills.

  2. The paper makes a sensible distinction between matching a teacher distribution and correcting remaining ranking inversions. Table 2 includes continued CS-OPD and unfiltered DPO controls, rather than comparing the second stage only against stopping training. Appendix C gives a clear derivation of the score-level objective and explicitly acknowledges that the reference does not preserve the full retrieval distribution and that finite training need not correct every inversion. These qualifications make the theoretical discussion more useful.

Critical weaknesses
  1. The comparisons do not yet isolate the two central design claims. For Stage I, Appendix B.2's on-policy pair baseline uses each annotated positive and the highest-scoring non-positive, whereas CS-OPD uses the student's Top-K set and does not insert missing positives. The support, number of comparisons, and supervision exposure therefore change along with the loss granularity. This is not a clean test of listwise versus pairwise alignment on the same retrieved candidates. The large gain over Rabtriever is also difficult to interpret mechanistically: A.4 explicitly describes an adaptation from a generative reranker to an embedding teacher, and the adapted baseline is substantially below InfoNCE. For Stage II, standard DPO and RD-DPO use the same objective and reference and differ in pair selection. Their comparison supports filtering, but not the choice of DPO over a simpler pairwise ranking loss. Appendix C discusses RankNet, MarginMSE, and hinge loss, but I could not find results using those objectives on the same retained pairs. This matters because the proposed contribution is primarily the combination of candidate selection, distribution matching, and residual correction.

  2. The evidence for the smaller effects and the practical meaning of on-policy training is incomplete. On ToolRet, RD-DPO adds only 0.35 NDCG@10, 0.76 Recall@10, and 0.74 completeness points over CS-OPD. No repeated-run uncertainty is reported, and the stage-wise and filtering analyses are limited to the 0.6B ToolRet setting. The paper also does not specify how frequently the full-catalog candidate embeddings/index and retrieved sets are refreshed as the student changes, or the training cost of doing so. These details determine how closely the training candidates track the evolving student and how expensive the approach is. Table 3 omits the reliability-weight parameters m0 and tau_m from Eq. (6), and there is no separate ablation of that weighting. As a result, the reported gains are promising, but the attribution and reproducibility are not yet strong enough for me.

Questions for authors
  1. Can you compare listwise KL with a pairwise objective using the same student-retrieved Top-K support, teacher, query weighting, and matched update budget? Please also clarify how much of the gap to Rabtriever reflects the embedding-teacher adaptation rather than the proposed listwise objective.

  2. On exactly the same reliable-disagreement pairs, how does score-level DPO compare with RankNet/BPR or a hinge-ranking loss? Separately, what happens with teacher filtering alone, difficulty filtering alone, and a size-matched random subset of the mined pairs? These comparisons would distinguish the objective from the selection procedure. Appendix C.5 refers to controlled objective comparisons, but I could not locate their numerical results.

  3. Please specify the catalog/index refresh schedule, whether candidate embeddings are recomputed with the current student, total training time including retrieval and teacher scoring, and the values of m0 and tau_m. Is the reliability weighting also applied in the controlled Stage I baselines? An unweighted CS-OPD result would help identify its contribution.

  4. Are the Stage II improvements stable across random seeds and on SkillRet? Please also define the evaluation query set and denominator for the reliable-pair disagreement and MRR@20 results in Figure 1: are they measured on held-out queries and a fixed candidate/pair set before and after training? This would make the large MRR change easier to interpret alongside the smaller aggregate improvements in Table 2.

5 comments

We thank the reviewer for raising these important points.

Regarding W1 and Q1: matched-support distillation and the Rabtriever adaptation

We thank the reviewer for this comment. On ToolRet with the 0.6B student, all-pairs KL forms all unordered candidate pairs within each retrieved Top-K candidate set and aligns the teacher and student binary softmax distributions for each pair. Pairwise losses are averaged within each query before batch aggregation. Online variants share candidate-ID sequences, teacher, query reliability weights, batch size, and optimizer-update budget.

Candidate constructionObjectiveN@10 ↑R@10 ↑C@10 ↑
OfflineAll-pairs KL49.1859.5648.85
OfflineListwise KL48.8359.5248.49
OnlineAll-pairs KL50.7461.0449.88
OnlineListwise KL51.0761.1750.26

The all-pairs variant retains the student-retrieved Top-K candidate construction of CS-OPD and replaces listwise KL with pairwise KL within that support. Its competitive performance shows that pairwise supervision can also transfer the teacher’s relative relevance information when incorporated into our candidate-construction framework. Furthermore, online student retrieval improves all three metrics for both objectives over their corresponding offline variants. These results support student-dependent candidate construction as an effective component across both alignment objectives, while distinguishing its contribution from the choice of listwise normalization.

Regarding Rabtriever, we implement its core distillation architecture and training procedure: the MLP predictor is trained while the student backbone is frozen. For comparability, we replace its reranker teacher with the same frozen embedding teacher used by the other distillation methods, while matching student initialization, training data, and evaluation protocol. Rabtriever’s original training formulation does not construct student-retrieved Top-K candidate sets for distillation. The additional Top-K all-pairs KL experiment therefore evaluates a different configuration: it retains our student-dependent candidate construction while replacing listwise alignment with pairwise alignment within that set. Together with the listwise results, the improvement of online all-pairs KL over its offline counterpart supports the effectiveness of student-dependent candidate supervision across both objectives.

Regarding W1 and Q2: objective choice and pair-selection ablations

We thank the reviewer for raising this important question. For the 0.6B student on ToolRet, we compare objectives from the same Stage-I checkpoint using identical reliable-disagreement pairs, batch size, and optimizer-update budget.

ObjectiveN@10 ↑R@10 ↑C@10 ↑
CS-OPD (Stage-I baseline)51.0761.1750.26
RankNet/BPR51.1361.5450.61
Hinge ranking51.1761.4850.56
MarginMSE51.1161.5250.48
Score-level DPO51.4261.9351.00

DPO achieves the highest scores on all three @10 metrics, exceeding the best alternative for each metric by 0.25/0.39/0.39 points. These results support our objective choice in the evaluated residual-correction setting. We will include these numerical comparisons in the revised manuscript.

We separately evaluated pair selection while retaining the same DPO objective, frozen Stage-I reference, batch size, and total number of optimizer updates. The size-matched random subset is sampled from the initial mined-pair pool and contains the same number of pairs as the combined-filtering subset.

Pair selectionN@10 ↑R@10 ↑C@10 ↑
CS-OPD (Stage-I baseline)51.0761.1750.26
All mined pairs50.7360.9449.85
Teacher filtering only50.8260.9249.90
Difficulty filtering only51.2961.4950.68
Size-matched random subset51.0161.2750.26
Teacher + difficulty filtering51.4261.9351.00

Difficulty filtering alone improves on Stage I; adding teacher validation yields a further 0.13/0.44/0.32 points on N@10, R@10, and C@10. Teacher filtering alone underperforms Stage I, whereas the combined procedure leads all three metrics and outperforms size-matched random sampling. These ablations support the effectiveness and complementarity of teacher validation and difficulty filtering, beyond sample reduction alone.

Regarding W2 and Q3: refresh schedule, reliability weighting, and training cost

We thank the reviewer for raising this important concern. To balance computational efficiency with keeping candidate supervision aligned with the evolving student, we refresh the index every 10 optimizer updates. At each refresh, we recompute the entire catalog’s embeddings using the current student and regenerate the queries’ Top-K candidate sets.

Yes, the same reliability weighting is applied in the controlled Stage-I baselines. We use m0=0 and tau_m=0.1. To assess its contribution, we additionally evaluate CS-OPD on ToolRet using the 0.6B student, setting r(q)=1 while keeping the remaining settings unchanged:

ConfigurationN@10 ↑R@10 ↑C@10 ↑
With reliability weighting51.0761.1750.26
Without reliability weighting50.0460.0149.40

Removing reliability weighting reduces N@10, R@10, and C@10 by 1.03, 1.16, and 0.86 percentage points, respectively, supporting its contribution in this setting.

Compared with standard offline distillation, our Stage-I CS-OPD adds approximately 30 minutes for the 0.6B student, increasing training time from 60 to 90 minutes. For the 4B student, it adds approximately 90 minutes, from 240 to 330 minutes. Stage II takes a further 10 minutes for pair preparation and 30 minutes for DPO training for the 0.6B student, and 20 and 60 minutes, respectively, for the 4B student. The complete two-stage pipeline therefore takes approximately 130 minutes for 0.6B and 410 minutes for 4B. All reported times include the complete operations within each stage, including catalog encoding and refresh, retrieval, teacher scoring, and pair preparation.

Regarding W2 and Q4: seed stability, SkillRet, and Figure 1

We thank the reviewer for this important point. We repeated Stage II with three random seeds for the 0.6B student, starting from a fixed CS-OPD checkpoint on each benchmark. All three metrics improve in every run.

DatasetStage-I N/R/C@10Stage-II N@10Stage-II R@10Stage-II C@10
ToolRet51.07 / 61.17 / 50.2651.45 ± 0.0461.82 ± 0.1150.90 ± 0.09
SkillRet80.65 / 87.23 / 78.1982.06 ± 0.3488.98 ± 0.3881.03 ± 0.76

Results are means ± sample standard deviations. The mean improvements are 0.38/0.65/0.64 points on ToolRet and 1.41/1.75/2.84 points on SkillRet. These results support consistent Stage-II gains across the tested seeds and extend the evidence to SkillRet. This experiment evaluates Stage-II randomness conditional on fixed Stage-I checkpoints.

All results in Figure 1 are measured on the validation set. In the original trajectory analysis, pairs are constructed from each checkpoint’s own retrieved Top-20. We additionally verified the disagreement reduction using a fixed pair set constructed once from the Stage-I CS-OPD checkpoint. For each query, pp is a labeled positive and nn is a non-positive retrieved in the Stage-I Top-20. We retain pairs for which the frozen teacher ranks pp above nn, including both pairs that Stage I orders correctly and those it inverts. Labeled positives outside the initial Top-20 are also included. The disagreement rate is pooled over pairs: its numerator counts student inversions, and its denominator is the total number of teacher-supported pairs. Both checkpoints are evaluated by rescoring exactly the same pairs, with candidate IDs, teacher eligibility, and the denominator held fixed.

MRR@20 is evaluated on the same validation queries before and after Stage II, using full-catalog retrieval with each checkpoint. For each query, we compute the reciprocal rank of the first annotated positive within the Top-20, assigning zero when no positive is retrieved, and average over all evaluation queries. The denominator therefore includes all evaluated queries, including those without a Top-20 hit. The query set and catalog remain fixed, while each checkpoint produces its own retrieved ranking. We will make this evaluation protocol explicit in the revised text and figure caption.

Additional comment on W1 and Q2: objective choice and pair-selection ablations on SkillRet

We thank the reviewer for this suggestion. We further evaluate objective choice and pair selection on SkillRet with the 0.6B student, starting all runs from the same Stage-I CS-OPD checkpoint. All metrics are percentages.

Objective comparison. We evaluate RankNet, hinge ranking, MarginMSE, and score-level DPO on reliable-disagreement training data:

ObjectiveN@10 ↑R@10 ↑C@10 ↑
CS-OPD (Stage-I checkpoint)80.6587.2378.19
RankNet80.9987.4078.19
Hinge ranking81.0487.3278.19
MarginMSE81.6487.7978.94
Score-level DPO82.0889.0581.03

DPO achieves the highest scores across all three metrics, improving N@10/R@10/C@10 over the Stage-I checkpoint by 1.43/1.82/2.84 percentage points, and exceeding MarginMSE by 0.44/1.26/2.09 points, respectively. These results extend the evidence for our objective choice to the evaluated SkillRet residual-correction setting.

Pair-selection ablations. With the DPO objective, we compare the combined selection procedure against unfiltered training, either filter alone, and a size-matched random subset:

Pair selectionN@10 ↑R@10 ↑C@10 ↑
CS-OPD (Stage-I checkpoint)80.6587.2378.19
All mined pairs80.9987.4378.26
Teacher filtering only81.0287.4378.23
Difficulty filtering only81.1287.4178.28
Size-matched random subset80.9987.3878.17
Teacher + difficulty filtering82.0889.0581.03

Combined filtering achieves the highest scores across all three metrics. It improves N@10/R@10/C@10 over unfiltered DPO by 1.09/1.62/2.77 points, and over size-matched random selection by 1.09/1.68/2.87 points. Together with the ToolRet results, these ablations support the effectiveness of combining teacher validation with difficulty filtering, with benefits beyond reducing the number of training pairs.

Reviewer_LUn6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the knowledge distillation problem in dense retrieval tools for LLM agents, pointing out two shortcomings of existing on-policy distillation: query-candidate level point-by-point supervision cannot model listwise competition within the candidate set, and distributed approximate matching leaves behind local order inversions. And the authors propose a two-stage framework, CS-OPD and RD-DPO to solve this problem.

Critical strengths
  1. Solid experimental foundation. The strongest on-policy baseline, Rabtriever, is compared at the same scale and covers sufficient ablation; training updates are only performed on students, and the entire process is frozen.

  2. The conclusions have a good reproducibility basis. And solid theoretical foundation.

Critical weaknesses
  1. Novelty is incremental. CS-OPD is essentially student-induced Top-K + listwise KL distillation, while RD-DPO uses preference optimization for retrieval ranking. Further elaboration on the methodology's novelty would enhance the persuasiveness of paper.

  2. Writing and presentation of results can be improved. For example, Figure 2 seems to require more manual adjustments.

Questions for authors

The questions are the same as the weekness, thanks.

3 comments

We thank the reviewer for the thoughtful and constructive feedback.

Regarding W1 and Q1: methodological contribution of CS-OPD and RD-DPO

We agree that prior work has established the value of retrieval distillation and preference optimization. Our contribution is a two-stage supervision design addressing two connected problems: teacher supervision becoming misaligned with the student's evolving retrieval behavior, and residual positive-negative inversions after distribution matching.

CS-OPD: student-dependent candidate-set supervision. MarginMSE transfers teacher-provided positive-negative margins, Score-Only KD matches centered score vectors over candidate rows, and Rabtriever reconstructs a generative reranker's contextualized representations through a JEPA-style predictor with auxiliary logit distillation. CS-OPD directly transfers a frozen embedding teacher's relevance distribution over the evolving student's retrieved Top-K candidates, with query reliability weighting. The support therefore follows the student's changing retrieval confusions, while the student retains its embedding-retriever architecture without an additional predictor or a generative reranker.

RD-DPO: reliable correction of residual ranking errors. Stage II starts from an already distilled retriever, so we focus its update budget on selected residual errors. Mining inversions alone does not establish teacher consistency or prioritize difficulty. Teacher validation keeps pairs where teacher and annotation agree; difficulty filtering excludes extreme conflicts and weak inversions closest to zero. Their combination selects teacher-supported errors within the chosen difficulty range.

The objective measures correction relative to Stage I. The frozen Stage-I reference provides each pair's starting preference, and DPO optimizes improvement from that point. This fits residual correction: it uses teacher-validated preferences without requiring the student to reproduce the teacher’s numerical margins, as MarginMSE requires. Whereas RankNet/BPR directly optimizes the current score margin, DPO optimizes its change from the Stage-I reference. Compared with hinge ranking, its gradient decreases smoothly with reference-relative improvement rather than becoming zero once a fixed margin threshold is reached.

To evaluate the objective choice, we compared alternative losses from the same Stage-I checkpoint using identical reliable-disagreement pairs. On ToolRet with the 0.6B student, DPO achieves 51.42 NDCG@10, compared with 51.13/51.17/51.11 for RankNet/BPR, hinge ranking, and MarginMSE, respectively. On SkillRet, DPO exceeds MarginMSE by 0.44/1.26/2.09 points in NDCG@10/Recall@10/Completeness@10. These results support our objective choice in the evaluated residual-correction setting.

The framework also improves first-rank retrieval. On ToolRet, our 4B student outperforms the InfoNCE-trained Tool-Embed (Tool-DE)-4B baseline by 5.26 percentage points in NDCG@1 and 3.41 points in Recall@1. On the same benchmark, it also exceeds the frozen 8B teacher on all three @10 metrics: 55.25/65.86/53.96 versus 53.60/64.05/52.42 in NDCG/Recall/Completeness, respectively. These results demonstrate the practical benefit of the supervision design with a smaller student.

Regarding W2 and Q2: writing and presentation of results

We thank the reviewer for this helpful suggestion. We have refined Figure 2 with consistent typography, better module alignment, and clearer connections between the two stages. We have also improved the writing and layout of the manuscript. We will incorporate these writing and presentation improvements into the revised manuscript.

Additional comment on W1 and Q1: theoretical motivation and Stage-II efficiency

We thank the reviewer for this important question. Regarding novelty, our Stage-II contribution lies in combining teacher-validated, difficulty-filtered residual-error selection with preference optimization relative to the frozen Stage-I checkpoint. This design concentrates post-distillation supervision on reliable ranking errors exposed by the student’s own retrieval behavior. Its motivation is the trade-off between sample quality and computational cost in contrastive supervision. Chuang et al. (2020) show that false negatives can introduce sampling bias and impair representation learning. Xu et al. (2022) discuss the trade-off between negative-sample quantity, informativeness, and acquisition cost. Informative negatives can improve learning, but expanding candidate pools or repeatedly mining hard negatives increases computation, while overly difficult or unreliable comparisons may introduce harmful supervision. Together, these studies indicate that contrastive learning can involve a large candidate space, and effective training depends not only on the number of comparisons but also on their reliability and relevance to the model’s current learning needs. These observations motivate concentrating post-distillation training on reliable comparisons that address the student’s remaining errors.

For an already distilled student, we mine positive–negative inversions from its retrieval results, validate the preference directions using the teacher, and apply difficulty filtering to retain comparisons appropriate for correction at this stage. These training pairs provide explicit relative-ranking supervision that directly addresses unresolved retrieval errors. DPO then optimizes preference changes relative to the frozen Stage-I model, adjusting the training signal according to each pair’s reference-relative improvement. Reliable pair selection and reference-relative optimization together provide a residual-error correction mechanism that concentrates supervision within a limited training budget. Pair selection determines which errors receive supervision, while reference-relative optimization adjusts their correction signal.

We conducted an additional comparison on ToolRet with the 0.6B student, starting both methods from the same Stage-I checkpoint and using its Top-20 retrieval results. InfoNCE uses all retrieved non-positive candidates as negatives, while RD-DPO trains on the selected residual inversions. Both methods use the same hardware and train for one epoch over their respective training data.

MethodData preparation (min)Total Stage-II time (min) ↓N@10 ↑R@10 ↑C@10 ↑
CS-OPD (Stage-I checkpoint)——50.8061.1550.11
Continued InfoNCE—161.7051.4161.8451.00
RD-DPO8.6223.1051.4261.9250.96

Retrieval metrics are reported as percentages. Total time includes data preparation and optimization, including retrieval and teacher-based filtering where applicable.

Total Stage-II time decreases from 161.70 to 23.10 minutes, a 7.00× speedup for the measured pipeline. Compared with continued InfoNCE, RD-DPO improves N@10 and R@10 by 0.01 and 0.08 percentage points, respectively. These results support the performance–cost advantage of the complete residual-correction procedure in this setting, including its data-selection and optimization stages.

References

Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning, 2020. URL https://arxiv.org/abs/2007.00224.

Lanling Xu, Jianxun Lian, Wayne Xin Zhao, Ming Gong, Linjun Shou, Daxin Jiang, Xing Xie, and Ji-Rong Wen. Negative sampling for contrastive representation learning: A review, 2022. URL https://arxiv.org/abs/2206.00212.

Dear Reviewer LUn6,

As the discussion period is coming to a close, we would appreciate your feedback on whether our responses have addressed your concerns. Please let us know if any further clarification is needed.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.