- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies knowledge distillation for retrieval-augmented generation under imperfect retrieval. The authors observe that retrieval-conditioned teacher predictions are not uniformly preferable to the teacher's parametric predictions: retrieved evidence can correct parametric errors, yet it may also introduce new errors. To address this issue, they propose Mixture RAG-KD, which constructs a convex combination of the parametric and retrieval-conditioned teacher distributions to form the distillation target. Specifically, the paper derives sufficient conditions characterizing when this intermediate mixture achieves lower target risk than either endpoint. It also develops global and instance-wise coefficient selection strategies based on forward KL and squared L2 risks. It also develops global and instance-wise coefficient selection strategies based on forward KL and squared L2 risks. Experiments use Gemma-3-12B-IT as the teacher and Gemma-3-1B/4B-IT as students, and show improvements over standard and retrieval-conditioned distillation on open-domain QA and retrieval-robustness benchmarks. Additional experiments examine noisy training retrieval and SFT-initialized students.
Critical strengths
S1. A well-motivated and practically relevant distillation problem.
The paper identifies a concrete failure mode of directly distilling retrieval-conditioned teacher predictions. Figure 2 provides empirical evidence that retrieval both corrects and introduces teacher errors, with 30.3% of examples correct only under retrieval conditioning and 7.3% correct only without retrieval conditioning. This motivates studying both distributions as potential supervision sources rather than treating retrieved evidence as uniformly beneficial.
The contribution matters because the reliability of teacher supervision becomes particularly important when students are trained to operate with imperfect retrieval at inference time.
S2. A simple and analytically tractable formulation.
Mixture RAG-KD has a straightforward objective, and the coefficient selection procedures are computationally manageable. The forward-KL formulation leads to a convex one-dimensional optimization problem, while squared-L2 risk admits a closed-form solution.
The analysis is valuable because it makes the mixture selection problem explicit and provides a clear mathematical interpretation of the proposed coefficients, even though the resulting theorem is limited to target-level risk.
Critical weaknesses
W1. Limited methodological novelty beyond conventional distribution interpolation.
The core method proposed in Section 3.2 constructs the distillation target as a convex combination of the parametric and retrieval-conditioned teacher distributions, . While this formulation is intuitive and empirically effective, linear interpolation between predictive distributions is a well-established technique in knowledge distillation and ensemble learning. Furthermore, the global and instance-wise coefficient selection strategies introduced in Section 3.3 are derived from standard forward-KL and squared- risk minimization, yielding conventional convex optimization and closed-form solutions rather than a fundamentally new optimization mechanism. Although applying these techniques to retrieval-conditioned distillation addresses a practically relevant problem, the paper does not sufficiently establish how the proposed formulation advances beyond existing intermediate-target distillation methods, such as DistiLLM, TAID, and AMiD, which are already discussed in Section 2.1. Consequently, the main contribution appears to be a relatively straightforward adaptation of established distribution-mixing techniques to the RAG-KD setting, rather than a substantial methodological advance.
W2. The theoretical guarantee does not directly address capacity-limited distillation.
Theorem 3.1 in Section 3.2 establishes that an intermediate mixture can achieve lower target risk than either endpoint under the assumption that the student hypothesis class contains every mixture distribution, i.e., for all . However, this assumption effectively eliminates the approximation error arising from limited student capacity, which is precisely the challenge motivating knowledge distillation in this paper. Under this assumption, the optimal student can exactly reproduce the mixture target, making the improvement in student risk a direct consequence of the improvement in target risk. For realistic compact students, however, a mixture distribution with lower target risk may not necessarily be easier to approximate, and its theoretical advantage may not translate into better student performance. Consequently, although Theorem 3.1 is mathematically valid under its stated assumptions, its guarantee does not directly address the capacity-constrained setting central to the paper's motivation, limiting its explanatory value for the empirical improvements reported in Section 4.
W3. The evidence for the superiority of instance-wise optimization is mixed.
Although Mixture RAG-KD consistently improves over Vanilla RAG-KD, the overall performance gains remain relatively modest. As shown in Table 1, the best-performing variants improve the General QA average by only 2.0 and 2.2 percentage points for the 1B and 4B students, respectively, while the improvements on RAG-Bench are 2.6 and 1.7 percentage points. Moreover, the best results across different benchmarks are achieved under different experimental settings, rather than a single consistent configuration, raising concerns about the robustness and generalizability of the proposed method. Given the relatively straightforward nature of the proposed method, these modest improvements raise questions about the practical significance of the results and whether they are sufficient to support a substantial methodological contribution.
W4. The experimental scope limits conclusions about generality.
The evaluation is concentrated on a single teacher family, relatively small supervised training sets, and short-answer QA tasks.
Questions for authors
Q1. Could the authors include stronger baselines to better establish the effectiveness of the proposed mixture strategy?
The current experiments primarily compare Mixture RAG-KD against standard distillation and its two endpoint variants. However, these comparisons may not be sufficient to establish the benefits of the proposed risk-based coefficient optimization. In particular, it would be valuable to include simpler adaptive baselines, such as instance-wise teacher selection based on gold-answer likelihood or confidence-based weighting of the two teacher distributions. These comparisons would help clarify whether the observed improvements stem from the proposed mixture optimization or can be achieved through simpler strategies using the same supervision signals.
Q2. Can retrieval robustness be evaluated using paired corruption interventions on fixed questions?
The current noise-level datasets are constructed by selecting examples according to teacher correctness. Could the authors instead hold the training questions and gold answers fixed while varying the relevance or correctness of their retrieved passages?
This would help distinguish the effect of retrieval corruption from changes in the composition of the training set. Evidence that Mixture RAG-KD preserves its advantage under such controlled interventions would strengthen the robustness claim.
11 comments
[W2. Idealized theoretical assumption]
First, we clarify the role of the realizability assumption in our theoretical analysis. Realizability is widely used in theoretical studies of imitation learning [1,2,3] and knowledge distillation [4,5] to analyze the effect of supervision quality separately from the model’s approximation limitations. In Theorem 3.1 of the submitted manuscript, this assumption is required only to transfer the teacher-level analysis into a student-level result. The teacher-level analysis remains valid and meaningful without this assumption.
Theorem 3.1. (revised, target-level). If and , then
To make this distinction explicit, we will separately present the teacher-level result as Theorem 3.1 and the student-level result under the realizability assumption as Corollary 3.2 in the revised manuscript
Next, we conduct additional analysis to examine the relationship among the target risk , the student risk , and the distillation risk to examine this transfer empirically. We evaluated forward-KL distillation using three student sizes (270M, 1B, and 4B), and mixture weights ( 0, 0.2, 0.45, 0.6, 0.8, 1.0, Global KL). The experimental results are in Figure.
-
As shown in Figure, the distillation risk is non-zero but remains small and nearly invariant across different values of . Specifically, across all evaluated settings. This confirms that realizability does not hold exactly in practice, consistent with the reviewer's point. As expected, this approximation error decreases as student capacity increases (270M->1B->4B). More importantly, for a fixed student size, the approximation error remains small and comparable across different values of . This indicates that the student absorbs different mixture targets with similar fidelity. As a result, differences in student quality across are driven primarily by differences in target quality rather than by differences in distillation difficulty.
-
The student risk obtained using Global KL is lower than any including the endpoints and .
-
The table below reports correlation coefficients between the student risk and either the target risk or the distillation risk across . Target risk exhibits a strong positive correlation with student risk across all student sizes and all correlation measures. This supports the hypothesis that a lower target risk leads to a lower student risk , even under limited capacity. Moreover, for every student size, the student risk is considerably more strongly associated with the target risk than with the distillation risk. This suggests that, in our setting, improving the quality of the mixture target contributes more to student quality than variations in distillation fidelity.
[Table. Correlation of Student Risk with Target Risk and Distillation Risk Across Mixture Coefficients.]
| Student | Measure | Pearson | Spearman | Kendall's tau |
|---|---|---|---|---|
| 270M | vs | 0.94 | 0.93 | 0.81 |
| 270M | vs | 0.55 | 0.46 | 0.43 |
| 1B | vs | 0.99 | 0.96 | 0.90 |
| 1B | vs | 0.68 | 0.32 | 0.24 |
| 4B | vs | 0.98 | 1.00 | 1.00 |
| 4B | vs | -0.07 | 0.29 | 0.24 |
[1] Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms, ICML 2024
[2] Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning, NeurIPS 2024
[3] Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning, NeurIPS 2025
[4] Towards Understanding Knowledge Distillation, ICML 2019
[5] AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distribution, ICLR 2026
(Continued below)
(Continued)
Finally, we examine whether the benefits of mixture supervision persist as student capacity decreases. To investigate the practical impact of limited student capacity, we compare Vanilla RAG-KD and Mixture RAG-KD across three student sizes (4B, 1B, and 270M), using 10K NQ training examples and three independent training runs.
[Table. Effect of student capacity on distillation performance (NQ-10K).]
Note. Accuracy (%), averaged over three seeds. Mixture RAG-KD uses instance-wise coefficient optimization across all student sizes. Δ (pp) denotes improvement over Vanilla RAG-KD.
| Student | Method | General QA | RAG-Bench |
|---|---|---|---|
| 4B | Vanilla RAG-KD | 55.11 | 50.58 |
| Mixture RAG-KD | 57.67 | 53.20 | |
| Δ (pp) | +2.56 | +2.62 | |
| 1B | Vanilla RAG-KD | 48.29 | 45.25 |
| Mixture RAG-KD | 51.20 | 48.02 | |
| Δ (pp) | +2.91 | +2.77 | |
| 270M | Vanilla RAG-KD | 41.10 | 37.70 |
| Mixture RAG-KD | 42.50 | 39.60 | |
| Δ (pp) | +1.40 | +1.90 |
Mixture RAG-KD consistently outperforms Vanilla RAG-KD across all three student sizes, including the capacity-limited 270M student. Although absolute accuracy decreases as student capacity becomes smaller, the benefits of mixture supervision remain positive on both General QA and RAG-Bench.
We acknowledge that the improvements are smaller for the 270M student (+1.40/+1.90 pp) than for the 1B (+2.91/+2.77 pp) and 4B (+2.56/+2.62 pp) students, suggesting that severely limited student capacity may constrain the magnitude of the gains.
In conclusion, these additional experimental results provide empirical evidence that improvements in supervision target quality can transfer into better downstream student performance even when the student cannot perfectly realize the teacher's target distribution. While this does not establish a general theoretical guarantee without realizability, it supports the practical relevance of our target-level analysis under limited student capacity.
We thank the reviewer for their thoughtful and constructive feedback. We have carefully considered the concerns and provide additional experimental evidence and clarifications in our responses below.
We acknowledge that linear interpolation and convex optimization are well-established techniques. However, our contribution lies not in introducing a new interpolation operator, but in addressing the supervision challenges arising from imperfect retrieval.
First, our method differs from existing KD methods in both the problem it addresses and the distributions it combines.
TAID and AMiD primarily address teacher–student distribution mismatch through intermediate or assistant distributions, while DistiLLM improves knowledge transfer through skew-KL objectives.
In contrast, Mixture RAG-KD addresses the unreliability of retrieval-conditioned teacher supervision caused by imperfect retrieval. As illustrated in Figure 2 of the main paper, retrieved passages can either improve teacher predictions or mislead the teacher into producing incorrect answers.
To address this problem, rather than interpolating between teacher and student distributions, we combine the parametric and retrieval-conditioned distributions of the same teacher. Crucially, we determine instance-wise mixing coefficients by minimizing gold-reference risk during training, allowing the supervision target to adapt to whether retrieval is helpful or misleading.
Thus, our methodological contribution is a risk-based, retrieval-aware supervision mechanism rather than a new interpolation operator.
Second, we empirically demonstrate the complementarity of our approach with existing KD methods.
We evaluate TAID, DistiLLM (SKL/SRKL), and AMiD under three supervision settings: Original KD, Vanilla RAG-KD, and Mixture RAG-KD. Experiments use 10K NQ training examples, Gemma-3-12B-IT as the teacher, Gemma-3-1B-IT as the student, and three independent training runs.
[Table. Compatibility of Mixture RAG-KD with existing distillation methods.]
Note. Each cell reports General QA / RAG-Bench accuracy (%). Supervision denotes the teacher target distribution. Student denotes the query-only (q0) or retrieval-conditioned (qc) student distribution. Mixture RAG-KD uses instance-wise L2 coefficient optimization.
Importantly, comparing Vanilla RAG-KD with Mixture RAG-KD isolates the additional benefit of supervision target construction while preserving the same student conditioning and underlying distillation objective. Our mixture improves General QA across all four KD objectives (+0.68 to +2.20 pp) and RAG-Bench in three of four configurations.
These results demonstrate that retrieval-aware supervision provides complementary benefits without modifying existing KD objectives.
Third, we provide empirical evidence supporting the intended mechanism of instance-wise coefficient selection. We analyze the selected coefficients on the NQ-10K training set according to the correctness of the parametric teacher p0 and retrieval-conditioned teacher pc.
[Table. Instance-wise mixing coefficients conditioned on teacher correctness.]
When only the retrieval-conditioned teacher is correct, 83.9% of instances receive α>0.8, assigning greater weight to retrieval-conditioned supervision. Conversely, when only the parametric teacher is correct, 67.3% receive α≤0.2, emphasizing parametric supervision.
These findings provide evidence that risk-based instance-wise mixing adaptively adjusts teacher supervision according to whether retrieved information is helpful or misleading.
Overall, our contribution goes beyond simply applying distribution interpolation to RAG-KD: we formulate the supervision problem arising from imperfect retrieval and address it through risk-based instance-wise target construction. The additional experiments demonstrate that this approach complements existing KD methods and provides further performance gains without changing their underlying objectives. We will clarify these distinctions and incorporate the additional results into the revised manuscript.