acceptodds
Under review as a conference paper at ICLR 2027

Mixture RAG-KD: Controlling Retrieval-Induced Shifts for Robust Knowledge Distillation

Abstract

Knowledge distillation (KD) transfers the capabilities of large language models (LLMs) to smaller student models, but the limited parametric capacity of these students can constrain achievable performance. We argue that students should be distilled not only to imitate a teacher's parametric behavior, but also to effectively utilize external knowledge through retrieval-augmented generation (RAG). To this end, we formulate RAG-aware Knowledge Distillation, a framework that integrates retrieval into KD so that student models can effectively utilize information from retrieved contexts. We first identify a limitation of Vanilla RAG-KD, which directly distills the teacher's retrieval-conditioned distribution: under imperfect retrieval, its effectiveness is sensitive to retrieval quality. To address this issue, we propose Mixture RAG-KD, which combines the complementary strengths of the teacher's parametric and retrieval-conditioned distributions. We theoretically show why mixture supervision can be preferable to relying on either teacher alone by characterizing conditions under which an intermediate mixture achieves lower target risk than both endpoints. Building on this analysis, we develop global and instance-wise strategies for selecting the mixing coefficient by minimizing forward-KL or squared- target risk. Experiments across general QA and diverse noisy-retrieval settings show that Mixture RAG-KD consistently improves over Vanilla RAG-KD and remains more robust as training-time retrieval noise increases.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Distillation, Retrieval-Augmented Generation, Large Language Models, Retrieval Robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mZK7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies knowledge distillation for retrieval-augmented generation under imperfect retrieval. The authors observe that retrieval-conditioned teacher predictions are not uniformly preferable to the teacher's parametric predictions: retrieved evidence can correct parametric errors, yet it may also introduce new errors. To address this issue, they propose Mixture RAG-KD, which constructs a convex combination of the parametric and retrieval-conditioned teacher distributions to form the distillation target. Specifically, the paper derives sufficient conditions characterizing when this intermediate mixture achieves lower target risk than either endpoint. It also develops global and instance-wise coefficient selection strategies based on forward KL and squared L2 risks. It also develops global and instance-wise coefficient selection strategies based on forward KL and squared L2 risks. Experiments use Gemma-3-12B-IT as the teacher and Gemma-3-1B/4B-IT as students, and show improvements over standard and retrieval-conditioned distillation on open-domain QA and retrieval-robustness benchmarks. Additional experiments examine noisy training retrieval and SFT-initialized students.

Critical strengths

S1. A well-motivated and practically relevant distillation problem.

The paper identifies a concrete failure mode of directly distilling retrieval-conditioned teacher predictions. Figure 2 provides empirical evidence that retrieval both corrects and introduces teacher errors, with 30.3% of examples correct only under retrieval conditioning and 7.3% correct only without retrieval conditioning. This motivates studying both distributions as potential supervision sources rather than treating retrieved evidence as uniformly beneficial.

The contribution matters because the reliability of teacher supervision becomes particularly important when students are trained to operate with imperfect retrieval at inference time.

S2. A simple and analytically tractable formulation.

Mixture RAG-KD has a straightforward objective, and the coefficient selection procedures are computationally manageable. The forward-KL formulation leads to a convex one-dimensional optimization problem, while squared-L2 risk admits a closed-form solution.

The analysis is valuable because it makes the mixture selection problem explicit and provides a clear mathematical interpretation of the proposed coefficients, even though the resulting theorem is limited to target-level risk.

Critical weaknesses

W1. Limited methodological novelty beyond conventional distribution interpolation.

The core method proposed in Section 3.2 constructs the distillation target as a convex combination of the parametric and retrieval-conditioned teacher distributions, Tα=(1−α)p+αpcT_\alpha=(1-\alpha)p+\alpha p_c. While this formulation is intuitive and empirically effective, linear interpolation between predictive distributions is a well-established technique in knowledge distillation and ensemble learning. Furthermore, the global and instance-wise coefficient selection strategies introduced in Section 3.3 are derived from standard forward-KL and squared-L2L_2​ risk minimization, yielding conventional convex optimization and closed-form solutions rather than a fundamentally new optimization mechanism. Although applying these techniques to retrieval-conditioned distillation addresses a practically relevant problem, the paper does not sufficiently establish how the proposed formulation advances beyond existing intermediate-target distillation methods, such as DistiLLM, TAID, and AMiD, which are already discussed in Section 2.1. Consequently, the main contribution appears to be a relatively straightforward adaptation of established distribution-mixing techniques to the RAG-KD setting, rather than a substantial methodological advance.

W2. The theoretical guarantee does not directly address capacity-limited distillation.

Theorem 3.1 in Section 3.2 establishes that an intermediate mixture can achieve lower target risk than either endpoint under the assumption that the student hypothesis class contains every mixture distribution, i.e., Tα∈QT_\alpha \in \mathcal Q for all α∈[0,1]\alpha \in [0,1]. However, this assumption effectively eliminates the approximation error arising from limited student capacity, which is precisely the challenge motivating knowledge distillation in this paper. Under this assumption, the optimal student can exactly reproduce the mixture target, making the improvement in student risk a direct consequence of the improvement in target risk. For realistic compact students, however, a mixture distribution with lower target risk may not necessarily be easier to approximate, and its theoretical advantage may not translate into better student performance. Consequently, although Theorem 3.1 is mathematically valid under its stated assumptions, its guarantee does not directly address the capacity-constrained setting central to the paper's motivation, limiting its explanatory value for the empirical improvements reported in Section 4.

W3. The evidence for the superiority of instance-wise optimization is mixed.

Although Mixture RAG-KD consistently improves over Vanilla RAG-KD, the overall performance gains remain relatively modest. As shown in Table 1, the best-performing variants improve the General QA average by only 2.0 and 2.2 percentage points for the 1B and 4B students, respectively, while the improvements on RAG-Bench are 2.6 and 1.7 percentage points. Moreover, the best results across different benchmarks are achieved under different experimental settings, rather than a single consistent configuration, raising concerns about the robustness and generalizability of the proposed method. Given the relatively straightforward nature of the proposed method, these modest improvements raise questions about the practical significance of the results and whether they are sufficient to support a substantial methodological contribution.

W4. The experimental scope limits conclusions about generality.

The evaluation is concentrated on a single teacher family, relatively small supervised training sets, and short-answer QA tasks.

Questions for authors

Q1. Could the authors include stronger baselines to better establish the effectiveness of the proposed mixture strategy?

The current experiments primarily compare Mixture RAG-KD against standard distillation and its two endpoint variants. However, these comparisons may not be sufficient to establish the benefits of the proposed risk-based coefficient optimization. In particular, it would be valuable to include simpler adaptive baselines, such as instance-wise teacher selection based on gold-answer likelihood or confidence-based weighting of the two teacher distributions. These comparisons would help clarify whether the observed improvements stem from the proposed mixture optimization or can be achieved through simpler strategies using the same supervision signals.

Q2. Can retrieval robustness be evaluated using paired corruption interventions on fixed questions?

The current noise-level datasets are constructed by selecting examples according to teacher correctness. Could the authors instead hold the training questions and gold answers fixed while varying the relevance or correctness of their retrieved passages?

This would help distinguish the effect of retrieval corruption from changes in the composition of the training set. Evidence that Mixture RAG-KD preserves its advantage under such controlled interventions would strengthen the robustness claim.

11 comments

We thank the reviewer for their thoughtful and constructive feedback. We have carefully considered the concerns and provide additional experimental evidence and clarifications in our responses below.


[W1. Methodological novelty beyond conventional distribution interpolation]

We acknowledge that linear interpolation and convex optimization are well-established techniques. However, our contribution lies not in introducing a new interpolation operator, but in addressing the supervision challenges arising from imperfect retrieval.

First, our method differs from existing KD methods in both the problem it addresses and the distributions it combines.

TAID and AMiD primarily address teacher–student distribution mismatch through intermediate or assistant distributions, while DistiLLM improves knowledge transfer through skew-KL objectives.

In contrast, Mixture RAG-KD addresses the unreliability of retrieval-conditioned teacher supervision caused by imperfect retrieval. As illustrated in Figure 2 of the main paper, retrieved passages can either improve teacher predictions or mislead the teacher into producing incorrect answers.

To address this problem, rather than interpolating between teacher and student distributions, we combine the parametric and retrieval-conditioned distributions of the same teacher. Crucially, we determine instance-wise mixing coefficients by minimizing gold-reference risk during training, allowing the supervision target to adapt to whether retrieval is helpful or misleading.

Thus, our methodological contribution is a risk-based, retrieval-aware supervision mechanism rather than a new interpolation operator.

Second, we empirically demonstrate the complementarity of our approach with existing KD methods.

We evaluate TAID, DistiLLM (SKL/SRKL), and AMiD under three supervision settings: Original KD, Vanilla RAG-KD, and Mixture RAG-KD. Experiments use 10K NQ training examples, Gemma-3-12B-IT as the teacher, Gemma-3-1B-IT as the student, and three independent training runs.

[Table. Compatibility of Mixture RAG-KD with existing distillation methods.]

Note. Each cell reports General QA / RAG-Bench accuracy (%). Supervision denotes the teacher target distribution. Student denotes the query-only (q0q_0) or retrieval-conditioned (qcq_c) student distribution. Mixture RAG-KD uses instance-wise L2L_2 coefficient optimization.

SettingSupervisionStudentTAIDDistiLLM (SKL)DistiLLM (SRKL)AMiD
Original KDp0p_0q0q_044.69 / 44.9044.38 / 44.5044.80 / 44.9344.71 / 44.37
+ Vanilla RAG-KDpcp_cqcq_c48.31 / 45.5049.38 / 46.6849.76 / 44.5749.11 / 45.98
+ Mixture RAG-KDTαT_{\alpha}qcq_c50.51 / 47.7750.47 / 46.1550.44 / 46.3050.04 / 47.58

Importantly, comparing Vanilla RAG-KD with Mixture RAG-KD isolates the additional benefit of supervision target construction while preserving the same student conditioning and underlying distillation objective. Our mixture improves General QA across all four KD objectives (+0.68 to +2.20 pp) and RAG-Bench in three of four configurations.

These results demonstrate that retrieval-aware supervision provides complementary benefits without modifying existing KD objectives.

Third, we provide empirical evidence supporting the intended mechanism of instance-wise coefficient selection. We analyze the selected coefficients on the NQ-10K training set according to the correctness of the parametric teacher p0p_0 and retrieval-conditioned teacher pcp_c.

[Table. Instance-wise mixing coefficients conditioned on teacher correctness.]

Parametric Teacher p0p_0RAG Teacher pcp_cSelected CoefficientProportion
IncorrectCorrectα>0.8\alpha>0.883.9%
CorrectIncorrectα≤0.2\alpha\leq0.267.3%

When only the retrieval-conditioned teacher is correct, 83.9% of instances receive α>0.8\alpha > 0.8, assigning greater weight to retrieval-conditioned supervision. Conversely, when only the parametric teacher is correct, 67.3% receive α≤0.2\alpha \leq 0.2, emphasizing parametric supervision.

These findings provide evidence that risk-based instance-wise mixing adaptively adjusts teacher supervision according to whether retrieved information is helpful or misleading.

Overall, our contribution goes beyond simply applying distribution interpolation to RAG-KD: we formulate the supervision problem arising from imperfect retrieval and address it through risk-based instance-wise target construction. The additional experiments demonstrate that this approach complements existing KD methods and provides further performance gains without changing their underlying objectives. We will clarify these distinctions and incorporate the additional results into the revised manuscript.

[W2. Idealized theoretical assumption]

First, we clarify the role of the realizability assumption in our theoretical analysis. Realizability is widely used in theoretical studies of imitation learning [1,2,3] and knowledge distillation [4,5] to analyze the effect of supervision quality separately from the model’s approximation limitations. In Theorem 3.1 of the submitted manuscript, this assumption is required only to transfer the teacher-level analysis into a student-level result. The teacher-level analysis remains valid and meaningful without this assumption.

Theorem 3.1. (revised, target-level). If Rˉ+′(0)<0\bar{\mathcal{R}}'_+(0) < 0 and Rˉ−′(1)>0\bar{\mathcal{R}}'_-(1) > 0, then R(p∗,Tα∗)<min⁡R(p∗,p),R(p∗,pc)\mathcal{R}(p^{\ast}, T_{\alpha^{\ast}}) < \min\\{\mathcal{R}(p^{\ast}, p), \mathcal{R}(p^{\ast}, p_c)\\}

To make this distinction explicit, we will separately present the teacher-level result as Theorem 3.1 and the student-level result under the realizability assumption as Corollary 3.2 in the revised manuscript

Next, we conduct additional analysis to examine the relationship among the target risk R(p∗,Tα)\mathcal{R}(p^{\ast}, T_{\alpha}), the student risk R(p∗,qα∗)\mathcal{R}(p^{\ast}, q_{\alpha}^\ast), and the distillation risk R(Tα,qα∗)\mathcal{R}(T_{\alpha}, q_{\alpha}^{\ast}) to examine this transfer empirically. We evaluated forward-KL distillation using three student sizes (270M, 1B, and 4B), and mixture weights (α=\alpha = 0, 0.2, 0.45, 0.6, 0.8, 1.0, Global KL). The experimental results are in Figure.

  • As shown in Figure, the distillation risk is non-zero but remains small and nearly invariant across different values of α\alpha. Specifically, R(Tα,qα∗)>0\mathcal{R}(T_\alpha, q_\alpha^\ast) > 0 across all evaluated settings. This confirms that realizability does not hold exactly in practice, consistent with the reviewer's point. As expected, this approximation error decreases as student capacity increases (270M->1B->4B). More importantly, for a fixed student size, the approximation error remains small and comparable across different values of α\alpha. This indicates that the student absorbs different mixture targets with similar fidelity. As a result, differences in student quality across are driven primarily by differences in target quality rather than by differences in distillation difficulty.

  • The student risk R(p∗,qα∗)\mathcal{R}(p^{\ast}, q_{\alpha}^{\ast}) obtained using Global KL is lower than any α\alpha including the endpoints R(p∗,q0∗)\mathcal{R}(p^{\ast}, q_{0}^{\ast}) and R(p∗,q1∗)\mathcal{R}(p^{\ast}, q_{1}^{\ast}).

  • The table below reports correlation coefficients between the student risk and either the target risk or the distillation risk across α\alpha. Target risk exhibits a strong positive correlation with student risk across all student sizes and all correlation measures. This supports the hypothesis that a lower target risk R(p∗,Tα)\mathcal{R}(p^{\ast}, T_{\alpha}) leads to a lower student risk R(p∗,qα∗)\mathcal{R}(p^{\ast}, q^{\ast}_{\alpha}), even under limited capacity. Moreover, for every student size, the student risk is considerably more strongly associated with the target risk than with the distillation risk. This suggests that, in our setting, improving the quality of the mixture target contributes more to student quality than variations in distillation fidelity.

[Table. Correlation of Student Risk with Target Risk and Distillation Risk Across Mixture Coefficients.]

StudentMeasurePearsonSpearmanKendall's tau
270MR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(p∗,Tα)R(p^{\ast}, T_{\alpha})0.940.930.81
270MR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(Tα,qα∗)R(T_{\alpha}, q_{\alpha}^{\ast})0.550.460.43
1BR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(p∗,Tα)R(p^{\ast}, T_{\alpha})0.990.960.90
1BR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(Tα,qα∗)R(T_{\alpha}, q_{\alpha}^{\ast})0.680.320.24
4BR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(p∗,Tα)R(p^{\ast}, T_{\alpha})0.981.001.00
4BR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(Tα,qα∗)R(T_{\alpha}, q_{\alpha}^{\ast})-0.070.290.24

[1] Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms, ICML 2024

[2] Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning, NeurIPS 2024

[3] Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning, NeurIPS 2025

[4] Towards Understanding Knowledge Distillation, ICML 2019

[5] AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distribution, ICLR 2026

(Continued below)

(Continued)

Finally, we examine whether the benefits of mixture supervision persist as student capacity decreases. To investigate the practical impact of limited student capacity, we compare Vanilla RAG-KD and Mixture RAG-KD across three student sizes (4B, 1B, and 270M), using 10K NQ training examples and three independent training runs.

[Table. Effect of student capacity on distillation performance (NQ-10K).]

Note. Accuracy (%), averaged over three seeds. Mixture RAG-KD uses instance-wise L2L_2 coefficient optimization across all student sizes. Δ (pp) denotes improvement over Vanilla RAG-KD.

StudentMethodGeneral QARAG-Bench
4BVanilla RAG-KD55.1150.58
Mixture RAG-KD57.6753.20
Δ (pp)+2.56+2.62
1BVanilla RAG-KD48.2945.25
Mixture RAG-KD51.2048.02
Δ (pp)+2.91+2.77
270MVanilla RAG-KD41.1037.70
Mixture RAG-KD42.5039.60
Δ (pp)+1.40+1.90

Mixture RAG-KD consistently outperforms Vanilla RAG-KD across all three student sizes, including the capacity-limited 270M student. Although absolute accuracy decreases as student capacity becomes smaller, the benefits of mixture supervision remain positive on both General QA and RAG-Bench.

We acknowledge that the improvements are smaller for the 270M student (+1.40/+1.90 pp) than for the 1B (+2.91/+2.77 pp) and 4B (+2.56/+2.62 pp) students, suggesting that severely limited student capacity may constrain the magnitude of the gains.

In conclusion, these additional experimental results provide empirical evidence that improvements in supervision target quality can transfer into better downstream student performance even when the student cannot perfectly realize the teacher's target distribution. While this does not establish a general theoretical guarantee without realizability, it supports the practical relevance of our target-level analysis under limited student capacity.

Reviewer_y1YCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mixture RAG-KD, which combines the same teacher’s predictive distributions with and without retrieved context as a distillation target. The mixing coefficient is selected through risk minimization using gold answers. The problem is practically relevant, but the current evidence does not sufficiently establish the contribution of the mixed distribution, the benefits learned by the student, or the statistical reliability of the improvements. Overall, the paper presents a reasonable and practical idea, but direct validation of the mixed teacher, student-level evidence of benefit transfer, and statistical support for the incremental gains remain insufficient.

Critical strengths
  1. The motivation is clear: retrieval can mislead the teacher, and directly distilling retrieval-conditioned predictions can propagate these errors to the student.
  2. The method is simple to implement, with additional computation primarily incurred during training. Positive results are reported across different student sizes and SFT initialization settings.
Critical weaknesses

1. The method is simple, and evidence for its necessity and effectiveness is insufficient. The core operation is a convex combination of two output distributions from the same teacher, with the mixing weight determined using gold answers. I have three concerns:

  • First, if the mixed teacher distribution provides better supervision, a direct validation would be to evaluate whether the mixture itself improves answer accuracy before training the student. The paper does not report this experiment. Such an evaluation should distinguish gold-dependent coefficient selection from a deployable inference setting.
  • Second, there is limited direct evidence that the student inherits the intended advantages of mixture supervision. Even if mixing improves the teacher distribution, can these benefits be transferred through distillation? How does matching the mixture teach the student both to use helpful evidence and to resist misleading evidence? Aggregate accuracy gains do not directly establish these behaviors.
  • Third, the results do not convincingly establish the value of risk-optimal, instance-wise mixing over a simple fixed mixture. In Table 1, fixed mixing achieves 46.3% on RAG-Bench for the 1B student, outperforming all adaptive variants; for the 4B student, it achieves 52.7%, matching or exceeding them. In Figure 7, instance-wise FKL/L2 improves over the best fixed coefficient by only 0.3/0.2 percentage points, without sufficient statistical evidence that these differences are reliable.

2. The theory relies on a strong assumption and does not explain distillation into capacity-limited students. The theorem assumes that the student hypothesis class can represent every mixture target TαT_\alpha, whereas the paper is motivated by limited student capacity. This assumption bypasses a central difficulty of practical distillation. Under the stated endpoint derivative conditions, the analysis establishes that an intermediate mixture is closer to the ideal target than either endpoint. It does not adequately explain why a better target leads to better task performance after approximation by a capacity-limited student, consistent with my second concern above.

3. The gains are modest, and statistical and mechanistic evidence remains limited. Although mixture distillation improves over Vanilla RAG-KD, the additional gains from instance-wise coefficient selection are small and inconsistent. In Table 1, instance-wise FKL improves average QA accuracy over fixed mixing by only 0.6/0.2 percentage points for the 1B/4B students, while slightly reducing average RAG-Bench accuracy in both cases. Confidence intervals or paired significance tests are needed to assess these differences.

4. Experimental coverage and baselines are insufficient to establish general applicability. Experiments focus on one model family, a single 12B teacher, two student sizes, 1,000 NQ training examples, and short-answer open-domain QA. This leaves applicability to larger training datasets, other model families, long-form generation, and multi-step reasoning unclear. Comparisons with established robust RAG training methods would also help establish the method’s value beyond the internal distillation baselines.

Questions for authors

please see the weakness

10 comments

We thank the reviewer for their thoughtful and constructive feedback. We have carefully considered the concerns and provide additional experimental evidence and clarifications in our responses below.

[W1a. Direct validation of mixture supervision]

To address this concern, we evaluated the mixed teacher distribution before student distillation, measuring both gold-reference KL risk and generated-answer accuracy. We further analyzed how mixing affects teacher predictions and the distribution of instance-wise mixing coefficients on NQ 10K.

First, the mixture distribution improves generated-answer accuracy while reducing gold-reference risk. We directly evaluated the teacher distributions on NQ 10K and three noisy retrieval datasets with noise ratios of 10%, 30%, and 50%. Table 1 reports generated-answer accuracy and KL risk, measured as KL(p∗∣Tα)\mathrm{KL}(p^{\ast}\\|T_\alpha). The mixing coefficient of TαT_\alpha is optimized for each instance by minimizing the gold-reference KL risk.

[Table 1. Direct evaluation of teacher distributions before student distillation.]

Note. Each entry reports answer accuracy (%, ↑) / gold-reference KL risk (↓). The last row shows the changes of TαT_\alpha relative to pcp_c.

Teacher distributionNQ 10KNoisy 10%Noisy 30%Noisy 50%
Parametric (p0p_0)37.0 / 1.8437.9 / 1.8147.9 / 1.6061.5 / 1.45
Retrieval-conditioned (pcp_c)60.1 / 1.3355.7 / 1.4245.0 / 1.4936.8 / 1.57
Mixture (TαT_\alpha)64.4 / 0.9562.5 / 0.9764.8 / 0.9070.6 / 0.83
Δ (Tα−pcT_\alpha-p_c)+4.3pp / −0.38+6.8pp / −0.45+19.8pp / −0.60+33.8pp / −0.74

As shown in Table 1, TαT_\alpha consistently achieves higher generated-answer accuracy and lower gold-reference KL risk than both teacher distributions across all datasets. This provides direct empirical validation that the mixture improves not only the gold-reference training objective but also the accuracy of generated teacher answers. Moreover, these improvements are consistently observed across noisy retrieval datasets, with larger accuracy gains and KL risk reductions relative to pcp_c at higher noise ratios.

Second, we further analyze where these accuracy improvements originate. We partitioned 10K NQ training examples into four groups based on the correctness of the retrieval-conditioned teacher (pcp_c) and the parametric teacher (p0p_0). To further understand how the mixture adjusts the relative contributions of the two teacher distributions, we analyzed the instance-wise mixing coefficients across three ranges: α≤0.2\alpha\leq0.2, 0.2<α≤0.80.2 < \alpha\leq0.8, and α>0.8\alpha > 0.8.

[Table 2. Teacher prediction accuracy and instance-wise mixing coefficient analysis on NQ 10K.]

Note. Each entry reports answer accuracy (%, ↑). The last column shows the percentage of examples in each mixing coefficient range.

Teacher correctness# Examplesp0p_0 Acc.pcp_c Acc.TαT_\alpha Acc.α\alpha distribution (≤0.2 / 0.2–0.8 / >0.8)
pcp_c ✓ / p0p_0 ✓3,013100.0100.099.329.9 / 7.4 / 62.7
pcp_c ✓ / p0p_0 ✗3,0000.0100.094.32.3 / 13.8 / 83.9
pcp_c ✗ / p0p_0 ✓691100.00.079.367.3 / 22.1 / 10.6
pcp_c ✗ / p0p_0 ✗3,2960.00.02.228.3 / 41.4 / 30.3
Overall10,00037.060.164.423.7 / 21.6 / 54.8

The analysis shows that TαT_\alpha recovers retrieval-associated errors while largely preserving correct answers obtained from helpful retrieved evidence. Specifically, when pcp_c is incorrect but p0p_0 is correct, TαT_\alpha recovers 79.3% of the correct answers. In this group, 67.3% of instances assign α≤0.2\alpha\leq0.2, placing greater weight on the parametric teacher distribution p0p_0. Conversely, when pcp_c is correct but p0p_0 is incorrect, TαT_\alpha preserves 94.3% of the correct answers. In this group, 83.9% of instances assign α>0.8\alpha > 0.8, placing greater weight on the retrieval-conditioned teacher distribution pcp_c.

These findings indicate that instance-wise risk minimization adjusts the relative contributions of the two teacher distributions to construct supervision targets that mitigate retrieval-associated errors while preserving the benefits of helpful retrieved evidence.

These results directly demonstrate that Mixture RAG-KD leverages the complementary knowledge of the two teachers to mitigate errors caused by misleading retrieved evidence while preserving the benefits of helpful retrieval, thereby improving teacher answer accuracy before student distillation.

[W1b. Transfer of mixture supervision to students]

To address the concern that the intended benefits of mixture supervision may not transfer to the student through distillation, we directly evaluate whether students trained with Mixture RAG-KD can leverage helpful retrieved evidence while resisting misleading evidence, rather than relying solely on aggregate accuracy.

First, Mixture RAG-KD provides adaptive supervision according to retrieval utility. We distinguish two cases based on teacher correctness: useful retrieval, where the retrieval-conditioned teacher pcp_c is correct but the parametric teacher p0p_0 is incorrect, and harmful retrieval, where p0p_0 is correct but pcp_c is incorrect. As demonstrated in our previous teacher-level analysis, instance-wise risk minimization assigns greater weight to pcp_c when retrieval is useful and to p0p_0 when retrieval is harmful. Specifically, 83.9% of useful cases assign α>0.8\alpha > 0.8, whereas 67.3% of harmful cases assign α≤0.2\alpha\leq0.2. This allows the mixture to construct supervision targets that emphasize helpful retrieval-conditioned predictions while reducing the influence of misleading ones.

Second, we directly examine whether these complementary properties are transferred to the student. We evaluate students trained with Parametric KD, Vanilla RAG-KD, and Mixture RAG-KD on useful and harmful retrieval cases from the NQ-10K training set, as well as on unseen examples.

[Table. Student performance under useful and harmful retrieval conditions.]

Note. Accuracy (%) is reported as mean ± standard deviation over three seeds. Avg. is the arithmetic mean of useful- and harmful-retrieval accuracies.

MethodNQ-10K Useful (3K)NQ-10K Harmful (691)Avg. ↑Unseen Useful (1K)Unseen Harmful (1K)Avg. ↑
Parametric KD24.9 ± 0.669.3 ± 1.047.133.9 ± 0.836.6 ± 0.935.3
Vanilla RAG-KD88.8 ± 0.514.0 ± 0.551.473.0 ± 0.821.1 ± 0.547.1
Mixture RAG-KD91.5 ± 0.269.6 ± 0.780.670.6 ± 0.630.8 ± 1.050.7

As seen from the table above, we demonstrate that the benefits of mixture supervision are also reflected in the behavior of the distilled student on the training examples. Specifically, Mixture RAG-KD improves accuracy on useful (+2.7pp from Vanilla RAG-KD), harmful retrieval (+0.3pp from Parametric KD), and average accuracy (+29.2pp from Vanilla RAG-KD).

Importantly, this trend also extends to unseen examples. Mixture RAG-KD largely preserves performance on useful retrieval cases while improving accuracy on harmful retrieval cases, achieving a 3.6pp improvement in average accuracy over Vanilla RAG-KD. This suggests that the benefits of mixture supervision are not limited to the training examples, but also extend to unseen examples, where the student achieves a better balance between leveraging helpful evidence and resisting misleading retrieval.

These findings support the transfer of mixture supervision's intended benefits to the student. Our teacher-level analysis demonstrates that the mixture constructs adaptive supervision targets according to retrieval utility, while our student-level analysis shows that these complementary properties are reflected in student behavior on both training and unseen examples.

[W1c & W3. Benefits of instance-wise mixing]

First, we clarify the motivation for risk-based instance-wise mixing. We emphasize that static mixing is itself part of our proposed mixture framework, while instance-wise coefficient optimization further extends this approach by adapting supervision to individual training examples. Static mixing requires training separate student models for multiple coefficients, such as α∈0.2,0.4,0.6,0.8\alpha\in\\{0.2,0.4,0.6,0.8\\}, and selecting an appropriate coefficient based on evaluation performance. This incurs additional training costs and requires a separate coefficient-selection procedure, with the optimal coefficient potentially varying across evaluation settings.

In contrast, our risk-based instance-wise mixing determines the mixing coefficient for each training instance by minimizing gold-reference risk during training. This eliminates the need for repeated training across multiple fixed coefficients or coefficient selection based on test-set performance.

Second, following the reviewer's suggestion, we conduct paired significance tests to evaluate the additional benefits of instance-wise mixing. In response to the concern that 1K NQ training examples were insufficient, we expanded the main experiments in Table 1 to 10K NQ examples. Using the results reported in Table 1 of the revised manuscript, we compare Static mixing (α=0.8\alpha=0.8), Instance-wise FKL, and Instance-wise L2L_2.

[Table. Paired significance tests of instance-wise mixing versus Static α=0.8\alpha=0.8 on NQ-10K].

Note. Entries report accuracy differences in percentage points (Instance − Static), computed from unrounded accuracies, with two-sided paired-test p-values in parentheses. Overall is the arithmetic mean of General QA and RAG-Bench accuracy.

MethodStudentGeneral QARAG-BenchOverall
Instance L21B+1.07 (0.021)−0.03 (0.977)+0.52 (0.162)
4B+1.04 (0.011)+0.40 (0.475)+0.72 (0.028)
Instance FKL1B+0.27 (0.610)−0.17 (0.809)+0.05 (0.908)
4B+1.31 (0.003)+0.62 (0.319)+0.96 (0.009)

The table above provides statistical evidence that instance-wise coefficient optimization can achieve higher performance over static mixing on General QA. However, the differences on the original RAG-Bench are not statistically significant. Therefore, we conduct additional evaluations to investigate how the benefits of instance-wise mixing vary across retrieval conditions.

Third, we extend our robustness evaluation to retrieval settings without gold passages. As described in the main text, The original RAG-Bench evaluates four retrieval conditions: Gold (gold passage only) and Relevant, Irrelevant, and Counterfactual (gold passage + noisy passage), assessing robustness to noisy retrieval when gold evidence is available. Thus, the original RAG-Bench evaluates whether models can effectively exploit gold evidence while remaining robust to additional noisy retrieval.

To examine robustness under a complementary retrieval setting, we additionally evaluate the same questions and retrieval-noise categories after removing the gold passage (Noisy passage only). This allows us to assess whether the benefits of instance-wise mixing persist when gold evidence is unavailable.

[Table. Robustness evaluation without gold passages.]

Note. Accuracy (%) is averaged over the Relevant, Irrelevant, and Counterfactual conditions without the gold passage and reported as mean ± standard deviation across three training seeds. Rows labeled Δ\Delta vs. Static report accuracy improvements in percentage points relative to Static mixing (α=0.8\alpha=0.8), computed from unrounded accuracies, with paired significance test p-values in parentheses.

StudentStatic α=0.8Instance FKLInstance L2L_2
1B20.18 ± 0.6523.04 ± 0.6723.40 ± 0.23
Δ vs. Static—+2.87 (p < 0.001)+3.22 (p < 0.001)
4B32.78 ± 0.6237.53 ± 0.3537.62 ± 0.31
Δ vs. Static—+4.76 (p < 0.001)+4.84 (p < 0.001)

The table above exhibits that the instance-wise mixing achieves substantially larger and statistically significant improvements. Also, these improvements are larger than those observed under the original Gold + Noise conditions. This suggests that the additional benefits of instance-wise coefficient selection become more pronounced in retrieval settings where gold passages are unavailable.

In conclusion, instance-wise mixing, which is one of ours methods, generalizes across diverse evaluation settings without relying on a fixed coefficient.

Reviewer_N91LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mixture RAG-KD to make knowledge distillation more robust to retrieval errors. Instead of relying only on retrieval-based teacher outputs, it combines the teacher’s parametric and retrieval-conditioned distributions using a mixing coefficient. The authors analyze when this mixture can improve student performance and propose global and instance-wise strategies to select the coefficient. Experiments on three open-domain QA benchmarks and RAG-Bench show that the method generally outperforms Vanilla RAG-KD and is more robust to retrieval noise. Results with SFT-initialized students also suggest that it can work alongside standard fine-tuning.

Critical strengths

 The motivation of the paper is clear and well-supported by the preliminary analysis showing how retrieval both corrects and corrupts parametric teacher predictions.  The proposed mixture formulation is simple, intuitive, and easy to plug into standard distillation pipelines. Solving for alpha via closed-form L2 projection or 1D FKL bisection is lightweight and adds almost no overhead once teacher logits are cached.  The empirical analysis of how the instance-wise alpha_i shifts under different noise ratios and teacher correctness regimes is also informative.

Critical weaknesses

 Limited training scale: Evaluating on only 1,000 NQ examples is too small. While small-sample distillation is supported by prior work, it remains unclear whether the proposed dynamics and empirical gains hold against larger-scale baselines.  Lack of model diversity: Experiments are confined entirely to the Gemma-3 family (12B to 1B/4B). Evaluating cross-family distillation is necessary to demonstrate that the method does not rely on idiosyncrasies of Gemma-3.  Idealized theoretical assumption: Theorem 3.1 relies on Assumption (1), which requires the student hypothesis class Q\mathcal{Q} to realize TαT_\alpha for every α\alpha. In practice, the student model's limited capacity is the very motivation for knowledge distillation, so this realizability assumption sidesteps the core bottleneck of distillation.  Missing baselines and questionable necessity of solvers: Despite discussing intermediate-target KD methods (e.g., DistiLLM, AMiD) and robust RAG techniques (e.g., RAAT) in Sections 2.1 and 2.2, none are compared empirically. Crucially, Table 1 shows that a simple static α=0.8\alpha = 0.8 matches and occasionally outperforms the risk-optimal solvers on RAG-Bench, which substantially weakens the practical motivation for deriving the FKL and L2L_2 solvers.

Questions for authors

 The experiments mainly use the Gemma-3 family. Have the authors tested whether Mixture RAG-KD generalizes to other teacher and student models?  The theoretical analysis assumes that the student can represent every mixture target. How does the method behave when this assumption does not hold, especially for smaller students?  Since the method requires both teacher distributions, how much extra time and memory does it require compared with Vanilla RAG-KD? Do the performance gains justify the additional cost?  Have the authors considered applying the mixing coefficient at the token level rather than using a single coefficient for the entire answer? This could be useful for multi-token answers where only some tokens depend on the retrieved context?

10 comments

We thank the reviewer for their thoughtful and constructive feedback. We have carefully considered the concerns and provide additional experimental evidence and clarifications in our responses below.

[W1. Training scale]

We agree that 1K examples alone cannot establish whether the proposed gains persist at larger scales. We therefore extended training to 10K NQ examples (10× the original setting), evaluating three student configurations (pretrained 1B, pretrained 4B, and SFT-initialized 1B) under the same experimental protocol, including three training random seeds.

[Table. Distillation performance across student configurations at the 10K training scale.]

Method1B│4B│SFT-1B
Gen Avg.RAG Avg.│Gen Avg.RAG Avg.│Gen Avg.RAG Avg.
Standard KD45.1845.25│55.9351.32│48.6746.32
Parametric-Teacher-KD (α=0\alpha=0)41.6243.10│44.9349.95│42.0741.38
Vanilla RAG-KD (α=1\alpha=1)48.2945.25│55.1150.58│49.2045.43
Empirical-Reference Distill48.8244.43│56.2950.45│49.1345.77
Mixture RAG-KD51.2048.02│57.6753.20│50.5347.70
Δ\Delta (pp; vs. Vanilla)+2.91+2.77│+2.56+2.62│+1.33+2.27

Note. Mixture RAG-KD uses instance-wise L2L_2 mixing throughout. Δ\Delta (pp) denotes the improvement over Vanilla RAG-KD.

The gains persist across all three student configurations at 10K. Mixture RAG-KD consistently outperforms all four baselines on both General QA and RAG-Bench. Compared with Vanilla RAG-KD, instance-wise L2L_2 mixing achieves improvements of +2.91/+2.77 pp for the 1B student, +2.56/+2.62 pp for the 4B student, and +1.33/+2.27 pp for the SFT-initialized 1B student on General QA/RAG-Bench, respectively.

We additionally report training-scale trends from 1K to 10K examples in Appendix (Figure), demonstrating that Mixture RAG-KD consistently maintains its performance advantage across different training scales.

Together, these results demonstrate that the benefits of mixture supervision are not limited to the original 1K setting and persist under a 10-fold increase in training data, across different student sizes and initialization conditions. We therefore adopt the 10K training setting as the main experimental setting in the revised manuscript and move the original 1K results to the appendix for comparison with prior small-data distillation.

[W2 & Q1. Model diversity]

We agree that experiments limited to Gemma-3 are insufficient to establish generalization across model families. To address this concern, we extended our experiments to three additional teacher–student configurations, using 10K NQ training examples and three independent training runs with different random seeds. Specifically, we investigate three aspects of generalization:

  1. Recent model families: We evaluate Qwen3.5-9B → 2B to examine whether Mixture RAG-KD remains effective on a recent model family.
  2. Smaller teacher models: We evaluate Llama-3.2-3B → 1B to investigate whether the benefits persist with a relatively small 3B teacher.
  3. Cross-family distillation: We evaluate Llama-3.1-8B → SmolLM3-3B to assess whether the method generalizes when the teacher and student belong to different model families.

[Table. Generalization of Mixture RAG-KD across model families.]

MethodQwen3.5Llama-3.2Cross-family
Standard KD54.29 / 50.05  49.51 / 48.22  56.16 / 52.62
Parametric-Teacher-KD (α=0\alpha=0)39.11 / 43.40  37.22 / 43.78  46.53 / 49.90
Vanilla RAG-KD (α=1\alpha=1)54.73 / 49.20  49.87 / 47.72  55.18 / 51.32
Empirical-Reference Distill55.18 / 49.70  51.71 / 46.65  56.64 / 53.30
Mixture RAG-KD56.58 / 51.15  51.96 / 49.78  57.27 / 54.55
Δ (pp; vs. Vanilla)+1.85 / +1.95  +2.09 / +2.06  +2.09 / +3.23

Note. General QA / RAG-Bench accuracy (%). Qwen3.5: 9B →\rightarrow 2B; Llama-3.2: 3B →\rightarrow 1B; Cross-family: Llama-3.1-8B →\rightarrow SmolLM3-3B. Mixture RAG-KD uses instance-wise L2L_2 mixing. Δ\Delta (pp) denotes improvement over Vanilla RAG-KD.

Mixture RAG-KD consistently outperforms all four distillation baselines on both General QA and RAG-Bench across all three configurations. These results show that the improvements persist on the recent Qwen3.5 family, remain effective with a relatively small 3B teacher, and extend to cross-family distillation. Notably, in the cross-family setting, Mixture RAG-KD achieves improvements of +2.09 pp on General QA and +3.23 pp on RAG-Bench over Vanilla RAG-KD.

Together with our Gemma-3 results, these findings provide empirical evidence that the benefits of mixture supervision are not limited to Gemma-3-specific characteristics but extend across recent model families, smaller teachers, and cross-family distillation settings.

[W3 &Q2. Idealized theoretical assumption ]

First, we clarify the role of the realizability assumption in our theoretical analysis. Realizability is widely used in theoretical studies of imitation learning [1,2,3] and knowledge distillation [4,5] to analyze the effect of supervision quality separately from the model’s approximation limitations. In Theorem 3.1 of the submitted manuscript, this assumption is required only to transfer the teacher-level analysis into a student-level result. The teacher-level analysis remains valid and meaningful without this assumption. Specifically, under the endpoint derivative conditions, the risk-optimal mixture target achieves lower target risk than either standalone endpoint target.

Theorem 3.1. (revised, target-level). If Rˉ+′(0)<0\bar{\mathcal{R}}'_+(0) < 0 and Rˉ−′(1)>0\bar{\mathcal{R}}'_-(1) > 0, then R(p∗,Tα∗)<min⁡R(p∗,p),R(p∗,pc)\mathcal{R}(p^{\ast}, T_{\alpha^{\ast}}) < \min\\{\mathcal{R}(p^{\ast}, p), \mathcal{R}(p^{\ast}, p_c)\\}

To make this distinction explicit, we will separately present the teacher-level result as Theorem 3.1 and the student-level result under the realizability assumption as Corollary 3.2 in the revised manuscript

Next, we conduct additional analysis to examine the relationship among the target risk R(p∗,Tα)\mathcal{R}(p^{\ast}, T_{\alpha}), the student risk R(p∗,qα∗)\mathcal{R}(p^{\ast}, q_{\alpha}^\ast), and the distillation risk R(Tα,qα∗)\mathcal{R}(T_{\alpha}, q_{\alpha}^{\ast}) to examine this transfer empirically. We evaluated forward-KL distillation using three student sizes (270M, 1B, and 4B), and mixture weights (α=\alpha = 0, 0.2, 0.45, 0.6, 0.8, 1.0, Global KL). The experimental results are in Figure.

  • The distillation risk is non-zero, but small and nearly invariant to α\alpha. Across all settings, R(p∗,qα∗)>0\mathcal{R}(p^{\ast}, q_{\alpha^{\ast}}) > 0. This confirms that realizability does not hold exactly in practice, consistent with the reviewer's point. As expected, this approximation error decreases as student capacity increases. More importantly, for a fixed student size, the approximation error remains small and comparable across different values of α\alpha. This indicates that the student absorbs different mixture targets with similar fidelity. As a result, differences in student quality across are driven primarily by differences in target quality rather than by differences in distillation difficulty.

  • The student risk R(p∗,qα∗)\mathcal{R}(p^{\ast}, q_{\alpha}^{\ast}) obtained using Global KL is lower than any α\alpha including the endpoints R(p∗,q0∗)\mathcal{R}(p^{\ast}, q_{0}^{\ast}) and R(p∗,q1∗)\mathcal{R}(p^{\ast}, q_{1}^{\ast}).

  • Target risk is strongly predictive of student risk. The table below reports correlation coefficients between the student risk and either the target risk or the distillation risk across α\alpha. Target risk exhibits a strong positive correlation with student risk across all student sizes and all correlation measures. This supports the hypothesis that a lower target risk R(p∗,Tα∗)\mathcal{R}(p^{\ast}, T_{\alpha^{\ast}}) leads to a lower student risk R(p∗,qα∗)\mathcal{R}(p^{\ast}, q_{\alpha^{\ast}}), even under limited capacity. Moreover, for every student size, the student risk is considerably more strongly associated with the target risk than with the distillation risk. This suggests that, in our setting, improving the quality of the mixture target contributes more to student quality than variations in distillation fidelity.

[Table. Correlation of Student Risk with Target Risk and Distillation Risk Across Mixture Coefficients.]

StudentMeasurePearsonSpearmanKendall's tau
270MR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(p∗,Tα)R(p^{\ast}, T_{\alpha})0.940.930.81
270MR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(Tα,qα∗)R(T_{\alpha}, q_{\alpha}^{\ast})0.550.460.43
1BR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(p∗,Tα)R(p^{\ast}, T_{\alpha})0.990.960.90
1BR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(Tα,qα∗)R(T_{\alpha}, q_{\alpha}^{\ast})0.680.320.24
4BR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(p∗,Tα)R(p^{\ast}, T_{\alpha})0.981.001.00
4BR(p∗,qα∗)R(p^{\ast}, q_{\alpha}^\ast) vs R(Tα,qα∗)R(T_{\alpha}, q_{\alpha}^{\ast})-0.070.290.24

[1] Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms, ICML 2024

[2] Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning, NeurIPS 2024

[3] Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning, NeurIPS 2025

[4] Towards Understanding Knowledge Distillation, ICML 2019

[5] AMiD: Knowledge Distillation for LLMs with α-mixture Assistant Distribution, ICLR 2026

(Continued below)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion Period

Dear AC and Reviewers,

We sincerely thank all reviewers (N91L, y1YC, mZK7) for their thoughtful and constructive feedback.

We appreciate the reviewers' recognition of the clear motivation and practical importance of addressing unreliable teacher supervision under imperfect retrieval (N91L, y1YC, mZK7), the simplicity and practicality of our approach (N91L, y1YC, mZK7), and its analytical tractability through convex optimization under forward KL and a closed-form solution under squared L2L_2 (N91L, mZK7).

We emphasize that our central contribution is a risk-based supervision framework for imperfect retrieval. We theoretically characterize when an intermediate teacher mixture achieves lower target risk than either endpoint (Sec. 3.2, Theorem 3.1) and develop global and instance-wise coefficient optimization strategies based on reference-risk minimization (Sec. 3.3, App. B).

We have uploaded a final revised manuscript. Below, we summarize the main concerns raised by the reviewers and our responses, with their locations in the manuscript.

  • Validity of mixture supervision (y1YC)

    • [Direct teacher-level validation] We verified that the mixture improves the accuracy of generated answers (App. D.1, Fig. 11) while reducing reference risk (Sec. 3.2, Fig. 5) before student distillation.
  • Theory and limited student capacity (N91L, y1YC, mZK7)

    • [Theory without realizability] We reformulated Theorem 3.1 as a target-level result without requiring realizability and separated the student-level guarantee into Corollary 3.2 (Sec. 3.2; proofs in App. A.1 and A.3).
    • [Empirical validation] Additional analyses across 270M, 1B, and 4B students show that student risk closely follows target risk even when exact realizability does not hold (Sec. 3.2, Fig. 6), and the gains persist for the 270M student (App. D.2).
  • Effectiveness of risk-based coefficient optimization

    • [Additional adaptive baselines (mZK7)] We compared instance-wise mixing against hard teacher selection and confidence-based weighting (App. D.8).
    • [Fixed mixing and statistical significance (N91L, y1YC, mZK7)] Instance-wise L2L_2 mixing significantly outperforms fixed mixing on General QA (p≤0.021p \le 0.021) without requiring a coefficient search, while achieving comparable performance on RAG-Bench. It also significantly outperforms Vanilla RAG-KD (p≤0.004p \le 0.004; Table 1, Fig. 9; responses to y1YC and mZK7).
  • Generalization and stronger baselines (N91L, y1YC, mZK7)

    • [Training scale and model diversity] We expanded training from 1K to 10K NQ examples and evaluated 270M–4B students, including SFT-initialized students (Sec. 4.1, Tables 1–2, App. D.2–D.3). We also added distillation experiments with Qwen3.5 and Llama-3.2, as well as cross-family distillation, empirically demonstrating the generalizability of our approach (Table 3, App. D.4).
    • [Task generalization] We extended evaluation to multi-hop QA and mathematical reasoning (App. D.7). The improvements persist across these settings using the same instance-wise L2L_2 mixing strategy.
    • [Existing KD and robust RAG methods] We added comparisons with TAID, DistiLLM, and AMiD (Fig. 7, App. D.5), as well as RAAT (App. D.6), demonstrating the complementary benefits of our retrieval-aware supervision framework. This is particularly valuable because our mixture only replaces the teacher target, so it can be combined with KD objectives that distill from a teacher distribution without modifying them.
  • Retrieval robustness and practical considerations

    • [Controlled retrieval corruption and mathematical reasoning (mZK7)] We demonstrated retrieval robustness through controlled experiments that varied retrieved passages while keeping questions and gold answers fixed (App. D.9). Furthermore, on GSM8K with noisy retrieval, Mixture RAG-KD significantly outperforms Vanilla RAG-KD (response to mZK7).
    • [Robustness without gold evidence (N91L, y1YC)] When gold passages are unavailable, instance-wise mixing significantly outperforms fixed mixing (p<0.001p<0.001; App. D.10 and response to y1YC).
    • [Training efficiency (N91L)] Despite approximately 34% additional training time, Mixture RAG-KD achieves +3.9/+3.7 pp over Vanilla RAG-KD on General QA/RAG-Bench under matched training-time budgets (App. D.12).
    • [Token-level mixing (N91L)] We investigated token-level coefficient optimization and discussed its limitations (App. D.11).

We appreciate the reviewers' continued engagement during the discussion period. We are encouraged that N91L and y1YC have explicitly indicated that their concerns have been addressed, and we have provided additional experimental evidence in response to mZK7's remaining questions.

Once again, we sincerely thank the AC and all reviewers.

Best regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.