acceptodds
Under review as a conference paper at ICLR 2027

DCRD: Retrieval-Grounded Curriculum Distillation for Imbalanced Multi-Domain Question Answering

Abstract

Knowledge distillation has become an effective approach for transferring the capabilities of large language models into smaller and more efficient students. However, existing distillation methods typically assume a single teacher and balanced training distributions, which limits their applicability to realistic question-answering scenarios involving heterogeneous domains with severe data imbalance. We present DCRD, a multi-teacher distillation framework for imbalanced multi-domain question answering that treats each domain–teacher pair as a distinct learning course and dynamically schedules training effort based on the student's recent validation accuracy on each course. The framework combines retrieval-augmented knowledge grounding—enriching teacher supervision with external evidence from Wikipedia and PubMed—with a curriculum scheduler that adaptively adjusts course sampling and training intensity according to the student's performance. Experiments on MedQA-USMLE, OpenBookQA, and StrategyQA show that our approach consistently improves performance across Flan-T5 model scales, reaching 43.29% average accuracy on Flan-T5-Large and outperforming teacher ensemble and MoE-based distillation baselines without additional inference cost. These results demonstrate that curriculum-aware multi-teacher distillation provides an effective and scalable solution for knowledge-intensive multi-domain reasoning. Our code is available at https://github.com/linda-2-hh/DCRD.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-teacher distillation, curriculum learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YPpLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DCRD (Dynamic Curriculum-RAG Distillation), a multi-teacher knowledge distillation framework designed for imbalanced multi-domain question answering (QA). DCRD treats each domain-teacher pair as a distinct learning course and integrates three core components: (1) retrieval-augmented distillation using external evidence from PubMed and Wikipedia to ground teacher supervision; (2) real-time learning progress tracking via moving-average distillation loss; and (3) a performance-driven curriculum scheduler that dynamically adjusts course sampling probabilities, local epochs, and gradient accumulation steps based on loss deficits. The authors evaluate DCRD across three QA benchmarks (MedQA-USMLE, OpenBookQA, and StrategyQA) using Flan-T5 models (Small, Base, Large) and test zero-shot generalization on five MMLU subjects, claiming superior performance over teacher ensembling and MoE routing baselines.

Critical strengths

Practical Problem Formulation and Intuitive Framework Design: The paper tackles a realistic challenge in deploying compact QA models: distilling knowledge from heterogeneous domain-specialized teachers under severe cross-domain data imbalance. Formulating domain-teacher pairs as courses and combining RAG grounding with loss-driven dynamic scheduling (modulating sampling frequency, local epochs, and gradient accumulation) is conceptually intuitive and well-motivated.

Critical weaknesses
  1. In Section 3.1 (lines 203–208) and Figure 10 (page 20), the authors explicitly state that the student input during training is constructed by concatenating the task instruction, retrieved documents, the question, AND the teacher's reasoning chain (Teacher Explanation), while training the student to generate only the final answer letter (A/B/C/D). However, during inference (Figures 7–9, pages 19–20), the teacher's explanation is absent from the input. Conditioning the student on the teacher's rationale as an input rather than training the student to generate it as a target contradicts the core principle of reasoning distillation. As shown in Tables 20 and 22, the teacher explanations explicitly conclude with the final answer (e.g., Answer: (D)), allowing the student to trivially copy the answer from the input prompt during training rather than learning to reason. This creates a severe train-test mismatch at inference time when teacher explanations are unavailable, which explains why DCRD underperforms simple Few-shot prompting on the 77M model (33.66% vs. 34.43% in Table 1) and exhibits erratic ablation behavior in Table 5.
  2. Table 1 states that the overall accuracy is a weighted average by test-set size (1273, 500, 687; total=2460), whereas Appendix B.1 (line 855) explicitly states it is computed as the macro-average over the three datasets. Worse, the reported numbers match neither. For Flan-T5-Large (31.26%, 60.00%, 53.28%), the weighted average is 43.25% and the macro-average is 48.18%, yet the abstract, Table 3, Table 6, and Table 13 report 43.29%, while Table 5 reports 43.25%. Similarly, in Table 2, the average of DCRD's five MMLU scores is 38.94%, not 43.54% (and the 0-shot average is missing).
  3. Algorithm 14 updates the student model twice per step (lines 15 and 23) and omits the scaling factors ΔE\Delta E and ΔA\Delta A from Equations (4) and (5).
  4. The paper includes a non-anonymized GitHub URL in the abstract (line 22) and reproducibility statement (line 497). It vaguely cites a 2020 philosophy paper (Floridi & Chiriatti, 2020) for the "GPT" teacher without specifying the model version. Numerous editing errors persist throughout, including inconsistent method naming (DYRD in Figures 3 & 6 vs. DCRD), broken LaTeX references (Table ?? on line 926, referring to Figures 7–9 as Tables 7–9 on lines 1011–1018), and scrambled column headers (Med / OB / Str) across columns in Table 15.
Questions for authors
  1. Section 3.1 and Figure 10 state that Teacher Explanation is concatenated into the student's input during training, whereas Figures 7–9 show it is absent during testing. Does your actual implementation feed the teacher's explanation as input rather than using it as the generation target? If so, how do you prevent the student from simply copying the answer from the teacher's rationale during training and suffering from distribution shift at test time?
  2. Why does Table 1 state that the overall score is a sample-weighted average (which yields 43.25% for Large) while Appendix B.1 claims it is a macro-average (which would be 48.18%), and why is this same metric reported as 43.29% in the Abstract, Table 3, and Table 6? Additionally, in Table 5, what is the exact difference between Teacher+student RAG (39.23%) and DCRD (full) (43.25%)?
    3.How is it possible that DCRD requires ~1.6 hours of training time (nearly 7x longer than LoRA's 14 minutes) with full-parameter updates, longer RAG-augmented sequences, and multiple local epochs, yet consumes less than a third of the total FLOPs (3.1 EFLOPs vs. 11 EFLOPs for LoRA)? Please clarify how FLOPs were measured.
1 comment
AuthorsOpenReview ↗

Response to Reviewer YPpL

Thank you for identifying these inconsistencies. We address the input protocol, score aggregation, and compute accounting separately.

1. Training and test inputs. The method description and Figure 10 specify that the student receives retrieved documents, the question, and the teacher explanation as training input; the teacher’s final answer is the generation target. Figures 7–9 specify test inputs containing retrieved knowledge, the question, and options, without a teacher explanation. Thus, the stated protocol has a train–test input difference.

The submission does not document a check that removes an explicit answer from the teacher explanation. If the explanation states the answer, the student could learn to copy it during training, even though that information is unavailable at test time. The reported “without RAG” experiment removes retrieval; it does not remove the explanation or mask answer mentions within it. We therefore cannot use the current ablations to quantify either explanation leakage or the resulting distribution shift.

2. Overall accuracy and Table 5. Table 1 defines overall accuracy as the number of correct predictions across all 2,460 test questions divided by 2,460. For the Large DCRD row, the displayed dataset accuracies give:

1273(31.26)+500(60.00)+687(53.28)2460=43.25%.\frac{1273(31.26)+500(60.00)+687(53.28)}{2460}=43.25\%.

Their unweighted mean is 48.18%. Appendix B.1’s statement that the main overall score is a macro-average conflicts with Table 1 and the displayed results. The applicable aggregation for these tables is the sample-weighted, pooled accuracy.

The Abstract, Table 3, and Table 6 instead report 43.29%. That value cannot be obtained from the three displayed dataset accuracies by the stated aggregation, and ordinary rounding of those three percentages cannot account for the 0.04-point difference. We cannot reconcile 43.25% and 43.29% as one result without the underlying prediction counts or evidence of a distinct run. The comparison should not mix them.

In Table 5, “Teacher+student RAG” scores 39.23% and “DCRD (full)” 43.25%, a 4.02-point difference. Both labels indicate retrieval on the teacher and student sides. The table does not specify a controlled change limited to one additional component: the full system also includes the multi-teacher course design and dynamic training schedule. Consequently, the 4.02-point difference is a system-level comparison, not an estimate of the isolated effect of curriculum scheduling or retrieval.

3. Training time and FLOPs. Table 9 reports approximately 1.6 hours and 3.1 EFLOPs for DCRD, versus 14 minutes and 11 EFLOPs for LoRA. The submission does not state how FLOPs were counted, whether they include forward and backward passes, actual sequence lengths and optimizer updates, teacher generation, retrieval, or preprocessing. It also does not establish that the two figures use the same measurement boundary.

Longer wall-clock time with fewer measured FLOPs is possible when retrieval, data preparation, or model calls cause idle time. That possibility does not validate these particular numbers. Given DCRD’s reported full-parameter training, longer inputs, and repeated local epochs, the claimed compute advantage cannot be assessed without the counting method and per-stage breakdown. We therefore regard the wall-clock figures as reported observations and do not infer from Table 9 that DCRD requires less total compute than LoRA.

Reviewer_6tmnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles the problem of large LLM distillation in the case of several subject areas, each with its own teacher and varying amounts of data. There are two parts of the method: first is retrieval, where documents from PubMed or Wikipedia and the teacher's explanation are added to the student's input. The second method is scheduler where the running loss is used to decide how often that particular course is sampled, how many epochs it gets and how many gradient steps are run. The student model here is flan T5 family and the paper runs experiments on question answering data sets like med QA, open book QA, strategy QA.

Critical strengths

The problem is very practical, frequently encountered in production scenarios. The proposed scheduler is simple and interpretable. The figures paper attaches does show a shifting effort between courses intuitively.

Critical weaknesses

The training input here includes the teacher explanation, but sometimes the example explanations also state the answer, for example, in Table 20. Given that the target is only the answer letter and not the explanation the copying can lead to a train-test mismatch as there is no explanation at this time.

I might be misunderstanding here, but the accuracies look very close to chance. For example, medQA is around 27%, but I believe we can get 25% from random guessing. Same for strategy QA at around 50%. So it is very hard to justify the strength of the baseline and the proposed method, or state confidently whether there is any improvement.

Questions for authors

I am slightly confused about this. Can the authors clarify if the training input contains the teacher explanation with the final answer? And what then constitutes the student input at test time? Did the authors explore the results when we removed the explanation or masked the answer during training?

Why are the BOLT and RTKD rows identical to Few-shot rows? Were those baselines run?

Can we correct Table 2’s averages? they seem to be computed wrong if I am understanding this correctly.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6tmn

Thank you for raising these points. The training prompt, repeated baseline scores, and Table 2 averages need to be interpreted carefully.

1. Teacher explanations and student inputs. During training, the student input contains the task instruction, retrieved documents, question, and a teacher-generated explanation. The supervision target is the teacher’s final answer. Appendix D.2 shows Teacher Explanation: {Teacher-generated Explanation} in the input and instructs the model to output the answer letter. The submission does not specify a check that removes an answer letter or an explicit final-answer statement if one appears inside the explanation. We therefore cannot rule out answer leakage through that field from the prompt specification alone.

At test time, the prompts in Appendix D.1 contain the question, options, and retrieved knowledge. They do not contain a teacher explanation or the correct answer. The student must generate an option letter.

The reported “without retrieval” ablation removes retrieved evidence; it is not an explanation-removal or answer-masking experiment. Neither of those two controls is reported. Consequently, the current ablations do not separate the effect of learning from a teacher’s reasoning from the effect of any answer information present in that reasoning.

2. BOLT and RTKD. The BOLT row matches Few-shot on MedQA, OpenBookQA, and StrategyQA at all three student scales. RTKD matches Few-shot on MedQA at all three scales, but its OpenBookQA and StrategyQA values differ. In particular, MedQA’s repeated 27.73% corresponds to 353 correct predictions out of 1,273; identical accuracy alone does not prove identical predictions.

The submitted materials do not provide per-question predictions, run identifiers, or implementation records that establish independent BOLT and RTKD runs for those matching entries. We cannot explain the BOLT-wide identity as a property of the method or confirm independent execution from the aggregate table. The corresponding comparative claims should therefore not rely on those matching rows as independently verified results.

3. Table 2 averages. We recalculated the averages from the five displayed subject scores using the stated sample counts: high-school chemistry 202, high-school physics 150, psychology 505, anatomy 131, and clinical knowledge 243. The total is 1,231, and each average is weighted by its subject count. Because the inputs are rounded to two decimals, the recomputed values are approximate to the last decimal place.

MethodReported averageRecomputed average
Zero-shot—42.32
CoT42.0842.16
LoRA43.4643.54
Self-KD41.2741.35
Ensemble43.0543.05
MoE39.3427.54
DCRD43.5443.54

The MoE entry is the material arithmetic error. Based on the displayed subject scores, DCRD and LoRA have the same rounded weighted average, 43.54%. Thus, Table 2 does not establish a unique highest average for DCRD.

Reviewer_B425OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses data imbalance in multi-domain knowledge distillation and proposes DCRD, a framework that combines retrieval augmentation with adaptive curriculum scheduling. It treats each domain–teacher pair as a training course and adjusts course sampling probabilities and training intensity based on the student's learning progress. The method is evaluated on three question-answering benchmarks with multiple student model sizes.

Critical strengths
  1. The motivation is reasonable: data imbalance is a real issue in multi-domain KD. The method directly targets this problem by adapting sampling and training effort, and the approach looks feasible to me.

  2. The experiments are fairly extensive, covering multiple student sizes, different imbalance settings, and a reasonably thorough set of component ablations and sensitivity analyses.

Critical weaknesses
  1. Several reported numbers raise concerns about the reliability of the results. In Table 1, Few-shot and BOLT have exactly the same scores on all three benchmarks at all three model sizes, total nine matching results, which are difficult to explain as coincidence alone. Table 2 also contains a clear error that the MoE average is reported as 39.34%, although all five subject scores fall between 23.66% and 28.67%. An average of those scores whether unweighted or weighted is impossible to exceed every individual score.

  2. The MedQA column in Table 1 contains 12 occurrences of 27.73% among 33 entries, an unusual concentration across different methods and model sizes. I checked the authors' released evaluation code and found that the answer parser (line 100-105 in test_med.py) checks for option letters anywhere in the uppercased output, starting with A. This means any output containing the letter “a” is parsed to A; for example Answer: B is parsed as A. Common answer formats could therefore cause many predictions to collapse to A. Meanwhile, always predicting A gives exact 27.73% accuracy (353/1,273) on the MedQA. These observations give me substantial reason to suspect an evaluation artifact.

  3. There are also some writing and experimental-detail issues, as noted in the questions below. One citation appears incorrect: the baseline labelled “prompt-KD” cites Gu et al. (2024), whose reference entry is MiniLLM. Please clarify which method was actually evaluated and correct the citation accordingly.

Questions for authors
  1. Could you provide the raw model responses and corresponding parsed answer choices for the repeated 27.73% MedQA results and the matching Few-shot/BOLT results? Alternatively, please provide the proportions of predictions assigned to each option, including invalid outputs, for each setting. It would also help to check whether these results change under strict answer extraction, explain the identical Few-shot/BOLT scores, and correct the MoE average in Table 2.

  2. Could you provide further details on the comparison settings, particularly how training budgets and access to teacher supervision and retrieval resources are matched between DCRD and the baselines?

  3. Could you provide further clarification of the data construction and evaluation protocols, particularly the imbalance settings, score aggregation or anything necessary?

1 comment
AuthorsOpenReview ↗

Response to Reviewer B425

Thank you for the careful reading. We address the numerical checks, comparison conditions, and evaluation protocol separately.

1. Repeated MedQA scores and Table 2. MedQA-USMLE contains 1,273 evaluation questions, so 27.73% corresponds to 353 correct answers. The recurrence of this value across methods and model sizes does not establish that they made the same predictions: identical totals can arise from different answer patterns. The submitted aggregate tables do not include raw responses, parsed choices, or counts for options A–D and invalid outputs. We therefore cannot provide those distributions or claim that the repeated scores survive strict answer extraction from the reported record.

Few-shot and BOLT have identical reported scores on all three datasets at each student scale. This is more than a matching rounded average, but the paper does not provide prediction-level outputs or a protocol explanation that establishes why they coincide. They should not be treated as independent evidence of equivalent methods without that check.

The MoE average in Table 2 is an arithmetic error. Using the stated MMLU subject sizes—202, 150, 505, 131, and 243—the sample-weighted average is:

=27.54\%.$$ The reported 39.34% is inconsistent with the five displayed subject scores. DCRD’s weighted average from its displayed scores is 43.54%. Correcting MoE’s average does not change their ordering. **2. Comparison resources and training budgets.** The trainable student baselines use Flan-T5 at the stated 77M, 250M, and 780M scales and the same three-domain data, according to Section 4.1. DCRD additionally uses domain-specific teacher explanations—GPT for MedQA, GLM-4.5-air for OpenBookQA, and DeepSeek-R1-0528 for StrategyQA—and retrieves three documents by default from PubMed or Wikipedia. Few-shot prompting and teacher ensembling are inference-time reference points, so their access to supervision and training compute is structurally different. The paper does not specify, baseline by baseline, whether the same teacher explanations and retrieved documents were available during training and inference. “Same multi-domain data” alone does not establish equal access to those resources. Nor does it demonstrate a matched number of optimizer updates or teacher calls. DCRD samples courses adaptively, varies local training intensity and gradient accumulation from 4 to 12, and can stop early; these choices alter its effective exposure to the data. Table 9 reports time and FLOPs, including approximately 1.6 hours and 3.1 EFLOPs for DCRD, but does not itemize teacher generation, retrieval, and indexing within those totals. The comparisons therefore measure the reported systems under their respective resource settings. They do not isolate an advantage under a fully matched teacher, retrieval, and compute budget. The within-DCRD ablations provide the more direct evidence for its components: on the Large student, removing retrieval changes the reported average from 43.29% to 39.87%, and replacing dynamic scheduling with uniform sampling changes it to 41.06%. **3. Data construction and score aggregation.** The main evaluation uses 1,273 MedQA-USMLE, 500 OpenBookQA, and 687 StrategyQA questions. Each domain is paired with its designated teacher; teacher-generated explanations are stored course-wise. The curriculum uses a moving loss-derived pseudo-accuracy to choose courses and training intensity. That quantity schedules training; it is not the test accuracy. The main “Average” is pooled accuracy across 2,460 questions, rather than an unweighted mean of three domain percentages: $$Acc_{\mathrm{overall}}= \frac{1273\,Acc_{\mathrm{MedQA}}+ 500\,Acc_{\mathrm{OpenBookQA}}+ 687\,Acc_{\mathrm{StrategyQA}}}{2460}.$$ Here each domain accuracy is expressed as a proportion. Thus MedQA contributes the largest share to the overall result. The MMLU average likewise weights its five subjects by their sample counts, totaling 1,231 questions. The imbalance study imposes domain sampling ratios from 1:1:1 to 1:8:16 in the order StrategyQA:MedQA-USMLE:OpenBookQA, while keeping the Flan-T5-Large architecture and stated training setup fixed. These ratios describe the constructed training imbalance; the test-set counts above remain the evaluation denominators. The paper does not give enough detail to recover the exact sampled training IDs, replacement policy, or per-course update counts at each level. Consequently, Table 12 supports comparison within the reported imbalance settings, but not an exact reconstruction of every training stream from the manuscript alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.