- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes DCRD (Dynamic Curriculum-RAG Distillation), a multi-teacher knowledge distillation framework designed for imbalanced multi-domain question answering (QA). DCRD treats each domain-teacher pair as a distinct learning course and integrates three core components: (1) retrieval-augmented distillation using external evidence from PubMed and Wikipedia to ground teacher supervision; (2) real-time learning progress tracking via moving-average distillation loss; and (3) a performance-driven curriculum scheduler that dynamically adjusts course sampling probabilities, local epochs, and gradient accumulation steps based on loss deficits. The authors evaluate DCRD across three QA benchmarks (MedQA-USMLE, OpenBookQA, and StrategyQA) using Flan-T5 models (Small, Base, Large) and test zero-shot generalization on five MMLU subjects, claiming superior performance over teacher ensembling and MoE routing baselines.
Critical strengths
Practical Problem Formulation and Intuitive Framework Design: The paper tackles a realistic challenge in deploying compact QA models: distilling knowledge from heterogeneous domain-specialized teachers under severe cross-domain data imbalance. Formulating domain-teacher pairs as courses and combining RAG grounding with loss-driven dynamic scheduling (modulating sampling frequency, local epochs, and gradient accumulation) is conceptually intuitive and well-motivated.
Critical weaknesses
- In Section 3.1 (lines 203–208) and Figure 10 (page 20), the authors explicitly state that the student input during training is constructed by concatenating the task instruction, retrieved documents, the question, AND the teacher's reasoning chain (Teacher Explanation), while training the student to generate only the final answer letter (A/B/C/D). However, during inference (Figures 7–9, pages 19–20), the teacher's explanation is absent from the input. Conditioning the student on the teacher's rationale as an input rather than training the student to generate it as a target contradicts the core principle of reasoning distillation. As shown in Tables 20 and 22, the teacher explanations explicitly conclude with the final answer (e.g., Answer: (D)), allowing the student to trivially copy the answer from the input prompt during training rather than learning to reason. This creates a severe train-test mismatch at inference time when teacher explanations are unavailable, which explains why DCRD underperforms simple Few-shot prompting on the 77M model (33.66% vs. 34.43% in Table 1) and exhibits erratic ablation behavior in Table 5.
- Table 1 states that the overall accuracy is a weighted average by test-set size (1273, 500, 687; total=2460), whereas Appendix B.1 (line 855) explicitly states it is computed as the macro-average over the three datasets. Worse, the reported numbers match neither. For Flan-T5-Large (31.26%, 60.00%, 53.28%), the weighted average is 43.25% and the macro-average is 48.18%, yet the abstract, Table 3, Table 6, and Table 13 report 43.29%, while Table 5 reports 43.25%. Similarly, in Table 2, the average of DCRD's five MMLU scores is 38.94%, not 43.54% (and the 0-shot average is missing).
- Algorithm 14 updates the student model twice per step (lines 15 and 23) and omits the scaling factors and from Equations (4) and (5).
- The paper includes a non-anonymized GitHub URL in the abstract (line 22) and reproducibility statement (line 497). It vaguely cites a 2020 philosophy paper (Floridi & Chiriatti, 2020) for the "GPT" teacher without specifying the model version. Numerous editing errors persist throughout, including inconsistent method naming (DYRD in Figures 3 & 6 vs. DCRD), broken LaTeX references (Table ?? on line 926, referring to Figures 7–9 as Tables 7–9 on lines 1011–1018), and scrambled column headers (Med / OB / Str) across columns in Table 15.
Questions for authors
- Section 3.1 and Figure 10 state that Teacher Explanation is concatenated into the student's input during training, whereas Figures 7–9 show it is absent during testing. Does your actual implementation feed the teacher's explanation as input rather than using it as the generation target? If so, how do you prevent the student from simply copying the answer from the teacher's rationale during training and suffering from distribution shift at test time?
- Why does Table 1 state that the overall score is a sample-weighted average (which yields 43.25% for Large) while Appendix B.1 claims it is a macro-average (which would be 48.18%), and why is this same metric reported as 43.29% in the Abstract, Table 3, and Table 6? Additionally, in Table 5, what is the exact difference between Teacher+student RAG (39.23%) and DCRD (full) (43.25%)?
3.How is it possible that DCRD requires ~1.6 hours of training time (nearly 7x longer than LoRA's 14 minutes) with full-parameter updates, longer RAG-augmented sequences, and multiple local epochs, yet consumes less than a third of the total FLOPs (3.1 EFLOPs vs. 11 EFLOPs for LoRA)? Please clarify how FLOPs were measured.
Response to Reviewer YPpL
Thank you for identifying these inconsistencies. We address the input protocol, score aggregation, and compute accounting separately.
1. Training and test inputs. The method description and Figure 10 specify that the student receives retrieved documents, the question, and the teacher explanation as training input; the teacher’s final answer is the generation target. Figures 7–9 specify test inputs containing retrieved knowledge, the question, and options, without a teacher explanation. Thus, the stated protocol has a train–test input difference.
The submission does not document a check that removes an explicit answer from the teacher explanation. If the explanation states the answer, the student could learn to copy it during training, even though that information is unavailable at test time. The reported “without RAG” experiment removes retrieval; it does not remove the explanation or mask answer mentions within it. We therefore cannot use the current ablations to quantify either explanation leakage or the resulting distribution shift.
2. Overall accuracy and Table 5. Table 1 defines overall accuracy as the number of correct predictions across all 2,460 test questions divided by 2,460. For the Large DCRD row, the displayed dataset accuracies give:
24601273(31.26)+500(60.00)+687(53.28)=43.25%.
Their unweighted mean is 48.18%. Appendix B.1’s statement that the main overall score is a macro-average conflicts with Table 1 and the displayed results. The applicable aggregation for these tables is the sample-weighted, pooled accuracy.
The Abstract, Table 3, and Table 6 instead report 43.29%. That value cannot be obtained from the three displayed dataset accuracies by the stated aggregation, and ordinary rounding of those three percentages cannot account for the 0.04-point difference. We cannot reconcile 43.25% and 43.29% as one result without the underlying prediction counts or evidence of a distinct run. The comparison should not mix them.
In Table 5, “Teacher+student RAG” scores 39.23% and “DCRD (full)” 43.25%, a 4.02-point difference. Both labels indicate retrieval on the teacher and student sides. The table does not specify a controlled change limited to one additional component: the full system also includes the multi-teacher course design and dynamic training schedule. Consequently, the 4.02-point difference is a system-level comparison, not an estimate of the isolated effect of curriculum scheduling or retrieval.
3. Training time and FLOPs. Table 9 reports approximately 1.6 hours and 3.1 EFLOPs for DCRD, versus 14 minutes and 11 EFLOPs for LoRA. The submission does not state how FLOPs were counted, whether they include forward and backward passes, actual sequence lengths and optimizer updates, teacher generation, retrieval, or preprocessing. It also does not establish that the two figures use the same measurement boundary.
Longer wall-clock time with fewer measured FLOPs is possible when retrieval, data preparation, or model calls cause idle time. That possibility does not validate these particular numbers. Given DCRD’s reported full-parameter training, longer inputs, and repeated local epochs, the claimed compute advantage cannot be assessed without the counting method and per-stage breakdown. We therefore regard the wall-clock figures as reported observations and do not infer from Table 9 that DCRD requires less total compute than LoRA.