acceptodds
Under review as a conference paper at ICLR 2027

Principal-Component Curriculum Learning for Multi-Source Knowledge Distillation

Abstract

Curriculum learning, which organizes training data from easy to hard, is increasingly used in knowledge distillation for large language models. Existing distillation curricula rank examples by estimated difficulty and thus decide when each example enters training. In practice, however, distillation data are mixtures drawn from multiple sources, and each update combines learning signals that differ in task, domain, and format. A curriculum for such data must therefore answer two questions: which examples the student learns first, and which it learns together. We propose principal-component curriculum learning (PCCL) to address both. PCCL orders batches by a weighted principal-loading score derived from per-example teacher gradients, so that the student first encounters the pool's dominant learning signal before more specialized ones. For multi-source pools, PCCL clusters examples with character n-gram TF–IDF features and KMeans and builds batches that mix examples across domains, so that each update draws on multiple sources. Experiments on single-source and multi-source distillation with Qwen and Llama students show that PCCL consistently improves ROUGE-L over random batching and reaches strong performance earlier in training.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Curriculum Learning, Knowledge Distillation, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PAoCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Principal-Component Curriculum Learning (PCCL) for knowledge distillation. It uses gradients from a frozen teacher to build a representation of relationships between training examples. Batches are ordered by their average normalized alignment with the leading principal direction of this representation. For multi-source data, the method also uses TF–IDF and KMeans to construct batches containing examples from different groups. Experiments on Qwen and Llama students report improvements in aggregate ROUGE-L over random training and several curriculum baselines. The paper also includes a component ablation, a convergence plot, and an analysis of batch size.

Critical strengths
  1. The paper separates two useful experimental choices. Treating batch membership and batch order separately makes the method easy to understand and allows their effects to be tested. Table 4 takes a useful first step by removing each component separately.
  2. The empirical evaluation goes beyond a single model or random baseline. The paper includes two model families, both single-source and multi-source settings, and several relevant curriculum and batching baselines. Reporting five-seed means and standard deviations in the main result tables is also a strength. These results provide initial evidence that the method can improve aggregate performance.
Critical weaknesses
  1. The technical contribution appears incremental, and the value of the specific scoring rule is not sufficiently established. Gradient-based data organization, curriculum ordering, and diverse batch construction all have substantial prior work, including methods discussed in the paper itself. The main additional idea appears to be the use of normalized alignment with the first principal direction as a batch-ordering score, combined with text-based grouping. A simple method can be valuable. My concern is that the experiments do not show why this particular score is preferable to simpler alternatives. For example, the Qwen single-source gain is 2.16 ROUGE-L over random, but only 0.55 over RCL. In Table 4, cross-group batching alone raises SQuAD from 70.0 to 71.5, while adding the proposed ordering raises it further to 71.8. This does not establish that the principal-component score is the main source of the improvement. To clarify its contribution, please compare the proposed score with random ordering, reversed score ordering, teacher-loss ordering, and gradient-norm ordering while keeping batch membership and the policy for reshuffling between epochs identical. For multi-source data, a simple baseline that balances the known data sources within each batch would also help establish whether text clustering adds value. Please report uncertainty for these comparisons.
  2. The explanation that high-scoring examples contain more general knowledge is not supported by direct evidence. The leading principal direction captures the largest variation in the teacher’s relational representation. This does not by itself establish that examples aligned with it contain more general knowledge or should be learned earlier. The direction could instead reflect differences in data source, response length, formatting, or gradient scale. In addition, relationships between teacher gradients need not match relationships between the student’s distillation gradients. The paper acknowledges that this is a curriculum prior. However, the interpretation of “shared information first, specialized information later” remains central to its motivation and conclusions. Removing the ordering component only tests whether that component helps; it does not explain why it helps. Please examine how the score relates to data source, response length, teacher loss, and gradient norm, and test whether its advantage remains after controlling for these factors. Comparisons with several random directions and other principal directions would help determine whether the leading direction is important. On a subset of examples, measuring student-gradient compatibility or transfer across held-out tasks could directly test the proposed explanation. If such evidence is unavailable, the claims should be narrowed to an empirical ordering heuristic rather than a demonstrated progression from general to specialized knowledge.
Questions for authors
  1. In Table 4, do the full method and the version without ordering use exactly the same batches? Is the batch order fixed or reshuffled across epochs in each condition, and are the reported ablations averaged over the same five seeds?
  2. What evidence supports interpreting high principal-loading scores as more general knowledge? Have you checked whether the ordering mainly reflects source identity, response length, teacher loss, or gradient norm?
Reviewer_9cukOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PCCL, a framework for organizing training examples during KD. The authors argue that existing curriculum-based KD primarily focuses on the order in which examples are presented, while ignoring which examples should be grouped together in the same batch during training, specifically when data come from different domains. PCCL computes per-example teacher gradients and uses principal-component analysis of their relationships to rank examples according to their alignment with the dominant gradient structure. Second, for multi-source datasets, it applies TF-IDF and KMeans clustering to construct batches containing examples from different groups. The resulting batches are ordered by their average principal-component score and used for standard knowledge distillation.

Critical strengths
  1. The paper considers both the order in which training examples are presented and which examples are combined within the same batch. This is a useful perspective for knowledge distillation in particular and training in general, where we have different sources of training data.

  2. The paper evaluates PCCL against several existing curriculum-learning and batch-selection methods.

Critical weaknesses
  1. The improvements of the method are relatively small. In the multi-source setting (Table 1), PCCL achieves 43.83 ROUGE-L compared with 43.08 for DPP and 43.05 for random batching. Furthermore, Table 4 shows that the full method does not consistently outperform its individual components. For example, on SAMSum, removing the ordering component improves performance from 39.0 to 39.1. These results raise questions about the practical benefit and complementarity of the proposed components.

  2. The paper does not include a full computational cost analysis of the proposed method to assess its efficiency compared to the competing approaches.

Questions for authors
  1. See weaknesses
  2. Can the authors evaluate PCCL with different training dataset sizes to determine whether its benefits persist as the data pool grows?
  3. Can the authors further clarify the novelty of this work?
Reviewer_daQQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Existing distillation curriculum decide only the order of each training example, typically by difficulty, even though distillation pools mix tasks and sources. This paper argues that which examples share a batch also matters. Based on this, it proposes PCCL: examples are scored by their loading on the dominant direction of the teacher's per-example gradient structure, high-scoring batches go first, and for multi-source pools a text-feature clustering step forces clusters to be mixed within a batch. Distilling 8B Qwen and Llama teachers into sub-1B students on 4,000-example pools, it beats random batching on pooled ROUGE-L by about 1 to 2 points.

Critical strengths
  1. The research question is practical, as knowledge distillation is widely used nowadays and studying the curriculum of training helps the general understanding of KD for the community.
  2. The proposed method is intuitive, straightforward and also somewhat efficient, considering the ordering signal is computed once from a frozen teacher and reused across student runs.
Critical weaknesses
  1. For me the biggest weakness of this paper is the gain is not sufficient, and the per-component evidence is thinner than the framing suggests. Outside single-source Qwen the margin over random is 0.78, 0.85 and 0.94 points, i.e. on the order of the seed standard deviations, with no significance tests. In the multi-source ablation (Table 4, reported without stds or an aggregate column), averaging the seven columns gives 38.3 for random, 38.6 for ordering alone, 38.8 for composition alone and 39.1 for full PCCL, so each component buys a few tenths, and ordering actually hurts GSM8K and SAMSum. The only substantial gain, single-source Qwen, is a setting where composition is not used at all.
  2. Evaluation and scale. ROUGE-L is used for GSM8K and code, and under GSM8K accuracy PCCL shows no gain (45.48 vs. 45.52 for random). StrategyQA sits between 48% and 52% for every method, i.e. chance for a binary task, and the dataset is not described in App.
Questions for authors

How much of the single-source gain survives against a fixed random order replayed every epoch, and what does the reversed order do?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.