- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Principal-Component Curriculum Learning (PCCL) for knowledge distillation. It uses gradients from a frozen teacher to build a representation of relationships between training examples. Batches are ordered by their average normalized alignment with the leading principal direction of this representation. For multi-source data, the method also uses TF–IDF and KMeans to construct batches containing examples from different groups. Experiments on Qwen and Llama students report improvements in aggregate ROUGE-L over random training and several curriculum baselines. The paper also includes a component ablation, a convergence plot, and an analysis of batch size.
Critical strengths
- The paper separates two useful experimental choices. Treating batch membership and batch order separately makes the method easy to understand and allows their effects to be tested. Table 4 takes a useful first step by removing each component separately.
- The empirical evaluation goes beyond a single model or random baseline. The paper includes two model families, both single-source and multi-source settings, and several relevant curriculum and batching baselines. Reporting five-seed means and standard deviations in the main result tables is also a strength. These results provide initial evidence that the method can improve aggregate performance.
Critical weaknesses
- The technical contribution appears incremental, and the value of the specific scoring rule is not sufficiently established. Gradient-based data organization, curriculum ordering, and diverse batch construction all have substantial prior work, including methods discussed in the paper itself. The main additional idea appears to be the use of normalized alignment with the first principal direction as a batch-ordering score, combined with text-based grouping. A simple method can be valuable. My concern is that the experiments do not show why this particular score is preferable to simpler alternatives. For example, the Qwen single-source gain is 2.16 ROUGE-L over random, but only 0.55 over RCL. In Table 4, cross-group batching alone raises SQuAD from 70.0 to 71.5, while adding the proposed ordering raises it further to 71.8. This does not establish that the principal-component score is the main source of the improvement. To clarify its contribution, please compare the proposed score with random ordering, reversed score ordering, teacher-loss ordering, and gradient-norm ordering while keeping batch membership and the policy for reshuffling between epochs identical. For multi-source data, a simple baseline that balances the known data sources within each batch would also help establish whether text clustering adds value. Please report uncertainty for these comparisons.
- The explanation that high-scoring examples contain more general knowledge is not supported by direct evidence. The leading principal direction captures the largest variation in the teacher’s relational representation. This does not by itself establish that examples aligned with it contain more general knowledge or should be learned earlier. The direction could instead reflect differences in data source, response length, formatting, or gradient scale. In addition, relationships between teacher gradients need not match relationships between the student’s distillation gradients. The paper acknowledges that this is a curriculum prior. However, the interpretation of “shared information first, specialized information later” remains central to its motivation and conclusions. Removing the ordering component only tests whether that component helps; it does not explain why it helps. Please examine how the score relates to data source, response length, teacher loss, and gradient norm, and test whether its advantage remains after controlling for these factors. Comparisons with several random directions and other principal directions would help determine whether the leading direction is important. On a subset of examples, measuring student-gradient compatibility or transfer across held-out tasks could directly test the proposed explanation. If such evidence is unavailable, the claims should be narrowed to an empirical ordering heuristic rather than a demonstrated progression from general to specialized knowledge.
Questions for authors
- In Table 4, do the full method and the version without ordering use exactly the same batches? Is the batch order fixed or reshuffled across epochs in each condition, and are the reported ablations averaged over the same five seeds?
- What evidence supports interpreting high principal-loading scores as more general knowledge? Have you checked whether the ordering mainly reflects source identity, response length, teacher loss, or gradient norm?