- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The author puts forward a completely opposite strategy to the previous paper - dissimilarity priority packing. DPP selects the farthest point on sample embedding, and fills each packet with samples that are not similar to each other. The author's contributions hope to use this completely opposite method to illustrate the previous work's idea that examples with similar semantics should be packaged. The authors released their harness and discussions implications for packaging method design. The authors trained on qwen3-0.6b and qwen3-1.7b, using alpaca-52k as the dataset.
Critical strengths
A strength is the block-diagonal attention ablation, which provides a useful mechanistic control. The disappearance of the policy ordering when cross-example attention is masked supports the interpretation that the observed composition effects depend on attention interactions in the tested setting.
Critical weaknesses
-
The paper’s main weakness is that its experiments do not sufficiently support the authors’ claims. The authors seek to challenge the following argument from prior work:
Compared with random packing, constructing training sequences from semantically related documents can provide more informative cross-document dependencies and reduce irrelevant-context interference.
Papers that argue for this view include, but are not limited to:
- In-Context Pretraining: Language Modeling Beyond Document Boundaries (Shi et al., ICLR 2024)
- Analysing The Impact of Sequence Composition on Language Model Pre-Training (Zhao et al., ACL 2024)
- SPLiCe: Structured Packing in LLM Training Improves Long Context Utilization (Staniszewski et al., AAAI 2025)
- Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs (Dong et al., NAACL 2025)
- LinkBERT: Pretraining Language Models with Document Links (Yasunaga et al., ACL 2022)
- SkyLadder: Better and Faster Pretraining via Context Window Scheduling (Zhu et al., NeurIPS 2025)
However, the experimental setup is insufficient to refute this argument. The primary concern is the use of Alpaca-52k. Earlier models, such as Llama 2, showed substantial improvements in instruction-following ability when fine-tuned on Alpaca-52k. Is there a comparable justification for fine-tuning Qwen3 on Alpaca-52k? The authors should establish whether this training provides meaningful benefits to Qwen3 before drawing broader conclusions about packing strategies.
- Furthermore, Random and TFP achieve the highest MMLU-mini accuracy, whereas DPP ranks second from the bottom among all methods. These results do not provide convincing evidence for DPP’s claimed advantage.
Questions for authors
- I'm curious. How far has SFT packaging gone now? I know that using pack in SFT two years ago is a novel and valuable method. It was meaningful to use a method that did not mask. But by 2026, the sequence packaging in SFT has been significantly upgraded. In particular, the scheme of using the attention mask to isolate different samples can now achieve almost no additional attention calculation overhead, and will not necessarily cause the quality of the model to decline. Is it valuable for the author to study this topic at this time?
- In addition, shouldn't LLM judge be the most normal Eval method for SFT alpaca-52k? Why does the author not use this method?