acceptodds
Under review as a conference paper at ICLR 2027

Does Neighbour Relatedness Matter? Dissimilarity-Priority Packing Contradicts the Similarity Hypothesis in Supervised Fine-Tuning

Abstract

Sequence packing—concatenating multiple fine-tuning examples into one fixed-length training sequence—is standard practice for supervised fine-tuning (SFT) of large language models. A recent line of work argues that pack composition matters and, specifically, that packs of related examples (high embedding similarity) improve downstream quality: Threshold Filtering Packing (TFP) reports gains from packing semantically similar neighbours together. We test this similarity hypothesis directly by proposing and evaluating the opposite policy, dissimilarity-priority packing (DPP), which greedily fills each pack with mutually dissimilar examples via farthest-point selection over example embeddings. Under a matched harness (same model, data, optimizer, step count, and an exactly matched token budget across all policies), we train Qwen3-0.6B and Qwen3-1.7B with four composition policies spanning the full similarity spectrum: DPP (dissimilar), random packing, TFP (threshold-similar), and a nearest-neighbour homogeneous control (HNP). Contrary to the similarity hypothesis, held-out response NLL is a monotone function of realized intra-pack similarity: DPP achieves the lowest NLL (1.5172±0.0014), random is next (1.5189±0.0005), a faithful re-implementation of TFP with its diversity filter intact sits mid-axis at 1.5245±0.0031, and both homogeneous policies (a nearest-neighbour control and an inverted-constraint variant that removes TFP’s diversity mechanism) are worst (1.5395–1.5413, a 0.024-nat gap to DPP). The harmful end of the axis is extreme homogeneity, and TFP’s “maintain sufficient diversity” mechanism is load-bearing: removing it triples the penalty. A block-diagonal attention control shows the effect is entirely mediated by cross-example attention: when leak is masked, all composition policies collapse to statistically indistinguishable quality. Composition choice is thus a real, measurable training signal—but its direction is opposite to the one the similarity hypothesis predicts, and its damage concentrates at the homogeneous extreme. We release our harness and discuss implications for packing-method design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sequence packing, supervised fine-tuning, TFP, curriculum, attention leak, SFT

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7P6KOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The author puts forward a completely opposite strategy to the previous paper - dissimilarity priority packing. DPP selects the farthest point on sample embedding, and fills each packet with samples that are not similar to each other. The author's contributions hope to use this completely opposite method to illustrate the previous work's idea that examples with similar semantics should be packaged. The authors released their harness and discussions implications for packaging method design. The authors trained on qwen3-0.6b and qwen3-1.7b, using alpaca-52k as the dataset.

Critical strengths

A strength is the block-diagonal attention ablation, which provides a useful mechanistic control. The disappearance of the policy ordering when cross-example attention is masked supports the interpretation that the observed composition effects depend on attention interactions in the tested setting.

Critical weaknesses
  1. The paper’s main weakness is that its experiments do not sufficiently support the authors’ claims. The authors seek to challenge the following argument from prior work:

    Compared with random packing, constructing training sequences from semantically related documents can provide more informative cross-document dependencies and reduce irrelevant-context interference.

Papers that argue for this view include, but are not limited to:

  • In-Context Pretraining: Language Modeling Beyond Document Boundaries (Shi et al., ICLR 2024)
  • Analysing The Impact of Sequence Composition on Language Model Pre-Training (Zhao et al., ACL 2024)
  • SPLiCe: Structured Packing in LLM Training Improves Long Context Utilization (Staniszewski et al., AAAI 2025)
  • Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs (Dong et al., NAACL 2025)
  • LinkBERT: Pretraining Language Models with Document Links (Yasunaga et al., ACL 2022)
  • SkyLadder: Better and Faster Pretraining via Context Window Scheduling (Zhu et al., NeurIPS 2025)

However, the experimental setup is insufficient to refute this argument. The primary concern is the use of Alpaca-52k. Earlier models, such as Llama 2, showed substantial improvements in instruction-following ability when fine-tuned on Alpaca-52k. Is there a comparable justification for fine-tuning Qwen3 on Alpaca-52k? The authors should establish whether this training provides meaningful benefits to Qwen3 before drawing broader conclusions about packing strategies.

  1. Furthermore, Random and TFP achieve the highest MMLU-mini accuracy, whereas DPP ranks second from the bottom among all methods. These results do not provide convincing evidence for DPP’s claimed advantage.
Questions for authors
  1. I'm curious. How far has SFT packaging gone now? I know that using pack in SFT two years ago is a novel and valuable method. It was meaningful to use a method that did not mask. But by 2026, the sequence packaging in SFT has been significantly upgraded. In particular, the scheme of using the attention mask to isolate different samples can now achieve almost no additional attention calculation overhead, and will not necessarily cause the quality of the model to decline. Is it valuable for the author to study this topic at this time?
  2. In addition, shouldn't LLM judge be the most normal Eval method for SFT alpaca-52k? Why does the author not use this method?
Reviewer_6n15OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is an empirical study of which examples should share a training pack. On Alpaca with two Qwen3 model sizes, it compares dissimilarity-priority, random, filtered-similarity, and homogeneous packing, then uses block-diagonal attention to investigate the poorer response NLL observed for extreme homogeneity.

Critical strengths

Pack composition is often treated as an implementation detail, so it is useful to study it directly. Two independently constructed homogeneous controls produce similar within-pack similarities and losses; filtered TFP falls between these controls and random packing (Table 1). Blocking cross-example attention reduces the NLL range across the four shared policies from 0.024 to 0.007 nats and changes their ordering (Tables 1 and 3). This is a worthwhile diagnostic, clearly presented in Figure 1. I also appreciate the restraint in treating DPP's small advantage over random as suggestive and making no claim of downstream improvement.

Critical weaknesses
  1. The clearest result overlaps with what motivated TFP in the first place. Dong et al. (2025) already report that packing related samples without their diversity mechanism performs worse. The more interesting finding here is that even filtered TFP loses its advantage. DPP itself offers only about 0.002 nats over random at 0.6B, and less at 1.7B, so the case for adopting it is still weak.
  2. The paper stops short of explaining this reversal or showing when to expect it. The 1.7B experiment repeats DPP/random/TFP, while the homogeneous controls and masking experiment are confined to 0.6B. Packs are fixed, and results are measured after 500 steps. It is therefore hard to tell whether the ordering persists or depends on this particular construction and stage of training. The mask experiment implicates cross-example attention, but does not explain why related context hurts. With NLL the only outcome showing a clear difference, I wanted a clearer account of either the conditions under which the finding holds or the reason it occurs.
Questions for authors

Do the existing checkpoints show when filtered TFP falls behind random packing, and whether that ordering persists? In Section 6.1, does the multiset check cover the examples actually visited within 2,000 packs, or the complete constructed partition?

Reviewer_dfHGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether semantic similarity among co-packed examples improves supervised fine-tuning. The authors propose Dissimilarity-Priority Packing (DPP), which greedily groups dissimilar examples using embedding-based farthest-point selection. Experiments on Qwen3-0.6B and Qwen3-1.7B suggest that dissimilar packing achieves lower held-out NLL than random and similarity-based packing. The authors further argue that this effect is mediated by cross-example attention.

Critical strengths

This paper investigates a practically relevant question in SFT that has received relatively limited systematic investigation.

Critical weaknesses
  1. The paper was extremely poorly written and confusing to read. The proposed method is overly simplistic and has no motivation.
  2. The experiments are extremely limited in terms of the size of the model and the scale of the data. The improvement of DPP over random packing seems to result in only a 0.0018 NLL on the 0.6B model. I'm not sure if this is an improvement or just due to the random seed.
Questions for authors

I do not have any questions for the authors. This paper does not meet the bar for ICLR and requires significant revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.