acceptodds
Under review as a conference paper at ICLR 2027

Dataset Alignment Over Diversity

Abstract

Scaling pre-training data volume has become the industry standard for improving language model performance, yet this approach is computationally wasteful when the goal is strong downstream performance on a specific task, especially when models are often pre-trained on diverse datasets rather than specialized training on datasets with overlapping domain coverage. We challenge this paradigm and propose Dataset Alignment Over Diversity (DAOD), a less compute dependent framework that predicts how well a pre-training dataset will transfer to a target downstream task without requiring any model training. DAOD scores candidates using a closed-form function of three geometric properties extracted from a randomly initialized model: centroid alignment between candidate and target embeddings, alignment friction (variance), and intra-dataset diversity. Across a benchmark of 8 candidate datasets evaluated against a mathematical reasoning target (OpenWebMath) and a second target validated across three architectures (Gemma-3 1B, LLaMA-3.2 1B, LFM-1.2B), DAOD reliably sorts candidates into the correct pre-training tier, capturing 95–98% of the achievable downstream performance gain while using only 25–38% of the total pre-training compute. The underlying score also shows strong rank correlation with ground-truth fine-tuning loss (maximum Spearman = 0.93 on OpenWebMath), though we note this correlation is computed over a modest number of candidates and is best interpreted alongside the tiering results, which are more robust to individual rank swaps. Together these results establish that data alignment, not volume, is the primary driver of efficient transfer.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Dataset pruning, Transferability Metric, Compute Efficiency, Transfer learning, Pre-training, Zero-shot dataset ranking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RiKxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Dataset Alignment Over Diversity proposes DAOD, a training-free approach to screening pretraining datasets for a known downstream target. A randomly initialized instance of the intended model architecture embeds samples from each candidate dataset and the target. The score combines cosine similarity between their centroids, the variance of sample-to-target similarities, and the standard deviation of pairwise similarities within each candidate dataset. It then groups candidate sources into three priority tiers for pretraining.

The language experiments consider eight candidate datasets for each of OpenWebMath and XSum, with reference rankings obtained by pretraining on each candidate and then fine-tuning on the target. On OpenWebMath with Gemma-3-1B-IT, DAOD achieves a Spearman correlation of 0.929 and selects NuminaMath as the top source. CC-News is reported as the best source selected for XSum across three architectures. Tiered mixture experiments evaluate the resulting selection strategy, while a preliminary ViT-B/16 experiment explores transfer to vision using only the alignment-minus-friction score.

Critical strengths
  1. Screening data before investing in pretraining is a useful problem, and DAOD offers a straightforward approach that is easy to inspect and implement. Equations 1 through 4 need only a randomly initialized model, forward passes, and basic statistics, with no gradient updates. Grouping sources into broad tiers also seems sensible when exact rankings may be noisy.

  2. The strongest empirical aspect is that the score is tested through training on selected mixtures, not just its correlation with a reference ranking. Table 1 provides the score components and empirical rank for every OpenWebMath candidate, Figures 1 and 2 show loss trajectories, and Table 3 evaluates progressively larger mixtures. The large loss reductions relative to no pretraining suggest that screening can be useful under this protocol, although the reported gain percentages need correction.

  3. I appreciate the attention to failure cases, which makes the limits of the approach easier to assess. Table 5 shows that diversity helps the OpenWebMath ranking but hurts the XSum ranking. Section 5 reports a severe failure on Alpaca involving HH-RLHF, where the best empirical source is ranked seventh. The ViT-B/16 experiment is explicitly preliminary and excludes the diversity term, rather than being presented as full cross-modal validation of the complete score.

Critical weaknesses
  1. The loss results are promising as evidence about optimization, but their implications for downstream generalization remain unclear. Section 3 defines transfer quality through fine-tuning loss after 500 pretraining steps and 500 fine-tuning steps, without saying whether this is training loss or loss on held-out examples. It is also unclear whether the examples used to construct the target centroid overlap with fine-tuning or evaluation data. No task metrics such as mathematical exact match or summarization ROUGE are reported either. I therefore cannot tell whether the observed ranking predicts held-out downstream utility.

  2. The savings are difficult to interpret because the gain percentages in Table 3 do not use a consistent denominator and the compute estimates are only a proxy. If the best OpenWebMath loss of 4.524 for Tier 1 plus 2 defines the full attainable gain, Tier 1 captures 92.46% rather than 95%. If the all-tier loss of 4.570 is used instead, Tier 1 captures 94.69%, but Tier 1 plus 2 captures 102.42% rather than 100%. A single definition would clarify how these rows should be compared. The compute percentages are proportional to dataset counts under equal step counts, rather than measured cost. Batch sizes, effective token counts, mixture construction, and GPU time or FLOPs are needed to assess comparable budgets and practical savings.

  3. A user applying DAOD to a new target still needs a way to choose the diversity weight before transfer outcomes are known. In Table 5, diversity raises the OpenWebMath correlation from 0.810 to 0.929 but lowers the XSum correlation from 0.786 to 0.619. Sections 2, 4, and 5 acknowledge that alpha should depend on target breadth and that there is no automatic rule. The Alpaca result shows why this matters, with HH-RLHF achieving the best empirical loss of 0.842 yet receiving a score of 0.589 and rank 7 of 8. Adding Tier 3 also improves XSum in Table 3, limiting a general recommendation to discard low-scoring sources. The open question is therefore how to decide prospectively between alpha equal to zero and alpha equal to one, not whether the reported choices were made after seeing the outcomes.

Questions for authors
  1. Are the losses in Tables 1, 3, and 4 training losses or held-out validation or test losses, and over which examples or steps are they averaged? How are target examples split among centroid construction, fine-tuning, and evaluation? Held-out task metrics would be helpful if available.

  2. Which single definition and denominator should be used for all OpenWebMath gain percentages in Table 3? How are the tier mixtures constructed and scheduled? Could you report batch sizes, effective token counts, and measured training costs for each row so that the savings can be compared on a consistent basis?

  3. How many independent scoring and training seeds were used, and how stable are the close candidate ranks and tier assignments? I would like to see uncertainty in scores and ranks, along with a matched comparison against unigram token KL covering both scoring cost and uncertainty in the correlation difference.

Reviewer_NsAyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to select candidate datasets for pre-training so that the final model performs well on a single target task. The proposed method, DAOD, is a simple, training-free method to rank available datasets. Each sample is expressed by the average of the final-layer outputs of a randomly initialized model, and the mean vector over 500 samples represents each dataset. The method's score combines three metrics per candidate dataset: alignment, the cosine similarity between its vector and that of the target; friction, how consistently its samples resemble the target; and diversity, how varied its samples are among themselves. They evaluate their method in two settings, with OpenWebMath and XSum as targets, each with eight candidate datasets, mostly small benchmark or fine-tuning datasets. The ranking is checked against the average loss on the target of a randomly initialized 1B model trained on each candidate dataset for 500 steps and then fine-tuned on the target for another 500 steps. On OpenWebMath, the score's ranking matches the loss ranking about as well as simply comparing token frequencies (rank correlation 0.929 vs 0.905); on XSum the correlation is lower (0.62). They also pre-train on several candidates together, and, by their measure, using only the top 2–3 datasets gives a loss close to using all of them.

Critical strengths

S1: Pre-training dataset selection is an important practical problem and the method is very cheap and simple.

Critical weaknesses

W1: The experiments cannot support the paper’s claims about pre-training data selection. Most candidates are small benchmark or fine-tuning datasets, and “pre-training” used is only 500 steps from a randomly initialized model (tokens used not reported, see Q1). Also, the differences among candidates mostly disappear after fine-tuning (as expected after such short pre-training), shrinking to about 0.3 overall and 0.1 within the top four candidates (Figure 1). So the ground truth is not reliable and the results cannot be transferred to pre-training choices (see Q2).

W2: The method is not shown to be better than other simple choices, even with the paper’s measurements. Its top choice is the same as the heuristic one from dataset names and descriptions alone (math data for the math target, news for the news target). On OpenWebMath, a simple token-frequency comparison does almost as well (0.929 vs 0.905), and with the setting the paper recommends by default (no diversity term) the method does worse (0.81). These baselines are only shown for OpenWebMath (Table 2). It is also not compared with existing data selection methods such as DSIR (Xie et al., 2023), which selects data for a target from n-gram counts without training (see Q3).

Questions for authors

Clarifications:

Q1: What batch size is used for the short pre-training and for fine-tuning, i.e. how many tokens are used in total?

Q2: Is the "average loss" the training loss over fine-tuning, or the loss on held-out data? Does the ranking change if you use the final loss instead, as in the vision experiment?

Q3: The alignment values in Table 1 look like low-precision (bfloat16) numbers, and the top three scores differ by less than one rounding step. Could their order come from rounding? Does the ranking hold with other random seeds?

Q4: In Table 3, if Tier 1+2 counts as 100%, Tier 1 on OpenWebMath is 92%, not 95%. Could you check? Also, Sec. 2.1 says adding Tier 3 always hurts, but on XSum it helps.

Reviewer_uwtEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DAOD, a training-free score for predicting the usefulness of a candidate pre-training dataset for a target downstream task. It extracts embeddings from a randomly initialized model and combines centroid alignment, the variance of sample-to-target similarities, and intra-dataset diversity. The score is evaluated against fine-tuning loss after short pre-training runs on 1B-scale models, primarily using OpenWebMath and XSum as targets. The authors report strong ranking correlation on OpenWebMath and propose a tiered selection strategy that captures most of the measured improvement with a fraction of the pre-training compute.

Critical strengths
  1. A practically relevant question and a simple approach. Estimating dataset usefulness before committing training compute is valuable. The proposed score is straightforward and requires no parameter updates. The paper also candidly discusses limitations, including format sensitivity and the inconsistent contribution of the diversity term across targets.
Critical weaknesses
  1. The training budget is too limited to support the broader conclusions. For the individual-candidate rankings, each randomly initialized 1B-scale model receives 500 pre-training steps with a batch size of 5, followed by 500 fine-tuning steps. It is unclear whether rankings measured this early remain predictive after substantial training. The paper provides no evidence that the rankings remain stable as the budget increases, yet draws broad conclusions about alignment versus data volume and pre-training compute savings.
  2. The main results measure training loss, and the selection comparisons lack key controls. In the supplementary code, “average loss” is the average over batches used for target fine-tuning. I could not find a held-out evaluation supporting the claims about mathematical reasoning or summarization performance. The tiered experiment trains on datasets sequentially: adding Tier 3 changes the data, the number of updates, and which data the model sees last. The resulting loss increase cannot distinguish harmful data from an order or forgetting effect. A simple token-KL baseline already obtains a correlation of 0.905 compared with DAOD’s 0.929 on OpenWebMath, so a reliable advantage over token-KL is not established.
Questions for authors
  1. The rankings are measured at a single, very short training budget and evaluated through training loss on the fine-tuning batches. Is there evidence from longer training or held-out evaluation that the rankings remain predictive and that the reported gains translate into reasoning or summarization performance?
  2. No equal-budget comparison against random or token-KL selection with mixed training data is reported. Do the claimed compute savings and the apparent harm of low-scoring data persist under these controls?

The research question is relevant, but the main claims rely on very short training runs, training-loss measurements, and selection experiments that lack key controls. These issues affect the central empirical evidence, so I do not support acceptance in the current form.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.