- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Dataset Alignment Over Diversity proposes DAOD, a training-free approach to screening pretraining datasets for a known downstream target. A randomly initialized instance of the intended model architecture embeds samples from each candidate dataset and the target. The score combines cosine similarity between their centroids, the variance of sample-to-target similarities, and the standard deviation of pairwise similarities within each candidate dataset. It then groups candidate sources into three priority tiers for pretraining.
The language experiments consider eight candidate datasets for each of OpenWebMath and XSum, with reference rankings obtained by pretraining on each candidate and then fine-tuning on the target. On OpenWebMath with Gemma-3-1B-IT, DAOD achieves a Spearman correlation of 0.929 and selects NuminaMath as the top source. CC-News is reported as the best source selected for XSum across three architectures. Tiered mixture experiments evaluate the resulting selection strategy, while a preliminary ViT-B/16 experiment explores transfer to vision using only the alignment-minus-friction score.
Critical strengths
-
Screening data before investing in pretraining is a useful problem, and DAOD offers a straightforward approach that is easy to inspect and implement. Equations 1 through 4 need only a randomly initialized model, forward passes, and basic statistics, with no gradient updates. Grouping sources into broad tiers also seems sensible when exact rankings may be noisy.
-
The strongest empirical aspect is that the score is tested through training on selected mixtures, not just its correlation with a reference ranking. Table 1 provides the score components and empirical rank for every OpenWebMath candidate, Figures 1 and 2 show loss trajectories, and Table 3 evaluates progressively larger mixtures. The large loss reductions relative to no pretraining suggest that screening can be useful under this protocol, although the reported gain percentages need correction.
-
I appreciate the attention to failure cases, which makes the limits of the approach easier to assess. Table 5 shows that diversity helps the OpenWebMath ranking but hurts the XSum ranking. Section 5 reports a severe failure on Alpaca involving HH-RLHF, where the best empirical source is ranked seventh. The ViT-B/16 experiment is explicitly preliminary and excludes the diversity term, rather than being presented as full cross-modal validation of the complete score.
Critical weaknesses
-
The loss results are promising as evidence about optimization, but their implications for downstream generalization remain unclear. Section 3 defines transfer quality through fine-tuning loss after 500 pretraining steps and 500 fine-tuning steps, without saying whether this is training loss or loss on held-out examples. It is also unclear whether the examples used to construct the target centroid overlap with fine-tuning or evaluation data. No task metrics such as mathematical exact match or summarization ROUGE are reported either. I therefore cannot tell whether the observed ranking predicts held-out downstream utility.
-
The savings are difficult to interpret because the gain percentages in Table 3 do not use a consistent denominator and the compute estimates are only a proxy. If the best OpenWebMath loss of 4.524 for Tier 1 plus 2 defines the full attainable gain, Tier 1 captures 92.46% rather than 95%. If the all-tier loss of 4.570 is used instead, Tier 1 captures 94.69%, but Tier 1 plus 2 captures 102.42% rather than 100%. A single definition would clarify how these rows should be compared. The compute percentages are proportional to dataset counts under equal step counts, rather than measured cost. Batch sizes, effective token counts, mixture construction, and GPU time or FLOPs are needed to assess comparable budgets and practical savings.
-
A user applying DAOD to a new target still needs a way to choose the diversity weight before transfer outcomes are known. In Table 5, diversity raises the OpenWebMath correlation from 0.810 to 0.929 but lowers the XSum correlation from 0.786 to 0.619. Sections 2, 4, and 5 acknowledge that alpha should depend on target breadth and that there is no automatic rule. The Alpaca result shows why this matters, with HH-RLHF achieving the best empirical loss of 0.842 yet receiving a score of 0.589 and rank 7 of 8. Adding Tier 3 also improves XSum in Table 3, limiting a general recommendation to discard low-scoring sources. The open question is therefore how to decide prospectively between alpha equal to zero and alpha equal to one, not whether the reported choices were made after seeing the outcomes.
Questions for authors
-
Are the losses in Tables 1, 3, and 4 training losses or held-out validation or test losses, and over which examples or steps are they averaged? How are target examples split among centroid construction, fine-tuning, and evaluation? Held-out task metrics would be helpful if available.
-
Which single definition and denominator should be used for all OpenWebMath gain percentages in Table 3? How are the tier mixtures constructed and scheduled? Could you report batch sizes, effective token counts, and measured training costs for each row so that the savings can be compared on a consistent basis?
-
How many independent scoring and training seeds were used, and how stable are the close candidate ranks and tier assignments? I would like to see uncertainty in scores and ranks, along with a matched comparison against unigram token KL covering both scoring cost and uncertainty in the correlation difference.