- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether data–task alignment can serve as a measurable third scaling axis of supervised fine-tuning loss alongside model size and training token budget. The authors develop an alignment score utilizing both SBERT similarity and ZIP-FIT compression similarity to rank training examples relative to a held-out target distribution. Experiments suggest that alignment benefits recur across settings.
Critical strengths
- The paper addresses a practically relevant question beyond demonstrating that data selection helps. Relating data relevance to model size and training budget is a worthwhile direction for model performance prediction.
- The primary experiment includes meaningful controlled comparisons. Fixed-token comparisons and variation in model size and token budget help distinguish composition effects from resource scaling.
- Component ablations, decontamination checks, and replication across targets and model families strengthen the study. The paper also acknowledges several important limitations.
Critical weaknesses
- Quantile selection can simultaneously change source composition, response length, diversity, etc. The paper lacks controlled comparisons between datasets with similar and token budgets but different compositions.
- Prior work already models performance under varying data composition. The paper should clarify what additional predictive capability is provided by the proposed alignment coordinate beyond these existing formulations.
- The Llama and cross-domain experiments primarily demonstrate selection benefits rather than transfer of the scaling law. Whether a calibrated model can accurately predict unseen configurations across model families and domains remains unresolved.
- Equation (1) assumes that alignment contributes an additive loss reduction with a coefficient independent of model size and token budget. Beyond describing this form as a first-order approximation, the paper provides little theoretical justification for these assumptions.
Questions for authors
Please address the concerns raised in the Weakness part above.
2 comments
I appreciate the authors' efforts. My major concerns have been addressed. This work falls around the acceptance threshold in my view, and I leave the final decision to the AC.
Author rebuttal to Reviewer 7MSi
We thank the reviewer for highlighting the distinction between predictive measurement and causal attribution. The revised paper sharpens this distinction and adds complete resource holdouts to test the predictive claim.
W1 response: The revision now reports the main distribution shifts. At 21M tokens, Q90 averages 280 tokens per example versus 368 for Random. NuminaMath contributes 46.9% of Q90 versus 85.9% of Random, OrcaMath contributes 24.6% versus 4.2%, and MetaMath contributes 18.2% versus 2.5%. The fixed-token intervention controls total training tokens and the trainer. α is a predictive summary of target relevance in the resulting distribution, not a causal decomposition of source, length, and diversity.
That predictive claim survives complete holdouts. The alignment law reduces leave-one-model-size-out RMSE from 0.0557 to 0.0321 and leave-one-token-budget-out RMSE from 0.0405 to 0.0200. The resource-multiplier alternative reaches 0.0473 and 0.0288, respectively. Alignment therefore adds predictive information under scale and budget shifts even though it compresses several properties of the selected distribution.
W2 response: Mixture laws model performance from named-source proportions. Our coordinate is target-relative and measured at the example level before downstream training, so it can compare examples within a source and subsets assembled from different sources. We then study its joint response with model size and fine-tuning tokens. The approaches are complementary: source weights describe how a dataset was assembled, while α measures how the realized dataset relates to a target. The revision sharpens this distinction without implying that mixture composition is unimportant.
W3 response: We do not present them as proof that the complete three-axis law transfers unchanged. Llama tests whether the fixed-token selection advantage survives a model-family change. Code tests target-relative selection and a four-size fixed-token candidate-law comparison. The independent math target provides the cleanest refit. After rescoring and retraining, its 16-configuration additive model reaches RMSE 0.0058 and remains BIC-best. The frozen gradient-selector study identifies the boundary: alignment remains predictive at every tested size, but its coefficient becomes size-dependent. This supports a transferable coordinate with regime-dependent calibration, not a universal formula.
W4 response: The additive term is a local first-order approximation to an unknown smooth response in α, not a claim of global linearity. It is compared empirically with no-alignment, multiplicative-resource, interaction, quadratic, and saturating alternatives. A saturating form improves in-range BIC, while extreme selection shows that monotone continuation beyond Q90 is unsafe. The crossed 48-configuration study also finds mild T/N dependence. The revision states the resulting scope directly: shared d is the minimal first-order description over the primary regime, and the stress tests identify where additional structure is needed.
The revised estimand, complete resource holdouts, data-mixture positioning, and transfer scope clarify what the alignment coordinate predicts without turning the paper into a causal-identification claim. If these changes address the reviewer's concerns about confounding, novelty, and transfer, we would appreciate reconsideration of the current assessment.