acceptodds
Under review as a conference paper at ICLR 2027

Data–Task Alignment as a Third Scaling Axis for Language-Model Fine-Tuning

Abstract

Scaling laws for language models usually predict loss from model size and training tokens while treating the data distribution as fixed. This assumption is especially limiting in fine-tuning, where examples at the same token budget can differ substantially in their relevance to the target task. Existing laws cannot quantify that difference. We introduce data–task alignment , a measurable property of a training set relative to a held-out target, and study it jointly with parameter count and fine-tuning tokens . Our primary experiment contains 51 controlled Qwen3 cells across four model sizes, six alignment conditions, and a token range. Within this grid, the first-order law predicts held-out loss with and RMSE . Omitting alignment changes both fitted compute exponents, so heterogeneous data composition can be misattributed to scaling with and . We then map the law's operating regime through a crossed-budget study that finds mild dependence on the token-to-parameter ratio. Selection beyond Q90 shows diminishing returns and eventual reversal as the eligible pool narrows. The alignment effect replicates on an independent math target, a code domain, Llama-3.2, and a frozen gradient selector, although the preferred functional form changes. Alignment therefore provides a useful third scaling coordinate whose interpretation remains tied to the target and measurement. These results connect task-specific data selection to finite-budget scaling by showing when better-targeted data can substitute for compute. They also identify how this return changes across regimes and eventually reverses when selection becomes too narrow.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Scaling laws, supervised fine-tuning, data selection, data–task alignment, data curation, language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7MSiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether data–task alignment can serve as a measurable third scaling axis of supervised fine-tuning loss alongside model size and training token budget. The authors develop an alignment score utilizing both SBERT similarity and ZIP-FIT compression similarity to rank training examples relative to a held-out target distribution. Experiments suggest that alignment benefits recur across settings.

Critical strengths
  1. The paper addresses a practically relevant question beyond demonstrating that data selection helps. Relating data relevance to model size and training budget is a worthwhile direction for model performance prediction.
  2. The primary experiment includes meaningful controlled comparisons. Fixed-token comparisons and variation in model size and token budget help distinguish composition effects from resource scaling.
  3. Component ablations, decontamination checks, and replication across targets and model families strengthen the study. The paper also acknowledges several important limitations.
Critical weaknesses
  1. Quantile selection can simultaneously change source composition, response length, diversity, etc. The paper lacks controlled comparisons between datasets with similar α\alpha and token budgets but different compositions.
  2. Prior work already models performance under varying data composition. The paper should clarify what additional predictive capability is provided by the proposed alignment coordinate beyond these existing formulations.
  3. The Llama and cross-domain experiments primarily demonstrate selection benefits rather than transfer of the scaling law. Whether a calibrated model can accurately predict unseen configurations across model families and domains remains unresolved.
  4. Equation (1) assumes that alignment contributes an additive loss reduction with a coefficient independent of model size and token budget. Beyond describing this form as a first-order approximation, the paper provides little theoretical justification for these assumptions.
Questions for authors

Please address the concerns raised in the Weakness part above.

2 comments
AuthorsOpenReview ↗

Author rebuttal to Reviewer 7MSi

We thank the reviewer for highlighting the distinction between predictive measurement and causal attribution. The revised paper sharpens this distinction and adds complete resource holdouts to test the predictive claim.

W1 summary: Quantile selection changes source composition, length, and diversity, so the study does not causally isolate alignment.

W1 response: The revision now reports the main distribution shifts. At 21M tokens, Q90 averages 280 tokens per example versus 368 for Random. NuminaMath contributes 46.9% of Q90 versus 85.9% of Random, OrcaMath contributes 24.6% versus 4.2%, and MetaMath contributes 18.2% versus 2.5%. The fixed-token intervention controls total training tokens and the trainer. α\alpha is a predictive summary of target relevance in the resulting distribution, not a causal decomposition of source, length, and diversity.

That predictive claim survives complete holdouts. The alignment law reduces leave-one-model-size-out RMSE from 0.05570.0557 to 0.03210.0321 and leave-one-token-budget-out RMSE from 0.04050.0405 to 0.02000.0200. The resource-multiplier alternative reaches 0.04730.0473 and 0.02880.0288, respectively. Alignment therefore adds predictive information under scale and budget shifts even though it compresses several properties of the selected distribution.

W2 summary: The paper should distinguish its contribution from existing data-mixture scaling laws.

W2 response: Mixture laws model performance from named-source proportions. Our coordinate is target-relative and measured at the example level before downstream training, so it can compare examples within a source and subsets assembled from different sources. We then study its joint response with model size and fine-tuning tokens. The approaches are complementary: source weights describe how a dataset was assembled, while α\alpha measures how the realized dataset relates to a target. The revision sharpens this distinction without implying that mixture composition is unimportant.

W3 summary: The Llama and code studies show selection benefits but do not establish transfer of a calibrated scaling law.

W3 response: We do not present them as proof that the complete three-axis law transfers unchanged. Llama tests whether the fixed-token selection advantage survives a model-family change. Code tests target-relative selection and a four-size fixed-token candidate-law comparison. The independent math target provides the cleanest refit. After rescoring and retraining, its 16-configuration additive model reaches RMSE 0.00580.0058 and remains BIC-best. The frozen gradient-selector study identifies the boundary: alignment remains predictive at every tested size, but its coefficient becomes size-dependent. This supports a transferable coordinate with regime-dependent calibration, not a universal formula.

W4 summary: The additive form and scale-independent coefficient have limited theoretical justification.

W4 response: The additive term is a local first-order approximation to an unknown smooth response in α\alpha, not a claim of global linearity. It is compared empirically with no-alignment, multiplicative-resource, interaction, quadratic, and saturating alternatives. A saturating form improves in-range BIC, while extreme selection shows that monotone continuation beyond Q90 is unsafe. The crossed 48-configuration study also finds mild T/NT/N dependence. The revision states the resulting scope directly: shared dd is the minimal first-order description over the primary regime, and the stress tests identify where additional structure is needed.

The revised estimand, complete resource holdouts, data-mixture positioning, and transfer scope clarify what the alignment coordinate predicts without turning the paper into a causal-identification claim. If these changes address the reviewer's concerns about confounding, novelty, and transfer, we would appreciate reconsideration of the current assessment.

I appreciate the authors' efforts. My major concerns have been addressed. This work falls around the acceptance threshold in my view, and I leave the final decision to the AC.

Reviewer_4ZmHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether how well fine-tuning data matches a target task can be measured and treated as a scaling variable, alongside model size (N) and training tokens (T). Each training example is scored by its similarity to a held-out target set (an average of embedding similarity and compression similarity), and a subset’s alignment α is the token-weighted average of these scores. The authors train Qwen3 models from 0.6B to 8B on subsets of different alignment and fit a Chinchilla-style law with an added dα term. At a fixed token budget, training on the top-scoring 10% of data lowers target loss by 6.7 to 11.2% compared with random data, and the added term roughly halves the fit error. Further experiments show that very aggressive selection (top 1-5%) backfires, and that the effect replicates on a second math target, code, Llama-3.2, and a gradient-based selector.

Critical strengths
  • The practical question is a good one. Fine-tuning budgets are almost always smaller than the available data, so choosing what fills the budget is a real decision, and putting it on the same scale as model size and tokens is useful.
  • I appreciate scaling laws being studied at a fine-tuning scale. Most of the field’s intuitions come from pretraining, and practitioners working at small scale need their own guidance.
  • The authors test where their own law breaks rather than only where it works. The extreme-selection experiment is well designed: matched Q90 runs at small token budgets show that the top-1% result is worse not merely because it has fewer tokens.
  • The paper is clear that alignment is relative to a target. Training on data matched to grade-school math helps GSM8K but hurts MATH-500, and the alignment score itself predicts this. This is honest and useful.
Critical weaknesses
  • The benefit may shrink with scale. The relative gain peaks at 4B (11.2%), drops at 8B (8.9%), and falls to 1.6% at 14B (Appendix H). Since the paper makes no case for a larger scale, it should state where this law applies and where existing laws take over. All experiments run at roughly 0.0006 to 0.07 tokens per parameter, far below the regime where Chinchilla-style intuitions were developed, and I would like this made explicit.
  • Part of the main result is true by design. The authors deliberately vary data quality at fixed N and T, so a law in N and T alone cannot explain the results; any data variable would improve the fit. The more interesting claim is that one alignment coefficient applies across all model sizes and budgets, i.e., alignment shifts loss by the same amount regardless of scale. I would like the paper to frame its contribution around that.
  • The value of selection is measured against the wrong comparisons. Every comparison is at a matched token budget, and no model is trained on the full pool. The practical question is whether selected data is worth more than more random data. Since Q90 and random runs exist from 2M to 42M tokens, the authors could report this directly (e.g., “21M selected tokens ~ X M random tokens”). Separately, the parameter-count exchange rate is computed from Q10, the worst-aligned subset, rather than random selection, which is what a practitioner would compare against.
  • The evidence outside the main math setting is thin. The code results show a small improvement, most cells use one seed, and the preferred form of the law changes depending on the selector.
  • Definitions are ambiguous. The quantile subsets are never precisely defined. Q90 appears to be the top 10%, but what are Q10 to Q70? For a paper whose core variable is a data property, subset construction should be unambiguous.
Questions for authors
  • If Q90 is the top 10% of the pool, does it contain enough tokens for the 42M-token runs? How were those subsets filled?
  • How does a DSIR-based α compare with the composite score, in both fit quality and transfer across scale?
  • Is the independent math target drawn from the same distribution as the original?
  • Do high-alignment subsets differ systematically in source dataset or example length
2 comments
AuthorsOpenReview ↗

Author rebuttal to Reviewer 4ZmH

We thank the reviewer for the careful reading and concrete questions. The revision makes the measured regime and subset construction explicit, adds the requested practical comparison, and tests the central restriction under complete resource holdouts.

W1 summary: The benefit may shrink with scale, the low-token fine-tuning regime is unclear, and improved fit over an N,TN,T-only law is partly guaranteed by varying composition.

W1 response: The primary range is Qwen3-0.6B through 8B and 2M through 42M fine-tuning tokens. Across the grid, T/NT/N ranges from 2.5×10−42.5\times10^{-4} to 7.0×10−27.0\times10^{-2}. We now state explicitly that this is a low-token supervised fine-tuning regime. The 14B result is reported as an outside-grid scope test, where the Q90 loss advantage remains positive but narrows to 1.6%.

Varying composition guarantees residual variation under an N,TN,T-only model. It does not guarantee that one scalar measured before training predicts that variation across scales and budgets. The shared coefficient is tested rather than assumed. On the 48-configuration crossed design, a smooth d(N)d(N) interaction is not favored over shared dd (ΔBIC=+3.9\Delta\mathrm{BIC}=+3.9), while mild d(T/N)d(T/N) dependence improves BIC by 3.7 points. Complete holdouts reduce RMSE from 0.05570.0557 to 0.03210.0321 when one size is absent and from 0.04050.0405 to 0.02000.0200 when one budget is absent. These results support a shared first-order coordinate over the measured regime, not strict invariance.

W2 summary: The paper should compare selected data with a larger random-data budget rather than only matching token counts or using Q10 as the practical reference.

W2 response: The existing runs provide this comparison directly. Q90 at 2M beats Random at 5M across all four sizes. Q90 at 5M beats Random at 10M, and Q90 at 10M beats Random at 21M, again across all four sizes. Q90 at 21M also beats Random at 42M for every size with a 42M run. Selected data outperforms 2.02.0-2.5×2.5\times more random tokens in all 15 observed comparisons. The revision reports this result and presents Random, rather than Q10, as the practitioner-facing reference.

W3 summary: Quantile construction is ambiguous, including whether Q90 can support 42M tokens.

W3 response: Q$q$ is a threshold condition, not a disjoint quantile bin. It retains examples at or above the pool's qqth score percentile, then samples without replacement with seed 42 until reaching the token budget. The reported α\alpha is the realized token-weighted mean rather than the quantile label. Q90 supports budgets through 21M only. There is no Q90-42M run, and the 42M endpoint is Random only. Section 4 now states each point explicitly.

W4 summary: Evidence for DSIR, an independent target, and transfer outside the main math setting is limited.

W4 response: At Qwen3-1.7B and 21M tokens on the same pool and target, the composite selector obtains loss 0.749, compared with 0.767 for DSIR and 0.828 for Random. We present this as one matched comparison, not a cross-scale DSIR law. The independent math target is the disjoint next 747 examples from the same official GSM8K training distribution. We recompute scores, rerun target-specific decontamination, rebuild subsets, and train 16 new configurations. This tests target-sample sensitivity within distribution. The broader gradient-selector experiment spans 0.6B, 1.7B, and 4B and shows that the coordinate remains useful even when its calibration changes.

W5 summary: High-alignment subsets may differ systematically in source composition or example length.

W5 response: They do. At 21M tokens, Q90 averages 280 tokens per example versus 368 for Random. NuminaMath contributes 46.9% of Q90 versus 85.9% of Random, while OrcaMath contributes 24.6% versus 4.2% and MetaMath 18.2% versus 2.5%. The revision reports this audit and sharpens the estimand. α\alpha is a predictive summary of the realized distribution's target relevance, which can include source, length, and content. The paper does not claim that α\alpha causally isolates relevance from those properties.

The revised regime statement, exact subset construction, direct random-token comparison, and source-and-length audit make the scope and practical value of the result substantially clearer. If these additions address the reviewer's concerns about generalization, construction, and practitioner relevance, we would appreciate reconsideration of the current score.

I thank the authors' responses. At this time, i'm leaning towards a weak acceptance but leave it to the AC.

Reviewer_GDc9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a data-task alignment metric: a token-weighted score combining SBERT and ZIP-FIT similarity between the training subset and a held-out math target. It incorporates this metric as an additive term to Chinchilla scaling laws, showing that it lowers RMSE over a 51-model scaling sweep. Two additional scaling sweeps show that incorporating the data-task alignment term (but with potentially different functional forms for the law) lowers RMSE over plain Chinchilla, and that filtering the train data to the 10% “most aligned” lowers loss vs random sampling for the same number of training tokens.

Critical strengths

Scaling laws which take into account data-task alignment is an interesting and valuable research direction.

Critical weaknesses

The contributions of the paper fall short of those expected for an ICLR paper. The scaling laws are only validated using R2 and RMSE (not even leave-one-out, as far as I can see), which is a very low bar. The value of scaling laws lies in their ability to extrapolate to larger scale runs. Do the laws hold when increasing data size by 100x, or model size by 10x? There is a fundamental tension between alpha and T as illustrated in Section 6.3, which suggests that the law can aid in deciding the optimal trade-off of the two terms, but this is not explored. One convincing validation of the law and the overall approach would be to use the fitted law to design the optimal training configuration of a much larger run, and see whether the predictions hold.

The paper concedes that the law’s functional form need not hold on other datasets/domains (Section 6.4), which limits the significance of the contributions. The proposed form also breaks down at very high alphas (Section 6.3), which is problematic since the optimal alpha according to the law might lie precisely in this region, thus limiting the ability of the law to be prescriptive of the training choices that should be made.

The paper pivots in Section 6.4-6.6 to evaluating the data-task alignment metric as a cratering for training data selection. These results say little about the paper's central claim that alignment is a scaling axis alongside N and T, since it lacks the scaling component altogether.

Questions for authors

NA

2 comments
AuthorsOpenReview ↗

Author rebuttal to Reviewer GDc9

We thank the reviewer for recognizing data-task alignment as a valuable research direction. The revised paper strengthens the predictive and prescriptive evidence while keeping the claim within the regime that the experiments measure.

W1 summary: The scaling law is evaluated mainly in sample, lacks meaningful extrapolation, and is not used to choose between data alignment and additional compute.

W1 response: We added complete resource holdouts using the paper's original fitting procedure. When one model size is absent from fitting, the alignment law obtains pooled RMSE 0.03210.0321, compared with 0.05570.0557 without alignment and 0.04730.0473 for the resource-multiplier alternative. When one token budget is absent, the corresponding RMSEs are 0.02000.0200, 0.04050.0405, and 0.02880.0288. Alignment therefore improves prediction when an entire scale or budget is unseen, not only on configurations used for fitting.

The observed token sweeps also answer the allocation question without extrapolating a fitted curve. At every tested size from 0.6B to 8B, Q90 at 2M, 5M, and 10M tokens outperforms Random at 5M, 10M, and 21M tokens, respectively. At 0.6B, 1.7B, and 4B, Q90 at 21M also outperforms Random at 42M. Selected data therefore outperforms 2.02.0-2.5×2.5\times more random tokens in all 15 directly observed comparisons.

The two-seed 14B experiment is an outside-grid scope test rather than part of the main fit. Q90 retains a positive mean advantage in held-out loss and GSM8K exact match, although the loss reduction narrows to 1.6%. We do not claim 10×10\times model-size or 100×100\times token extrapolation from a grid spanning 13×13\times in parameters and 21×21\times in tokens. The supported claim is stronger than an in-sample fit and narrower than universal extrapolation: alignment improves held-out prediction across the measured fine-tuning regime and remains useful under complete scale and budget holdouts.

W2 summary: The tension between alignment and token count is not developed into a resource-dependent analysis or practical decision rule.

W2 response: A 48-configuration crossed experiment measures all six alignment conditions at 5M and 21M tokens across four model sizes. A smooth size-only interaction is not favored over the shared coefficient (ΔBIC=+3.9\Delta\mathrm{BIC}=+3.9), while a mild T/NT/N interaction improves BIC by 3.7 points. The revised paper now frames the additive term as a first-order in-range law, states the measured T/NT/N regime, and reports the direct selected-versus-more-random comparisons above. These results map the resource dependence without presenting one coefficient as universal.

W3 summary: Breakdown under extreme selection and limited scaling evidence outside the main setting weaken the claim that alignment is a prescriptive scaling axis.

W3 response: Q95-Q99 is a deliberate stress test outside the fitted Q10-Q90 range. Q99 at 2.8M tokens is worse than Q90 at only 2M tokens, so its reversal is not explained by the smaller token budget alone. The resulting prescription is to optimize alignment only within validated support, not to extrapolate toward α=1\alpha=1.

The transfer studies answer separate questions. The independent math target repeats scoring, decontamination, subset construction, and training, and its 16-configuration additive fit reaches RMSE 0.00580.0058. Code and Llama test whether the direction of fixed-token selection transfers. The frozen gradient selector tests a different alignment measurement across scale and finds that alignment remains predictive while its coefficient becomes size-dependent. The revised paper therefore distinguishes transfer of the coordinate from transfer of one coefficient or functional form.

The complete resource holdouts, observed token-substitution results, and explicit support boundary strengthen both the predictive validation and prescriptive value of the paper. If these analyses address the reviewer's concerns about extrapolation, allocation, and transfer, we would appreciate reconsideration of the current assessment.

Thank you for the response. The authors now report RMSE under leave-one-model-size-out (and leave-one-token-budget-out), which is a step in the right direction. Otherwise, the limitations highlighted in my review still stand: there is no evidence that the law extrapolates beyond 2× in model size, nor that it can be used to arrive at better training configurations. Its functional form also changes across settings, and it breaks down at high alignment, which is precisely where the law would steer data selection. I don't think the contributions are sufficient to warrant acceptance, and therefore I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.