acceptodds
Under review as a conference paper at ICLR 2027

Best Before Post-Training? The Shelf Life of Pretraining Mixture Rankings

Abstract

The pretraining data mixture strongly influences a language model's capabilities, yet prior research on data mixture optimization typically selects candidates before observing their post-training performance. This practice rests on the untested assumption that pretraining rankings survive post-training. In this paper, we conduct a thorough experimental study over 106 models of 1.5B parameters that differ only in their pretraining mixtures and share identical mid-training and supervised fine-tuning (SFT) data and configurations. Our results demonstrate that knowledge rankings change across these stages, while math rankings largely persist but are diluted by a fixed-weight composite of pretraining scores. Yet these scores retain useful information: pretraining GSM8K alone predicts post-SFT rankings better than the composite for knowledge, math, and code. We therefore introduce MixSelector, a ridge-regression method that uses a subset of post-trained candidates to learn how to weight the pretraining scores together with the mixture composition and then ranks the remaining candidates. On held-out candidates, its ranking agrees with the post-SFT ranking at a mean Spearman correlation of 0.72 across capability families, against 0.10 for the composite, while maintaining a smaller lead over the best single score. With only ten post-trained candidates, it retains most of this advantage. Fixed pretraining rankings thus have a limited shelf life, but recalibrating pretraining signals on a few post-trained models suffices to align mixture selection with post-training performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pretrain Data, Datamix selection, Supervised Fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ETCcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares 106 models with different pretraining mixtures. All use the same 1.5B architecture and training budget. It tracks their rankings through mid-training and SFT. MixSelector then combines pretraining scores and mixture weights to predict post-SFT performance.

Critical strengths

The controlled comparison is useful. It shows which pretraining signals remain informative after SFT, and where a fixed composite fails. The small-label results are also encouraging. Within this pool, post-training a few candidates already helps rank the rest.

Critical weaknesses
  1. The large-model use case is unclear. RegMix [1] tests selected mixtures on larger models, including 7B. DoReMi [3] transfers weights from a 280M proxy to an 8B model. These are pretraining results. Here, all candidates are 1.5B. There is no equivalent test for mixtures chosen by post-SFT performance. This matters if small-model search is meant to avoid training many large candidates.

  2. The gain over D-only is modest. In Table 1, mean correlation is 0.72 versus 0.70. Yet Regret@1 is worse: 0.77 versus 0.26. That gap comes from instruction following. Random cross-validation favors MixSelector, so D-only is not a consistent winner either. I would like to know when the extra features actually improve the final choice.

Overall Assessment

The contribution is mainly empirical. Related work already covers mixture regression [1] and post-SFT prediction [2]. I find the controlled study useful and lean toward acceptance. The large-scale use case remains open.

Questions for authors
  1. Would users reuse an existing candidate pool, or train small proxies for a larger model? Please separate the additional cost from the full search cost. Include candidate pretraining, mid-training, SFT, and evaluation.

  2. A useful transfer test would select mixtures at 1.5B. Compare MixSelector with a strong pretraining-score baseline and D-only. Then test those mixtures at a larger scale, with the same budget and SFT recipe. Mixture rankings can change with scale and training length [4]. Does selecting for post-SFT performance still help after scaling?

  3. How sensitive are the top choices to the post-training seed? A few reruns on close candidates would help. The current bootstrap intervals do not measure this variation.

References

[1] Liu et al. RegMix: Data Mixture as Regression for Language Model Pre-training. ICLR 2025.

[2] Zeng et al. Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?. 2025.

[3] Xie et al. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. NeurIPS 2023.

[4] Ye et al. Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance. ICLR 2025.

3 comments

Dear Reviewer,

Thank you for the thoughtful review and for the supportive assessment. We address your questions with a full cost account taken from our training and evaluation logs, a family-level analysis of when the extra features help, and a clarification of the intended workflow.

Response to W1 / Q1 / Q2: the selector stays at proxy scale; mixture transfer is untested

  • Intended use: the selector re-ranks the pool of proxy candidates that a mixture search already trains. It is not a model to transfer across scales. Our 106 candidates are such a pool, the checkpoints of one multi-round RegMix search. Its first round proposed 68 candidates, and later rounds proposed 38 more based on earlier evaluations (Appendix A).
  • The workflow: post-train a few early candidates with the target post-training recipe, fit MixSelector, and rank the remaining and future candidates. Our evaluation follows this order, fitting on the 68 early candidates and testing on the 38 later ones (§3.5). With 10 post-trained candidates, median held-out ρ is 0.63, against 0.09 for the composite and 0.50 for the best single metric (Table 19).
  • As you note, RegMix and DoReMi test pretraining-selected mixtures at larger scale. Whether post-SFT-selected mixtures transfer in the same way is the open question you raise, and we have not tested it. Our own finding that rankings shift between training stages makes a shift with scale a real risk. If small proxies are trained specifically to choose a mixture for a larger model, MixSelector's benefit depends on that transfer.
  • We agree that your proposed test (select at 1.5B, then pretrain, mid-train and SFT the selected mixtures at a larger scale) is a valid and interesting check of the use case. It is not feasible within the discussion period: at 1.5B alone, mid-training, SFT and evaluation cost 255.6 GPU-hours per candidate, before any larger-scale pretraining. If compute allows, we will consider this as future work; we do not promise it for the camera-ready revision.
  • Revision: we will add an explicit "intended workflow" paragraph to §3.5. We will also state in §4 that our evidence covers selection within a fixed-scale search and that cross-scale transfer of post-SFT-selected mixtures is untested.

Response to W2: extra features help on knowledge and code, not on IF

  • On knowledge and code, D+P ranks better than D-only (math is tied, 0.91 vs 0.90), and on all three it picks a top candidate of the same quality on the main split. On instruction following no method is reliable, and D-only's lower IF regret drives the 4-family gap. Held-out, 38 candidates:
Familyρ D+Pρ D-onlyRegret@1 D+P, zRegret@1 D-only, z
Knowledge0.880.730.020.02
Math0.910.900.170.17
Code0.880.840.310.31
Instruction following0.210.322.580.55
Mean excluding IF0.170.17
  • The 4-family Regret@1 difference (0.77 vs 0.26 z) has a 95% interval of [-0.20, 0.66] and comes entirely from IF.
  • Across 50 repeats of 5-fold cross-validation, excluding IF, D+P improves ρ by +0.042 ± 0.007 (mean ± SD of the paired difference across repeats) and lowers Regret@1 from 0.24 to 0.11 z. On IF it is slightly worse in ρ (-0.021 ± 0.025).
  • By label budget (200 draws each), D+P helps most at the extremes. At n = 10, median ρ is 0.63 vs 0.57 (higher in 78.5% of draws), and median non-IF Regret@1 is 0.22 vs 0.38 z. At n = 45, D+P has higher ρ in 76.5% of draws, and it is higher on the single n = 68 split. At n = 15 to 30 the two are close (54.0% to 62.5% of draws), and non-IF regret is tied.
  • Short answer to your question: the extra features improve ranking quality on knowledge and code (+0.15 and +0.03 ρ on the main split; 0.877 vs 0.844 on code) and help top-1 selection mainly at small budgets. They do not help instruction following.
  • Revision: we will add this per-family table and the budget breakdown to §3.5, report the non-IF mean, and frame the claim as ranking quality.

(Continued, 2/2)

Response to Q1: additional cost is n x 255.6 GPU-hours

  • Per-candidate cost from our job accounting (GPU-hours as logged; card types not normalised):
StageMedian GPU-h per candidateFull search (pretraining-score selection)Additional for MixSelector
Candidate pretrainingnot separately loggedall 106none
Base evaluation (17 metrics)10.7 to 16.2all 106none
Mid-training (YaRN)167.6selected mixture onlyn candidates
SFT to E1 (8 x H200)32.4selected mixture onlyn candidates
Post-SFT evaluation55.6 (upper bound, includes re-runs)selected mixture onlyn candidates
Fitting the selectorseconds on one CPUnoneonce
  • The additional cost is therefore n x 255.6 GPU-hours (167.6 + 32.4 + 55.6), 2,556 GPU-hours at n = 10. Post-training all 106 candidates would cost 27,094 GPU-hours.
  • Relative to post-training all 106 (which needs no base evaluation), MixSelector at n = 10 avoids 90.6% of the post-training cost, or 82.8% to 85.8% if the base evaluation of all candidates is also charged to it. At n = 68, the saving is 35.8%, or 28.1% to 31.1% with base evaluation charged.
  • Candidate pretraining is identical across strategies, so it does not enter the additional cost.
  • Corrections: SFT ran on H200, not H100 as stated in the paper (card types for mid-training and evaluation are not recorded in our logs, so we do not normalise them). The 55.6 figure is an upper bound because it includes re-runs.
  • Revision: this table will replace the SFT-only figure in Appendix J and the Ethics statement.

Response to Q3: seed variance is unmeasured; we will rerun close candidates

  • We agree that our bootstrap intervals do not capture this. Each candidate was fine-tuned once (seed 42, Table 20).
  • What we do measure is narrower. Item-level reliability of the post-SFT rankings is at least 0.93 on six of eight benchmarks (Table 5). Rankings within a run are stable from E1 to E5 (ρ ≥ 0.93 on six of eight benchmarks, Table 9). Neither of these measures seed variation.
  • We have not rerun SFT with additional seeds, so seed variance of the top-1 pick is not yet measured. We will report the gap between the top candidates in each family so that readers can see which choices are close.
  • Revision: we will state single-seed post-training as a limitation in §4, and we will rerun SFT with additional seeds for the closest top candidates per family in the revision.

Response to the Overall remark: a primarily empirical contribution

RegMix (Liu et al., 2025a)Zeng et al. (2025)This work
Stage predictedpretraining (proxy-model loss or benchmark score)fine-tuning outcomes (as cited in §2)post-SFT capability, per family
What is learnedregression from mixture weightspairwise comparisons among fixed-size models, aggregated for selectionridge from domain shares and 17 pretraining metrics
Candidate variationmixtureobjectives or data (more than the mixture)mixture only, with identical mid-training and SFT
  • We agree that the contribution is primarily empirical. The controlled 106-model study, in which only the mixture varies through a shared mid-training and SFT pipeline, shows which pretraining signals survive post-training (math) and which do not (knowledge), that the fixed composite misranks math and code, and that instruction following is not predictable from pretraining signals. The method is intentionally minimal: a small post-SFT calibration set is enough to correct these failures, and the best of eight alternative regressors matches ridge within uncertainty (PLS minus ridge: +0.007 [-0.025, 0.033]).

Dear Reviewer ETCc,

Thank you again for your thoughtful and supportive review. As the discussion period closes on November 18, we would like to briefly recap how our response addressed your questions:

  1. Intended workflow and transfer (W1 / Q1 / Q2). The selector re-ranks the proxy candidates that a mixture search already trains, and our held-out evaluation follows this order (fit on 68 early candidates, test on 38 later ones). We agree that your proposed cross-scale test is valid and interesting. It is not feasible within the discussion period, so we will state in §4 that the cross-scale transfer of post-SFT-selected mixtures is untested.

  2. When the extra features help (W2). On knowledge and code, D+P ranks better than D-only; math is tied. The extra features help top-1 selection mainly at small budgets and do not help instruction following.

  3. Cost and seeds (Q1 / Q3). The additional cost is n x 255.6 GPU-hours per post-trained candidate (2,556 at n = 10). Post-training is single-seed; we will state this as a limitation and rerun SFT with additional seeds for the closest top candidates in the revision.

We also agreed that the contribution is primarily empirical and clarified how it differs from RegMix and Zeng et al. (2025).

Is there anything that remains unclear, or anything we could clarify further before the discussion closes? If our responses have resolved your questions, we would be grateful if your final assessment could reflect this.

Best regards,

The authors

Reviewer_2KGJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether pretraining data-mixture rankings predict performance after mid-training and SFT. Using 106 1.5B models with different pretraining mixtures and a shared training pipeline, it finds that ranking persistence varies by capability and that a fixed composite score can dilute useful signals. It proposes MixSelector, a ridge regression combining Base benchmark scores and mixture shares with post-SFT supervision. Held-out experiments and calibration-budget analyses show improved ranking prediction, although instruction following remains poorly predicted.

Critical strengths
  1. The controlled study of 106 models provides useful evidence about how mixture rankings behave across training stages. Holding architecture, training budget, and post-training recipe fixed makes the comparisons informative.

  2. The analysis goes beyond correlation to measure selection regret and shortlist quality. Testing on later search candidates and examining smaller calibration budgets also makes the evaluation relevant to practical mixture search.

Critical weaknesses
  1. Practical applicability to large-model pretraining remains unproven. All experiments use one model size and token budget, with no evidence that either the fitted selector or its mixture recommendations transfer across scales. Calibration also requires mid-training and SFT on multiple candidates. The paper does not establish whether the selection gains justify the full additional cost.

  2. The added value over well-designed Base evaluation is limited. Capability-specific evaluation aligned with downstream objectives is a natural alternative to the fixed composite. Domain-only ridge already reaches 0.70 mean correlation versus 0.72 for MixSelector, and achieves lower mean top-1 regret. Moreover, Base evaluation uses few-shot CoT, likelihood scoring, and other protocols, while SFT evaluation uses zero-shot generation with a chat template. The observed reshuffling therefore mixes training effects with evaluation-protocol changes, weakening its attribution to post-training itself.

Questions for authors
  1. Is the intended large-model workflow to transfer a selector fitted on small proxies, or to recalibrate at the target scale? What evidence and full calibration-cost accounting support this workflow?

  2. Under the same calibration budget, what advantage remains over capability-specific Base evaluation with benchmarks selected for consistency with post-SFT performance?

  3. How much of the same-benchmark ranking change remains when prompting and scoring protocols are kept consistent across stages? Can this be checked using the existing checkpoints?

5 comments
AuthorsOpenReview ↗

Response to W2 / Q3: knowledge reorder persists without few-shot or 16-sample scoring

Dear Reviewer,

Thank you for the detailed and constructive review. Using checkpoints and evaluations we already had, we ran the cross-stage check you asked for in Q3, the same-budget comparison with capability-specific base evaluation you asked for in Q2, and a full calibration-cost account for Q1. Results and clarifications follow.

Response to W2 / Q3: stage-specific protocols are standard practice, and the knowledge reorder persists without few-shot or 16-sample scoring

  • Evaluating base and post-trained checkpoints under different protocols is standard practice in open technical reports: base models with few-shot prompts, post-trained models through the chat template, mostly zero-shot and with thinking where available (Team OLMo et al., 2024, arXiv:2501.00656, App. A.1 vs A.2; Yang et al., 2025, arXiv:2505.09388, §3.3 vs §4.6; DeepSeek-AI, 2024, arXiv:2412.19437, §4.4.1 vs §5.3.1; Ettinger et al., 2025, arXiv:2512.13961, Table 46 vs Table 16). §3.2 follows this practice by design, because it measures the ranking change a practitioner faces when selecting on base-model scores and deploying the post-trained model. We agree that P to E1 alone does not show how much of the change the protocol accounts for, so we ran the check below.
  • Check on existing checkpoints. Every candidate had also been evaluated after mid-training and before SFT (call it E0) on MMLU, BBH, GSM8K and IFEval, with the same items, the same zero-shot prompt text (IFEval: the system prompt is inlined at E0), grader, answer parser, max-token budget and decoding (one sample, temperature 0.6, top-p 0.95) as E1. Going from E0 to E1 therefore removes the few-shot prompting and 16-sample scoring used at P, and the mid-training stage. The SFT update remains, together with the chat template and the thinking format (IFEval has thinking off at both stages).
  • Spearman ρ with E1, all 106 candidates, 95% intervals from 2,000 candidate-bootstrap resamples; the paired difference uses the same resamples:
Benchmarkρ(P → E1)ρ(E0 → E1)ρ(E0 → E1) minus ρ(P → E1)Top-10 overlap, E0 → E1
MMLU0.760.51 [0.35, 0.65]-0.25 [-0.39, -0.11]4
BBH0.500.60 [0.46, 0.71]+0.10 [-0.07, 0.27]1
GSM8K0.930.87 [0.79, 0.92]-0.06 [-0.11, -0.02]7
IFEvalnot in the P suite0.19 [-0.02, 0.38]n/a2
Mean of MMLU, BBH, GSM8K0.890.82 [0.74, 0.88]-0.07 [-0.13, -0.02]6
Mean of all fourn/a0.78 [0.69, 0.85]n/a5
  • The single-benchmark P → E1 values are those of §3.2. Answer to Q3: with few-shot prompting and multi-sample scoring removed, rankings are not more stable on any benchmark. No paired difference is significantly above zero. The selective pattern of §3.2 remains: math rankings persist (GSM8K 0.87), knowledge rankings reorder (MMLU 0.51, BBH 0.60), and IFEval rankings are barely preserved (0.19). The 38 held-out candidates show the same direction (MMLU 0.46, BBH 0.29, GSM8K 0.72).
  • Limitations of this check: E0 uses one sample per item (P: 16), so its noisier scores may lower ρ(E0 → E1); the chat template and thinking format still differ; and all results use one SFT seed.
  • Why we do not force one identical prompt across stages: in a one-model check, base-style few-shot prompts made the SFT model write new questions instead of answering and skip its thinking mode, so an identical prompt introduces its own confound. We therefore compare each stage under its standard protocol, as the reports above do.
  • Revision: MixSelector's held-out evaluation is unaffected, as it is fit on and scored against post-SFT outcomes under the deployed protocol. In §3.2 we will state explicitly that each stage is evaluated under its standard protocol, following the open reports above, and that the knowledge reordering persists when few-shot prompting and multi-sample scoring are removed (E0 to E1). We will add the E0 to E1 table to Appendix E and note in §4 that the chat template, thinking format and number of samples still differ between E0 and E1.
AuthorsOpenReview ↗

Continue of our response 2/3

Response to W2 / Q2: a lead at n = 68, a tie at n = 10

  • We built the baselines you describe on the same split and budgets: label-free capability-matched pretraining groups (math group or the composite as the proxy for code and IF, which have no pretraining counterpart) and, at the same label budget, the best group, the average of the top 1 to 3 metrics chosen on the labelled candidates, and non-negative least squares (NNLS) over the four groups.
  • Held-out results at n = 68 (38 candidates, 2,000 shared test resamples):
MethodUses post-SFT labelsMean ρTop-10Regret@1 (z)Regret@3 (z)
Composite Z (paper)no0.103.001.431.17
Aligned groups, composite for code and IFno0.314.251.110.87
Aligned groups, math group for code and IFno0.636.500.800.34
Best groupyes0.556.001.140.34
Top-3 metric averageyes0.596.000.360.36
NNLS over 4 groupsyes0.615.750.650.36
D-only ridgeyes0.705.750.260.21
MixSelector (D+P)yes0.726.250.770.36
  • At n = 68, MixSelector exceeds the strongest of these baselines (aligned groups with the math proxy) by +0.091 [0.033, 0.159] in mean ρ.
  • At n = 10 the difference is -0.014 [-0.081, 0.057], a tie, and across budgets and metrics only 2 of 24 paired intervals favour MixSelector. Top-3 averaging and NNLS also have lower Regret@1 than MixSelector at n = 68 (0.36 and 0.65 vs 0.77 z). MixSelector's own Regret@1 is dominated by instruction following (Table 16).
  • One note on the strongest baseline: choosing the math group as the code proxy relies on a post-SFT finding (pretraining GSM8K is the best single predictor of code, Table 11). Without that knowledge, using the composite as the code and IF proxy, mean ρ is 0.31.
  • Revision: we will add this table and the budget curves to Appendix I, and narrow the abstract and §3.5 to ranking quality, without claiming consistent superiority over aligned base evaluation.

Response to W1 / Q1: the selector stays at proxy scale; mixture transfer is untested

  • Direct answer to Q1: neither option as stated. The selector is fit and applied at proxy scale, inside the mixture search; what would move to the target scale is the selected mixture, as in RegMix.
  • We agree that whether the selected mixtures keep their advantage at a larger scale is a valid and important question. Our own finding that rankings shift between training stages makes a shift with scale a real risk, so we do not claim that post-SFT-selected mixtures transfer. A transfer test needs pretraining, mid-training and SFT of several mixtures at a larger scale; at 1.5B, mid-training, SFT and evaluation already cost 255.6 GPU-hours per candidate, before any larger-scale pretraining, which is not feasible within the discussion period. If compute allows, we will consider this as future work; we do not promise it for the camera-ready revision.
  • The intended workflow is recalibration within the search, not transferring a fitted selector across scales. Mixture searches such as RegMix already train many small proxies over rounds. In our search, 68 candidates were proposed in the first round and 38 in later rounds that built on earlier evaluations (Appendix A). MixSelector adds one step: post-train a small subset of the early candidates with the target post-training recipe, fit the selector, and rank the rest. Our held-out evaluation mirrors this, fitting on early candidates and ranking the 38 later ones (§3.5). With 10 post-trained candidates, median ρ is 0.63, against 0.09 for the composite (Table 19).
  • Revision: we will add an "intended workflow" paragraph to §3.5 and state in §4 that cross-scale transfer is untested.
AuthorsOpenReview ↗

Continue of our response 3/3

Response to W1 / Q1: calibration costs 255.6 GPU-hours per candidate

  • Measured cost per candidate, from our job accounting (GPU-hours as logged; card types not normalised):
StageMedian GPU-h per candidateNote
Pretrainingnot separately loggedidentical across strategies, so it cancels in the comparison
Mid-training (YaRN, 8 GPUs)167.6included in the calibration cost
SFT to E1 (8 x H200)32.4
Post-SFT evaluation, 8 benchmarks55.6historical allocations including re-runs, so an upper bound
Base evaluation, 17 metrics10.7 to 16.2needed by any base-score selection; not needed by D-only
Fitting MixSelectorseconds on one CPU
  • Calibration cost per labelled candidate is therefore 255.6 GPU-hours (167.6 + 32.4 + 55.6). Pretraining is common to all strategies, and the finally selected mixture is post-trained in every strategy, so neither enters the comparison.
  • For reference, the cost of calibrating on n of the 106 candidates relative to post-training every candidate (106 x 255.6, 27,094 GPU-hours, which needs no base evaluation); the last column also charges MixSelector the base evaluation of all 106:
nPost-training cost (GPU-h)Saving, post-training onlySaving incl. base evaluation of all 106
102,55690.6%82.8% to 85.8%
205,11281.1%73.3% to 76.4%
4511,50257.5%49.7% to 52.8%
6817,38135.8%28.1% to 31.1%
  • The practical comparison is with pretraining-score selection, which post-trains only its top pick: the extra cost is n x 255.6 GPU-hours (2,556 at n = 10; 17,381 at n = 68). At n = 10 this raises median held-out ρ from 0.09 (composite) and 0.50 (best single metric) to 0.63 (Table 19). A label-free aligned-group baseline ties it at n = 10 (Comment 2/3), but only when one already knows from post-SFT data that the math group is the right proxy for code; with the composite as proxy it reaches 0.31. A clear gain over that strongest baseline appears at n = 68 (+0.091 [0.033, 0.159]). We will state this trade-off in §3.5.
  • Against D-only ridge, the extra cost is the base evaluation (10.7 to 16.2 GPU-hours per candidate), which base-model mixture searches run anyway.
  • Two corrections: SFT ran on H200, not H100 as the paper states, and the 55.6 evaluation figure is an upper bound because it includes re-runs.
  • Revision: this table will replace the SFT-only figure in Appendix J and the Ethics statement.

Response to W2: D-only has lower mean regret; the gap is IF

  • You are right that D-only has lower mean top-1 regret (0.26 vs 0.77 z). The gap comes entirely from instruction following: excluding IF, both reach 0.17 z, with top picks of identical quality on knowledge, math and code (4-family difference 0.51 z, 95% interval [-0.20, 0.66]).
  • Under 50 repeats of 5-fold cross-validation (Table 18), D+P has higher ρ (0.80 vs 0.77, all 50 repeats) and lower Regret@1 (0.41 vs 0.56 z); paired difference -0.150 ± 0.078 z (mean ± SD across repeats).
  • At n = 10, D+P has median ρ 0.63 vs 0.57 and is higher in 78.5% of draws (Table 19). At n = 15 to 30 the advantage is small.
  • Revision: we will narrow the claim to ranking quality and report the non-IF regret alongside the 4-family mean.
Reviewer_e8hROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies that the ranking of different data mixtures can change greatly after the SFT stage. It uses some groups of benchmarks as signals for pretraining capability and finds that different benchmarks have different prediction abilities for different post-training capabilities. Furthermore, they propose MixSelector to use ridge regression over the data mixture and pretraining composite to predict the SFT performance. It shows improvement over the best pretraining signal.

Critical strengths
  1. This paper identifies an interesting and practical observation that the ranking of data mixtures can change greatly after SFT.
  2. It proposes a simple method to use ridge regression to predict the ranking after SFT.
Critical weaknesses
  1. The improvement is marginal. As shown in Table 1, the MixSelector only slightly outperforms the D-only or P-only baselines and largely falls behind in terms of regret.
  2. The ablation study is far from ready. For example, the SFT mixture and data composite can greatly affect the conclusions, and the necessity of ridge regression is questionable.
Questions for authors
  1. Provide ablation studies on different SFT mixtures and data composites.
  2. Try different regression approaches and explain the necessity of ridge regression.
  3. Deeper analysis: I believe that using a regression approach can not help in understanding the ranking shift or greatly improving prediction accuracy.
4 comments
AuthorsOpenReview ↗

Response to W1/Q1, W2/Q2

Dear Reviewer,

Thank you for the careful review. We ran the ablations you asked for on the existing 106 models: eight alternative regressors, label-free and label-using alternatives for combining pretraining scores at the same label budget, and a family-level decomposition of the Regret@1 gap. The results are below.

Response to W1: small D-only gain; we claim ranking quality

  • What holds: post-SFT calibration improves substantially on pretraining-score selection (mean ρ 0.72 vs 0.10 for the composite and 0.59 for the best single metric; paired difference over the best single metric 0.13 [0.05, 0.23], Table 1, §3.5).
  • What we concede: the gain of D+P over D-only (0.70) and P-only (0.68) is small, and its mean Regret@1 is higher than D-only's (0.77 vs 0.26 z). We will state this directly in §3.5 and will not claim that D+P improves top-1 selection over D-only overall.
  • The Regret@1 gap comes entirely from instruction following (IF). Paired bootstrap on the 38 held-out candidates, 2,000 shared resamples:
Familyρ D+Pρ D-onlyRegret@1 D+P, z (pp)Regret@1 D-only, z (pp)
Knowledge0.880.730.02 (0.6)0.02 (0.6)
Math0.910.900.17 (2.0)0.17 (2.0)
Code0.880.840.31 (2.4)0.31 (2.4)
Instruction following0.210.322.58 (6.1)0.55 (1.3)
Mean of 4 families0.720.700.770.26
Mean excluding IF0.170.17
  • On knowledge, math and code, both methods pick the same-quality top candidate. The 4-family Regret@1 difference is 0.51 z with a 95% interval of [-0.20, 0.66], which includes zero. No method ranks IF reliably (every 95% interval for IF includes zero; §3.5 and Appendix I.2).
  • Across 50 repeats of 5-fold cross-validation over all 106 candidates (Table 18), D+P has both higher ρ (0.80 vs 0.77, higher in all 50 repeats) and lower Regret@1 (0.41 vs 0.56 z). The paired difference across repeats is +0.026 ± 0.007 in ρ and -0.150 ± 0.078 z in Regret@1 (mean ± SD across repeats).
  • With only 10 labelled candidates (200 draws, Table 19), D+P has median ρ 0.63 vs 0.57 for D-only and is higher in 78.5% of draws. Its median non-IF Regret@1 is 0.22 vs 0.38 z, and its regret is no worse than D-only's in 66.0% of draws.
  • At intermediate budgets (n = 15 to 30) the ρ advantage is small (D+P higher in 54.0% to 62.5% of draws) and non-IF regret is tied.
  • Revision: §3.5 will report the non-IF mean alongside the 4-family mean and frame the claim as ranking quality.

Response to W2 / Q2: ridge is a simple default; PLS matches it

  • We refit the selector with eight alternative regressors under the identical protocol. Held-out mean Spearman, D+P features, paired 95% interval against the composite (2,000 shared test resamples; stochastic models over 5 seeds):
RegressorMean ρGain over composite [95% CI]
Ridge (MixSelector)0.717+0.614 [0.397, 0.821]
PLS0.724+0.620 [0.405, 0.824]
Kernel ridge (RBF)0.719+0.616 [0.396, 0.821]
Random forest0.715 ± 0.012+0.612 [0.394, 0.805]
Gradient boosting0.711 ± 0.012+0.607 [0.395, 0.798]
Elastic net0.674+0.571 [0.362, 0.773]
Pairwise RankSVM0.663+0.560 [0.351, 0.766]
  • The best alternative, PLS, differs from ridge by +0.007 [-0.025, 0.033]. The gain comes from post-SFT calibration, not from the choice of regressor.
  • Why ridge: the features are collinear by construction. The 8 domain shares sum to one (Appendix A), so with an intercept the design matrix is exactly singular, and the math-reasoning-trace share correlates at 0.83 with the pretraining math group (Appendix H). At a budget of 10 labels there are more features (25) than labels, so unpenalised least squares has no unique solution. Ridge with a leave-one-out penalty is the simplest well-posed choice and has a single hyperparameter.
  • Disclosures: Lasso and the Gaussian process produce constant predictions for at least one family on the main split (Lasso: in its single fit; GP: in 1 of 5 seeds), so their Spearman is undefined. Counting undefined correlations as zero, the GP is clearly worse than ridge (-0.29 [-0.46, -0.14]). PLS has lower median Regret@1 than ridge for n ≥ 15 (0.40 vs 0.77 z at n = 68) but not at n = 10 (0.74 vs 0.72).
  • Revision: we will add this table to Appendix I, describe ridge as a simple default within uncertainty, and report PLS as a variant.
AuthorsOpenReview ↗

Continue of our response 2/2

(Continued, 2/2)

Response to W2 / Q1: composites ablated; a different SFT recipe is untested

  • Data composite. We read "data composite" as the composite of pretraining scores used for selection; if you meant the SFT data composition, please see the SFT-recipe point below, and we would welcome a correction. We compared ways of combining pretraining scores on the same split: label-free (equal-weight, PCA, math-plus-knowledge, capability-aligned groups) and label-using at the same budget (best group, top 1 to 3 metric average, non-negative least squares over the four groups).
MethodUses post-SFT labelsMean ρ (38 held-out)
PC1 of 17 metricsno-0.16
Equal weight over 17 metricsno-0.03
Composite Z (paper)no0.10
Math + knowledge groupsno0.51
Capability-aligned groups (math group for code and IF)no0.63
Best groupyes0.55
Best single metricyes0.59
Top-2 / top-3 metric averageyes0.60 / 0.59
NNLS over 4 groupsyes0.61
MixSelector (D+P ridge)yes0.72
  • Every fixed composite ranks poorly, as the paper concludes; capability alignment recovers much of the gap. At n = 68 MixSelector exceeds the strongest alternative (aligned groups) by +0.091 [0.033, 0.159]. At n = 10 the two are tied (-0.014 [-0.081, 0.057]), and across budgets and metrics only 2 of 24 paired intervals favour MixSelector. The strongest of these baselines uses the math group as the code proxy, which itself relies on post-SFT data (Table 11); with the composite as proxy it reaches 0.31.
  • Revision: we will report the n = 10 tie and the 2 of 24 count in Appendix I, and remove any claim of consistent superiority over capability-aligned baselines from the abstract and §3.5.
  • SFT mixture. We agree that a different SFT recipe is a valid and interesting test, and we have not run one. It needs a new SFT run and full evaluation per candidate (32.4 H200 GPU-hours of SFT each, plus evaluation), not feasible within the discussion period. If compute allows, we will consider this as future work; Within the fixed recipe the target is stable (E1 and E5 rankings agree at ρ ≥ 0.93 on six of eight benchmarks, Table 9). The paper states the scope as "under a given SFT recipe" (§1); we will keep this scope visible in the abstract and conclusion and list alternative SFT datasets and recipes as a limitation in §4.

Response to Q3: analyses, not the ridge fit, explain the ranking shift

  • We agree that the ridge selector itself does not explain why rankings shift; the analyses it builds on do (§3.2 to §3.4):
    • Which capabilities shift, beyond item noise or prompting alone: same-benchmark ρ from P to E1 is 0.93 and 0.94 for GSM8K and MATH500, but 0.76, 0.79 and 0.50 for MMLU, MMLU-Pro and BBH. Reliability-adjusted agreement is 0.53 [0.37, 0.69] for BBH versus 0.94 and 0.98 for the math benchmarks (Table 6). On checkpoints taken after mid-training and evaluated zero-shot with the same prompt text and grader as after SFT, knowledge rankings still reorder through SFT (ρ 0.51 on MMLU, 0.60 on BBH) while math persists (0.87 on GSM8K), over all 106 candidates. The chat template and thinking format still differ between these two evaluations.
    • Why the composite fails, and what composition adds beyond scores: half of the composite's weight goes to the commonsense and open-QA groups, which correlate at most 0.29 with any post-SFT family (Table 11). Residual associations that survive Benjamini-Hochberg correction (Table 15) include code share +0.42 and web share -0.31 with residual code capability, and math-corpus share -0.37 with residual knowledge and -0.32 with residual math.
  • The fitted selector is interpretable (Appendix I.4). For code, its largest weights are GSM8K (0.37) and code share (0.35), with negative web (-0.24) and synthetic QA (-0.20) weights. For IF, the penalty shrinks every weight to about 0.05, so no pretraining feature carries a usable IF signal.
  • Accuracy gains over current practice are large where it fails (Table 16, held-out ρ):
FamilyComposite ZBest single metricMixSelector
Knowledge0.580.720.88
Math0.180.880.91
Code-0.070.730.88
Instruction following-0.280.020.21
  • In benchmark points, the top pick's loss falls from 11.3 to 2.0 pp on math and from 6.5 to 2.4 pp on code (random: 12.1 and 7.7 pp).
  • On math alone the gain over the best single metric is small (0.03 [-0.05, 0.15], Appendix I.2), because GSM8K already ranks math well.
  • Revision: we will move a short summary of the fitted weights into §3.5.
AuthorsOpenReview ↗

Follow-up response

Dear Reviewer e8hR,

Thank you again for your review. As the discussion period closes on November 18, we would like to briefly recap how our response addressed your main concerns:

  1. Small gain over D-only (W1). On knowledge, math and code, D+P and D-only pick top candidates of identical quality (non-IF Regret@1 0.17 z for both). The Regret@1 gap comes entirely from instruction following, which no method ranks reliably. The revision will claim ranking quality rather than better top-1 selection overall.

  2. Choice of ridge (W2 / Q2). We refit the selector with eight alternative regressors. The best one, PLS, matches ridge within uncertainty (+0.007 [-0.025, 0.033]), so the gain comes from post-SFT calibration rather than from the regressor. Ridge will be described as a simple, well-posed default.

  3. Composites, SFT recipe and interpretability (Q1 / Q3). We compared label-free and same-budget composites; every fixed composite ranks poorly, and we now report that MixSelector ties the strongest aligned baseline at n = 10 and leads it at n = 68. We have not tested a different SFT recipe and will state this as a limitation. We also pointed to the analyses in §3.2 to §3.4 that explain the ranking shift, and to the fitted weights in Appendix I.4.

Are there any concerns that remain open, or anything we could clarify further before the discussion closes? If our responses have addressed your concerns, we would be grateful if you would consider updating your score.

Best regards,

The authors

Thank you for the thorough and thoughtful rebuttal. I appreciate the additional experiments and the authors' willingness to acknowledge the limitations.

However, I feel that some of the new results actually reinforce my original concerns. In particular, the limited improvement over D-only and the lack of consistent gains in Regret@1. The robustness across different SFT recipes also remains untested. I will maintain my original score of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.