- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper compares 106 models with different pretraining mixtures. All use the same 1.5B architecture and training budget. It tracks their rankings through mid-training and SFT. MixSelector then combines pretraining scores and mixture weights to predict post-SFT performance.
Critical strengths
The controlled comparison is useful. It shows which pretraining signals remain informative after SFT, and where a fixed composite fails. The small-label results are also encouraging. Within this pool, post-training a few candidates already helps rank the rest.
Critical weaknesses
-
The large-model use case is unclear. RegMix [1] tests selected mixtures on larger models, including 7B. DoReMi [3] transfers weights from a 280M proxy to an 8B model. These are pretraining results. Here, all candidates are 1.5B. There is no equivalent test for mixtures chosen by post-SFT performance. This matters if small-model search is meant to avoid training many large candidates.
-
The gain over D-only is modest. In Table 1, mean correlation is 0.72 versus 0.70. Yet Regret@1 is worse: 0.77 versus 0.26. That gap comes from instruction following. Random cross-validation favors MixSelector, so D-only is not a consistent winner either. I would like to know when the extra features actually improve the final choice.
Overall Assessment
The contribution is mainly empirical. Related work already covers mixture regression [1] and post-SFT prediction [2]. I find the controlled study useful and lean toward acceptance. The large-scale use case remains open.
Questions for authors
-
Would users reuse an existing candidate pool, or train small proxies for a larger model? Please separate the additional cost from the full search cost. Include candidate pretraining, mid-training, SFT, and evaluation.
-
A useful transfer test would select mixtures at 1.5B. Compare MixSelector with a strong pretraining-score baseline and D-only. Then test those mixtures at a larger scale, with the same budget and SFT recipe. Mixture rankings can change with scale and training length [4]. Does selecting for post-SFT performance still help after scaling?
-
How sensitive are the top choices to the post-training seed? A few reruns on close candidates would help. The current bootstrap intervals do not measure this variation.
References
[1] Liu et al. RegMix: Data Mixture as Regression for Language Model Pre-training. ICLR 2025.
[2] Zeng et al. Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?. 2025.
[3] Xie et al. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. NeurIPS 2023.
[4] Ye et al. Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance. ICLR 2025.
3 comments
(Continued, 2/2)
Response to Q1: additional cost is n x 255.6 GPU-hours
- Per-candidate cost from our job accounting (GPU-hours as logged; card types not normalised):
| Stage | Median GPU-h per candidate | Full search (pretraining-score selection) | Additional for MixSelector |
|---|---|---|---|
| Candidate pretraining | not separately logged | all 106 | none |
| Base evaluation (17 metrics) | 10.7 to 16.2 | all 106 | none |
| Mid-training (YaRN) | 167.6 | selected mixture only | n candidates |
| SFT to E1 (8 x H200) | 32.4 | selected mixture only | n candidates |
| Post-SFT evaluation | 55.6 (upper bound, includes re-runs) | selected mixture only | n candidates |
| Fitting the selector | seconds on one CPU | none | once |
- The additional cost is therefore n x 255.6 GPU-hours (167.6 + 32.4 + 55.6), 2,556 GPU-hours at n = 10. Post-training all 106 candidates would cost 27,094 GPU-hours.
- Relative to post-training all 106 (which needs no base evaluation), MixSelector at n = 10 avoids 90.6% of the post-training cost, or 82.8% to 85.8% if the base evaluation of all candidates is also charged to it. At n = 68, the saving is 35.8%, or 28.1% to 31.1% with base evaluation charged.
- Candidate pretraining is identical across strategies, so it does not enter the additional cost.
- Corrections: SFT ran on H200, not H100 as stated in the paper (card types for mid-training and evaluation are not recorded in our logs, so we do not normalise them). The 55.6 figure is an upper bound because it includes re-runs.
- Revision: this table will replace the SFT-only figure in Appendix J and the Ethics statement.
Response to Q3: seed variance is unmeasured; we will rerun close candidates
- We agree that our bootstrap intervals do not capture this. Each candidate was fine-tuned once (seed 42, Table 20).
- What we do measure is narrower. Item-level reliability of the post-SFT rankings is at least 0.93 on six of eight benchmarks (Table 5). Rankings within a run are stable from E1 to E5 (ρ ≥ 0.93 on six of eight benchmarks, Table 9). Neither of these measures seed variation.
- We have not rerun SFT with additional seeds, so seed variance of the top-1 pick is not yet measured. We will report the gap between the top candidates in each family so that readers can see which choices are close.
- Revision: we will state single-seed post-training as a limitation in §4, and we will rerun SFT with additional seeds for the closest top candidates per family in the revision.
Response to the Overall remark: a primarily empirical contribution
| RegMix (Liu et al., 2025a) | Zeng et al. (2025) | This work | |
|---|---|---|---|
| Stage predicted | pretraining (proxy-model loss or benchmark score) | fine-tuning outcomes (as cited in §2) | post-SFT capability, per family |
| What is learned | regression from mixture weights | pairwise comparisons among fixed-size models, aggregated for selection | ridge from domain shares and 17 pretraining metrics |
| Candidate variation | mixture | objectives or data (more than the mixture) | mixture only, with identical mid-training and SFT |
- We agree that the contribution is primarily empirical. The controlled 106-model study, in which only the mixture varies through a shared mid-training and SFT pipeline, shows which pretraining signals survive post-training (math) and which do not (knowledge), that the fixed composite misranks math and code, and that instruction following is not predictable from pretraining signals. The method is intentionally minimal: a small post-SFT calibration set is enough to correct these failures, and the best of eight alternative regressors matches ridge within uncertainty (PLS minus ridge: +0.007 [-0.025, 0.033]).
Dear Reviewer ETCc,
Thank you again for your thoughtful and supportive review. As the discussion period closes on November 18, we would like to briefly recap how our response addressed your questions:
-
Intended workflow and transfer (W1 / Q1 / Q2). The selector re-ranks the proxy candidates that a mixture search already trains, and our held-out evaluation follows this order (fit on 68 early candidates, test on 38 later ones). We agree that your proposed cross-scale test is valid and interesting. It is not feasible within the discussion period, so we will state in §4 that the cross-scale transfer of post-SFT-selected mixtures is untested.
-
When the extra features help (W2). On knowledge and code, D+P ranks better than D-only; math is tied. The extra features help top-1 selection mainly at small budgets and do not help instruction following.
-
Cost and seeds (Q1 / Q3). The additional cost is n x 255.6 GPU-hours per post-trained candidate (2,556 at n = 10). Post-training is single-seed; we will state this as a limitation and rerun SFT with additional seeds for the closest top candidates in the revision.
We also agreed that the contribution is primarily empirical and clarified how it differs from RegMix and Zeng et al. (2025).
Is there anything that remains unclear, or anything we could clarify further before the discussion closes? If our responses have resolved your questions, we would be grateful if your final assessment could reflect this.
Best regards,
The authors
Dear Reviewer,
Thank you for the thoughtful review and for the supportive assessment. We address your questions with a full cost account taken from our training and evaluation logs, a family-level analysis of when the extra features help, and a clarification of the intended workflow.
Response to W1 / Q1 / Q2: the selector stays at proxy scale; mixture transfer is untested
Response to W2: extra features help on knowledge and code, not on IF