acceptodds
Under review as a conference paper at ICLR 2027

Provable Checkpoint Reuse for RegMix Proxy Swarms

Abstract

The composition of pre-training data is a primary determinant of language-model quality, which makes mixture optimization a standard stage of large-scale training pipelines. Existing regression-based mixture search methods pioneered by a RegMix-style pipeline fit a surrogate to a swarm of proxy-model runs and therefore train many candidate mixtures from scratch even though their cumulative compositions may overlap heavily. Therefore, we introduce the Mixture Training Reuse (MTR) problem, which aims to realize a fixed family of target data mixtures at minimum training cost by treating every target mixture as a cumulative process to its terminal composition and sharing intermediate checkpoints across runs. For the arbitrary-order problem MTR-a where the order of training examples does not matter, we show that the best shared checkpoint for a set of targets is their coordinatewise minimum, which reduces the continuous checkpoint search to laminar optimization over target subsets; building on this reduction, we prove NP-completeness and a tight -approximation reuse-gain guarantee for a greedy agglomeration algorithm. For the random-order problem MTR-r, we design a descendant-aware greedy planner that admits a checkpoint only when every descendant target satisfies a feasibility test measured by total variation (TV). Our evaluation covers both synthetic mixture families and real-data training. On tractable synthetic instances, MTR-a recovers of the training savings attained by the exact dynamic-programming optimum ( versus ). On a -mixture Essential-Web proxy swarm, MTR-a saves of training tokens, while TV-constrained MTR-r saves with a closer compositional match to independent training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data mixture optimization, approximation algorithms, language model training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vtJfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the issue of optimizing the data mixing ratio during the pre-training stage of large-scale language models, and proposes a framework for the problem of mixed training reuse. Existing methods, when searching for the optimal data mixing ratio, usually require training a large number of candidate models from scratch, ignoring the significant data overlap between different candidate schemes. The authors model the process of finding the optimal mixing ratio as a cumulative training process that shares intermediate checkpoints, and define two specific settings: arbitrary order and random order. Under the arbitrary order setting, the authors prove that the optimal shared checkpoint is the minimum coordinate of each target mixing ratio, thereby transforming the continuous optimization problem into a layered combinatorial optimization problem, and prove that this problem is non-deterministic polynomial complete in the computational complexity theory. At the same time, a greedy aggregation algorithm with a 50% approximation guarantee is proposed. In the random order setting, the authors introduce a feasibility test based on total variation to constrain the training trajectory. Experimental results show that this method can significantly reduce the training token consumption on synthetic datasets and real network data proxy clusters, while maintaining similar model performance to independent training.

Critical strengths
  1. Current research on data mixing strategies for large models mainly focuses on finding the optimal ratio. However, this paper keenly captures the significant computational redundancy in the training process of the proxy model cluster and proposes to reduce the overall training cost by using checkpoint replay. This is a research direction with great engineering value that has not been fully explored yet.

  2. The author not only ingeniously transforms the continuous checkpoint search under any order setting into a layered combinatorial optimization problem, but also rigorously proves the NP-completeness of this problem through reduction from the triangular decomposition problem. Moreover, the guarantee of half-approximate reuse benefits provided for the greedy aggregation algorithm demonstrates a solid theoretical foundation.

Critical weaknesses

First, from a global perspective, I believe this article is a typical incremental piece. It has a strong AI-generated flavor, and its own innovation points are rather limited. Moreover, the experiments are also quite limited.

Then, The experimental design has significant limitations and scale flaws. The verification mainly relies on very small-scale proxy models and synthetic datasets. The proportion of training tokens saved observed on models with less than one million parameters raises serious doubts as to whether this can be directly extrapolated to large-scale pre-training scenarios with billions or hundreds of billions of parameters. The lack of end-to-end verification in the real industrial-grade large model training pipeline makes the claimed computational cost savings seem unrealistic.

Questions for authors

Lack of discussion with:

  1. DoGE: Domain Reweighting with Generalization Estimation

  2. Fast Data Mixture Optimization via Gradient Descent

The schemes of the RegMix series are very slow and require training a large number of small proxy models. They are slow and costly. There is a large gap between the proxy models and the actual deployed models. These schemes are not as good as the two references mentioned above.

Reviewer_dm3vOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper clarified how to reduce cost of RegMix-style data mixture search by reusing checkpoints. The authors formulate training as a tree structure. They propose MTR-a for arbitrary data ordering and MTR-r with constraints on training composition. The authors present an NP-completeness analysis and a tight 1/2 approximation guarantee on reuse savings for MTR-a. On 128 Essential-Web proxy mixtures, MTR-r reduces token consumption by 49.0% and GPU-hours by 48.3%. Its loss-ranking correlation with independent training is 0.952. Experiments use approximately one-million-parameter proxies; the effect on final large-model performance remains untested.

Critical strengths

The paper addresses a concrete source of repeated computation in mixture search. The checkpoint tree formulation turns this overlap into an explicit planning problem and provides a systematic alternative to training every proxy independently.

The authors report actual GPU-hour savings, not just token savings. MTR-r reduces GPU-hours by 48.3% while maintaining a proxy-loss rank correlation of 0.952 across six paired seeds, providing evidence that reuse can reduce cost .

Critical weaknesses

The laminar reformulation omits a valid source of reuse. Section 4 excludes the full target set from the gain calculation, whereas Section 5 represents it as a non-root shared checkpoint. For targets (0.8,0.2) and (0.6,0.4), a shared checkpoint at (0.6,0.2) gives cost 1.2, but the Section 4 definition gives zero gain and cost 2. The equivalence in Theorem 1 therefore does not hold as written.

The evidence for preserving mixture-selection quality is limited. The real data experiments use 0.98M parameter proxies and evaluate agreement with independent proxy training. A high overall rank correlation does not establish that the best candidates remain correctly ranked or that the selected mixtures retain their value when used to train larger models.

Questions for authors

How does Theorem 1 account for a nonzero checkpoint shared by all targets? Please reconcile these conventions and clarify whether the correction affects the subsequent approximation guarantee.

Reviewer_HeuwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reduces the cost of RegMix proxy training by sharing checkpoints across candidate data mixtures. It formalizes this as a tree optimization problem, proves NP-completeness and a tight 1/2 approximation guarantee for arbitrary-order reuse, and introduces Forward-TV constraints to limit changes in data exposure. Experiments include synthetic planning benchmarks and 128 Essential-Web mixtures, where the constrained method saves 49% of training tokens and 48.3% of GPU-hours.

Critical strengths
  1. The paper addresses a practical bottleneck in mixture search. Hierarchical checkpoint sharing achieves substantial measured GPU savings, including checkpoint save/load costs.

  2. The arbitrary-order formulation has clear theoretical substance: the coordinatewise-minimum checkpoint structure, hardness result, and tight greedy approximation guarantee provide a principled basis for reuse planning.

Critical weaknesses
  1. Preserving the quality of mixture selection is not established. MTR-r achieves a Spearman correlation of 0.9516, but the selected composition still differs from independent training by an L1 distance of 0.1615 and a maximum domain deviation of 6.52 percentage points. The paper does not evaluate models trained on these selected mixtures, so the effect on actual performance remains unknown.

  2. Real-training evidence is limited to 0.98M-parameter proxies. The theoretical guarantee concerns arbitrary-order token reuse; the practical method relies on Forward-TV to control training distortion. Its reliability at a representative proxy scale remains untested, and the threshold sweep reports planning savings without corresponding training-quality measurements.

Questions for authors
  1. Have you compared the actual validation loss or downstream performance of models trained on the mixtures selected by MTR-r and Strict? How large is the performance gap?

  2. Do you have evidence beyond 0.98M-parameter proxies that the savings and mixture-ranking agreement persist?

  3. How was the Forward-TV threshold of 0.20 chosen? What is the measured tradeoff between computational savings and selection quality as this threshold changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.