- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the issue of optimizing the data mixing ratio during the pre-training stage of large-scale language models, and proposes a framework for the problem of mixed training reuse. Existing methods, when searching for the optimal data mixing ratio, usually require training a large number of candidate models from scratch, ignoring the significant data overlap between different candidate schemes. The authors model the process of finding the optimal mixing ratio as a cumulative training process that shares intermediate checkpoints, and define two specific settings: arbitrary order and random order. Under the arbitrary order setting, the authors prove that the optimal shared checkpoint is the minimum coordinate of each target mixing ratio, thereby transforming the continuous optimization problem into a layered combinatorial optimization problem, and prove that this problem is non-deterministic polynomial complete in the computational complexity theory. At the same time, a greedy aggregation algorithm with a 50% approximation guarantee is proposed. In the random order setting, the authors introduce a feasibility test based on total variation to constrain the training trajectory. Experimental results show that this method can significantly reduce the training token consumption on synthetic datasets and real network data proxy clusters, while maintaining similar model performance to independent training.
Critical strengths
-
Current research on data mixing strategies for large models mainly focuses on finding the optimal ratio. However, this paper keenly captures the significant computational redundancy in the training process of the proxy model cluster and proposes to reduce the overall training cost by using checkpoint replay. This is a research direction with great engineering value that has not been fully explored yet.
-
The author not only ingeniously transforms the continuous checkpoint search under any order setting into a layered combinatorial optimization problem, but also rigorously proves the NP-completeness of this problem through reduction from the triangular decomposition problem. Moreover, the guarantee of half-approximate reuse benefits provided for the greedy aggregation algorithm demonstrates a solid theoretical foundation.
Critical weaknesses
First, from a global perspective, I believe this article is a typical incremental piece. It has a strong AI-generated flavor, and its own innovation points are rather limited. Moreover, the experiments are also quite limited.
Then, The experimental design has significant limitations and scale flaws. The verification mainly relies on very small-scale proxy models and synthetic datasets. The proportion of training tokens saved observed on models with less than one million parameters raises serious doubts as to whether this can be directly extrapolated to large-scale pre-training scenarios with billions or hundreds of billions of parameters. The lack of end-to-end verification in the real industrial-grade large model training pipeline makes the claimed computational cost savings seem unrealistic.
Questions for authors
Lack of discussion with:
-
DoGE: Domain Reweighting with Generalization Estimation
-
Fast Data Mixture Optimization via Gradient Descent
The schemes of the RegMix series are very slow and require training a large number of small proxy models. They are slow and costly. There is a large gap between the proxy models and the actual deployed models. These schemes are not as good as the two references mentioned above.