- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies multi-domain language-model pretraining and argues that a data-mixture ratio conflates two distinct controls: domain exposure, which determines how many tokens from each domain are observed, and optimization scale, which determines how strongly each domain contributes to parameter updates. Through matched-exposure experiments on knowledge and mathematics data, the authors show that changing domain loss coefficients while keeping the sampled mixture fixed can produce a better observed knowledge–reasoning trade-off than direct mixing or stage-wise training.
Building on this observation, the paper proposes LARE, an online domain-level reweighting rule that sets each domain coefficient proportional to the inverse square root of its smoothed loss and normalizes the mixture-averaged coefficient to one. A stylized noisy logistic-regression analysis is used to motivate the inverse-square-root exponent through a loss–gradient-noise relationship. The empirical evaluation covers multiple data ratios, three base optimizers, knowledge/mathematics/code mixtures, 30B–240B training-token budgets, and models ranging from 124M to 1.5B parameters. The reported gains are particularly large on mathematical reasoning and code benchmarks, while knowledge performance is generally maintained in the principal settings.
Critical strengths
-
A clean formulation and controlled isolation of an overlooked training variable. The paper clearly separates domain exposure from optimization scale and tests this distinction under matched model configurations, token budgets, sampled mixtures, and training streams. The comparison among stage-wise training, direct mixing, and fixed loss reweighting provides convincing evidence that domain token counts alone do not determine the final capability trade-off. This is consequential because most LLM data-mixture work primarily optimizes sampling proportions rather than studying loss scale at fixed exposure.
-
A simple adaptive method with broad empirical support. LARE requires only running domain-loss statistics and introduces little conceptual or implementation complexity. Its behavior is evaluated across AdamW, Muon, and ActGD; several knowledge-to-mathematics ratios; two- and three-domain mixtures; multiple model sizes; and training budgets up to 240B tokens. The mathematical-reasoning and code gains are large and directionally consistent across many settings, making the method potentially useful even if the theoretical account is only approximate.
Critical weaknesses
-
The adaptive reweighting baseline set is too limited to establish the specific contribution of LARE. Adaptive task/domain loss weighting is well established in multi-task learning, and the paper itself cites uncertainty weighting, gradient balancing, multi-objective optimization, multi-domain sampling/loss-weight analyses, and dynamic loss-based sample reweighting for LLM pretraining. However, the empirical comparison is primarily against uniform weighting and a fixed coefficient grid. Without direct comparisons to representative adaptive weighting methods, it is unclear whether the gains arise from the particular inverse-square-root rule or from adaptive domain reweighting more generally. The absence of exponent and gradient-based ablations further limits the novelty claim.
-
The assumptions connecting the theoretical result to LARE are not validated in actual LLM training. The loss–gradient-noise theorem concerns low-noise binary logistic regression with isotropic Gaussian features, an unconstrained classifier at its population optimum, and comparable effective dimensions and hidden constants across domains. LARE instead substitutes online current cross-entropy for the unknown optimum loss and applies the rule throughout nonstationary autoregressive pretraining. Cross-entropy scales can also differ across knowledge, mathematics, and code because of intrinsic entropy, tokenization, masking, and domain partition choices. The paper does not directly verify the predicted gradient-RMS relationship in its trained models. In addition, most reported cells are single training runs, so the small MMLU differences do not currently establish that knowledge performance is consistently improved rather than statistically unchanged.
Questions for authors
-
How does LARE compare with representative adaptive loss-weighting methods such as GradNorm, uncertainty weighting, Dynamic Weight Averaging, gradient-norm balancing, and existing loss-based sample reweighting for LLM pretraining? Such comparisons are necessary to determine whether the benefit is specific to LARE rather than to adaptive reweighting in general.
-
Do the assumptions motivating the inverse-square-root rule approximately hold during real LLM pretraining? In particular, can the authors measure per-domain gradient covariance or RMS over training and test whether it is proportional to the square root of the corresponding domain loss? How different are the effective dimensions and proportionality constants across knowledge, mathematics, and code?
-
How sensitive is LARE to the exponent and the substitution of online loss for optimum loss? Please compare rules of the form for several values of , as well as a rule based directly on measured gradient norms or variances.
-
At a fixed parameter state, reweighting with coefficients has the same expected gradient direction as a direct mixture with proportions , up to a global scale. Can the authors compare reweighting against these exact effective mixture ratios rather than only the relatively sparse 7:1, 3:1, and 1:1 grid? For example, 7:1 exposure with a 1:2 loss ratio corresponds to an effective gradient ratio of approximately 3.5:1. This comparison would clarify whether the empirical frontier expansion arises from fixed exposure and gradient variance rather than an insufficiently dense remix baseline.
-
Are the headline results stable across independent training seeds? Please report training variability for the small MMLU changes and for the claim that all five benchmarks improve at the 8× budget. It would also be helpful to confirm the longer-budget, three-domain result with a standard optimizer in addition to ActGD.
-
How sensitive is LARE to the definition and granularity of a domain? If a knowledge corpus is divided into two subdomains, or if domains have substantially different tokenization and irreducible cross-entropy, the normalization and resulting weights may change even when the underlying data are unchanged. Please characterize this sensitivity and clarify what domain partition the method assumes in practice.
-
What exactly is accelerated by the “Fast LLM Pretraining” claim? Please report the runtime and memory overhead of LARE and, where possible, compare tokens or wall-clock time required to reach matched downstream performance against the actual mixture- or coefficient-tuning pipeline used as the alternative.
5 comments
The second response to Reviewer CLGx
Q3: In addition, most reported cells are single training runs, so the small MMLU differences do not currently establish that knowledge performance is consistently improved rather than statistically unchanged.
A3: We interpret the 0.01 pp MMLU difference in the 120B three-domain comparison as essentially unchanged. The +0.83 pp difference at 240B is also a single-run observation. Our compute budget did not allow us to repeat the long-budget experiments with additional seeds, so we cannot quantify training-seed variability. We will qualify the joint-improvement claim accordingly. In the reported runs, Figure 7 (Appendix D.6) shows that LARE’s GSM8K advantage over direct mixing persists across later checkpoints at both 120B and 240B. The reasoning advantage is therefore sustained over the course of training in both comparisons.
Q4: How sensitive is LARE to the exponent and the substitution of online loss for optimum loss? Please compare rules of the form for several values of , as well as a rule based directly on measured gradient norms or variances.
A4: We compare LARE with two alternative online weighting rules in 30B-token ActGD runs, holding the sampling ratio fixed within each comparison.
Inverse Loss scales the recent decrease in domain cross-entropy by the inverse of its slow estimate. With and denoting the fast and slow loss EMAs, the score before normalization is
Approximate Inverse Loss uses the slow EMA of the domain’s prediction-residual energy: For small token cross-entropy , the squared residual on the target class satisfies , which motivates the approximation. Appendix G.2, Equations (47)–(49), gives the definitions and the shared normalization, clipping, and weight smoothing.
| Weighting rule | K:M = 7:1 | K:M = 3:1 | K:M = 1:1 |
|---|---|---|---|
| Direct mixture | 31.83 / 27.82 | — | 31.53 / 54.89 |
| LARE | 31.99 / 34.50 | 31.56 / 47.38 | 30.55 / 59.74 |
| Inverse Loss | 31.11 / 41.47 | — | 30.87 / 55.65 |
| Approximate Inverse Loss | 32.00 / 33.59 | 31.31 / 44.73 | 30.89 / 50.34 |
Figure 10 shows that Inverse Loss pulls ahead of LARE on GSM8K after warmup at 7:1, finishing 6.97 pp higher on GSM8K and 0.88 pp lower on MMLU. At 1:1, LARE achieves the highest final GSM8K score. Figure 11 shows that Approximate Inverse Loss tracks LARE closely at 7:1, while LARE finishes higher on both benchmarks at 3:1. These comparisons vary the full online weighting rule. An exponent-only sweep and comparisons using optimum loss or measured gradient statistics remain open.
Q5: At a fixed parameter state, reweighting with coefficients has the same expected gradient direction as a direct mixture with proportions , up to a global scale. Can the authors compare reweighting against these exact effective mixture ratios rather than only the relatively sparse 7:1, 3:1, and 1:1 grid? For example, 7:1 exposure with a 1:2 loss ratio corresponds to an effective gradient ratio of approximately 3.5:1. This comparison would clarify whether the empirical frontier expansion arises from fixed exposure and gradient variance rather than an insufficiently dense remix baseline.
A5: After masking, the coefficients on the domain mean gradients are proportional to , where is the domain’s share of valid targets. Combining the realized shares in Appendix C.3 (Table 4) with the 30B Muon results in Appendix D.4 (Tables 9–10), we obtain two near matches to the 1:1 direct mixture.
| K:M exposure | K:M loss weights | Math share of coefficients | MMLU (%) | GSM8K (%) |
|---|---|---|---|---|
| 1:1 | 1:1 | 43.58% | 31.67 | 59.89 |
| 7:1 | 1:7 | 43.56% | 31.19 | 48.98 |
| 3:1 | 1:3 | 43.57% | 31.75 | 56.71 |
The recipes reach different endpoints despite their nearly identical post-warmup coefficients on the domain mean gradients. Direct mixing gives the highest GSM8K score at this operating point, while 3:1 exposure with 1:3 weights gives slightly higher MMLU. The runs differ in exposure and warmup history, so this comparison cannot isolate gradient variance as the cause. We will limit the frontier claim to the tested recipes. Our compute budget did not allow us to run the proposed 3.5:1 direct-mixture experiment.
The third response to Reviewer CLGx
Q6: Are the headline results stable across independent training seeds? Please report training variability for the small MMLU changes and for the claim that all five benchmarks improve at the 8× budget. It would also be helpful to confirm the longer-budget, three-domain result with a standard optimizer in addition to ActGD.
A6: All five final scores are higher in the reported 240B LARE run in Table 2, including +0.83 pp on MMLU, +17.06 pp on GSM8K, and +9.00 pp on MATH. These are observations from one matched pair. Our compute budget precluded independent repeats and a standard-optimizer replication at the longer budget. Table 9 shows that the reasoning gains extend to AdamW and Muon at 30B. The 240B evidence is specific to ActGD.
Q7: How sensitive is LARE to the definition and granularity of a domain? If a knowledge corpus is divided into two subdomains, or if domains have substantially different tokenization and irreducible cross-entropy, the normalization and resulting weights may change even when the underlying data are unchanged. Please characterize this sensitivity and clarify what domain partition the method assumes in practice.
A7: LARE uses a fixed assignment of sources to weighting groups and computes one coefficient per group from its mean loss over valid targets. Splitting a group can change the coefficients when the resulting groups have different losses.
To test this sensitivity, we kept the 1.5B model, ActGD optimizer, and 30B-token training stream fixed. We compared three semantic domains, six groups formed by ranking the 18 sources by their calibration loss at the end of warmup, and 18 individually weighted sources. Appendix G.1 reports the final scores in Table 15 and the training curves in Figure 9.
| Weighting partition | MMLU (%) | GSM8K (%) |
|---|---|---|
| Direct mixture | 32.47 | 14.25 |
| LARE, 3 semantic domains | 32.20 | 22.97 |
| LARE, 6 groups based on calibration loss | 32.74 | 24.03 |
| LARE, 18 individual sources | 32.35 | 22.67 |
All three partitions improve GSM8K by 8.42–9.78 pp over direct mixing, with MMLU within 0.3 pp of the baseline. The six-group partition gives the highest observed scores on both benchmarks, showing that the choice of partition can affect the outcome even at fixed exposure.
Q8: What exactly is accelerated by the “Fast LLM Pretraining” claim? Please report the runtime and memory overhead of LARE and, where possible, compare tokens or wall-clock time required to reach matched downstream performance against the actual mixture- or coefficient-tuning pipeline used as the alternative.
A8: LARE adds little training overhead. In the 30B ActGD comparison at 7:1 exposure (Table 16), logged training time increased by only 0.20%, from 27.42 to 27.48 hours. In the separate three-domain comparison (Table 15), direct mixing and LARE both recorded approximately 68.29 GiB of peak GPU memory.
In the 7:1 comparison, LARE first matched or exceeded the direct run’s final MMLU and GSM8K scores at 25.69B tokens, compared with 29.88B tokens for direct mixing at the first qualifying evaluated checkpoints. This is about 14% fewer tokens to reach both benchmark targets. LARE also selects its coefficients within the training run, avoiding a separate fixed-weight search.
The first response to Reviewer CLGx
We thank the reviewer for the detailed feedback. We have added experiments on domain granularity and alternative weighting rules (Appendix G.1–G.2, Tables 15–16 and Figures 9–11). Below, we also report a comparison at nearly matched effective mixture ratios and measurements of LARE’s training overhead.
A1: The cited methods can complement LARE. Mixture selection determines how often each domain is sampled, and sample-level reweighting adjusts the contributions of individual examples within a domain. LARE adjusts the overall loss coefficient for each domain at fixed exposure. We can therefore apply LARE to a mixture selected by an existing method and combine it with within-domain sample weighting. Task-balancing methods such as GradNorm, DWA, and uncertainty weighting use different signals. Their weights could also be incorporated as an additional factor alongside LARE’s domain coefficients. Our compute budget did not allow the additional pretraining runs needed for direct comparisons with these methods or for testing their combinations with LARE.
A2: We evaluated LARE directly in autoregressive LLM pretraining. In the matched 120B- and 240B-token experiments in Table 2, LARE computes domain weights from online cross-entropy and improves mathematical reasoning over direct mixing. These results demonstrate the practical effectiveness of the rule in the pretraining setting studied here.
Theorem 1 motivates the inverse-square-root exponent through a loss–gradient-noise relationship at the optimum of a stylized logistic model. The proposed gradient measurements would test whether that relationship also holds along the LLM training trajectory and how its proportionality constants vary across domains. We no longer retain the intermediate checkpoints needed for this analysis, and our compute budget did not allow us to rerun pretraining to collect these measurements.