acceptodds
Under review as a conference paper at ICLR 2027

Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting

Abstract

Large language models often acquire specialized capabilities through successive pretraining stages, but these gains can come at the cost of forgetting earlier knowledge. In this work, we investigate whether such trade-offs can be improved by changing how strongly each domain contributes to learning while keeping its data exposure fixed. We find that, in controlled knowledge–mathematics experiments, loss reweighting produces a single checkpoint that surpasses the best knowledge and reasoning scores attained along either stage trajectory. This reveals an independent control that data-mixture ratios alone do not capture. Building on this finding, we propose LARE (Loss-Adaptive Reweighting under fixed Exposure), which automatically adjusts domain weights from running loss statistics. Motivated by a theoretical analysis of gradient noise, its normalized inverse-square-root rule balances relative loss scales without changing the sampled mixture or requiring additional runs to search for weights. Across knowledge, mathematics, and code, LARE improves all five evaluated benchmarks for a 1.5B model trained on 240B tokens, averaging +7.84 pp over a matched direct mixture, including gains of +17.06 pp on GSM8K and +9.00 pp on MATH. These findings demonstrate that adjusting domain loss scales can improve joint capabilities without additional domain exposure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Pretraining, Data Mixture, Multi-stage Training, Loss Reweighting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CLGxOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-domain language-model pretraining and argues that a data-mixture ratio conflates two distinct controls: domain exposure, which determines how many tokens from each domain are observed, and optimization scale, which determines how strongly each domain contributes to parameter updates. Through matched-exposure experiments on knowledge and mathematics data, the authors show that changing domain loss coefficients while keeping the sampled mixture fixed can produce a better observed knowledge–reasoning trade-off than direct mixing or stage-wise training.

Building on this observation, the paper proposes LARE, an online domain-level reweighting rule that sets each domain coefficient proportional to the inverse square root of its smoothed loss and normalizes the mixture-averaged coefficient to one. A stylized noisy logistic-regression analysis is used to motivate the inverse-square-root exponent through a loss–gradient-noise relationship. The empirical evaluation covers multiple data ratios, three base optimizers, knowledge/mathematics/code mixtures, 30B–240B training-token budgets, and models ranging from 124M to 1.5B parameters. The reported gains are particularly large on mathematical reasoning and code benchmarks, while knowledge performance is generally maintained in the principal settings.

Critical strengths
  1. A clean formulation and controlled isolation of an overlooked training variable. The paper clearly separates domain exposure from optimization scale and tests this distinction under matched model configurations, token budgets, sampled mixtures, and training streams. The comparison among stage-wise training, direct mixing, and fixed loss reweighting provides convincing evidence that domain token counts alone do not determine the final capability trade-off. This is consequential because most LLM data-mixture work primarily optimizes sampling proportions rather than studying loss scale at fixed exposure.

  2. A simple adaptive method with broad empirical support. LARE requires only running domain-loss statistics and introduces little conceptual or implementation complexity. Its behavior is evaluated across AdamW, Muon, and ActGD; several knowledge-to-mathematics ratios; two- and three-domain mixtures; multiple model sizes; and training budgets up to 240B tokens. The mathematical-reasoning and code gains are large and directionally consistent across many settings, making the method potentially useful even if the theoretical account is only approximate.

Critical weaknesses
  1. The adaptive reweighting baseline set is too limited to establish the specific contribution of LARE. Adaptive task/domain loss weighting is well established in multi-task learning, and the paper itself cites uncertainty weighting, gradient balancing, multi-objective optimization, multi-domain sampling/loss-weight analyses, and dynamic loss-based sample reweighting for LLM pretraining. However, the empirical comparison is primarily against uniform weighting and a fixed coefficient grid. Without direct comparisons to representative adaptive weighting methods, it is unclear whether the gains arise from the particular inverse-square-root rule or from adaptive domain reweighting more generally. The absence of exponent and gradient-based ablations further limits the novelty claim.

  2. The assumptions connecting the theoretical result to LARE are not validated in actual LLM training. The loss–gradient-noise theorem concerns low-noise binary logistic regression with isotropic Gaussian features, an unconstrained classifier at its population optimum, and comparable effective dimensions and hidden constants across domains. LARE instead substitutes online current cross-entropy for the unknown optimum loss and applies the rule throughout nonstationary autoregressive pretraining. Cross-entropy scales can also differ across knowledge, mathematics, and code because of intrinsic entropy, tokenization, masking, and domain partition choices. The paper does not directly verify the predicted gradient-RMS relationship in its trained models. In addition, most reported cells are single training runs, so the small MMLU differences do not currently establish that knowledge performance is consistently improved rather than statistically unchanged.

Questions for authors
  1. How does LARE compare with representative adaptive loss-weighting methods such as GradNorm, uncertainty weighting, Dynamic Weight Averaging, gradient-norm balancing, and existing loss-based sample reweighting for LLM pretraining? Such comparisons are necessary to determine whether the benefit is specific to LARE rather than to adaptive reweighting in general.

  2. Do the assumptions motivating the inverse-square-root rule approximately hold during real LLM pretraining? In particular, can the authors measure per-domain gradient covariance or RMS over training and test whether it is proportional to the square root of the corresponding domain loss? How different are the effective dimensions and proportionality constants across knowledge, mathematics, and code?

  3. How sensitive is LARE to the exponent and the substitution of online loss for optimum loss? Please compare rules of the form wi∝Li−αw_i\propto L_i^{-\alpha} for several values of α\alpha, as well as a rule based directly on measured gradient norms or variances.

  4. At a fixed parameter state, reweighting with coefficients wiw_i has the same expected gradient direction as a direct mixture with proportions mi′∝miwim'_i\propto m_iw_i, up to a global scale. Can the authors compare reweighting against these exact effective mixture ratios rather than only the relatively sparse 7:1, 3:1, and 1:1 grid? For example, 7:1 exposure with a 1:2 loss ratio corresponds to an effective gradient ratio of approximately 3.5:1. This comparison would clarify whether the empirical frontier expansion arises from fixed exposure and gradient variance rather than an insufficiently dense remix baseline.

  5. Are the headline results stable across independent training seeds? Please report training variability for the small MMLU changes and for the claim that all five benchmarks improve at the 8× budget. It would also be helpful to confirm the longer-budget, three-domain result with a standard optimizer in addition to ActGD.

  6. How sensitive is LARE to the definition and granularity of a domain? If a knowledge corpus is divided into two subdomains, or if domains have substantially different tokenization and irreducible cross-entropy, the normalization and resulting weights may change even when the underlying data are unchanged. Please characterize this sensitivity and clarify what domain partition the method assumes in practice.

  7. What exactly is accelerated by the “Fast LLM Pretraining” claim? Please report the runtime and memory overhead of LARE and, where possible, compare tokens or wall-clock time required to reach matched downstream performance against the actual mixture- or coefficient-tuning pipeline used as the alternative.

5 comments
AuthorsOpenReview ↗

The first response to Reviewer CLGx

We thank the reviewer for the detailed feedback. We have added experiments on domain granularity and alternative weighting rules (Appendix G.1–G.2, Tables 15–16 and Figures 9–11). Below, we also report a comparison at nearly matched effective mixture ratios and measurements of LARE’s training overhead.

Q1: The adaptive reweighting baseline set is too limited to establish the specific contribution of LARE. Adaptive task/domain loss weighting is well established in multi-task learning, and the paper itself cites uncertainty weighting, gradient balancing, multi-objective optimization, multi-domain sampling/loss-weight analyses, and dynamic loss-based sample reweighting for LLM pretraining. However, the empirical comparison is primarily against uniform weighting and a fixed coefficient grid. Without direct comparisons to representative adaptive weighting methods, it is unclear whether the gains arise from the particular inverse-square-root rule or from adaptive domain reweighting more generally. The absence of exponent and gradient-based ablations further limits the novelty claim.

How does LARE compare with representative adaptive loss-weighting methods such as GradNorm, uncertainty weighting, Dynamic Weight Averaging, gradient-norm balancing, and existing loss-based sample reweighting for LLM pretraining? Such comparisons are necessary to determine whether the benefit is specific to LARE rather than to adaptive reweighting in general.

A1: The cited methods can complement LARE. Mixture selection determines how often each domain is sampled, and sample-level reweighting adjusts the contributions of individual examples within a domain. LARE adjusts the overall loss coefficient for each domain at fixed exposure. We can therefore apply LARE to a mixture selected by an existing method and combine it with within-domain sample weighting. Task-balancing methods such as GradNorm, DWA, and uncertainty weighting use different signals. Their weights could also be incorporated as an additional factor alongside LARE’s domain coefficients. Our compute budget did not allow the additional pretraining runs needed for direct comparisons with these methods or for testing their combinations with LARE.

Q2: The assumptions connecting the theoretical result to LARE are not validated in actual LLM training. The loss–gradient-noise theorem concerns low-noise binary logistic regression with isotropic Gaussian features, an unconstrained classifier at its population optimum, and comparable effective dimensions and hidden constants across domains. LARE instead substitutes online current cross-entropy for the unknown optimum loss and applies the rule throughout nonstationary autoregressive pretraining. Cross-entropy scales can also differ across knowledge, mathematics, and code because of intrinsic entropy, tokenization, masking, and domain partition choices. The paper does not directly verify the predicted gradient-RMS relationship in its trained models.

Do the assumptions motivating the inverse-square-root rule approximately hold during real LLM pretraining? In particular, can the authors measure per-domain gradient covariance or RMS over training and test whether it is proportional to the square root of the corresponding domain loss? How different are the effective dimensions and proportionality constants across knowledge, mathematics, and code?

A2: We evaluated LARE directly in autoregressive LLM pretraining. In the matched 120B- and 240B-token experiments in Table 2, LARE computes domain weights from online cross-entropy and improves mathematical reasoning over direct mixing. These results demonstrate the practical effectiveness of the rule in the pretraining setting studied here.

Theorem 1 motivates the inverse-square-root exponent through a loss–gradient-noise relationship at the optimum of a stylized logistic model. The proposed gradient measurements would test whether that relationship also holds along the LLM training trajectory and how its proportionality constants vary across domains. We no longer retain the intermediate checkpoints needed for this analysis, and our compute budget did not allow us to rerun pretraining to collect these measurements.

AuthorsOpenReview ↗

The second response to Reviewer CLGx

Q3: In addition, most reported cells are single training runs, so the small MMLU differences do not currently establish that knowledge performance is consistently improved rather than statistically unchanged.

A3: We interpret the 0.01 pp MMLU difference in the 120B three-domain comparison as essentially unchanged. The +0.83 pp difference at 240B is also a single-run observation. Our compute budget did not allow us to repeat the long-budget experiments with additional seeds, so we cannot quantify training-seed variability. We will qualify the joint-improvement claim accordingly. In the reported runs, Figure 7 (Appendix D.6) shows that LARE’s GSM8K advantage over direct mixing persists across later checkpoints at both 120B and 240B. The reasoning advantage is therefore sustained over the course of training in both comparisons.

Q4: How sensitive is LARE to the exponent and the substitution of online loss for optimum loss? Please compare rules of the form wi∝Li−αw_i\propto L_i^{-\alpha} for several values of α\alpha, as well as a rule based directly on measured gradient norms or variances.

A4: We compare LARE with two alternative online weighting rules in 30B-token ActGD runs, holding the sampling ratio fixed within each comparison.

Inverse Loss scales the recent decrease in domain cross-entropy by the inverse of its slow estimate. With Fi,tF_{i,t} and Si,tS_{i,t} denoting the fast and slow loss EMAs, the score before normalization is ri,t=max⁡(Si,t−Fi,t,0)max⁡(Si,t,ϵ).r_{i,t}=\frac{\max(S_{i,t}-F_{i,t},0)}{\max(S_{i,t},\epsilon)}.

Approximate Inverse Loss uses the slow EMA E^i,t\widehat E_{i,t} of the domain’s prediction-residual energy: ri,t=(E^i,t+ϵ)−1/2.r_{i,t}=(\widehat E_{i,t}+\epsilon)^{-1/2}. For small token cross-entropy qq, the squared residual on the target class satisfies (1−e−q)2≈q2(1-e^{-q})^2\approx q^2, which motivates the approximation. Appendix G.2, Equations (47)–(49), gives the definitions and the shared normalization, clipping, and weight smoothing.

Weighting ruleK:M = 7:1K:M = 3:1K:M = 1:1
Direct mixture31.83 / 27.82—31.53 / 54.89
LARE31.99 / 34.5031.56 / 47.3830.55 / 59.74
Inverse Loss31.11 / 41.47—30.87 / 55.65
Approximate Inverse Loss32.00 / 33.5931.31 / 44.7330.89 / 50.34

Figure 10 shows that Inverse Loss pulls ahead of LARE on GSM8K after warmup at 7:1, finishing 6.97 pp higher on GSM8K and 0.88 pp lower on MMLU. At 1:1, LARE achieves the highest final GSM8K score. Figure 11 shows that Approximate Inverse Loss tracks LARE closely at 7:1, while LARE finishes higher on both benchmarks at 3:1. These comparisons vary the full online weighting rule. An exponent-only sweep and comparisons using optimum loss or measured gradient statistics remain open.

Q5: At a fixed parameter state, reweighting with coefficients wiw_i has the same expected gradient direction as a direct mixture with proportions mi′∝miwim'_i\propto m_iw_i, up to a global scale. Can the authors compare reweighting against these exact effective mixture ratios rather than only the relatively sparse 7:1, 3:1, and 1:1 grid? For example, 7:1 exposure with a 1:2 loss ratio corresponds to an effective gradient ratio of approximately 3.5:1. This comparison would clarify whether the empirical frontier expansion arises from fixed exposure and gradient variance rather than an insufficiently dense remix baseline.

A5: After masking, the coefficients on the domain mean gradients are proportional to πiwi\pi_i w_i, where πi\pi_i is the domain’s share of valid targets. Combining the realized shares in Appendix C.3 (Table 4) with the 30B Muon results in Appendix D.4 (Tables 9–10), we obtain two near matches to the 1:1 direct mixture.

K:M exposureK:M loss weightsMath share of coefficientsMMLU (%)GSM8K (%)
1:11:143.58%31.6759.89
7:11:743.56%31.1948.98
3:11:343.57%31.7556.71

The recipes reach different endpoints despite their nearly identical post-warmup coefficients on the domain mean gradients. Direct mixing gives the highest GSM8K score at this operating point, while 3:1 exposure with 1:3 weights gives slightly higher MMLU. The runs differ in exposure and warmup history, so this comparison cannot isolate gradient variance as the cause. We will limit the frontier claim to the tested recipes. Our compute budget did not allow us to run the proposed 3.5:1 direct-mixture experiment.

AuthorsOpenReview ↗

The third response to Reviewer CLGx

Q6: Are the headline results stable across independent training seeds? Please report training variability for the small MMLU changes and for the claim that all five benchmarks improve at the 8× budget. It would also be helpful to confirm the longer-budget, three-domain result with a standard optimizer in addition to ActGD.

A6: All five final scores are higher in the reported 240B LARE run in Table 2, including +0.83 pp on MMLU, +17.06 pp on GSM8K, and +9.00 pp on MATH. These are observations from one matched pair. Our compute budget precluded independent repeats and a standard-optimizer replication at the longer budget. Table 9 shows that the reasoning gains extend to AdamW and Muon at 30B. The 240B evidence is specific to ActGD.

Q7: How sensitive is LARE to the definition and granularity of a domain? If a knowledge corpus is divided into two subdomains, or if domains have substantially different tokenization and irreducible cross-entropy, the normalization and resulting weights may change even when the underlying data are unchanged. Please characterize this sensitivity and clarify what domain partition the method assumes in practice.

A7: LARE uses a fixed assignment of sources to weighting groups and computes one coefficient per group from its mean loss over valid targets. Splitting a group can change the coefficients when the resulting groups have different losses.

To test this sensitivity, we kept the 1.5B model, ActGD optimizer, and 30B-token training stream fixed. We compared three semantic domains, six groups formed by ranking the 18 sources by their calibration loss at the end of warmup, and 18 individually weighted sources. Appendix G.1 reports the final scores in Table 15 and the training curves in Figure 9.

Weighting partitionMMLU (%)GSM8K (%)
Direct mixture32.4714.25
LARE, 3 semantic domains32.2022.97
LARE, 6 groups based on calibration loss32.7424.03
LARE, 18 individual sources32.3522.67

All three partitions improve GSM8K by 8.42–9.78 pp over direct mixing, with MMLU within 0.3 pp of the baseline. The six-group partition gives the highest observed scores on both benchmarks, showing that the choice of partition can affect the outcome even at fixed exposure.

Q8: What exactly is accelerated by the “Fast LLM Pretraining” claim? Please report the runtime and memory overhead of LARE and, where possible, compare tokens or wall-clock time required to reach matched downstream performance against the actual mixture- or coefficient-tuning pipeline used as the alternative.

A8: LARE adds little training overhead. In the 30B ActGD comparison at 7:1 exposure (Table 16), logged training time increased by only 0.20%, from 27.42 to 27.48 hours. In the separate three-domain comparison (Table 15), direct mixing and LARE both recorded approximately 68.29 GiB of peak GPU memory.

In the 7:1 comparison, LARE first matched or exceeded the direct run’s final MMLU and GSM8K scores at 25.69B tokens, compared with 29.88B tokens for direct mixing at the first qualifying evaluated checkpoints. This is about 14% fewer tokens to reach both benchmark targets. LARE also selects its coefficients within the training run, avoiding a separate fixed-weight search.

Reviewer_kTgUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies loss reweighting independently of data sampling in multi-domain LLM pretraining. It proposes LARE, a simple method that adjusts domain weights using running losses while keeping the data mixture fixed. Experiments on knowledge, mathematics, and code show gains over direct mixing.

Critical strengths
  1. The method is simple and requires no additional training runs to select loss weights.

  2. The paper clearly distinguishes how much data the model sees from each domain from how strongly that domain influences optimization, providing a clear motivation for the method.

Critical weaknesses
  1. ActGD is introduced abruptly in the experimental section, despite being separate from the central loss-reweighting method. I find the decision to build the larger-budget experiments around this new optimizer insufficiently motivated, and it makes the experimental narrative harder to follow.
  2. The paper tests a few mixture ratios but gives little justification for choosing them. It is unclear whether LARE would still offer comparable gains over a well-tuned data mixture.
  3. The experiments cover only three broad domains: knowledge, mathematics, and code. The larger-budget experiments use only a 1.5B model, ActGD, and knowledge-heavy mixtures. These settings are too limited to establish how broadly the method applies.
  4. There are no experimental comparisons with existing adaptive reweighting methods. Although fixed-weight sweeps are included at 30B tokens, the larger-budget experiments compare only against direct mixing.
  5. The results come from single training runs, with no uncertainty estimates. Some MMLU gains are extremely small, including just 0.01 percentage points in Table 2. Without repeated runs, there is little basis for treating these differences as reliable improvements rather than random variation.
Questions for authors
  1. Why introduce ActGD in a paper focused on loss reweighting, and why use it for all larger-budget experiments? Could you reproduce at least one 120B or 240B comparison with AdamW or Muon to show that the gains also hold with an established optimizer?
  2. How were the mixture ratios chosen? Would LARE still provide comparable gains when applied to a mixture selected through systematic tuning, rather than the few manually specified ratios tested here?
  3. How sensitive is LARE to the definition and number of domains? Have you tested finer-grained domain splits or mixtures that are not knowledge-heavy at larger training budgets? The current three-domain setup leaves this unclear.
  4. Could you compare LARE with existing adaptive loss-reweighting methods, particularly those discussed in the related work? Also, could you include fixed-weight baselines in the 120B/240B experiments, using weights selected at 30B if full-budget tuning is too expensive? This would help distinguish the benefit of LARE’s adaptive rule from the benefit of reweighting alone.
3 comments
AuthorsOpenReview ↗

The first response to Reviewer kTgU

We thank the reviewer for the constructive suggestions. In response, we have added experiments comparing three weighting partitions on the same training stream (Appendix G.1, Table 15 and Figure 9). Below, we clarify the rationale for our optimizer and mixture choices and discuss the scope of the longer-budget results.

Q1: ActGD is introduced abruptly in the experimental section, despite being separate from the central loss-reweighting method. I find the decision to build the larger-budget experiments around this new optimizer insufficiently motivated, and it makes the experimental narrative harder to follow.

Why introduce ActGD in a paper focused on loss reweighting, and why use it for all larger-budget experiments? Could you reproduce at least one 120B or 240B comparison with AdamW or Muon to show that the gains also hold with an established optimizer?

A1: We developed ActGD during this work and found that it gave the best MMLU and GSM8K results in our 30B direct-mixture comparison (Section 4.2, Table 1). It also uses a single set of optimizer hyperparameters for the transformer blocks, embeddings, and output head. These were the reasons we chose ActGD for the larger-budget experiments. At 30B and 7:1 exposure, LARE also improves GSM8K by 11.75 pp with AdamW and 12.13 pp with Muon (Table 9). Our compute budget precluded repeating the 120B or 240B comparison with either optimizer.

Q2: The paper tests a few mixture ratios but gives little justification for choosing them. It is unclear whether LARE would still offer comparable gains over a well-tuned data mixture.

How were the mixture ratios chosen? Would LARE still provide comparable gains when applied to a mixture selected through systematic tuning, rather than the few manually specified ratios tested here?

A2: The 7:1, 3:1, and 1:1 ratios allocate 12.5%, 25%, and 50% of sampled tokens to specialized data, spanning knowledge-heavy to equal exposure. The three-domain experiments keep the same total specialized share and split it evenly between math and code. Among the ActGD direct mixtures in Table 8, 1:1 already gives the highest GSM8K score. At that exposure, LARE adds 4.85 pp on GSM8K with a 0.94 pp decrease in MMLU. This shows a further reasoning gain at a strong direct-mixture operating point, with a knowledge trade-off. A comparison on a mixture selected through systematic tuning for both metrics remains open.

Q3: The experiments cover only three broad domains: knowledge, mathematics, and code. The larger-budget experiments use only a 1.5B model, ActGD, and knowledge-heavy mixtures. These settings are too limited to establish how broadly the method applies.

How sensitive is LARE to the definition and number of domains? Have you tested finer-grained domain splits or mixtures that are not knowledge-heavy at larger training budgets? The current three-domain setup leaves this unclear.

A3: We tested finer-grained weighting partitions using the same 1.5B model, ActGD optimizer, and 30B-token training stream. We compared three semantic domains, six groups formed by ranking the 18 sources by calibration loss at the end of warmup, and 18 individually weighted sources. Appendix G.1 reports the final scores in Table 15 and the training curves in Figure 9.

Weighting partitionMMLU (%)GSM8K (%)
Direct mixture32.4714.25
LARE, 3 semantic domains32.2022.97
LARE, 6 groups based on calibration loss32.7424.03
LARE, 18 individual sources32.3522.67

All three partitions improve GSM8K by 8.42–9.78 pp over direct mixing, with MMLU within 0.3 pp of the baseline. The six-group partition gives the highest observed scores on both benchmarks. These results extend the evidence to finer partitions at 32.71B tokens.

Q4: There are no experimental comparisons with existing adaptive reweighting methods. Although fixed-weight sweeps are included at 30B tokens, the larger-budget experiments compare only against direct mixing.

Could you compare LARE with existing adaptive loss-reweighting methods, particularly those discussed in the related work?

A4: To our knowledge, the cited methods do not provide a directly matched baseline for adapting domain-level loss weights online during the target LLM pretraining run under fixed sampling proportions. DoReMi and DoGE select sampling ratios for the target run. FastMix uses loss weighting in its proxy formulation to search for those ratios. Mixture selection is orthogonal to LARE in this sense: we can train on the mixture selected by any of these methods and apply LARE within that fixed mixture.

AuthorsOpenReview ↗

The second response to Reviewer kTgU

Q5: The results come from single training runs, with no uncertainty estimates. Some MMLU gains are extremely small, including just 0.01 percentage points in Table 2. Without repeated runs, there is little basis for treating these differences as reliable improvements rather than random variation.

A5: We agree that the 0.01 pp MMLU difference in the 120B three-domain experiment is best described as essentially unchanged. In the same run, LARE improves GSM8K by 14.79 pp and MATH by 6.40 pp. Figure 7 (Appendix D.6) shows that the GSM8K advantage over direct mixing persists across later checkpoints at both 120B and 240B. Our compute budget did not allow us to repeat the long-budget experiments with additional seeds.

Q6: Also, could you include fixed-weight baselines in the 120B/240B experiments, using weights selected at 30B if full-budget tuning is too expensive? This would help distinguish the benefit of LARE’s adaptive rule from the benefit of reweighting alone.

A6: Table 10 compares LARE with fixed coefficients at 30B. Transferring a selected coefficient would test how much of the long-budget benefit can be recovered with a scale chosen at a smaller budget. We were unable to add that comparison at 120B or 240B within the available compute budget.

Thank you for the clarification and the additional experiments, but several of my main concerns remain.

  1. My concern is not whether ActGD performs well, but why all long-budget evidence for LARE is built on another new and largely independent method. The paper itself states that ActGD and LARE act at different points in the pipeline; although the paired comparisons remain internally valid, using ActGD throughout the 120B/240B experiments adds another factor and makes the results harder to compare with established pretraining work.

  2. I understand why fixing the sampling ratio. However, the practical motivation remains unclear. The paper motivates LARE partly by avoiding repeated coefficient search runs, but many modern mixture selection methods also avoid exhaustive full scale sweeps through proxy losses, generalization estimates, or smaller runs. To establish practical value, the paper should show either that fixed-exposure loss reweighting is more effective or efficient than a representative mixture-selection method under comparable cost, or that LARE provides additional gains on top of a systematically tuned mixture, i.e., tuned mixture + Direct versus tuned mixture + LARE. Without either result, the paper shows that loss scale is an independent control under manually chosen mixtures, but not that LARE offers a clear practical advantage over existing mixture optimization.

  3. I do not find the claim that there is no “directly matched” baseline convincing. The paper itself cites GradNorm, uncertainty weighting, and dynamic loss-based sample reweighting as related methods, and at least one representative method, most naturally a GradNorm-style domain-weighting baseline, could be adapted to the same fixed training stream. The current results therefore cannot separate the benefit of reweighting itself from the benefit of LARE’s adaptive rule.

  4. I agree that the large GSM8K and MATH gains are unlikely to be explained by noise alone, and that the 0.010.01 pp MMLU difference should be treated as unchanged. However, all final scores are computed once; if multiple seeds are too expensive, the authors should at least evaluate several neighboring late-training checkpoints and report their mean and variation, rather than relying on one final checkpoint or only a smoothed trajectory, to reduce sensitivity to checkpoint selection and avoid overinterpreting tiny differences. I think this is essential for a rigorous pretraining study, especially when evaluating relatively weak models.

Reviewer_eLN3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper separates domain exposure from domain loss scaling in language-model pretraining. Its controlled experiments show that reweighting losses while keeping sampled data fixed can improve the knowledge-reasoning trade-off. LARE selects normalized inverse-square-root weights from smoothed domain losses without a separate coefficient sweep. Stylized analyses distinguish remixing from reweighting and motivate the exponent. The 1.5B model trained on 240B tokens improves the five-benchmark average by 7.84 percentage points over matched direct mixing.

Critical strengths
  1. Within each controlled setting, runs reuse the same frozen stream, sample order, and loss masks (Appendix C.3, p. 24). These controls allow the stage-schedule and fixed-weight comparisons to test whether equal exposure can produce different capability trade-offs. The linear-regression result helps explain this difference under its explicitly restricted assumptions.

  2. LARE improves mathematical reasoning across all nine optimizer-mixture settings in Tables 8 to 9. In the matched 240B-token comparison, it improves GSM8K by 17.06 points, MATH by 9.00, and both code benchmarks (Table 2). The fixed-weight grids and smaller-model results help put these gains in context. Finding competitive weights within one run is a practical benefit even if a retrospectively selected grid point performs better. The token-budget-matched results support a sample-efficiency benefit; they do not establish end-to-end wall-clock acceleration (Appendix F.4).

Critical weaknesses
  1. The experiments establish the benefit of loss reweighting more clearly than the benefit of continuing to adapt the weights. Figure 5 shows a relatively stable weight ratio after warmup. In Table 10 (p. 28), fixed 1:3 weighting with AdamW obtains 31.64/35.86 MMLU/GSM8K versus LARE's 31.40/33.66, while the larger-budget experiments compare only against unit weighting. The controller may remain useful for selecting a scale, but the contribution of subsequent updates is uncertain. Theorem 1 motivates the exponent near a stylized optimum and leaves this empirical distinction unresolved.

  2. Evidence for simultaneous knowledge and reasoning improvements is less firm than the evidence for reasoning gains. Results are single runs (§4.1), and MMLU increases by only 0.01 point in the 120B-token three-domain setting (Table 2). The paper also reports knowledge decreases at reasoning-heavy mixtures (Tables 8 to 9). These results qualify the joint-improvement claim: the large reasoning gains are convincing, whereas the small positive MMLU differences need uncertainty information to carry much evidential weight.

Questions for authors
  1. What existing evidence distinguishes continued weight adaptation from selecting a stable scale early in training? If a frozen-after-warmup comparison is already available, it would help assess this distinction. Otherwise, please acknowledge the limitation when describing the controller's contribution.
  2. Is uncertainty information available for the small MMLU differences? Please distinguish evaluation uncertainty from training-seed variability and qualify joint-improvement claims accordingly.
1 comment
AuthorsOpenReview ↗

The response to Reviewer eLN3

We thank the reviewer for the constructive feedback and for recognizing the reasoning gains. We have added comparisons with alternative weighting rules at fixed exposure in Appendix G.2. Below, we also discuss what the training curves show about weight stability and the reasoning gains over training.

Q1: The experiments establish the benefit of loss reweighting more clearly than the benefit of continuing to adapt the weights. Figure 5 shows a relatively stable weight ratio after warmup. In Table 10 (p. 28), fixed 1:3 weighting with AdamW obtains 31.64/35.86 MMLU/GSM8K versus LARE's 31.40/33.66, while the larger-budget experiments compare only against unit weighting. The controller may remain useful for selecting a scale, but the contribution of subsequent updates is uncertain. Theorem 1 motivates the exponent near a stylized optimum and leaves this empirical distinction unresolved.

What existing evidence distinguishes continued weight adaptation from selecting a stable scale early in training? If a frozen-after-warmup comparison is already available, it would help assess this distinction. Otherwise, please acknowledge the limitation when describing the controller's contribution.

A1: LARE’s practical benefit is to select a competitive weight ratio within a single training run, without a separate sweep. In the Muon run in Figure 5, the math-to-knowledge weight ratio averages 2.30 from step 25,000 onward and ends at 2.41. LARE reaches an endpoint comparable to nearby fixed-weight settings in Table 10. As the reviewer notes, the fixed 1:3 setting with AdamW achieves higher scores on both benchmarks.

We have yet to test a variant that selects weights from the warmup loss statistics and then freezes them. Theorem 1 motivates the exponent in a stylized model, and Table 2 measures the longer-budget gains over unit weighting. We will clarify that the current evidence supports automatic selection of a useful loss scale, while the added benefit of continuing to update it after warmup remains unmeasured.

Q2: Evidence for simultaneous knowledge and reasoning improvements is less firm than the evidence for reasoning gains. Results are single runs (§4.1), and MMLU increases by only 0.01 point in the 120B-token three-domain setting (Table 2). The paper also reports knowledge decreases at reasoning-heavy mixtures (Tables 8 to 9). These results qualify the joint-improvement claim: the large reasoning gains are convincing, whereas the small positive MMLU differences need uncertainty information to carry much evidential weight.

Is uncertainty information available for the small MMLU differences? Please distinguish evaluation uncertainty from training-seed variability and qualify joint-improvement claims accordingly.

A2: The paper describes the +0.39 pp MMLU gain in the 120B knowledge–math experiment as modest. In the 120B three-domain experiment, MMLU is essentially unchanged for the reported checkpoints (33.52 to 33.53), while GSM8K and MATH improve by 14.79 and 6.40 pp, respectively. Tables 8–9 also show that the MMLU effect varies with the mixture.

Figure 7 (Appendix D.6) shows that LARE’s GSM8K advantage persists across many checkpoints and generally widens over training at both 120B and 240B. The reasoning gain is visible throughout the later stages of training, beyond the final scores in Table 2. Meanwhile, Our compute budget precluded repeating the 120B and 240B pretraining runs with additional seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.