acceptodds
Under review as a conference paper at ICLR 2027

JADORE: Online Data Mixing Through the Lens of Optimization Geometry

Abstract

Training data mixtures greatly impact the generalization performance of language models. Existing gradient-based mixing methods rely on raw gradients, ignoring the optimizer dynamics. We introduce *JAcobian DOmain REweighting* (JADORE), an online data mixing framework that estimates the local effect of mixtures on optimization geometry. It defines the domain scores based on the derivative of the optimizer mapping from raw gradients to actual updates. We derive the action of the optimizer's Jacobian for multiple update geometries and show its efficient computation. To ensure scalability, we develop a single backward pass implementation for both training and scoring. Extensive experiments demonstrate accuracy improvements across diverse benchmarks in both pretraining and fine-tuning on multiple datasets. These gains are consistent across optimizers and model scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Online Data Mixing, Domain Reweighting, Optimizer-Aware Learning, Language Model Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_DiyKOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a new method for online data mixing: JADORE.

Data mixing is the problem of assigning weights to different subsets of data during training so as to maximize the resulting LLM's score on some held-out evaluation.

JADORE develops a gradient-based method for online data mixing. It builds on prior work: Dynamic Gradient Alignment (DGA). The intuitive idea is to update mixing weights using the gradient of the validation loss after one optimizer update. DGA approximates this gradient as gradient alignment, which derives from a Taylor-expansion argument after taking one SGD step from the training data. JADORE improves on DGA by considering how different optimizers transform the gradient before applying an update. The core idea is simple: incorporate the Jacobian of how the optimizer transforms the gradient into your update. The authors then execute this idea well with a number of details to make the implementation fast.

They validate JADORE against a number of relevant baselines on standard setups used for data mixing, in both zero-shot and supervised fine-tuning frameworks.

[1]: "Dynamic gradient alignment for online data mixing" (Fan et al., 2024)

Critical strengths

The paper's core idea is simple and well-executed. The natural improvement for DGA is to incorporate how adaptive optimizers transform the gradient. Using the Jacobian is a sensible approach. The authors take this solid idea, derive the necessary equations for the most popular optimizers, come up with a fast streaming implementation, and then evaluate it against the important baselines from the literature.

Beyond this well-designed and executed approach, the paper is also well-written and easy to follow.

Critical weaknesses

The paper's primary weakness is that performance improvements are small; however, I believe this weakness is more a function of the problem's difficulty: despite active research, automatic data mixing methods have not found substantial improvements in these setups (e.g., zero-shot performance on SlimPajama). While the improvements are small, they appear fairly consistent, and I believe this consistency gives reasonable confidence that the method works.

Questions for authors

A few questions I was curious about:

  1. In Eq. 6: Is the KL-penalty necessary to prevent all mixing weight from going towards a single domain? IIUC, since you're linearizing the loss w.r.t. the mixing weights, then when you take the minimum over the simplex it would have to occur at a vertex (i.e., all weight on one domain).
  2. In Eq. 6: Why is the exponential moving average (via beta) necessary, and does it relate to my point above?
  3. To align the gradients between train and val, you need to evaluate the gradient on val. Your implementation computes gradients for train and val in the same batch. Is there an advantage to this approach over the simpler one of first computing the gradient for train then for val, or alternatively computing gradients for train and val on separate machines? Does your approach use less overall compute?
3 comments

We are grateful for the reviewer’s encouraging feedback and pleased that they found the core idea clear and well executed. We respond to their questions below.

Q1. The performance improvements are small.

The improvement over Uniform is even larger after adjusting some benchmarks as suggested by Reviewer hXUK, reaching 1.83- 2.94 pp on the 1.07B model.

We thank the reviewer for recognizing the consistency of the improvements. JADORE achieves the highest average accuracy among the evaluated methods in each of four optimizers across model scales 124M and 1.07B pretraining and 7B LLaMA SFT (Tables 2–3, 7–8). On SlimPajama at 1.07B, the gains are 1.83–2.94 percentage points over Uniform and 0.95–1.35 over DGA for each optimizer. We have further expanded the evaluation to include Qwen3-8B SFT (Table 11) and extended the OOD test to eight benchmarks (Tables 17-18), showing consistent improvements. These results strengthen the empirical case for optimizer-aware data mixing by demonstrating consistent aggregate improvements across optimizers, model scales, and training stages.

Q2. In Eq. 6: Is the KL-penalty necessary to prevent all mixing weight from going towards a single domain? IIUC, since you're linearizing the loss w.r.t. the mixing weights, then when you take the minimum over the simplex it would have to occur at a vertex (i.e., all weight on one domain).

The reviewer's understanding is correct. Without the KL penalty, the linearized subproblem in Eq. 6 returns a one-hot proposal whenever a single domain has the highest score. Primal averaging smooths the actual sampling weights, but repeated one-hot proposals for the same domain can still drive the mixture toward that vertex. The KL penalty gives a softmax proposal with positive weight on every domain.

Our choice of a uniform KL anchor also helps prevent the mixture from concentrating on a few domains during training. A common exponentiated-gradient update in data mixing (e.g., DoReMi, DoGE, and DGA) uses the current mixture αt\alpha_t as the KL anchor to limit changes in α\alpha. Over repeated updates, the mixture may concentrate on the highest-scoring domains. JADORE uses a fixed uniform anchor instead, so fixed scores yield the same softmax proposal at each step. We added a detailed discussion in Appendix D.3.

Q3. In Eq. 6: Why is the exponential moving average (via beta) necessary, and does it relate to my point above?

Primal averaging (the exponential moving average) in Eq. 6 stabilizes online reweighting. A small step size 1−βα1-\beta_\alpha dampens fluctuations from stochastic score estimates and limits each update's change to the mixture based on the local first-order sensitivities in Eq. 4. This indeed relates to Q2 because the fixed uniform KL anchor makes the proposal α^\hat{\alpha} independent of the current mixture αt\alpha_t for a given score vector. Primal averaging then controls how far αt\alpha_t moves toward that proposal.

Q4. To align the gradients between train and val, you need to evaluate the gradient on val. Your implementation computes gradients for train and val in the same batch. Is there an advantage to this approach over the simpler one of first computing the gradient for train then for val, or alternatively computing gradients for train and val on separate machines? Does your approach use less overall compute?

Thanks for this very good question. The main advantage of combining the training and validation examples in one batch is lower memory usage in layerwise scoring. During backpropagation at layer ℓ\ell, the training activations and backward signals (aiℓ,biℓa_i^\ell,b_i^\ell) and the Jacobian-transformed validation gradient (Jℓ⊤rℓJ^{\ell \top} r^\ell) needed for Eq. 11 are available. We compute that layer’s contribution to the domain scores and release the scoring intermediates before moving to the next layer, as we discussed in Appendix B. Separate passes would need to retain the relevant quantities from the first pass across layers until the second pass reaches them. Specifically, we would need to store aiℓ,biℓa_i^\ell,b_i^\ell for all layers and all training samples if train gradient was computed first, and Jℓ⊤rℓJ^{\ell \top} r^\ell for all layers if val gradient was computed first.

Another potential advantage is execution efficiency even at the same FLOPs: the joint pass might improve wall-clock efficiency by amortizing kernel launches and increasing GPU utilization by fusing the small validation batch into the training kernels. Running the passes on separate machines is also feasible and can overlap their computation, with the trade-off of using additional devices and communicating the quantities needed for scoring. As discussed in Appendix B, the suggested implementation strategy can exploit the specific training setup and hardware.

Thank you for your sincere engagement throughout the rebuttal.

I appreciate your answers to my questions. The discussion of your computational approach's advantage over a more naive implementation is particularly helpful. Readers weighing whether or not to adopt the more advanced approach may appreciate a brief sentence about its motivation in the main text (i.e., memory savings and increased implementation-level efficiency).

Having read your rebuttal and the other reviewers' comments, I will maintain my score.

AuthorsOpenReview ↗

Response to Reviewer DiyK

Dear Reviewer DiyK,

Thank you for your constructive feedback and recognition of our work.

Following your suggestion, we have added a brief motivation in Section 2.3 and expanded Appendix B to explain how joint batching lowers scoring memory through layerwise buffer release and improves execution efficiency.

Best regards,

The authors

Reviewer_4iWUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JADORE, an optimizer-aware online data mixing framework for language model training. Unlike existing gradient-based methods that rely on raw gradient alignment, JADORE uses the Jacobian of the optimizer's gradient-to-update mapping to estimate domain utility. The authors derive Jacobian-based scoring rules for different optimizer geometries and develop an efficient single-backward implementation to reduce computational overhead. Experiments across pretraining and supervised fine-tuning demonstrate improvements over existing data mixing baselines.

Critical strengths

The Jacobian-based formulation provides a principled way to incorporate optimizer geometry into domain reweighting, therefore, the motivation is clear.

Critical weaknesses
  1. For Signum, JADORE computes domain scores using the Jacobian of a tanh surrogate rather than the actual sign-based update. This weakens the interpretation of the proposed score as the marginal effect on the optimizer update. Further justification of the surrogate approximation would strengthen the theoretical framework.
  2. Another concern is that the improvement over OPUS is limited, at most 0.5%. This raises the concern of whether introducing Jacobian for optimizer-awareness introduces actual benefit. Reporting loss curves comparison with OPUS would be great, or report some variance.
  3. The experiment scale is limited. From-scratch pretraining experiments are limited to a maximum of a 1B model, and FT is limited to the rather old llama 2 model.
Questions for authors
  1. Could the authors provide more insight into how domain score rankings differ across optimizers? An analysis of score correlations or ranking differences could help better illustrate the effect of optimizer geometry on domain selection.
3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We appreciate that the reviewer found our use of the Jacobian to incorporate optimizer geometry into domain reweighting well motivated. We address the remaining concerns below.

Q1. For Signum, JADORE computes domain scores using the Jacobian of a tanh surrogate rather than the actual sign-based update. This weakens the interpretation of the proposed score as the marginal effect on the optimizer update. Further justification of the surrogate approximation would strengthen the theoretical framework.

As we discussed in line 240, the hard-sign map ΦSign(g)=sign⁡(m(g))\Phi_{\mathrm{Sign}}(g)=\operatorname{sign}(m(g)) is locally constant away from sign changes and discontinuous at mi=0m_i=0. Its classical Jacobian is therefore zero wherever all momentum coordinates are nonzero, so it provides no first-order signal for domain rankings and requires smoothing. Smooth sign surrogates also have precedent in binary-network training [1] and optimizer differentiation [2]. Within our LMO framework (Eq. 8), tanh⁡(m/τs)=arg⁡max⁡∥u∥∞≤1{⟨m,u⟩+τs∑ih(1+ui2)}\tanh(m/\tau_s) = \arg\max_{\|u\|_\infty\le 1}\{\langle m,u\rangle + \tau_s\sum_i h(\frac{1+u_i}{2})\} with binary entropy h(q)=−qlog⁡q−(1−q)log⁡(1−q)h(q)=-q\log q-(1-q)\log(1-q). Thus, the surrogate is a smooth regularization of the sign update. We have added this discussion to Appendix A.5 in the revised manuscript.

Q2. Improvement over OPUS.

As the reviewer suggested, we additionally plot SlimPajama 124M validation-loss trajectories for OPUS-P in Figure 9 (Appendix D.1). JADORE maintains consistently lower loss throughout training, achieving ~10× larger final loss reductions over Raw than OPUS-P:

Table 4iWU-1: Validation loss of Slimpajama 124M.

MethodAdamMuonRMNPSignum
Raw3.32513.18243.62023.3799
OPUS-P3.31943.1808——
JADORE3.26563.16693.37933.3140

Furthermore, we emphasize that our Jacobian gives a more accurate prediction of the actual update compared with the frozen preconditioner OPUS uses. Specifically, the frozen preconditioners of OPUS are PAdam=κ1Diag⁡(1/q),PMuon=γ(1+μ)L(0).P_{\mathrm{Adam}}=\kappa_1\operatorname{Diag}(1/q), \quad P_{\mathrm{Muon}}=\gamma(1+\mu)L^{(0)}. Compared with JADORE, PAdamP_{\mathrm{Adam}} drops the second term in Adam-JVG and PMuonP_{\mathrm{Muon}} ignores both normalization and subsequent Newton-Schulz factors for Muon.

We have added a detailed discussion comparing OPUS-P and JADORE in Appendix D.1, along with a concrete example. To isolate the effect of the optimizer Jacobian, we compare how accurately OPUS-P and JADORE predict the update change caused by a gradient perturbation across 1000 paired synthetic cases. JADORE achieves higher mean cosine similarity and lower mean relative ℓ2\ell_2 error for both Adam and Muon at all three perturbation sizes. For example, at the 0.5% perturbation setting, Muon’s mean cosine similarity rises from 0.168 to 0.960, and its mean relative ℓ2\ell_2 error falls from 5.87 to 0.25 compared with OPUS-P.

[1] Qin et al. "Forward and backward information retention for accurate binary neural networks.", CVPR, 2020.

[2] Huang et al. "Optimizer amalgamation.", ICLR, 2022.

AuthorsOpenReview ↗

Rebuttal (2/2)

Q3. The experiment scale is limited. From-scratch pretraining experiments are limited to a maximum of a 1B model, and FT is limited to the rather old llama 2 model.

We choose LLaMA-2-7B for SFT to follow the established evaluation setting (Wang et al., 2024) with the same backbone, LESS instruction data, and five-shot evaluation on the same nine MMLU subjects. To verify transfer to a modern, higher-capability model, we have conducted additional SFT experiments on Qwen3-8B, in the same LoRA configuration as in Appendix C.1.2, training for 1000 steps, and retuning the learning rate to the new model. JADORE consistently improves 5-shot MMLU accuracy over Uniform across all four optimizers (Table 11, Appendix C.2):

Table 4iWU-2: Qwen3-8B SFT on MMLU.

OptimizerMethodSoc.Pol.Alg.Anat.Astr.Eth.Clin.Bio.Chem.Avg.
AdamUniform85.0787.0061.0071.8588.1678.0078.1188.8957.0077.23
AdamJADORE86.0787.0062.0071.8587.5078.0078.8788.8959.0077.68
MuonUniform86.0789.0063.0069.6388.1665.0078.8788.8956.0076.07
MuonJADORE85.5788.0066.0069.6388.1669.0079.2590.2856.0076.88
RMNPUniform85.5787.0062.0071.1188.1676.0078.4988.8956.0077.03
RMNPJADORE86.0789.0061.0071.8588.8276.0078.8788.8956.0077.39
SignumUniform85.5787.0061.0071.1188.8278.0078.4989.5853.0076.96
SignumJADORE85.5787.0060.0072.5988.8278.0079.2589.5858.0077.64

Regarding the pretraining scale, our 1.07B-model SlimPajama comparison alone comprises 96 from-scratch pretraining runs (4 optimizers ×\times 8 methods ×\times 3 seeds, 21.44B tokens each; 2.06T total tokens). This made 1.07B our largest feasible controlled scale given the available compute budget. Crucially, scaling from 124M to 1.07B, JADORE's gains over uniform weights increase across all four optimizers and it keeps its advantage over the strongest baselines at every scale. The same holds for OOD generalization across eight unseen benchmarks (Tables 17 and 18). Additionally, unlike proxy-based methods (e.g., DoReMi, DoGE) that assume mixture weights transfer from tiny proxies to target models, JADORE computes JVGs directly on the target model, making the formulation robust to model scale.

Q4. Could the authors provide more insight into how domain score rankings differ across optimizers? An analysis of score correlations or ranking differences could help better illustrate the effect of optimizer geometry on domain selection.

Thank you for the suggestion. We have added a pairwise Spearman analysis to compare domain-score rankings across all six optimizer pairs for the pretraining and SFT runs shown in Figures 3 and 5, respectively. The table summarizes the mean correlation, and the new Figure 8 shows the temporal evolution.

Table 4iWU-2: Mean Spearman correlation between optimizers.

Optimizer pairPretraining <15kPretraining 15k–18kPretraining >18kSFT overall
Adam–Muon0.9810.8930.742−0.040
Adam–RMNP0.9860.9740.798−0.020
Adam–Signum0.9890.9800.8890.000
Muon–RMNP0.9860.9180.7260.440
Muon–Signum0.9900.9290.7980.080
RMNP–Signum0.9940.9950.8100.440

Domain rankings reflect training stage and optimizer geometry. Earlier in training, broad web corpora (C4, CC) dominate validation-loss reduction, yielding rank consensus. At the end of training, rankings separate geometrically: Adam and Signum (both coordinate-wise diagonal JVGs) stay closest; orthogonalizing singular directions in Muon reduces dominant web domains and amplifies distinct technical domains (StackExchange and GitHub rise), and RMNP (row-wise JVG) sits between Muon and coordinate-wise methods. In SFT, rankings exhibit much lower correlations due to optimizer-specific phase shifts; notably, Muon sharply shifts rank 1 from FLAN-v2 to CoT in the second half of training, whereas other optimizers follow different trajectories. Note that, because Eq. (6) uses continuous scores rather than ranks alone, geometry can alter weights even when rankings match.

In addition, we switch the domain weights between Adam and Muon in SFT experiments, showing that optimizer-matched domain weights are better. The detailed discussion can be found in Appendix D.2. This further supports that optimizer geometry should be considered in data mixing.

Thanks the response of author. I dont have further question and raise my score.

Reviewer_rwznOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

JADORE adjusts training data mixtures using the optimizer’s gradient-to-update Jacobian. It scores each domain by estimating how increasing its weight would reduce reference-set loss after one update. Scores are computed every step. Mixture weights are updated every 100 steps in the main setting.

The paper derives scoring rules for Adam, Muon, RMNP, and a smooth surrogate for Signum. Training and scoring share a backward pass to reduce cost. Experiments cover pretraining up to 1.07B parameters and LoRA fine-tuning of a 7B model. JADORE achieves the highest average accuracy among the tested methods across these settings.

Critical strengths

The scoring rule has a clear motivation. Raw-gradient alignment ignores how the optimizer transforms gradients into updates. JADORE accounts for this transformation. The scoring ablations help isolate its contribution from other parts of the method.

The implementation reduces the cost of online scoring. It shares training computations and avoids separate backward passes for each domain. The paper also tests less frequent scoring and scoring only the last block. Experiments across four optimizers, pretraining, and fine-tuning show that the method is useful across several settings.

Critical weaknesses

The practical cost at larger scales is not established. Detailed timing and memory results are limited to the 124M model. Full scoring adds roughly 7–8% step time and 4.5–6.9% peak memory in that setting. These costs are incurred every step, despite mixture updates occurring every 100 steps. Lower-cost variants are promising, but their cost and accuracy trade-offs are not established at larger scales.

This matters because scoring competes with training for both compute and memory. The same resources could support more training tokens or a larger microbatch. The paper does not show whether JADORE reaches a target loss or accuracy sooner in wall-clock time. Results at a fixed token budget therefore do not establish an end-to-end efficiency gain.

The loss evaluation needs a clearer separation from mixture selection. The reference set guides the mixture updates. It is unclear whether Figure 2 measures loss on this same set or on a separate held-out set. I could not find an independent language-model test loss or perplexity evaluation. The downstream results support the method’s usefulness, but do not show whether gains on the target distribution come at a cost elsewhere. The OOD results partly address this concern, though they cover only the 124M model and report no uncertainty.

Questions for authors

What are the step-time and peak-memory overheads for the 1.07B pretraining and 7B LoRA experiments? Please report both full scoring and the lower-cost variant you would recommend in practice.

Under the same hardware and wall-clock budget, does JADORE reach a target loss or accuracy sooner than the strongest baselines? Please compare time to a fixed number of steps, time to target held-out loss, and time to target accuracy. The loss should be measured on a fixed set that is not used for mixture selection. Baselines should be able to use the available memory for a larger microbatch where this improves throughput.

What exact dataset is used for Figure 2? Is it disjoint from the reference set used to update mixture weights? Please also clarify the benchmark splits used for reference-set construction and final evaluation.

Do you have loss or perplexity results on an independent held-out corpus, including results by domain? These would help assess whether the target-task gains involve regressions elsewhere.

Questions 1 and 3 concern the experimental setup and reporting. Questions 2 and 4 request additional evidence that could change my assessment.

The scoring rule and implementation are useful contributions. The accuracy results support the method in the tested settings. However, the evidence for practical efficiency remains incomplete. Cost measurements at larger scales and wall-clock comparisons would materially strengthen the paper.

I am close to a weak acceptance. The method is well motivated, and the accuracy results are promising. My main concerns are the cost at larger scales and the separation between mixture selection and evaluation. I would be happy to raise my score if the authors’ responses and supporting results address these concerns.

5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for finding our approach well motivated and our accuracy results promising, and we address the remaining concerns below.

Q1. What are the step-time and peak-memory overheads for the 1.07B pretraining and 7B LoRA experiments?

Following the reviewer's suggestion, we add the step-time and peak-memory overheads over Uniform at the larger scales for both the 1.07B pretraining and 7B LoRA SFT experiments in Tables rwzn-1,2 below and in Appendix C.3 in the revised manuscript. We observe that the overheads of 1.07B models are consistent with the 124M pretraining experiments. JADORE adds negligible GPU memory in LoRA fine-tuning, and the higher step-time overhead in SFT is mainly due to a larger reference-to-training batch ratio.

Table rwzn-1: Time and memory usage for 1.07B model pretraining.

OptimizerSequential (time)Full-layer JADORE (time / memory)Last-block JADORE (time / memory)
Adam+40.70%+9.57% / +3.81%+2.44% / +2.04%
Muon+37.77%+9.13% / +9.30%+2.27% / +3.03%
RMNP+38.18%+9.71% / +5.10%+3.10% / +2.35%
Signum+38.57%+9.24% / +3.07%+2.95% / +2.13%

Table rwzn-2: Time and memory usage for 7B LoRA SFT.

OptimizerSequential (time)Full-layer JADORE (time / memory)Last-block JADORE (time / memory)Uniform (time-matched) / JADORE avg accuracy
Adam+19.36%+16.20% / +0.38%+5.99% / +0.01%46.49/46.82
Muon+19.39%+16.39% / +1.14%+5.34% / +0.04%47.04/47.24
RMNP+20.76%+17.24% / +1.15%+5.76% / +0.04%46.62/47.25
Signum+19.32%+15.79% / +0.38%+5.75% / +0.01%45.30/46.36

We also compare JADORE with a time-matched Uniform baseline in both pretraining and SFT. In each setting, we train Uniform for additional steps to match Full-layer JADORE’s wall-clock training time. We show summarized results for SFT in Table rwzn-2, and detailed pretraining results can be found in Q2 below. JADORE achieves higher average accuracy over time-matched Uniform in both settings, and the complete table can be found in Table 16 of the revised manuscript.

The results show that: (i) our efficient scoring implementation significantly reduces computational time over standard sequential scoring, and (ii) in practice even the full-layer JADORE scoring is feasible for large-scale pretraining and fine-tuning yielding accuracy gains at time-matched budgets.

AuthorsOpenReview ↗

Rebuttal (2/3)

Q2. Under the same hardware and wall-clock budget, does JADORE reach a target loss or accuracy sooner than the strongest baselines?

To provide a comprehensive assessment of the end-to-end practical efficiency, we have performed the rigorous wall-clock and compute-matched comparisons. In Table-rwzn-3, we measured time-to-target, i.e., total runtime to reach the final held-out loss achieved by the Uniform baseline. We report time-to-target and speedup over Uniform for the two strongest baselines (DGA and ADO) as well.

Table rwzn-3: Time-to-target (min) and speedup over Uniform for 1.07B pretraining.

OptimizerUniformDGAADOJADORE
Adam240 (0%)200 (−16.7%)203 (−15.3%)161 (−32.9%)
Muon265 (0%)232 (−12.6%)227 (−14.5%)167 (−37.1%)
RMNP249 (0%)164 (−34.3%)211 (−15.3%)125 (−49.7%)
Signum236 (0%)215 (−9.1%)200 (−15.3%)140 (−40.8%)

These results demonstrate that JADORE's optimization-aware data mixing translates into substantial compute savings at fixed target. JADORE accelerates convergence to the target held-out loss, achieving up to a 49.7% reduction in time-to-target relative to Uniform.

To further validate the method's practical gains, we compare accuracy in time-matched and compute-matched scenarios. Time-matched Uniform trains for additional steps to match JADORE’s wall-clock training time. Compute-matched Uniform uses a larger training batch to account for the compute JADORE spends processing reference samples. We report the benchmark results in Table rwzn-4 below.

Table rwzn-4: Time-matched and compute-matched accuracy (%) for 124M SlimPajama pretraining.

MethodARC-CARC-EC.QAHellaSwagMMLUOBQAPIQASIQAW.G.Avg.
AdamUniform (Time-matched)21.6734.7429.8427.3926.8524.6059.1035.2151.4634.54
AdamUniform (Compute-matched)22.6135.0633.0927.4326.3627.2058.6535.3150.8335.17
AdamJADORE23.2937.4035.6328.8126.7626.1362.2136.9851.9936.58
MuonUniform (Time-matched)20.4234.5330.1727.6226.5425.0759.2535.3650.7534.41
MuonUniform (Compute-matched)21.1635.9433.9128.4027.6224.6060.8336.0351.0735.51
MuonJADORE24.1538.3336.4930.3427.6727.4063.2436.9350.8637.27
RMNPUniform (Time-matched)20.4234.1830.9027.6426.5824.2759.5235.7652.1434.60
RMNPUniform (Compute-matched)21.8433.9629.5727.3526.8625.8058.4335.7752.0934.63
RMNPJADORE20.9536.3934.1928.2227.2826.7361.5236.3951.9335.96
SignumUniform (Time-matched)21.1034.2630.8826.9426.9725.3359.1035.6951.6234.65
SignumUniform (Compute-matched)21.0833.0829.7327.2026.7324.4058.6035.9851.1434.21
SignumJADORE22.2737.2934.2328.3027.0125.9361.1436.3950.8335.93

JADORE achieves higher nine-benchmark average accuracy than both Uniform baselines for all four optimizers. Averaged across optimizers, the gains are 1.89 pp at matched wall-clock time and 1.56 pp at matched compute. In practical scenarios, JADORE thus offers two complementary benefits to users: reaching a target loss with substantially less training time, or achieving higher accuracy under the same computational budget.

Q3. What exact dataset is used for Figure 2? Is it disjoint from the reference set used to update mixture weights? Please also clarify the benchmark splits used for reference-set construction and final evaluation.

We report the validation loss trajectories in Figure 2 on a separate set disjoint from the reference set used for domain scoring, constructed using the same procedure as the reference set.

In addition, we use separate benchmark splits to generate retrieval queries for reference-set construction and to evaluate the trained models. For ARC-Challenge, ARC-Easy, OpenBookQA, and MMLU, we draw retrieval queries from the validation splits and evaluate on the full test splits. For HellaSwag, PIQA, Social IQa, WinoGrande, and CommonsenseQA, we draw retrieval queries from the training splits and evaluate on the full validation splits.

AuthorsOpenReview ↗

Rebuttal (3/3)

Q4. Do you have loss or perplexity results on an independent held-out corpus, including results by domain? These would help assess whether the target-task gains involve regressions elsewhere.

Alongside our existing OOD evaluation on 124M checkpoints (Table 17), we evaluate our 1.07B checkpoints on (i) the disjoint SlimPajama test split and (ii) additional 4 OOD benchmarks (BLiMP, LAMBADA, MathQA, CodeMMLU).

We evaluate the perplexity on the SlimPajama test split (Table rwzn-5, and Appendix C.9 in the revised manuscript). JADORE yields substantial (4.7%–6.3%) reduction in average perplexity across every optimizer. Cross-domain rebalancing under ∑dαd=1\sum_d \alpha_d = 1, i.e., prioritizing high-leverage domains at a minor frequency cost on others, is consistent with prior data mixing literature (e.g., ODM, ADO, DOGE). Nevertheless, the losses of the downweighted domains differ by only +0.05–0.09+0.05\text{--}0.09 nats, thanks to JADORE's late-stage weight recovery on ArXiv, GitHub, and StackExchange (Figure 7).

Table rwzn-5: Test perplexity on 1.07B model on SlimPajama.

OptimizerMethodArXivBooksC4CCGitHubStackExchangeWikipediaAvg.
AdamUniform4.79117.56721.61319.2052.3274.4505.33110.755
AdamJADORE5.05916.73818.93917.3782.4534.7355.78310.155
MuonUniform4.60416.31619.99917.7922.2614.2705.01610.037
MuonJADORE4.86516.15617.25116.3332.3794.5325.4609.568
RMNPUniform5.33121.43126.42123.5472.5505.0466.51112.977
RMNPJADORE5.61620.35422.86221.0822.7045.3917.13112.163
SignumUniform4.98719.01923.38320.7892.4084.6575.76811.573
SignumJADORE5.34318.25220.13518.5572.5514.9826.26410.869

Importantly, JADORE did not overfit to the target and did not cause downstream regressions, as shown below. In Table rwzn-6, we report extra downstream evaluations on additional four OOD benchmarks excluded from reference-set construction on the larger 1.07B model against Uniform and the strongest baseline.

Table rwzn-6: Additional 1.07B OOD evaluations and full 8-task OOD average (%).

OptimizerMethodMathQACodeMMLUBLiMPLAMBADA8-Task OOD Avg.
AdamUniform23.7939.4982.9741.6742.09
AdamDGA24.1536.9483.4545.8642.94
AdamJADORE24.4239.3183.8544.3443.17
MuonUniform25.4638.2282.5946.9043.11
MuonDGA25.2335.6782.2749.4143.51
MuonJADORE24.8238.2282.8549.5444.08
RMNPUniform23.3535.0380.7333.2038.97
RMNPDGA23.6537.5881.3338.3541.05
RMNPJADORE24.8239.4981.9736.2341.49
SignumUniform24.0539.4982.6640.4241.08
SignumDGA24.2534.3983.1642.1741.23
SignumJADORE24.5637.5883.6941.3042.21

JADORE achieves the highest average 8-task zero-shot average among all six methods on every optimizer (+0.97+0.97 to +2.52+2.52 pp over Uniform), further confirming that the gains do not cause systemic regressions elsewhere. This shows that simply statically increasing, e.g., Arxiv and GitHub weights without considering the impact of optimization geometry does not necessarily yield consistent gains on the downstream benchmarks. The complete table can be found in Table 18.

Reviewer_hXUKOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

JADORE performs online domain reweighting using optimizer-Jacobian scores, sd=⟨gd,J⊤r⟩s_d=\langle g_d,J^\top r\rangle, and a shared backward pass. Experiments cover pretraining and LoRA fine-tuning with four optimizers. The pretraining reference set is retrieved using target-benchmark validation queries.

Critical strengths
  1. The optimizer-Jacobian formulation gives a clear local interpretation of domain utility and accounts for transformations overlooked by raw-gradient alignment.
  2. Experiments span two corpora and four optimizers, with three-seed statistics for the main comparisons. At 1.07B, average accuracy improves by 1.30-2.19 percentage points over Uniform (Table 2).
  3. The shared-backward implementation adds approximately 7-8% step time in the measured 124M setting (Table 11).
Critical weaknesses
  1. Reference-set dependence and comparison scope. The reference set is built from the validation queries of the evaluated benchmarks (Appendix C.1.1), but its domain composition and sensitivity are not reported. The splits used for retrieval queries and final evaluation also need to be explicitly clarified for each pretraining benchmark; without this, it is impossible to determine whether they influenced the training process. Gains over the Uniform baseline conflate algorithmic improvements with access to target information. Without a benchmark-independent reference-set comparison, the general applicability to pretraining remains unclear. For targeted adaptation, a simple target-focused baseline with the same information access (the information from the validation dataset) and a matched budget is also needed.

  2. Unmeasured capability trade-offs. The GitHub and ArXiv weights drop substantially from their initial allocations for much of training (Figures 3 and 7). Although ArXiv weights are not a direct measure of mathematical content, these reallocations call for a more thorough evaluation of mathematical and code capabilities. The existing BBH and abstract-algebra results are not sufficient to determine whether gains on the target tasks come at the expense of mathematical problem solving or code generation. Improvements on the selected benchmark suite therefore do not by themselves establish that the data mixtures are better overall.

  3. Mismatch between benchmarks and model capability. Despite following approximately 1x or 2x Chinchilla budgets, even the 1.07B models score around 23-24% on MMLU, 19-20% on CommonsenseQA, and 22-26% on ARC-C, all near chance level. Small differences here provide limited evidence of improved knowledge or reasoning. PIQA and HellaSwag gains remain meaningful, but broader claims require tasks with measurable competence, more capable models, or longer training that produces informative performance. The issue is evaluation sensitivity, not compliance with scaling-law budgets.

Questions for authors
  1. What is the reference set's domain composition, and how are retrieval queries separated from final evaluation samples? Does the advantage persist with a benchmark-independent reference set and against a simple target-focused baseline using the same target information and a matched budget?
  2. Can you report mathematics and code performance, alongside fixed-weight per-domain held-out losses, to assess whether the learned mixtures trade away other capabilities?
  3. Can you provide one representative comparison where the relevant benchmarks are reliably above chance, using suitable tasks, a more capable model, or longer training? For longer training, intermediate checkpoints would help assess whether gains persist.
3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for highlighting our interpretation of domain utility and the scope of our experiments, and we address the remaining concerns below.

Q1. Reference set and comparison scope.

We clarify that we use separate benchmark splits to generate retrieval queries for reference set construction and to evaluate the trained models. In particular, for ARC-Challenge, ARC-Easy, OpenBookQA, and MMLU, retrieval queries are drawn from the validation splits and we evaluate on the full test splits; for HellaSwag, PIQA, Social IQa, WinoGrande, and CommonsenseQA, retrieval queries are drawn from the training splits and we evaluate on the full validation splits. This follows the official benchmark protocols and the public availability of labels. In all cases, retrieval and evaluation samples remain disjoint. The computed reference set is composed by 0.28% Arxiv, 0.96% Book, 64.24% C4, 31.61% CC, 0.92% Github, 1.59% StackExchange, and 0.39% Wikipedia.

To address the reviewer's concern, we are glad to provide additional results using a benchmark-independent reference set. We construct this reference set by randomly selecting samples in the validation split of SlimPajama itself; thus, the domain compositions are proportional to their size. Even with a completely random proxy (no benchmark-aligned retrieval), JADORE consistently improves average accuracy over all baselines (Table 25 in Appendix C.8). Table hXUK-1 below list its average accuracy with online data mixing baselines for comparison.

Table hXUK-1: Average accuracy comparison with benchmark-independent JADORE.

ADOODMAioliDGA (target-focused)JADORE (random DR\mathcal D_R)
Adam35.1634.4034.0635.2136.14
Muon35.8235.1635.4835.8436.76
RMNP34.6834.4733.8435.4135.63
Signum34.6734.2534.2334.3635.66

Our additional experiments show that JADORE’s performance varies little across the reference-set sizes. We report these results in Table 24 of the revised manuscript.

For targeted adaptation, we emphasize that we have already included target-focused baselines in the main paper. DGA in Tables 2 and 3 and Raw and OPUS-P in Table 4 test domain gradient alignment to the reference set. Note that the same reference set is used for all these baselines. In Slimpajama 124M, DGA, Raw and OPUS-P have almost the same overhead as ours. JADORE achieves the better peformance over them. For the 1.07B model, we show that JADORE reaches the target loss faster than DGA in response to Q2 of Reviewer rwzn.

Q2. Capability trade-offs.

Following the reviewer's suggestion, to directly investigate potential trade-offs, we conducted additional evaluations of the 1.07B checkpoints on MathQA [1] and CodeMMLU [2], which assess mathematical problem solving and code understanding, respectively. Note that these benchmarks were not included in constructing the reference set, making these evaluations an out-of-target stress test for capability retention. We report zero-shot accuracy across all four optimizers in Table hXUK-2 below.

Table hXUK-2: Math and code benchmark accuracy (%) on SlimPajama 1.07B.

MethodMathQACodeMMLU
AdamUniform23.7939.49
AdamJADORE24.4239.31
MuonUniform25.4638.22
MuonJADORE24.8238.22
RMNPUniform23.3535.03
RMNPJADORE24.8239.49
SignumUniform24.0539.49
SignumJADORE24.5637.58
4-Opt. MeanUniform24.1638.06
4-Opt. MeanJADORE24.6638.65

Remarkably, we observe that the final performance remains competitive, showing a higher optimizer mean on both MathQA and CodeMMLU, despite Arxiv and GitHub being dynamically downweighted during training.

As shown in Figure 7, JADORE does not permanently suppress these domains; instead, it prioritizes broad linguistic foundations (C4/CC) earlier and substantially increases ArXiv, GitHub, and StackExchange weights during the late decay stage. Conversely, Uniform forces a fixed 1/71/7 allocation on every domain regardless of optimization dynamics.

We additionally report the held-out losses on the training domains in Appendix C.9 in the revised manuscript, and JADORE yields substantial (4.7%–6.3%) reduction over Uniform in average perplexity across optimizers. It shows that simply statically increasing, e.g., Arxiv and GitHub weights without considering the impact of optimization geometry does not necessarily yield consistent gains on the downstream benchmarks.

[1] Amini et al. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019

[2] Nguyen et al. CodeMMLU: A multi-task benchmark for assessing code understanding & reasoning capabilities of CodeLLMs. ICLR, 2025

AuthorsOpenReview ↗

Rebuttal (2/2)

Q3. Representative comparison where the relevant benchmarks are above chance.

We thank the reviewer for pointing this out, and we agree that informative (above-chance) results on benchmarks are important for comparison. We therefore re-evaluate our checkpoints on MMLU, CommonsenseQA and ARC-C using a consistent zero-shot full-answer cloze protocol with PMI calibration, retaining accuracy as the metric. This established approach reduces reliance on answer-letter prediction and adjusts for answer-string priors [3,4]. The results on 1.07B models in Table 2 show clearer competence on CommonsenseQA and MMLU and modestly above-chance performance on ARC-C. For example, JADORE improves Muon over Uniform from 27.13% to 29.91% on ARC-Challenge, from 41.11% to 46.15% on CommonsenseQA, and from 29.08% to 30.22% on MMLU. After fixing the evaluation, the improvement of JADORE over Uniform is even larger, specifically +2.16, +2.56, +2.94, +1.83 pp for Adam, Muon, RMNP, and Signum, respectively.

We update all pretraining tables using this protocol. At 124M, CommonsenseQA and MMLU scores show informative comparison, while ARC-Challenge remains difficult for this small model. JADORE retains the highest average accuracy for each optimizer on both SlimPajama and The Pile (Tables 7 and 8).

As the reviewer suggested, we train 124M SlimPajama models longer for up to 4×4\times Chinchilla token budget and evaluate four checkpoints from each run. The average accuracy is reported below and shows that JADORE achieves consistent improvement on benchmarks over training.

Table hXUK-3: Average accuracy of intermediate checkpoints for longer training.

OptimizerMethod1×1\times2×2\times3×3\times4×4\times
AdamUniform34.6534.8935.4435.58
AdamJADORE35.3036.9437.3838.49
MuonUniform34.9335.4336.1436.41
MuonJADORE36.4737.3637.9238.07
RMNPUniform33.9134.7735.3735.89
RMNPJADORE35.9236.6937.2137.27
SignumUniform34.2334.7135.1335.67
SignumJADORE35.4536.9037.6137.83

We revise the OOD evaluation in a same way and add MathQA and CodeMMLU to the OOD tables (Tables 17, 18). The results also support the strength of JADORE.

[3] Gu et al. "Olmes: A standard for language model evaluations." In Findings of the Association for Computational Linguistics: NAACL, 2025.

[4] Holtzman et al. "Surface form competition: Why the highest probability answer isn’t always right." In Proceedings of the 2021 conference on empirical methods in natural language processing, 2021.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer hXUK,

Thank you again for your time and feedback. As the discussion period closes in one day, we would greatly appreciate hearing whether our rebuttal addresses your concerns. We are also happy to clarify if you have any further questions.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of contributions and rebuttal updates

Dear Area Chair and Reviewers,

Thank you for the constructive discussion.

We would like to highlight that JADORE provides a principled and efficient framework for optimizer-aware online data mixing. It derives domain utility from the optimizer’s gradient-to-update Jacobian and integrates scoring into training via a shared, layerwise backward pass. Our original study spans four optimizers, two pretraining corpora, 124M and 1.07B models, and LLaMA-2-7B SFT, supported by three-seed main comparisons and controlled scoring ablations.

During the rebuttal period, we have strengthened our empirical evidence to address the reviewers’ principal concerns:

  • Practical efficiency. We extend our overhead analysis to larger scales (1B pretraining and 7B SFT). Additional time- and compute-matched comparisons against Uniform confirm JADORE’s practical efficiency and accuracy gains. At the 1.07B scale, JADORE reaches Uniform’s final validation loss in approximately 33–50% less training time, outperforming the strongest baselines in time-to-target across all four optimizers.

  • Consistent gains and broader applicability. Updated evaluations and longer pretraining runs further demonstrate JADORE’s consistent improvements, including a 1.83–2.94 pp gain over Uniform at 1.07B. These gains hold even when using a benchmark-independent reference set, and Qwen3-8B SFT results extend these benefits to a stronger, modern backbone.

  • Broader generalization. We add per-domain test perplexity evaluations and extend our eight-task OOD evaluation (including math and code) to the 1.07B scale. JADORE reduces average test perplexity and improves average OOD accuracy across all four optimizers. We also analyze these aggregate gains alongside the domain-level perplexity trade-offs in data mixing.

  • Deeper analysis and discussion. Detailed comparisons with OPUS’s preconditioners show that Jacobian-based scores predict optimizer-update changes more accurately. Domain ranking analyses and cross-optimizer mixture-swapping experiments further validate the necessity of tailoring data mixtures to the specific training optimizer. Finally, we expand our discussions on mixture updates, the shared-backward implementation, and the Signum surrogate.

We have revised the manuscript to incorporate additional results, clarify evaluation protocols and data splits, and refine the scope of our claims. Note that all revisions are highlighted in blue.

Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.