- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a new method for online data mixing: JADORE.
Data mixing is the problem of assigning weights to different subsets of data during training so as to maximize the resulting LLM's score on some held-out evaluation.
JADORE develops a gradient-based method for online data mixing. It builds on prior work: Dynamic Gradient Alignment (DGA). The intuitive idea is to update mixing weights using the gradient of the validation loss after one optimizer update. DGA approximates this gradient as gradient alignment, which derives from a Taylor-expansion argument after taking one SGD step from the training data. JADORE improves on DGA by considering how different optimizers transform the gradient before applying an update. The core idea is simple: incorporate the Jacobian of how the optimizer transforms the gradient into your update. The authors then execute this idea well with a number of details to make the implementation fast.
They validate JADORE against a number of relevant baselines on standard setups used for data mixing, in both zero-shot and supervised fine-tuning frameworks.
[1]: "Dynamic gradient alignment for online data mixing" (Fan et al., 2024)
Critical strengths
The paper's core idea is simple and well-executed. The natural improvement for DGA is to incorporate how adaptive optimizers transform the gradient. Using the Jacobian is a sensible approach. The authors take this solid idea, derive the necessary equations for the most popular optimizers, come up with a fast streaming implementation, and then evaluate it against the important baselines from the literature.
Beyond this well-designed and executed approach, the paper is also well-written and easy to follow.
Critical weaknesses
The paper's primary weakness is that performance improvements are small; however, I believe this weakness is more a function of the problem's difficulty: despite active research, automatic data mixing methods have not found substantial improvements in these setups (e.g., zero-shot performance on SlimPajama). While the improvements are small, they appear fairly consistent, and I believe this consistency gives reasonable confidence that the method works.
Questions for authors
A few questions I was curious about:
- In Eq. 6: Is the KL-penalty necessary to prevent all mixing weight from going towards a single domain? IIUC, since you're linearizing the loss w.r.t. the mixing weights, then when you take the minimum over the simplex it would have to occur at a vertex (i.e., all weight on one domain).
- In Eq. 6: Why is the exponential moving average (via beta) necessary, and does it relate to my point above?
- To align the gradients between train and val, you need to evaluate the gradient on val. Your implementation computes gradients for train and val in the same batch. Is there an advantage to this approach over the simpler one of first computing the gradient for train then for val, or alternatively computing gradients for train and val on separate machines? Does your approach use less overall compute?
3 comments
Thank you for your sincere engagement throughout the rebuttal.
I appreciate your answers to my questions. The discussion of your computational approach's advantage over a more naive implementation is particularly helpful. Readers weighing whether or not to adopt the more advanced approach may appreciate a brief sentence about its motivation in the main text (i.e., memory savings and increased implementation-level efficiency).
Having read your rebuttal and the other reviewers' comments, I will maintain my score.
Response to Reviewer DiyK
Dear Reviewer DiyK,
Thank you for your constructive feedback and recognition of our work.
Following your suggestion, we have added a brief motivation in Section 2.3 and expanded Appendix B to explain how joint batching lowers scoring memory through layerwise buffer release and improves execution efficiency.
Best regards,
The authors
Rebuttal
We are grateful for the reviewer’s encouraging feedback and pleased that they found the core idea clear and well executed. We respond to their questions below.
The improvement over Uniform is even larger after adjusting some benchmarks as suggested by Reviewer hXUK, reaching 1.83- 2.94 pp on the 1.07B model.
We thank the reviewer for recognizing the consistency of the improvements. JADORE achieves the highest average accuracy among the evaluated methods in each of four optimizers across model scales 124M and 1.07B pretraining and 7B LLaMA SFT (Tables 2–3, 7–8). On SlimPajama at 1.07B, the gains are 1.83–2.94 percentage points over Uniform and 0.95–1.35 over DGA for each optimizer. We have further expanded the evaluation to include Qwen3-8B SFT (Table 11) and extended the OOD test to eight benchmarks (Tables 17-18), showing consistent improvements. These results strengthen the empirical case for optimizer-aware data mixing by demonstrating consistent aggregate improvements across optimizers, model scales, and training stages.
The reviewer's understanding is correct. Without the KL penalty, the linearized subproblem in Eq. 6 returns a one-hot proposal whenever a single domain has the highest score. Primal averaging smooths the actual sampling weights, but repeated one-hot proposals for the same domain can still drive the mixture toward that vertex. The KL penalty gives a softmax proposal with positive weight on every domain.
Our choice of a uniform KL anchor also helps prevent the mixture from concentrating on a few domains during training. A common exponentiated-gradient update in data mixing (e.g., DoReMi, DoGE, and DGA) uses the current mixture αt as the KL anchor to limit changes in α. Over repeated updates, the mixture may concentrate on the highest-scoring domains. JADORE uses a fixed uniform anchor instead, so fixed scores yield the same softmax proposal at each step. We added a detailed discussion in Appendix D.3.
Primal averaging (the exponential moving average) in Eq. 6 stabilizes online reweighting. A small step size 1−βα dampens fluctuations from stochastic score estimates and limits each update's change to the mixture based on the local first-order sensitivities in Eq. 4. This indeed relates to Q2 because the fixed uniform KL anchor makes the proposal α^ independent of the current mixture αt for a given score vector. Primal averaging then controls how far αt moves toward that proposal.
Thanks for this very good question. The main advantage of combining the training and validation examples in one batch is lower memory usage in layerwise scoring. During backpropagation at layer ℓ, the training activations and backward signals (aiℓ,biℓ) and the Jacobian-transformed validation gradient (Jℓ⊤rℓ) needed for Eq. 11 are available. We compute that layer’s contribution to the domain scores and release the scoring intermediates before moving to the next layer, as we discussed in Appendix B. Separate passes would need to retain the relevant quantities from the first pass across layers until the second pass reaches them. Specifically, we would need to store aiℓ,biℓ for all layers and all training samples if train gradient was computed first, and Jℓ⊤rℓ for all layers if val gradient was computed first.
Another potential advantage is execution efficiency even at the same FLOPs: the joint pass might improve wall-clock efficiency by amortizing kernel launches and increasing GPU utilization by fusing the small validation batch into the training kernels. Running the passes on separate machines is also feasible and can overlap their computation, with the trade-off of using additional devices and communicating the quantities needed for scoring. As discussed in Appendix B, the suggested implementation strategy can exploit the specific training setup and hardware.