- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies multi-task SFT of LLMs when tasks have substantially different amounts of training data. The authors propose AMD, which jointly fine-tunes two copies of the same base model using different task-sampling temperatures (τ): one with τ=1, where tasks are sampled proportionally to their dataset sizes, and another with τ=3, which oversamples tasks with less available data. After an initial warmup phase of standard SFT (a little over an epoch), each model of the pair is fine-tuned with a weighted combination of SFT loss and a distillation loss toward its counterpart’s predicted token distribution. The combination is weighted differently per task and distillation direction and the weights are updated periodically (every 1k steps). The reweighting is based on the validation accuracy of three pairs of models that are produced by fine-tuning the current pair for 1% of an epoch after either decreasing, increasing or keeping unchanged all the weights. Before these trial runs, the training state of both models is saved. Each task and direction chooses to increase, decrease or maintain its weight by looking at which trial run gave the student model the highest accuracy on that task, so the resulting combination of weights may never have been tested. Fine-tuning then discards the three trial runs and resumes from the saved state with the new weights. Finally, the two produced models can be merged by parameter averaging. Experiments are conducted on Qwen3-0.6B-Base, Llama-3.2-1B and Llama-3.1-8B (LoRA). The models are fine-tuned on a mixture of math, medicine, law domains and general instruction data. The fine-tuned models are evaluated across six downstream benchmarks (two per domain, none for the general data). On macro-average, their method is reported to outperform SFT baselines that use the same sampling and five tested task-balancing fine-tuning methods. The merged model their method produces is reported to further raise the average, to 2.91 points over multi-task SFT on average across the three backbones. From their ablations on Llama-3.2-1B, the authors conclude that adapting the weights per task and distillation direction over the course of fine-tuning is better (than fixed or tied weights). Across the three backbones, the best adaptive model outperforms the best fixed-weight model by 0.26–0.52 points. Ablations, seed repeats and the merging analysis are performed only on Llama-3.2-1B.
Critical strengths
S1: Mutual distillation between models trained with different task-sampling temperatures gives a higher average score than the SFT baselines on all three backbones, and on 1B it also beats mutual distillation using the same sampling. So, on 1B the comparison suggests that the gain does not come from the extra compute alone.
Critical weaknesses
W1: It is not clear that the paper's weight-update rule helps.
(a) The benefit of adapting the weights (per task and per direction) during training, one of the paper’s central contributions, is not clearly above seed noise. The only experiment with several seeds is for Llama-3.2-1B (Table 5) : over three seeds, Adaptive's average gain over the best fixed-weight version (Fixed-Uni) is 0.25 points for the empirical-sampling model, less than its own seed std (0.38), and Fixed-Uni wins one of the three runs. For the smoothed-sampling model, Adaptive wins all three runs, but only by 0.03–0.30 points. The bigger gaps on Qwen and Llama-8B are from single-seed runs and the fixed weights for those models were never tuned for them (only swept for the 1B model, Fig. 4) (see Q1, Q5).
(b) There is no direct evidence that the update rule carries any signal. The update is based on the comparison between three models. The same pair of models is finetuned three separate times for 1% of an epoch, so the resulting models have only slightly different weights and are evaluated on accuracy, which is known to fluctuate across steps (Heineman et al., 2025). On the 1B model, where the accuracy is low or close to chance, the decision could be based on just very few questions of the 100-200 in total. The paper doesn’t compare with random updates at the same intervals and doesn’t report the probe scores that were measured and used, so it’s possible decisions are driven by noise (see Q2).
W2: Joint training is never compared with sequential training, and the headline gain includes the effect of merging.
(a) There is no evidence that training the two models together is advantageous compared to training them in sequence. The paper motivates joint training by saying the supervision "evolves during training", but there is no experiment to support that, since there is no sequential baseline where a finished model is the teacher (see Q4).
(b) The headline gain is confounded by the effect of merging. The +2.91 in the abstract compares the merged AMD model with one SFT model. On Llama-3.2-1B, the only backbone where the performance of merging two SFT models is reported, almost half of the gain comes from this merging alone (+1.25 of +2.70, Table 6). Other fine-tuned pairs of τ=1 and τ=3 are also never merged (see Q3, Q6).
Questions for authors
Q1: Could you run Adaptive and Fixed-Uni on Llama-3.1-8B, where all benchmarks are well above chance, and report the average over 3 seeds? On Llama-3.2-1B, could the main tables also report the 3-seed average?
Q2: Could you add a baseline that changes the weights at random (decrease, keep or increase) at the same intervals, on Llama-3.2-1B with 3 seeds? Reporting the probe scores behind each update would also help to get an idea of how noisy the score currently is.
Q3: Could you report merged versions of the fine-tuned models you have: the two SFT models on Qwen3-0.6B and Llama-3.1-8B, the two Fixed-Uni models on all backbones, and on Llama-3.2-1B the two runs of each prior method (Table 8)? This needs no extra training.
Q4: Could you compare AMD with two simpler baselines on Llama-3.2-1B (3 seeds, averaged)? (a) A new model distilled from the Smoothed SFT model you already have, to see whether the teacher needs to be trained at the same time. (b) A single model that switches from smoothed to empirical sampling during training (Cooldown, Li et al., 2025b), to see whether the gain needs distillation or comes from seeing both data mixtures instead.
Q5: Could you also report the task loss (the negative log-likelihood of the reference answers on held-out data) for the main comparisons? On the smaller models (Qwen3-0.6B and Llama-3.2-1B), accuracy on some benchmarks (e.g. MedQA) is close to chance, and a task loss would be more informative (Heineman et al., 2025).
Q6: DEM (Ram et al., 2024) addresses the problem of unequal data by training one model per task and merging them, at roughly the cost of a single SFT run. Could you compare with that on Llama-3.2-1B, to see whether a simpler merging approach could give similar gains?
Q7: Could the authors confirm that the math and law validation examples do not overlap with the test sets?
3 comments
Q5 · Reference-answer NLL
Thank you for suggesting reference-answer NLL. Adaptive achieves lower mean NLL than SFT in both sampling branches on both smaller backbones. The medical and legal domain means show the same reduction, indicating improved modeling of the corresponding reference answers.
Qwen3-0.6B-Base
Adaptive achieves lower NLL in all 12 benchmark–branch comparisons.
| Method | Sampling | GSM8K | Math-500 | MedQA-US | MedBullets | LawBench | CMMLU-Law | Mean NLL ↓ |
|---|---|---|---|---|---|---|---|---|
| Multi-task SFT | τ=1 | 0.83 | 0.80 | 1.77 | 1.61 | 3.58 | 1.15 | 1.62 |
| Smoothed SFT | τ=3 | 1.06 | 0.82 | 1.95 | 1.70 | 4.07 | 1.17 | 1.79 |
| Adaptive | τ=1 | 0.79 | 0.77 | 1.64 | 1.40 | 3.45 | 1.09 | 1.52 |
| Adaptive | τ=3 | 0.84 | 0.78 | 1.59 | 1.35 | 3.47 | 1.10 | 1.52 |
Llama-3.2-1B
Adaptive achieves lower NLL in 11 of 12 benchmark–branch comparisons, including MedQA-US in both branches.
| Method | Sampling | GSM8K | Math-500 | MedQA-US | MedBullets | LawBench | CMMLU-Law | Mean NLL ↓ |
|---|---|---|---|---|---|---|---|---|
| Multi-task SFT | τ=1 | 1.14 | 1.04 | 3.74 | 1.83 | 2.86 | 1.43 | 2.01 |
| Smoothed SFT | τ=3 | 1.23 | 1.13 | 4.93 | 2.04 | 3.09 | 1.84 | 2.38 |
| Adaptive | τ=1 | 1.11 | 1.02 | 2.73 | 1.54 | 2.70 | 1.47 | 1.76 |
| Adaptive | τ=3 | 1.11 | 1.06 | 2.11 | 1.65 | 3.00 | 1.45 | 1.73 |
Q6 · Domain experts and DEM
Merged Adaptive achieves higher average performance than merged DEM on Llama-3.2-1B.
| Method | Math avg. | Medicine avg. | Law avg. | Average performance |
|---|---|---|---|---|
| Multi-task SFT | 13.47 | 29.98 | 25.04 | 22.83 |
| +Merged | 16.10 | 32.65 | 23.49 | 24.08 |
| Domain experts | 15.04 | 29.71 | 21.05 | 21.93 |
| +DEM | 11.10 | 21.54 | 21.81 | 18.15 |
| Adaptive | 16.46 | 31.71 | 26.93 | 25.03 |
| +Merged | 15.88 | 33.89 | 26.84 | 25.53 |
Training each task separately does not necessarily produce stronger domain experts. We found that experts for lower-resource tasks reached performance plateaus early. Further training may increase overfitting risk, while separate training may forgo positive transfer across tasks. Directly merging these experts may also introduce conflicts. How to mitigate conflicts when merging domain experts remains a worthwhile research question.
Q7 · Validation/test separation
The mathematics and law validation examples do not overlap with their corresponding test sets. We constructed the mathematics, medicine, and law validation subsets from the training split of NuminaMath-1.5, the MedQA validation set, and the objective evaluation data of DISC-Law-Eval, respectively. We will correct the descriptions of the mathematics and law validation data sources in Appendix A.1 of the revised paper.
| Task | Data source | Validation examples |
|---|---|---|
| Mathematics | NuminaMath-1.5 (train) | 100 |
| Medicine | MedQA (validation set) | 100 |
| Law | DISC-Law-Eval (objective) | 200 |
Thank you for your thoughtful feedback on our paper. In our rebuttal, we have responded to your concerns and provided additional experiments and analyses. We would appreciate your thoughts on our responses, particularly whether any questions remain unresolved. We would be happy to clarify any points or provide further information during the remaining discussion period.
We thank the reviewer for recognizing the benefits of mutual distillation across different task-sampling distributions. We also appreciate the suggestions for testing the adaptive updates and comparing simpler alternatives.
Q1 · Consistency of gains across seeds
On Llama-3.1-8B, Adaptive outperforms Fixed-Uni in both sampling branches across seeds 42, 43, and 44. These results support reproducible gains from adaptive weight updates in this setting.
We will also report three-seed averages in the Llama-3.2-1B main results tables of the revised paper.
Q2 · Value of validation feedback
Validation feedback provides useful guidance for weight updates. On Llama-3.2-1B, Random-update randomly chooses decrease, keep, or increase at the same update intervals as Adaptive. Across all three seeds, Adaptive achieves higher average performance than Random-update in both sampling branches.
We will present the probe scores at each update in a figure in the revised paper to make the results easier to interpret.
Q3 · Performance after model merging
Adaptive retains its advantage in average performance when the baselines are also merged. The merged Adaptive model outperforms merged SFT and Fixed-Uni on all three backbones, as well as the four merged prior methods on Llama-3.2-1B.
Llama-3.2-1B
Q4 · Joint versus sequential training
On Llama-3.2-1B, both Adaptive branches outperform frozen-teacher distillation and Cooldown across all three seeds. The frozen-teacher baseline tests whether distillation from an already trained Smoothed SFT model can achieve comparable performance. Cooldown tests whether a single model can achieve comparable gains by switching from smoothed to empirical sampling without distillation. These results support continued collaboration during training.