acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Mutual Distillation for Balanced Multi-Task Post-Training of Large Language Models

Abstract

Multi-task post-training of large language models (LLMs) aims to improve performance across tasks with unequal amounts of training data. Existing methods focus primarily on balancing task contributions during single-model training. Different task-balancing strategies can produce models with complementary strengths, creating opportunities for mutual distillation. However, the usefulness of cross-model supervision can vary across tasks, transfer directions, and stages of training. We propose Adaptive Mutual Distillation (AMD), a collaborative post-training framework that jointly trains two models with different task-balancing strategies. AMD evaluates candidate adjustments to distillation weights through short training probes shared across tasks, then uses task-wise validation scores to select an adjustment for each task and transfer direction. Across six benchmarks and three LLM backbones, both AMD models achieve higher average benchmark scores than supervised fine-tuning (SFT) baselines trained with the same sampling strategies. They also outperform the task-balancing methods evaluated in our experiments. Merging the two trained models can further improve their average benchmark score while yielding a single model for inference. The merged models outperform multi-task SFT by an average of 2.91 points across the three backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multi-task learning, post-training

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_F1kjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multi-task SFT of LLMs when tasks have substantially different amounts of training data. The authors propose AMD, which jointly fine-tunes two copies of the same base model using different task-sampling temperatures (τ): one with τ=1, where tasks are sampled proportionally to their dataset sizes, and another with τ=3, which oversamples tasks with less available data. After an initial warmup phase of standard SFT (a little over an epoch), each model of the pair is fine-tuned with a weighted combination of SFT loss and a distillation loss toward its counterpart’s predicted token distribution. The combination is weighted differently per task and distillation direction and the weights are updated periodically (every 1k steps). The reweighting is based on the validation accuracy of three pairs of models that are produced by fine-tuning the current pair for 1% of an epoch after either decreasing, increasing or keeping unchanged all the weights. Before these trial runs, the training state of both models is saved. Each task and direction chooses to increase, decrease or maintain its weight by looking at which trial run gave the student model the highest accuracy on that task, so the resulting combination of weights may never have been tested. Fine-tuning then discards the three trial runs and resumes from the saved state with the new weights. Finally, the two produced models can be merged by parameter averaging. Experiments are conducted on Qwen3-0.6B-Base, Llama-3.2-1B and Llama-3.1-8B (LoRA). The models are fine-tuned on a mixture of math, medicine, law domains and general instruction data. The fine-tuned models are evaluated across six downstream benchmarks (two per domain, none for the general data). On macro-average, their method is reported to outperform SFT baselines that use the same sampling and five tested task-balancing fine-tuning methods. The merged model their method produces is reported to further raise the average, to 2.91 points over multi-task SFT on average across the three backbones. From their ablations on Llama-3.2-1B, the authors conclude that adapting the weights per task and distillation direction over the course of fine-tuning is better (than fixed or tied weights). Across the three backbones, the best adaptive model outperforms the best fixed-weight model by 0.26–0.52 points. Ablations, seed repeats and the merging analysis are performed only on Llama-3.2-1B.

Critical strengths

S1: Mutual distillation between models trained with different task-sampling temperatures gives a higher average score than the SFT baselines on all three backbones, and on 1B it also beats mutual distillation using the same sampling. So, on 1B the comparison suggests that the gain does not come from the extra compute alone.

Critical weaknesses

W1: It is not clear that the paper's weight-update rule helps.

(a) The benefit of adapting the weights (per task and per direction) during training, one of the paper’s central contributions, is not clearly above seed noise. The only experiment with several seeds is for Llama-3.2-1B (Table 5) : over three seeds, Adaptive's average gain over the best fixed-weight version (Fixed-Uni) is 0.25 points for the empirical-sampling model, less than its own seed std (0.38), and Fixed-Uni wins one of the three runs. For the smoothed-sampling model, Adaptive wins all three runs, but only by 0.03–0.30 points. The bigger gaps on Qwen and Llama-8B are from single-seed runs and the fixed weights for those models were never tuned for them (only swept for the 1B model, Fig. 4) (see Q1, Q5).

(b) There is no direct evidence that the update rule carries any signal. The update is based on the comparison between three models. The same pair of models is finetuned three separate times for 1% of an epoch, so the resulting models have only slightly different weights and are evaluated on accuracy, which is known to fluctuate across steps (Heineman et al., 2025). On the 1B model, where the accuracy is low or close to chance, the decision could be based on just very few questions of the 100-200 in total. The paper doesn’t compare with random updates at the same intervals and doesn’t report the probe scores that were measured and used, so it’s possible decisions are driven by noise (see Q2).

W2: Joint training is never compared with sequential training, and the headline gain includes the effect of merging.

(a) There is no evidence that training the two models together is advantageous compared to training them in sequence. The paper motivates joint training by saying the supervision "evolves during training", but there is no experiment to support that, since there is no sequential baseline where a finished model is the teacher (see Q4).

(b) The headline gain is confounded by the effect of merging. The +2.91 in the abstract compares the merged AMD model with one SFT model. On Llama-3.2-1B, the only backbone where the performance of merging two SFT models is reported, almost half of the gain comes from this merging alone (+1.25 of +2.70, Table 6). Other fine-tuned pairs of τ=1 and τ=3 are also never merged (see Q3, Q6).

Questions for authors

Q1: Could you run Adaptive and Fixed-Uni on Llama-3.1-8B, where all benchmarks are well above chance, and report the average over 3 seeds? On Llama-3.2-1B, could the main tables also report the 3-seed average?

Q2: Could you add a baseline that changes the weights at random (decrease, keep or increase) at the same intervals, on Llama-3.2-1B with 3 seeds? Reporting the probe scores behind each update would also help to get an idea of how noisy the score currently is.

Q3: Could you report merged versions of the fine-tuned models you have: the two SFT models on Qwen3-0.6B and Llama-3.1-8B, the two Fixed-Uni models on all backbones, and on Llama-3.2-1B the two runs of each prior method (Table 8)? This needs no extra training.

Q4: Could you compare AMD with two simpler baselines on Llama-3.2-1B (3 seeds, averaged)? (a) A new model distilled from the Smoothed SFT model you already have, to see whether the teacher needs to be trained at the same time. (b) A single model that switches from smoothed to empirical sampling during training (Cooldown, Li et al., 2025b), to see whether the gain needs distillation or comes from seeing both data mixtures instead.

Q5: Could you also report the task loss (the negative log-likelihood of the reference answers on held-out data) for the main comparisons? On the smaller models (Qwen3-0.6B and Llama-3.2-1B), accuracy on some benchmarks (e.g. MedQA) is close to chance, and a task loss would be more informative (Heineman et al., 2025).

Q6: DEM (Ram et al., 2024) addresses the problem of unequal data by training one model per task and merging them, at roughly the cost of a single SFT run. Could you compare with that on Llama-3.2-1B, to see whether a simpler merging approach could give similar gains?

Q7: Could the authors confirm that the math and law validation examples do not overlap with the test sets?

3 comments

We thank the reviewer for recognizing the benefits of mutual distillation across different task-sampling distributions. We also appreciate the suggestions for testing the adaptive updates and comparing simpler alternatives.

Q1 · Consistency of gains across seeds

On Llama-3.1-8B, Adaptive outperforms Fixed-Uni in both sampling branches across seeds 42, 43, and 44. These results support reproducible gains from adaptive weight updates in this setting.

MethodSamplingSeed 42Seed 43Seed 44Mean ± Std.
Fixed-Uniτ=145.5945.2846.1545.68 ± 0.44
Fixed-Uniτ=346.4446.0245.9746.14 ± 0.26
Adaptiveτ=146.6647.0646.6246.78 ± 0.24
Adaptiveτ=346.8946.6446.6146.71 ± 0.15
+ Merged-47.4047.3147.1947.30 ± 0.11

We will also report three-seed averages in the Llama-3.2-1B main results tables of the revised paper.

Q2 · Value of validation feedback

Validation feedback provides useful guidance for weight updates. On Llama-3.2-1B, Random-update randomly chooses decrease, keep, or increase at the same update intervals as Adaptive. Across all three seeds, Adaptive achieves higher average performance than Random-update in both sampling branches.

MethodSamplingSeed 42Seed 43Seed 44Mean ± Std.
Random-updateτ=124.1123.8324.2424.06 ± 0.21
Random-updateτ=324.1323.7923.9423.95 ± 0.17
Adaptiveτ=125.0324.3224.4424.60 ± 0.38
Adaptiveτ=324.5424.7024.5224.59 ± 0.10

We will present the probe scores at each update in a figure in the revised paper to make the results easier to interpret.

Q3 · Performance after model merging

Adaptive retains its advantage in average performance when the baselines are also merged. The merged Adaptive model outperforms merged SFT and Fixed-Uni on all three backbones, as well as the four merged prior methods on Llama-3.2-1B.

Llama-3.2-1B

Merged methodSFTPCGradFAMOCoBaMFTCoderFixed-UniAdaptive
Average performance24.0824.2024.4424.1923.5724.9825.53
Merged methodQwen3-0.6B-BaseLlama-3.1-8B
SFT41.7945.94
Fixed-Uni42.5846.70
Adaptive43.7547.40

Q4 · Joint versus sequential training

On Llama-3.2-1B, both Adaptive branches outperform frozen-teacher distillation and Cooldown across all three seeds. The frozen-teacher baseline tests whether distillation from an already trained Smoothed SFT model can achieve comparable performance. Cooldown tests whether a single model can achieve comparable gains by switching from smoothed to empirical sampling without distillation. These results support continued collaboration during training.

MethodSampling / schemeSeed 42Seed 43Seed 44Mean ± Std.
Frozen-teacher distillationFixed Smoothed SFT teacher23.4622.1322.4622.68 ± 0.69
CooldownSmoothed → empirical sampling19.5420.5620.4820.19 ± 0.56
Adaptiveτ=125.0324.3224.4424.60 ± 0.38
Adaptiveτ=324.5424.7024.5224.59 ± 0.10

Q5 · Reference-answer NLL

Thank you for suggesting reference-answer NLL. Adaptive achieves lower mean NLL than SFT in both sampling branches on both smaller backbones. The medical and legal domain means show the same reduction, indicating improved modeling of the corresponding reference answers.

Qwen3-0.6B-Base

Adaptive achieves lower NLL in all 12 benchmark–branch comparisons.

MethodSamplingGSM8KMath-500MedQA-USMedBulletsLawBenchCMMLU-LawMean NLL ↓
Multi-task SFTτ=10.830.801.771.613.581.151.62
Smoothed SFTτ=31.060.821.951.704.071.171.79
Adaptiveτ=10.790.771.641.403.451.091.52
Adaptiveτ=30.840.781.591.353.471.101.52

Llama-3.2-1B

Adaptive achieves lower NLL in 11 of 12 benchmark–branch comparisons, including MedQA-US in both branches.

MethodSamplingGSM8KMath-500MedQA-USMedBulletsLawBenchCMMLU-LawMean NLL ↓
Multi-task SFTτ=11.141.043.741.832.861.432.01
Smoothed SFTτ=31.231.134.932.043.091.842.38
Adaptiveτ=11.111.022.731.542.701.471.76
Adaptiveτ=31.111.062.111.653.001.451.73

Q6 · Domain experts and DEM

Merged Adaptive achieves higher average performance than merged DEM on Llama-3.2-1B.

MethodMath avg.Medicine avg.Law avg.Average performance
Multi-task SFT13.4729.9825.0422.83
+Merged16.1032.6523.4924.08
Domain experts15.0429.7121.0521.93
+DEM11.1021.5421.8118.15
Adaptive16.4631.7126.9325.03
+Merged15.8833.8926.8425.53

Training each task separately does not necessarily produce stronger domain experts. We found that experts for lower-resource tasks reached performance plateaus early. Further training may increase overfitting risk, while separate training may forgo positive transfer across tasks. Directly merging these experts may also introduce conflicts. How to mitigate conflicts when merging domain experts remains a worthwhile research question.

Q7 · Validation/test separation

The mathematics and law validation examples do not overlap with their corresponding test sets. We constructed the mathematics, medicine, and law validation subsets from the training split of NuminaMath-1.5, the MedQA validation set, and the objective evaluation data of DISC-Law-Eval, respectively. We will correct the descriptions of the mathematics and law validation data sources in Appendix A.1 of the revised paper.

TaskData sourceValidation examples
MathematicsNuminaMath-1.5 (train)100
MedicineMedQA (validation set)100
LawDISC-Law-Eval (objective)200

Thank you for your thoughtful feedback on our paper. In our rebuttal, we have responded to your concerns and provided additional experiments and analyses. We would appreciate your thoughts on our responses, particularly whether any questions remain unresolved. We would be happy to clarify any points or provide further information during the remaining discussion period.

Reviewer_3ibJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a mutual learning framework to address data imbalance in multi-task training, where tasks with larger datasets tend to dominate. It trains two models using different task-sampling strategies and allows them to learn from each other. Experiments across several math, medical, and legal benchmarks show that both models outperform existing methods.

Critical strengths
  • The paper deals with a common but realistic problem: uneven task data. The idea of using two differently trained models to help each other is simple and intuitive.

  • The paper includes extensive experiments across three model sizes and compares the proposed method against a wide range of baselines. The method consistently outperforms all of them, even when the baselines are given twice as much training time.

Critical weaknesses
  • AMD needs two models in memory, an extra forward pass through the partner on every batch, and three probe runs with state save/restore at each weight update. The paper reports no memory information. The baselines are single-model, so the comparison is not cost-matched.

  • The two models train on different batches, so the partner often has not trained on the sample it is scoring. The paper does not explain what knowledge it provides in that case. It also never ablates shared versus separate batches, and does not report how often the two models see the same sample. The gains may come from task-level differences, but this is never tested.

  • Parameter averaging gives the best results, but the paper does not show that the two models remain in a compatible region of weight space or that distillation draws them closer. The gain is shown empirically only.

  • The paper does not specify the KL direction, whether full-vocabulary logits are used, or whether the teacher side is detached.

Questions for authors
  • What are the GPU memory, time per step, and total training time of AMD compared with the single-model baselines?

  • Adaptive updates use only 100 to 200 validation examples per task. How sensitive are the selected weights to validation set size and noise?

4 comments

We thank the reviewer for recognizing the practical relevance of the problem, the intuitive design of our mutual learning framework, and its consistent performance gains. We also appreciate the constructive suggestions.

W1 / Q1 · Training cost and performance

To make training costs broadly comparable, we train single-model baselines for 6 epochs and AMD for 3 epochs per model on Llama-3.2-1B. FAMO is the strongest single-model baseline in this setting, and AMD still achieves the highest average performance (Table 10). The table summarizes resource usage and performance measured on 8 × NVIDIA A800 80GB GPUs.

MethodEpochs/modelPeak memory (GiB/GPU)Step time (s)Performance
Multi-task SFT631.960.92323.05
FAMO631.961.22224.16
Adaptive371.932.05425.03

In our experimental setting, five probe rounds add approximately 0.15 epochs of training. Assuming the same average step time as regular training, the training portion of these probes adds approximately 5% to the time for the regular three-epoch training budget.

We will add these resource measurements and experimental settings to the revised paper.

W2 · Knowledge transfer with separate batches

Both models have already trained on the full training set during SFT warmup before mutual distillation begins. Empirical and smoothed sampling give them different task mixtures, and the partner's predictions on the current batch convey task knowledge accumulated through its training history.

The cross-temperature (1,3) pair outperforms the same-temperature controls with shared or separate batches, supporting mutual learning across different task mixtures. On Llama-3.2-1B, we add a (1,1) shared-batch control to the comparisons in Table 4(a). At (1,1), both models score higher with shared batches than with separate batches.

Sampling temperatures(1, 1)(1, 1)(3, 3)(1, 3)
BatchesSeparateSharedSeparateSeparate
Branch average performance22.50 / 22.4522.59 / 23.3323.18 / 22.2224.07 / 23.92

We sampled 100 cross-temperature mutual-distillation steps and measured a same-step sample overlap of 3.5%. Of the distinct examples used by each model, 75.5% are also used by its partner within this window. Shared examples are therefore mostly encountered at different training steps.

W3 · Parameter compatibility and model merging

Thank you for the suggestion. Our method centers on collaborative training of multiple models, and we merge the trained models for deployment as a single model. We assess their parameter compatibility through linear interpolation.

The evaluated Adaptive branches are compatible for parameter averaging. At λ=0.5, equal-weight merging lowers test NLL from 1.76/1.73 to 1.69 and improves performance from 25.03/24.54 to 25.53. At every tested interior point, NLL remains below the higher endpoint value, supporting a low-loss connection between the branches. Adaptive also outperforms Baseline at every tested mixing weight.

Here, λ is the parameter-mixing weight of the τ=3 model, with 1−λ assigned to the τ=1 model.

Model pairMetricλ=0λ=0.2λ=0.4λ=0.5λ=0.6λ=0.8λ=1
BaselineNLL2.012.102.202.212.242.312.38
AdaptiveNLL1.761.751.711.691.681.681.73
BaselinePerformance22.8324.0623.8024.0824.2022.9422.97
AdaptivePerformance25.0325.0725.2025.5325.4024.5224.54

W4 · Distillation loss implementation

In each direction, we compute DKL(pteacher∥pstudent)D_{\mathrm{KL}}(p_{\mathrm{teacher}}\Vert p_{\mathrm{student}}) using full-vocabulary distributions, with teacher-side gradients stopped. The loss is averaged over response tokens. We apologize for the confusion caused by not stating these details explicitly in the main text.

Q2 · Sensitivity and effectiveness of validation feedback

Thank you for the suggestion. Larger representative validation sets generally help reduce sampling noise. We held the Llama-3.2-1B candidate models fixed and resampled the validation sets. Agreement with the actions selected using 100/100/200 mathematics/medicine/law examples is 91.88% at 95/95/192 and 85.28% at 90/90/180.

To test the value of feedback in the original setting, Random-update replaces score-based action selection with random choices at the same intervals. Adaptive achieves higher average performance than Random-update in both branches for all three seeds, showing reproducible gains from validation-guided weight selection.

MethodSamplingSeed 42Seed 43Seed 44Mean ± Std.
Random-updateτ=124.1123.8324.2424.06 ± 0.21
Random-updateτ=324.1323.7923.9423.95 ± 0.17
Adaptiveτ=125.0324.3224.4424.60 ± 0.38
Adaptiveτ=324.5424.7024.5224.59 ± 0.10

Thank you for your thoughtful feedback on our paper. In our rebuttal, we have responded to your concerns and provided additional experiments and analyses. We would appreciate your thoughts on our responses, particularly whether any questions remain unresolved. We would be happy to clarify any points or provide further information during the remaining discussion period.

Reviewer_3ibJOpenReview ↗

Reply to the Authors

Thanks to the authors for their detailed response and for addressing my concerns. I think adding these details into the paper will provide better clarity. I will increase my score from 2 to 3.

Comments on the paper

3 comments

We thank the reviewer for recognizing the benefits of collaboration between models with different sampling strategies and the evidence supporting these benefits.

Q1. Additional benefit of adaptive updates after merging.

On Llama-3.2-1B, merged Adaptive outperforms merged Fixed-Uni across all three seeds, with a mean paired gain of 0.53 points. This comparison supports the benefit of adaptive updates even after both methods undergo merging.

Merged modelSeed 42Seed 43Seed 44Mean ± Std.
Fixed-Uni24.9824.8225.0424.95 ± 0.12
Adaptive25.5325.6125.2825.47 ± 0.17

We also conducted training with three seeds on Llama-3.1-8B. Adaptive outperforms Fixed-Uni in both sampling branches across all three seeds.

MethodSamplingSeed 42Seed 43Seed 44Mean ± Std.
Fixed-Uniτ=145.5945.2846.1545.68 ± 0.44
Fixed-Uniτ=346.4446.0245.9746.14 ± 0.26
Adaptiveτ=146.6647.0646.6246.78 ± 0.24
Adaptiveτ=346.8946.6446.6146.71 ± 0.15

Q2. Validation data sources and separation from test sets.

We constructed the mathematics, medicine, and law validation subsets by sampling from the NuminaMath-1.5 training split, MedQA, and DISC-Law-Eval objective evaluation data, respectively. The mathematics and law validation examples are disjoint from their corresponding test sets. We will correct the descriptions of their sources in the revised Appendix A.1.

TaskData sourceValidation examples
MathematicsNuminaMath-1.5 (train)100
MedicineMedQA (validation)100
LawDISC-Law-Eval (objective)200

Thanks for authors' reply, most of my concerns are sloved, and i decide to remain my socre.

Thank you for taking the time to review our responses. We are glad that they have addressed most of your concerns. We appreciate your constructive feedback and thoughtful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.