acceptodds
Under review as a conference paper at ICLR 2027

USA: Update-aware SAM for Cross-domain On-Policy Disitllation of Language Agents

Abstract

On-policy distillation instils multi-turn agentic reasoning through dense token-level supervision on the student's own trajectories, but a single domain saturates early, so further supervision has to be drawn from other domains. Multi-domain data mixing is the most direct way of incorporating them, at the cost of conflicts between their data distributions and of retraining the entire model whenever one domain is revised. Model merging avoids both by distilling every domain independently and fusing the resulting task vectors afterwards. We find instead that the benefit polarizes across domain pairs: on those exhibiting negative transfer, every merging operator we evaluate falls below the single-domain reference. We attribute this to cross-domain update coupling, where a substantial fraction of coordinates is updated comparably by both domains and a merge can therefore displace them by as much as their own updates. To overcome this limitation, we propose USA, which converts per-parameter update magnitudes measured during a brief warm-up into per-coordinate perturbation radii, reducing curvature precisely on the coordinates that carry most of the merging displacement. Experiments across mathematics, science and code at two student scales show USA strongest in all six transfer directions, ahead of the single-domain reference by more than four points on average, and reverse the negative transfer of the conflicting pairs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Agent Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UzzrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether language agents trained separately on math, code, and science can be merged into a model that performs better than the single-domain models. The authors find that merging does not always help: code and math usually benefit each other, whereas math and science are prone to negative transfer.

To address this, the authors propose Update-aware SAM (USA). Specifically, each domain first runs 10 steps of on-policy distillation (OPD), and the importance of each parameter is determined by the magnitude of its update. Parameters with larger updates are then assigned a larger SAM perturbation radius. Subsequent training requires the model to keep a low loss within this non-uniform perturbation neighborhood, with the goal of making these parameters more robust to the displacement introduced by merging. Each domain is still trained independently, and the final model is obtained by plain weight averaging, without sharing data or gradients across domains.

USA achieves strong results on all six transfer directions (across the math, code, and science domains) for both students (Qwen3-1.7B and Qwen3-4B), with average improvements of 4.55 and 4.01 percentage points over single-domain OPD, respectively.

Critical strengths
  1. The problem being addressed is valuable. Compared with methods that require repeated joint multi-domain retraining, USA preserves the modular workflow of training each expert separately, while making the experts more suitable for merging already at training time.

  2. The empirical results are strong and convincing. USA is effective not only on domain pairs that naturally exhibit positive transfer, but also improves the originally conflicting math–science pair. For example, for the 1.7B student on Math → Science, single-domain OPD reaches 48.42, plain weight averaging drops to 45.44, while USA reaches 53.50. For the 4B student in the same direction, the three numbers are 61.71, 59.02, and 67.04, respectively.

  3. The experimental coverage and reporting are generally thorough. The main experiments and the code-interpreter experiments are run with five independent seeds, and both the main table and the per-benchmark tables report uncertainty. The appendix further extends the setting to up to five source domains and evaluates TIES, DARE, and AdaMerging. The search-agent experiments use different teacher scales and tool environments, and USA outperforms the compared baselines on all six directions without re-tuning its hyperparameters. These extensions show that the conclusions do not hinge on a single domain pair or a single merging operator.

  4. The method is simple to implement, and the actual training overhead is reported. The method itself is concise (the two forward/backward passes of USA reuse the same batch of student trajectories and teacher scores, without requiring an additional full rollout), and it is well supported by the experiments.

Critical weaknesses

The scope of the evidence for broader generalization and multi-domain capability retention remains somewhat limited.

Both the students and the teachers in the main experiments come from the Qwen3 family, and the main ablations and mechanistic analyses focus on the 1.7B student. It is currently unclear whether the update-magnitude-driven scaling transfers equally well to other backbones.

Questions for authors

see weakness

Reviewer_1CyeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Update-aware Sharpness Aware Minimization (SAM) to address the negative interference in multi-domain on-policy distillation (OPD). It use parameter-update magnitude during an initial warm-up phrase to adaptively allocate the flatness constraint. Experiments on the cross-domain transfer setting between three domains reports improvements over existing OPD and model merging baseline.

Critical strengths
  • Addressing negative interference in multi-domain and cross-domain OPD is a timely and important research problem.
Critical weaknesses
  • The paper is not well-written or structured, making it difficult to follow. Several abbreviations and technical terms are not properly introduced, e.g. SAM in the abstract.
  • The writing is also inconsistent. The paper motivates the problem through multi-domain OPD in the abstract, then switches to cross-domain OPD in the introduction. The preliminary briefly discuss multi-turn OPD, then simply reduce the training objective (Eq. 2) to single turn-OPD. The main method and experiment primarily focuses on cross-domain model merging.
  • Similarly, the notation is also inconsistent. For example, the indices kk and tt are reused for interaction turns, token positions, and domains across Sections 2 and 3.
  • The related work is inadequate, mainly focusing on OPD and ignoring multi-domain and cross-domain OPD, the main focus of the paper. There is a longer related work section in the appendix, but it also does not discuss multi-domain OPD.
  • Given these issues, the paper is difficult to follow, making it challenging to properly assess its technical soundness. Substantial revisions are needed to improve clarity, better articulate its contributions.
Questions for authors

See weakness

Reviewer_H45bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Update-aware SAM (USA) to make independently distilled language agents easier to merge across domains. USA uses parameter changes during a short warm-up to assign larger sharpness-aware perturbations to heavily updated parameters, aiming to make them more tolerant of merging. Contributions include this training method, an analysis linking merging interference to local curvature, and empirical evidence connecting overlapping updates with negative transfer.

Across math, science, and coding at two student scales, USA performs best in all six transfer directions, improving over single-domain distillation by roughly 4–4.6 percentage points on average. Five-run experiments, component ablations, alternative merging operators, and additional search-agent experiments support the approach.

Critical strengths
  1. A targeted training-time solution to merging interference. USA uses early parameter updates to concentrate sharpness-aware regularization where merging is expected to cause disruption, while preserving independent domain training. The curvature analysis motivates this design, and ablations against uniform SAM, randomized scales, and alternative scaling rules support the importance of update-aware allocation. This makes the contribution more convincing than a generic application of SAM.

  2. Consistent empirical gains with meaningful controls. USA leads all six transfer directions at both student scales, with five-run results showing average gains of roughly 4–4.6 percentage points over single-domain distillation. Comparisons separating single-domain training gains from merging gains, together with tests across merging operators, strengthen the evidence that USA improves cross-domain compatibility rather than merely producing stronger individual experts.

Critical weaknesses
  1. The proposed mechanism is only partially validated. The coupling statistic ignores update signs and local curvature, both of which determine whether merging is harmful. Moreover, the paper acknowledges that its aggregate coupling plot does not directly test the assumption concerning update-salient coordinates. The local curvature bound motivates USA, but does not establish that its scaling rule reduces actual merging loss. This weakens the causal explanation despite convincing performance gains.

  2. Generality remains uncertain. All experiments use Qwen3 models, and the main results cover only three domains with two relatively small student scales. The search-agent extension broadens the setting but uses single-run evaluations. Because USA relies on early updates identifying the parameters needing protection, evidence across different model families and update dynamics is important to establish its broader applicability.

Questions for authors
  1. Clarification of the mechanism: Have you directly tested Assumption 1 on the update-salient coordinates, rather than through the aggregate coupling distribution? Does coupling predict merging degradation after accounting for update signs and curvature? This would strengthen the proposed explanation of negative transfer.

  2. Clarification of the warm-up estimate:How reliably do the update magnitudes after the short warm-up identify the coordinates carrying the eventual merging displacement? How sensitive are the gains to warm-up length? This determines whether USA’s fixed perturbation geometry rests on a reliable early signal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.