acceptodds
Under review as a conference paper at ICLR 2027

Beyond Token-Average Loss: Rebalancing Control and Binding in Agent Policy Distillation

Abstract

Distilling multi-step agent policies into autoregressive language models requires mastering two coupled decisions: high-level operational control and granular argument binding. Under standard action serialization, control opcodes span only one or two tokens, whereas arguments, identifiers, and syntax dominate sequence length. Consequently, standard token-mean cross-entropy subordinates critical control decisions to argument formatting, leading to brittle failures where a single mispredicted opcode derails an execution trajectory. To address this structural imbalance, our action-group-normalized objective () equalizes supervisory weight between control opcodes and argument bindings at each step. In a controlled factorial study on a shared parameter-binding-adapted Qwen3-4B backbone, we evaluate standard cross-entropy () versus balanced across expert demonstrations and matched replay substitution, strictly matching sample exposure and update budgets across three seeds. During evaluation, student policies operate fully autonomously, generating opcodes and arguments end-to-end without external guidance, policy masks, or runtime heuristics. The balanced objective yields double-digit improvements in closed-loop task success: +12.22 percentage points on replay data (95% task-cluster bootstrap interval: [9.17, 15.42], conditional on observed seeds) and +10.56 points on expert data. Fully autonomous balanced students achieve 94.17% (replay) and 95.00% (expert) mean success, matching the 95.00% point estimate of an external teacher-guided reference. In contrast, expanding offline recovery-state coverage yields no benefit over expert demonstrations under standard objectives, demonstrating that balanced supervisory allocation governs policy competence where offline state diversity alone provides no benefit. Finally, stress tests under joint tool-and-control shifts map the operational boundaries of frozen policies. Code and recorded outcomes are provided in the supplementary materials.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model agents, agent policy distillation, structured action generation, tool use, loss reweighting, imitation learning, closed-loop evaluation, multi-agent coordination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Z8qoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a group-normalized loss that equally weights opcode prediction and remaining action-token supervision to address token-length imbalance in agent policy distillation. A controlled 2×2 study on an adapted Qwen3-4B backbone, comparing loss formulations and expert versus replay training data across three seeds, reports 10.56–12.22 percentage-point improvements on synthetic partition-sum tasks, supplemented by failure diagnostics and distribution-shift tests.

Critical strengths

The matched 2×2 design controls training exposure and optimization budgets, providing useful evidence for the effect of loss reweighting within the studied setting.

Critical weaknesses
  1. Insufficient novelty and missing competitive controls. The objective is a two-group weighted cross-entropy loss, closely related to existing segment-level reweighting approaches such as FunReason[1], yet no tuned opcode-weighted baseline or weighting sweep is reported. The experiments therefore do not establish a methodological advantage beyond increasing supervision on a short, task-critical span.

  2. Claims substantially exceed the demonstrated scope. All positive results come from one parameter-binding-adapted Qwen3-4B backbone and a single synthetic partition-sum task family, while the joint-shift evaluation yields zero success across all tested conditions. This evidence supports a local training improvement, but not the claimed resolution of a fundamental bottleneck in agent policy distillation.

  3. The primary effect is highly seed-sensitive. One seed contributes 77 of the 88 net additional successes underlying the headline replay improvement; the other two seeds improve by only 1.67 and 2.92 percentage points. With only three training seeds and confidence intervals conditional on those seeds, the evidence is insufficient to establish the magnitude and reliability of the claimed stabilization.

[1] FunReason: Enhancing Large Language Models' Function Calling via Self-Refinement Multiscale Loss and Automated Data Refinement. ACL findings 2026

Questions for authors
  1. Methodological distinction: Are comparisons against tuned opcode-weighted cross-entropy or alternative group weights available, and what evidence distinguishes the proposed normalization from existing segment-level loss reweighting?

  2. Scope and initialization: How was the shared parameter-binding parent trained, and what evidence supports extending the conclusions beyond this adapted backbone and the partition-sum task family?

  3. Seed robustness: How sensitive is the primary replay estimate to individual training seeds, and are additional independent runs or seed-aware uncertainty estimates available to substantiate the claimed stabilization?

Reviewer_yUNwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how the training loss should to be allocated between action selection and argument generation during policy distillation for agents. The objective proposed treats the average loss on opcode tokens as equally important as the loss on other action tokens. Here an empirical 2 x 2 study is proposed to compare this objective against traditional cross-entropy on expert demonstrations and their respective replay counterparts.

Critical strengths

1 The controls help isolate the effect of changing the objective. In the datasets provided, the loss comparison is conducted on the same initialisation, same training samples, same sample ordering, and same number of training iterations. Therefore, the effect of altering the objective becomes apparent.

2 The improvements are evaluated based on the successful execution. Evaluation consists of the proper task completion and confirmed termination. Improvements can be seen in both the datasets. In the expert dataset, the improvement is substantial with all three seeds. This presents the evidence that the improvement is not limited to the poor replay baseline run.

Critical weaknesses

1 Earlier work have already been providing some extra weight to the key elements of the output. There is a lack of comparison with the proposed method. The authors need to make it clear why their scheme adding an equal weight to all the parts of the output is valuable compared to the previous schemes. FunReason, which has been referenced in Section 2, averages loss values across different output categories. In this work, equal weighting of opcode and other token loss values is used. An equivalent experiment using a set opcode-token weight would help understant whether the improvements depend on example-specific category weighting or stronger opcode supervision. A comparison of the schemes can be helpful here.

2 The vast majority of the gains from the replay data can be attributed to one bad training run (refering to the likes of Seed 37 contributing 77 of 88 net additional successes). Moreover, performing an experiment for one approach to replace examples is not enough to conclude that recovery examples are not useful. These results support narrower conclusions and implicaitons than the claim that the approach prevents failure.

Questions for authors
  • Does equal weighting perform better than a simpler fixed additional weight to actions?
  • Are the examples used for recovery relevant to the problems observed during evaluation?
  • Is the improvement sustainable after further training? Are there small post-hoc runs that can show these results ?
Reviewer_YBNiOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes L_BAL, a within-example objective for supervised distillation of serialized agent policies. Each supervised action is partitioned into an opcode-token group O_i (the control verb) and a complement group R_i (arguments, handles, JSON syntax, EOS); cross-entropy is averaged within each group and the two group means are averaged with fixed 0.5/0.5 weights (Eq. 3). The motivating measurement is that under token-mean CE the opcode group receives a mean per-example objective coefficient of ~3.36% (range 2.63%–6.25%) while L_BAL fixes it at 50% (Eq. 2, Fig. 2). The authors run a 2×2 factorial — L_CE vs L_BAL × expert table D₀ vs a length-matched replay-substitution table D₁ — from a shared "parameter-binding-adapted" Qwen3-4B parent with LoRA/NF4, three seeds, matched sample exposure and update budget (336 AdamW updates, one epoch), and evaluate fully autonomous closed-loop task success on a procedural partition-sum coordination task (120 held-out bases × 2 resource environments = 240 episodes per condition). Balanced students reach 94.17% on replay (from 81.94%) and 95.00% on expert data (from 84.44%), i.e. +12.22 and +10.56 pp, with a task-cluster bootstrap interval of [9.17, 15.42] on the replay contrast. Supporting analyses report a null effect of state-coverage expansion, a per-opcode failure taxonomy, a 60-pair counterfactual opcode-preference probe, and a 0/96 outcome for all nine frozen conditions under a joint tool-and-control shift. The claimed contributions are an explicit formulation of the opcode/complement length asymmetry, a controlled factorial isolating supervisory allocation from training-state distribution, and a mapping of operational boundaries.

Critical strengths

Exposure, update budget, and per-row token-length buckets are held fixed across arms; the replay rows are length-matched to the expert rows they displace; and the four cells share one parent checkpoint, one runtime, and one evaluation bank. Two consequences follow that bear directly on the assessment. First, the comparison is not confounded by the usual distillation artefacts (data volume, sequence length, compute), which is rare in this genre and is what makes the negative result on D₁ interpretable at all. Second, §6.3 volunteers the single most incriminating statistic in the paper — "Seed 37 accounted for 77 of the 88 net additional successes" — and Figure 4b publishes the per-seed deltas. The authors therefore already report the quantity that falsifies their headline; what is missing is the inference drawn from it, not the data. This matters because it means the core problems below are about interpretation and two cheap missing experiments, not about a study that would need rebuilding from scratch.

Critical weaknesses

The contribution claim — that group normalization specifically is what matters — is never tested against the baseline that would falsify it, and the same objection disposes of the novelty claim. L_BAL is algebraically a two-group inverse-frequency reweighting: it replaces |O_i|/N and |R_i|/N with ½ and ½. A static per-opcode-token weight w — equivalently, k-fold duplication of opcode tokens in a token-mean loss, a one-line change — induces coefficient w/(w|O_i|+|R_i|), and over the paper's own measured range (2.63%–6.25%, a 2.4× spread) a single w brings nearly every row into a tight band around 50%; a swept λ≈15 lands every row between roughly 39% and 94%. The §4.2 defence — "no global scalar multiplier applied to L_CE can establish consistent intra-example balance" — is an argument about exactness, not about efficacy, and it is being used as a licence to skip the baseline. Without that arm, "any loss that raises the opcode gradient by ~15× does this" remains fully live, and that hypothesis is a boring optimisation fact rather than a contribution. The same gap disposes of the novelty question concretely: §2 already cites Octopus v2 [2404.01744], FunReason [2505.20192] and GUI-Libra [2602.22190] as action-aware token reweighting, all of which upweight low-frequency action tokens; the stated differentiator ("a general structural imbalance … evaluated through controlled factorial") is a claim about evaluation, not mechanism. None of the three is implemented as a fifth arm under the identical harness, and an uncited antecedent making the same structural observation in the RL setting [2605.14558] exists. Two related measurement failures sit underneath: the load-bearing Fig. 2 plots nominal loss coefficients while §4.2 explicitly concedes these are "not measured empirical shares of realized gradients" — under LoRA+NF4+Adam the coefficient-to-gradient-share map is not the identity — and no teacher-forced per-group accuracy is reported anywhere, so the paper jumps from a coefficient claim to a control claim with no measurement in between. The §6.4 failure taxonomy (70 of 82 failures = premature WAIT) is computed for CE/replay/seed 37 alone and generalised in the abstract. Finally, the central negative result does not survive its own design: D₁ touches 1,536 of 5,376 rows (28.6%), only 326 of the replacement states are novel relative to D₀, only 237 come from failed episodes, Table 1 itself admits the interesting recovery states are "Not covered," and the source rollout succeeds 0/240 — so D₁ differs in label provenance and difficulty, not only coverage. The contrast is −2.50 pp with CI [−5.97, 0.83]; §6.2 nonetheless reads it as "slightly impaired," and the abstract elevates it to "expanding offline recovery-state coverage yields no benefit," a causal claim the manipulation cannot carry.

Questions for authors
  1. What is the seed-level uncertainty on the primary contrast? Please report a two-way bootstrap over tasks and training runs, or a hierarchical model with seed as a random effect, alongside the existing task-cluster interval, for all six Fig. 3b contrasts. Given per-seed replay deltas of {+1.67, +32.08, +2.92} pp and a seed-level interval of roughly [−30, +55], on what basis is "double-digit improvements" stated in the abstract unqualified? If you regard §6.3's 77-of-88 attribution as a variance-reduction finding rather than a mean shift, please say so explicitly and re-title the claim accordingly.
  2. Was the replay contrast pre-specified as primary before the runs, or designated after inspecting Fig. 4? If pre-specified, state where; if not, label all six contrasts exploratory. Relatedly, which arm would you promote if forced to choose the one with a defensible seed-level interval — and would that change the abstract's headline from +12.22 to the expert-arm figure?
  3. Decompose the effect by branch count. Reconstructing Fig. 5, the replay contrast is ≈0 pp at n=2, ≈+6.7 pp at n=3, and ≈+30 pp at n=4 — the only branch count never seen in training. Report the in-distribution (n∈{2,3}) and out-of-distribution (n=4) contrasts separately in the abstract and main table, state whether n=4 is intended as in-distribution or as a generalisation probe, and add an n=4-trained CE arm to separate "better control learning" from "better structural extrapolation" to unseen branch counts.
  4. Does a scalar per-opcode-token weight with w chosen once reproduce most of the effect, and does α = 0.5 matter? L_BAL's only hyperparameter is never swept; α ∈ {0.1, 0.25, 0.5, 0.75, 0.9} on at least the expert arm, 3 seeds, same protocol, would establish whether "balanced" or merely "α large enough to matter" is doing the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.