- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BATON for LLM agent reinforcement learning. BFA reallocates within-trajectory weights using feedback likelihoods under executed and alternative actions, while TMN averages trajectory-level losses uniformly. Experiments show improvements over GRPO and GiGPO across three benchmarks.
Critical strengths
The framework clearly separates within-trajectory attribution from across-trajectory aggregation and integrates easily with existing learners.
BFA uses the policy itself for feedback scoring, requiring neither an auxiliary model nor additional environment interactions.
The evaluation covers multiple model sizes and four training seeds, with component ablations and useful controls for alternative weighting schemes.
Critical weaknesses
Episode-uniform evaluation does not itself justify length-normalizing policy gradients, since token summation follows from trajectory likelihood. When should TMN be preferable to the standard episodic policy gradient?
BFA relies on policy likelihoods to model environment feedback. Although the attribution diagnostics are useful, robustness to inaccurate predictions and misleading lexical overlap needs further investigation.
Several paired confidence intervals in Table 9 include zero when comparing BATON with the stronger individual component. More seeds would strengthen evidence for combining both axes.
Questions for authors
Refer to weakness.
5 comments
Response to Reviewer heBL(2/3)
Thank you for asking us to examine feedback prediction errors and misleading lexical overlap. We have added an action–feedback matching diagnostic and feedback text-perturbation experiments in Appendix L, Tables 22 and 23 (p. 25), with the main findings summarized in Section 4.6 (p. 9). The corresponding reviewer-specific changes are highlighted in dark orange; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.
W2.1: Attribution under feedback prediction errors.
We conduct an environment-replay diagnostic to test whether the policy’s feedback likelihoods identify the corresponding candidate action. Among 1,000 sampled decisions, 804 have distinguishable candidate feedback. The policy assigns higher likelihood to the observed feedback under the actual action in 616 cases, yielding 76.62% matching accuracy.
Using rollout-based reference advantages from Appendix E, BFA reduces conditional conflict from 19.95% to 16.65% on these distinguishable cases. The group-level analysis shows how prediction errors affect attribution:
- In correctly matched cases, conditional conflict decreases by 1.27 percentage points.
- In incorrectly matched cases, conditional conflict increases by 2.18 percentage points, while this group’s resolved-mass contribution to the full batch decreases from 6.522% to 3.199%.
The aggregate improvement therefore combines better allocation within correctly matched cases with redistribution of coefficient mass across groups.
W2.2: Sensitivity to misleading lexical overlap.
We conduct text-perturbation experiments on 120 trajectories containing 2,000 decisions. Holding the model, histories, candidate actions, and reference labels fixed, we add length-matched irrelevant text or lexical distractors favoring either candidate, then recompute feedback likelihoods and complete-trajectory normalized weights.
| Feedback condition | Weight Spearman | Conditional conflict |
|---|---|---|
| Uniform | — | 17.98% |
| Original BFA | 1.0000 | 14.91% |
| Length-matched irrelevant text | 0.9821 | 15.04% |
| Distractor favoring the executed action | 0.9314 | 15.39% |
| Distractor favoring the alternative action | 0.9073 | 15.61% |
Weight correlations are measured against original BFA, and conditional conflict uses the same reference labels across conditions.
The targeted distractors increase conditional conflict by 0.48 and 0.70 percentage points relative to original BFA. Under both perturbations, BFA retains lower conditional conflict than Uniform’s 17.98%.
Response to Reviewer heBL(3/3)
Thank you for suggesting additional training seeds to assess the consistency of the component gains. We have added the complete component ablation on two supplementary training seeds in Appendix F, Table 15 (p. 21), alongside the original four-seed paired statistics in Table 14 (p. 20). The supplementary findings are summarized in Section 4.4 (p. 8). The corresponding changes are highlighted in dark orange. All references below refer to the revised manuscript.
W3: Stability of the component gains across training seeds.
The main component ablation uses four training seeds. We additionally report two supplementary seeds across both host learners, all three environments, and the corresponding model sizes. Each cell below lists the two seed results separately, with the same seed order across component settings. All values are percentages.
| Component setting | ALFWorld 1.5B All | ALFWorld 7B All | WebShop 1.5B Score | WebShop 1.5B Succ. | WebShop 7B Score | WebShop 7B Succ. | SearchQA 3B Avg. | SearchQA 7B Avg. |
|---|---|---|---|---|---|---|---|---|
| GRPO | 73.1 / 75.4 | 76.9 / 79.1 | 76.8 / 74.3 | 58.2 / 55.8 | 80.2 / 77.6 | 67.4 / 64.8 | 39.2 / 36.8 | 43.7 / 41.9 |
| GRPO + BFA | 84.3 / 83.6 | 86.6 / 84.3 | 82.7 / 81.4 | 68.2 / 66.4 | 83.5 / 82.0 | 70.4 / 68.2 | 41.8 / 39.6 | 45.3 / 43.4 |
| GRPO + TMN | 83.6 / 80.6 | 84.3 / 82.8 | 80.5 / 78.8 | 65.2 / 62.8 | 82.1 / 80.6 | 68.6 / 66.2 | 40.6 / 38.4 | 44.1 / 42.2 |
| GRPO + BATON | 85.8 / 85.1 | 88.1 / 85.8 | 85.4 / 84.8 | 69.4 / 67.2 | 84.2 / 82.3 | 71.2 / 68.8 | 42.3 / 39.4 | 47.1 / 44.8 |
| GiGPO | 87.3 / 85.8 | 91.8 / 89.6 | 84.5 / 82.9 | 69.0 / 65.4 | 87.3 / 85.2 | 77.0 / 72.8 | 43.6 / 40.5 | 48.6 / 45.9 |
| GiGPO + BFA | 94.0 / 92.5 | 94.8 / 92.5 | 89.0 / 87.5 | 74.6 / 72.2 | 90.2 / 88.9 | 80.2 / 76.8 | 43.7 / 41.2 | 49.6 / 46.9 |
| GiGPO + TMN | 91.8 / 89.6 | 92.5 / 90.3 | 86.8 / 85.1 | 72.8 / 70.4 | 88.5 / 86.8 | 77.6 / 74.8 | 44.2 / 41.7 | 48.8 / 46.1 |
| GiGPO + BATON | 94.8 / 93.3 | 95.5 / 94.0 | 90.8 / 89.6 | 77.2 / 75.4 | 90.1 / 89.5 | 81.0 / 77.4 | 44.0 / 42.4 | 50.2 / 47.1 |
Across all 16 configurations, BATON’s two-seed mean exceeds the stronger of BFA-only and TMN-only. At the individual-seed level, BATON outperforms both components in 29 of 32 comparisons; in the remaining three, it trails the stronger component by 0.1–0.2 percentage points.
These additional paired results support the consistency of the combined method’s gains across training seeds.
Response to Reviewer heBL(1/3)
Thank you for asking us to clarify TMN’s motivation and relationship to episodic policy gradients. We have revised the Abstract (p. 1) to foreground the observed trajectory-length imbalance and clarified the aggregation choice and update analysis in Section 3.4 (pp. 5–6). Controlled normalization comparisons are summarized in Section 4.6, Table 5 (p. 10), with full results in Appendix C, Table 7 (p. 17). The corresponding reviewer-specific changes are highlighted in dark orange; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.
W1.1: Motivation for TMN.
TMN is motivated by the trajectory-length imbalance observed during agent training. Figure 1 shows that failed trajectories are often longer, and the longest 25% of trajectories receive 60.3% of the aggregation mass in ALFWorld and 42.2% in WebShop under global token averaging.
TMN averages policy-surrogate terms within each complete trajectory and then averages uniformly across trajectories. This changes the outer coefficients from token-count-proportional weights to equal weights per task attempt.
The resulting coefficient concentration can be quantified by the weight-based effective sample size. For a batch of B trajectories, global token averaging gives:
ESS=∑iNi2(∑iNi)2,
whereas TMN’s uniform outer coefficients give ESS=B. Episode-uniform evaluation shares this aggregation unit; the training motivation is the observed concentration of coefficients on longer trajectories.
W1.2: Relationship to episodic policy gradients and applicable settings.
Standard episodic policy gradients sum token-level contributions because the trajectory likelihood factorizes over tokens. TMN introduces trajectory-specific scaling by 1/Ni in the host learner’s surrogate.
With the rollout batch, advantages, and attribution weights held fixed, Proposition 2 gives the exact gradient difference:
gflat−gTMN=NˉCovB(N,g),
where gi is the gradient of trajectory i’s token-averaged surrogate. This identity isolates the length-dependent weighting removed by TMN. When all trajectories have equal valid policy-token counts, the two reducers coincide.
We directly compare TMN with global-token averaging and constant-denominator reduction, which retain within-trajectory token summation. Using Qwen2.5-1.5B, GRPO, and two supplementary training seeds, equal-budget validation-based learning-rate selection gives:
These comparisons support TMN in the tested agent-training settings, where trajectory lengths vary substantially and failures tend to be longer. Its gains persist both with and without BFA. TMN controls aggregation across complete trajectories, while BFA redistributes coefficients within each trajectory and preserves its total coefficient mass.