acceptodds
Desk rejected at ICLR 2027

Dual-Axis Policy Optimization for LLM Agents: Bayesian Feedback Attribution and Trajectory Mass Normalization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

A reinforcement-learning objective for LLM agents makes two weighting choices that are usually left implicit: which decisions inside a trajectory receive credit, and how complete trajectories are weighted in the batch. We make both choices explicit. For the first, we propose Bayesian Feedback Attribution (BFA): we score the observed environment feedback under the executed action and under a counterfactual action sampled from the same policy, and use the resulting posterior lift to reallocate the host learner’s advantages across decisions while conserving each trajectory’s total coefficient mass. The policy model itself serves as the feedback model, so no learned dynamics model, return predictor, or auxiliary loss is required. For the second, we show that global token averaging weights trajectories by a size-biased transform of the episode distribution, and we derive the exact finite-batch discrepancy to uniform per-trajectory averaging as a length–gradient covariance term. Trajectory Mass Normalization (TMN) weights episodes uniformly, matching how benchmarks evaluate: one vote per episode. The resulting framework, BATON, introduces no new hyperparameters and no additional environment interaction, and drops into GRPO and GiGPO. On ALFWorld, WebShop, and search-augmented question answering across 1.5B–7B models, each component improves its host on its own, and the combination is best or tied for best in every comparison we ran, at 10–17% additional wall-clock cost measured on ALFWorld at the 1.5B scale.

Primary area:
reinforcement learning
Keywords:
Agentic RL, reinforcement learning, large language model

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_heBLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BATON for LLM agent reinforcement learning. BFA reallocates within-trajectory weights using feedback likelihoods under executed and alternative actions, while TMN averages trajectory-level losses uniformly. Experiments show improvements over GRPO and GiGPO across three benchmarks.

Critical strengths

The framework clearly separates within-trajectory attribution from across-trajectory aggregation and integrates easily with existing learners.

BFA uses the policy itself for feedback scoring, requiring neither an auxiliary model nor additional environment interactions.

The evaluation covers multiple model sizes and four training seeds, with component ablations and useful controls for alternative weighting schemes.

Critical weaknesses

Episode-uniform evaluation does not itself justify length-normalizing policy gradients, since token summation follows from trajectory likelihood. When should TMN be preferable to the standard episodic policy gradient?

BFA relies on policy likelihoods to model environment feedback. Although the attribution diagnostics are useful, robustness to inaccurate predictions and misleading lexical overlap needs further investigation.

Several paired confidence intervals in Table 9 include zero when comparing BATON with the stronger individual component. More seeds would strengthen evidence for combining both axes.

Questions for authors

Refer to weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer heBL(1/3)

Thank you for asking us to clarify TMN’s motivation and relationship to episodic policy gradients. We have revised the Abstract (p. 1) to foreground the observed trajectory-length imbalance and clarified the aggregation choice and update analysis in Section 3.4 (pp. 5–6). Controlled normalization comparisons are summarized in Section 4.6, Table 5 (p. 10), with full results in Appendix C, Table 7 (p. 17). The corresponding reviewer-specific changes are highlighted in dark orange; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.

W1.1: Motivation for TMN.

TMN is motivated by the trajectory-length imbalance observed during agent training. Figure 1 shows that failed trajectories are often longer, and the longest 25% of trajectories receive 60.3% of the aggregation mass in ALFWorld and 42.2% in WebShop under global token averaging.

TMN averages policy-surrogate terms within each complete trajectory and then averages uniformly across trajectories. This changes the outer coefficients from token-count-proportional weights to equal weights per task attempt.

The resulting coefficient concentration can be quantified by the weight-based effective sample size. For a batch of BB trajectories, global token averaging gives:

ESS=(∑iNi)2∑iNi2,\mathrm{ESS}=\frac{\left(\sum_i N_i\right)^2}{\sum_i N_i^2},

whereas TMN’s uniform outer coefficients give ESS=B\mathrm{ESS}=B. Episode-uniform evaluation shares this aggregation unit; the training motivation is the observed concentration of coefficients on longer trajectories.

W1.2: Relationship to episodic policy gradients and applicable settings.

Standard episodic policy gradients sum token-level contributions because the trajectory likelihood factorizes over tokens. TMN introduces trajectory-specific scaling by 1/Ni1/N_i in the host learner’s surrogate.

With the rollout batch, advantages, and attribution weights held fixed, Proposition 2 gives the exact gradient difference:

gflat−gTMN=Cov⁡B(N,g)Nˉ,g_{\mathrm{flat}}-g_{\mathrm{TMN}}=\frac{\operatorname{Cov}_B(N,g)}{\bar N},

where gig_i is the gradient of trajectory ii’s token-averaged surrogate. This identity isolates the length-dependent weighting removed by TMN. When all trajectories have equal valid policy-token counts, the two reducers coincide.

We directly compare TMN with global-token averaging and constant-denominator reduction, which retain within-trajectory token summation. Using Qwen2.5-1.5B, GRPO, and two supplementary training seeds, equal-budget validation-based learning-rate selection gives:

ReductionBFAALFWorld successWebShop success
Global token meanNo74.0%57.8%
Constant denominatorNo80.5%63.1%
TMNNo82.8%65.1%
Global token meanYes84.7%67.4%
Constant denominatorYes84.3%67.4%
TMNYes86.0%69.0%

These comparisons support TMN in the tested agent-training settings, where trajectory lengths vary substantially and failures tend to be longer. Its gains persist both with and without BFA. TMN controls aggregation across complete trajectories, while BFA redistributes coefficients within each trajectory and preserves its total coefficient mass.

AuthorsOpenReview ↗

Response to Reviewer heBL(2/3)

Thank you for asking us to examine feedback prediction errors and misleading lexical overlap. We have added an action–feedback matching diagnostic and feedback text-perturbation experiments in Appendix L, Tables 22 and 23 (p. 25), with the main findings summarized in Section 4.6 (p. 9). The corresponding reviewer-specific changes are highlighted in dark orange; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.

W2.1: Attribution under feedback prediction errors.

We conduct an environment-replay diagnostic to test whether the policy’s feedback likelihoods identify the corresponding candidate action. Among 1,000 sampled decisions, 804 have distinguishable candidate feedback. The policy assigns higher likelihood to the observed feedback under the actual action in 616 cases, yielding 76.62% matching accuracy.

Using rollout-based reference advantages from Appendix E, BFA reduces conditional conflict from 19.95% to 16.65% on these distinguishable cases. The group-level analysis shows how prediction errors affect attribution:

  • In correctly matched cases, conditional conflict decreases by 1.27 percentage points.
  • In incorrectly matched cases, conditional conflict increases by 2.18 percentage points, while this group’s resolved-mass contribution to the full batch decreases from 6.522% to 3.199%.

The aggregate improvement therefore combines better allocation within correctly matched cases with redistribution of coefficient mass across groups.

W2.2: Sensitivity to misleading lexical overlap.

We conduct text-perturbation experiments on 120 trajectories containing 2,000 decisions. Holding the model, histories, candidate actions, and reference labels fixed, we add length-matched irrelevant text or lexical distractors favoring either candidate, then recompute feedback likelihoods and complete-trajectory normalized weights.

Feedback conditionWeight SpearmanConditional conflict
Uniform—17.98%
Original BFA1.000014.91%
Length-matched irrelevant text0.982115.04%
Distractor favoring the executed action0.931415.39%
Distractor favoring the alternative action0.907315.61%

Weight correlations are measured against original BFA, and conditional conflict uses the same reference labels across conditions.

The targeted distractors increase conditional conflict by 0.48 and 0.70 percentage points relative to original BFA. Under both perturbations, BFA retains lower conditional conflict than Uniform’s 17.98%.

AuthorsOpenReview ↗

Response to Reviewer heBL(3/3)

Thank you for suggesting additional training seeds to assess the consistency of the component gains. We have added the complete component ablation on two supplementary training seeds in Appendix F, Table 15 (p. 21), alongside the original four-seed paired statistics in Table 14 (p. 20). The supplementary findings are summarized in Section 4.4 (p. 8). The corresponding changes are highlighted in dark orange. All references below refer to the revised manuscript.

W3: Stability of the component gains across training seeds.

The main component ablation uses four training seeds. We additionally report two supplementary seeds across both host learners, all three environments, and the corresponding model sizes. Each cell below lists the two seed results separately, with the same seed order across component settings. All values are percentages.

Component settingALFWorld 1.5B AllALFWorld 7B AllWebShop 1.5B ScoreWebShop 1.5B Succ.WebShop 7B ScoreWebShop 7B Succ.SearchQA 3B Avg.SearchQA 7B Avg.
GRPO73.1 / 75.476.9 / 79.176.8 / 74.358.2 / 55.880.2 / 77.667.4 / 64.839.2 / 36.843.7 / 41.9
GRPO + BFA84.3 / 83.686.6 / 84.382.7 / 81.468.2 / 66.483.5 / 82.070.4 / 68.241.8 / 39.645.3 / 43.4
GRPO + TMN83.6 / 80.684.3 / 82.880.5 / 78.865.2 / 62.882.1 / 80.668.6 / 66.240.6 / 38.444.1 / 42.2
GRPO + BATON85.8 / 85.188.1 / 85.885.4 / 84.869.4 / 67.284.2 / 82.371.2 / 68.842.3 / 39.447.1 / 44.8
GiGPO87.3 / 85.891.8 / 89.684.5 / 82.969.0 / 65.487.3 / 85.277.0 / 72.843.6 / 40.548.6 / 45.9
GiGPO + BFA94.0 / 92.594.8 / 92.589.0 / 87.574.6 / 72.290.2 / 88.980.2 / 76.843.7 / 41.249.6 / 46.9
GiGPO + TMN91.8 / 89.692.5 / 90.386.8 / 85.172.8 / 70.488.5 / 86.877.6 / 74.844.2 / 41.748.8 / 46.1
GiGPO + BATON94.8 / 93.395.5 / 94.090.8 / 89.677.2 / 75.490.1 / 89.581.0 / 77.444.0 / 42.450.2 / 47.1

Across all 16 configurations, BATON’s two-seed mean exceeds the stronger of BFA-only and TMN-only. At the individual-seed level, BATON outperforms both components in 29 of 32 comparisons; in the remaining three, it trails the stronger component by 0.1–0.2 percentage points.

These additional paired results support the consistency of the combined method’s gains across training seeds.

Reviewer_SVnfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes two simple and effective methods, BFA and TMN, to improve credit assignment and trajectory weighting in long-horizon agent reinforcement learning. The proposed methods are well motivated, and the experimental results demonstrate their effectiveness across different settings.

Critical strengths
  1. Simple and effective methods. Both BFA and TMN are straightforward and effective. The motivations are reasonable, and the authors clearly explain the problems and their proposed solutions.

  2. Comprehensive experiments. The experiments cover multiple benchmarks, model scales, and RL algorithms. The results and ablation studies provide convincing evidence of the effectiveness of the proposed methods.

Critical weaknesses
  1. Limited theoretical justification for BFA. The theoretical motivation behind BFA appears somewhat naive. In particular, using the policy model's likelihood of environment feedback as a proxy for action influence relies on assumptions that are not sufficiently justified.

  2. Limited novelty of TMN. Although TMN is simple and effective, it essentially replaces global token averaging with uniform trajectory-level averaging. This is a rather straightforward modification, and it is unclear whether it constitutes a sufficiently novel academic contribution.

Questions for authors

Please refer to the "Critical Weaknesses". Overall, I think this paper could get an accept

2 comments
AuthorsOpenReview ↗

Response to Reviewer SVnf(1/2)

Thank you for asking us to strengthen BFA’s theoretical justification and validate its attribution signal. We have clarified the prior-to-lift and lift-to-weight derivation in Section 3.3 (pp. 4–5), added an action–feedback matching diagnostic in Appendix L, Table 22 (p. 25), and added a return-based attribution diagnostic in Section 4.6, Table 4 (p. 9). The corresponding reviewer-specific changes are highlighted in purple; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.

W1: Theoretical justification for BFA and the validity of policy feedback likelihoods as an attribution signal.

1. Theoretical basis: from feedback likelihoods to training weights.

BFA uses the frozen policy’s feedback likelihoods to update support over candidate actions. Equations (6)–(8) construct an equal-prior candidate set containing the executed action and one independently sampled alternative. Let f0f_0 and f1f_1 denote the likelihoods of the same observed feedback under these candidates. The executed action’s posterior-to-prior lift is:

e=2f0f0+f1=2σ(s0−s1).e=\frac{2f_0}{f_0+f_1}=2\sigma(s_0-s_1).

This ratio measures how observing the feedback changes support for the executed action under the policy-provided feedback model.

Proposition 1(ii) connects this evidence to the training allocation. For a trajectory with original token-proportional decision shares btb_t, the BFA allocation uniquely maximizes:

∑tμtlog⁡et−DKL(μ∥b),\sum_t \mu_t\log e_t-D_{\mathrm{KL}}(\mu\|b),

over the probability simplex. Its solution is:

μt∗=btet∑sbses.\mu_t^{*}=\frac{b_t e_t}{\sum_s b_s e_s}.

Thus, BFA adjusts the original allocation using feedback evidence while preserving total trajectory coefficient mass. The host learner supplies the advantage and update sign; BFA determines how its coefficients are distributed across decisions.

2. Empirical validation of the feedback signal.

The environment-replay diagnostic in Table 22 tests whether feedback scoring identifies the corresponding candidate action. Among 1,000 sampled decisions, 804 have distinguishable candidate feedback. The policy correctly matches the action and feedback in 616 cases (76.62%). On these distinguishable cases, BFA reduces conditional conflict with rollout-estimated reference advantages from 19.95% to 16.65%.

The return-based diagnostic in Table 4 evaluates 120 trajectories containing 2,000 decisions, estimating reference advantages from final task returns using 16 Q and 16 V continuations per decision. Lift correlates more strongly with reference-advantage magnitude than length-normalized executed likelihood (Spearman 0.263 vs. 0.114), and BFA reduces overall conditional conflict from 17.98% to 14.91%. When goal progress occurs at step 4 or later, conflict decreases from 22.61% to 19.52%.

3. Controlled training evidence for candidate-relative scoring.

With TMN held fixed, Appendix D, Table 10 (p. 18) compares uniform allocation, executed-likelihood weighting, and BFA:

WeightingALFWorld successWebShop success
Uniform82.6%63.8%
Executed likelihood only83.8%65.3%
BFA85.8%68.3%

BFA achieves higher success rates in both environments. Together, the Bayesian construction specifies the feedback evidence, Proposition 1 establishes the conserved allocation rule, and the diagnostics and controlled training comparisons support the usefulness of candidate-relative feedback scoring.

AuthorsOpenReview ↗

Response to Reviewer SVnf(2/2)

Thank you for asking us to clarify TMN’s contribution. We have revised Section 3.4 (pp. 5–6) and Section 5 (p. 10) to explain its update analysis, relationship to sequence-mean reduction, and integration with BFA. Training length statistics appear in Appendix C, Table 8 (p. 17), and controlled normalization comparisons are summarized in Section 4.6, Table 5 (p. 10), with full results in Appendix C, Table 7 (p. 17). The corresponding reviewer-specific changes are highlighted in purple; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.

W2: TMN’s relationship to existing reduction schemes and its contribution.

TMN applies sequence-mean reduction to complete multi-turn trajectories. When a sequence is already defined as a complete trajectory, the reduction forms coincide. Our contribution is to analyze this aggregation choice under the length imbalance observed in agent training and integrate it with Bayesian feedback attribution in a controlled, empirically evaluated framework.

1. Exact analysis of the aggregation choice.

Under global token averaging, each trajectory’s outer coefficient is proportional to its valid policy-token count. Proposition 2 characterizes the gradient difference introduced by uniform complete-trajectory aggregation:

gflat−gTMN=Cov⁡B(N,g)Nˉ,g_{\mathrm{flat}}-g_{\mathrm{TMN}}=\frac{\operatorname{Cov}_B(N,g)}{\bar N},

where gig_i is the gradient of trajectory ii’s token-averaged surrogate, with the rollout batch, advantages, and attribution weights held fixed.

Training statistics from four GRPO seeds, covering 72,000 trajectories per environment, quantify the associated length imbalance:

Environment / modelFailed/successful mean token-length ratioMean ESS/BTMN gain over GRPO
ALFWorld / Qwen2.5-1.5B2.0130.6416+9.5 pp
WebShop / Qwen2.5-1.5B1.7160.6788+6.9 pp
SearchQA / Qwen2.5-3B1.3750.7701+1.4 pp

ESS/B measures outer-weight concentration and is averaged within and across seeds from actual training batches. TMN’s uniform outer coefficients yield ESS/B = 1. The gains come from Table 3 and use ALFWorld/WebShop success and SearchQA average exact match. Greater length imbalance and weight concentration accompany larger TMN gains across these settings.

2. Complementary roles within BATON.

TMN controls each complete trajectory’s outer share, while BFA redistributes its internal shares using feedback and preserves its total coefficient mass. The token-proportional reference allocation,

bi,t=ni,tNi,b_{i,t}=\frac{n_{i,t}}{N_i},

also serves as the KL-regularization reference for BFA in Proposition 1(ii). This connects the aggregation choice to the within-trajectory allocation rule.

3. Controlled comparisons of reduction schemes.

With Qwen2.5-1.5B, GRPO, and two supplementary training seeds, equal-budget validation-based learning-rate selection gives:

ReductionALFWorld successWebShop success
Global token mean74.0%57.8%
Constant denominator80.5%63.1%
Turn mean81.7%63.5%
TMN82.8%65.1%

Turn mean also averages trajectories uniformly but assigns equal mass to decisions within each trajectory. TMN achieves higher success rates in both environments, supporting its token-proportional internal allocation. With BFA, TMN also improves ALFWorld/WebShop success over constant-denominator reduction from 84.3%/67.4% to 86.0%/69.0%.

Reviewer_uGyeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BATON, which modifies agent reinforcement learning along two axes: Bayesian Feedback Attribution reallocates learning weight across decisions by comparing how well the policy predicts observed feedback under the executed action versus a sampled alternative; Trajectory Mass Normalization gives each complete episode equal weight rather than weighting longer episodes more heavily. The paper provides a formal characterization of the attribution rule and an exact covariance expression for the difference between token- and episode-level averaging.

Four-seed experiments with GRPO and GiGPO on ALFWorld, WebShop, and search-augmented QA across 1.5B–7B models show improvements from both components, with their combination best or tied across all ablation comparisons. Gains accompany shorter evaluation trajectories, with 10–17% additional training time measured in the 1.5B ALFWorld setting.

Critical strengths
  1. A clear separation of two consequential weighting choices. BATON distinguishes credit allocation within trajectories from weighting across trajectories, allowing each to be analyzed and modified independently. The mass-conserving attribution rule and exact length–gradient covariance identity make the design interpretable and clarify what changes relative to the host learner.

  2. Strong controlled empirical validation. Four-seed experiments span two host algorithms, multiple model scales, and three agent settings. Component ablations and feedback-weighting controls support contributions from both axes, while improved success accompanies shorter evaluation trajectories. This strengthens confidence that the gains reflect useful learning improvements rather than increased interaction or generation budgets.

Critical weaknesses
  1. Feedback predictability is an imperfect basis for credit assignment. BFA measures how strongly an action explains the next observation, rather than its contribution to eventual success. The policy’s feedback likelihoods are also not guaranteed to model the environment accurately. Although the controls support empirical usefulness, they do not establish when this proxy reliably assigns credit, particularly with delayed consequences or lexical retrieval feedback. This limits confidence in the method’s central interpretation and broader applicability.

  2. The theoretical justification for trajectory normalization is narrower than the motivation suggests. The covariance identity precisely characterizes the change in surrogate weighting, but does not establish that uniform averaging of token-normalized episode losses better optimizes expected episode return. TMN also closely resembles existing sequence-mean reduction, applied to complete interactions. Consequently, its conceptual novelty and theoretical support for improved optimization are limited; the case rests primarily on the empirical results.

Questions for authors
  1. Clarification on attribution validity: Beyond the reported helpful-action diagnostic, does BFA lift predict actions’ contribution to eventual return, particularly when consequences are delayed or feedback is highly predictable but unhelpful? This would clarify when feedback influence provides useful credit allocation.

  2. Clarification on TMN’s justification: Is the claim that TMN matches episode-level evaluation intended only as a weighting analogy, or is there a formal connection to optimizing expected episode return? How does its contribution differ from applying existing sequence-mean reduction to complete trajectories?

2 comments
AuthorsOpenReview ↗

Response to Reviewer uGye(1/2)

Thank you for asking how feedback attribution relates to eventual task return. We have added a return-alignment diagnostic in Section 4.6, Table 4 (p. 9), an analysis of highly predictable actions with negative reference advantages in Appendix K, Table 21 (p. 24), and lexical-perturbation experiments in Appendix L, Table 23 (p. 25). The corresponding reviewer-specific changes are highlighted in dark green; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.

W1/Q1: Connecting feedback attribution to eventual return.

BFA uses next-step feedback to allocate the host learner’s advantage across decisions. The host advantage supplies the update sign, while the posterior lift determines the within-trajectory allocation. We evaluate this allocation against reference advantages estimated from final task returns.

1. Alignment with eventual return across progress delays.

On 120 trajectories containing 2,000 decisions, we estimate QQ and VV using 16 continuations each, following Appendix E, and compute:

A^ref=Q^−V^.\widehat{A}^{\mathrm{ref}}=\widehat{Q}-\widehat{V}.

We group decisions by the median time to first goal progress across their QQ continuations. Correlations use all decisions in each group; conditional conflict follows Appendix E’s resolved criterion and mass-weighted calculation.

Time to first subsequent goal progressSpearman: lift vs. abs(reference advantage)Spearman: executed likelihood vs. abs(reference advantage)Uniform conditional conflictBFA conditional conflict
All decisions0.2630.11417.98%14.91%
Step 10.3920.28712.76%9.14%
Steps 2–30.2780.12617.38%13.87%
Step 4 or later0.146−0.01822.61%19.52%
No progress observed0.0310.07421.95%21.10%

Overall, lift correlates more strongly with reference-advantage magnitude than length-normalized executed likelihood (0.263 vs. 0.114). The correlation weakens with delayed or unobserved progress, while BFA reduces conditional conflict in every group, including from 22.61% to 19.52% when progress occurs at step 4 or later.

2. Highly predictable but return-disadvantageous actions.

From the same data, we select 93 decisions whose length-normalized executed likelihood is in the top 20% and whose reference-advantage 95% interval lies entirely below zero.

Decision subsetUniform allocation massExecuted-likelihood allocation massBFA allocation mass
Highly predictable, negative reference advantage4.73%6.48%3.16%

All masses use the complete sample’s total trajectory mass as the denominator. Executed-likelihood weighting increases this subset’s share, whereas BFA reduces it. This comparison supports the value of candidate-relative feedback information beyond absolute predictability.

3. Sensitivity to lexical interference.

On the same trajectories, lexical distractors favoring the executed or alternative action yield BFA conditional conflict rates of 15.39% and 15.61%, respectively, compared with 14.91% for original BFA and 17.98% for Uniform. These perturbations quantify the effect of lexical interference and the attribution improvement retained under the tested conditions.

AuthorsOpenReview ↗

Response to Reviewer uGye(2/2)

Response to Reviewer uGye(2/2)

Thank you for asking us to clarify TMN’s theoretical interpretation and contribution. We have revised Section 3.4 (pp. 5–6) to explain its aggregation structure and update analysis, and Section 5 (p. 10) to clarify its relationship to sequence-mean reduction. Training length statistics appear in Appendix C, Table 8 (p. 17), and the normalization comparisons are summarized in Section 4.6, Table 5 (p. 10), with full results in Appendix C, Table 7 (p. 17). The corresponding reviewer-specific changes are highlighted in dark green; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.

Q2.1: Is TMN’s alignment with episode-level evaluation an analogy or a formal connection?

The alignment is a correspondence in aggregation weights: episode-level evaluation counts complete task attempts uniformly, while TMN applies uniform outer coefficients to token-averaged trajectory surrogates.

Our theoretical analysis characterizes how this aggregation choice changes the surrogate objective and its gradient. First, the size-biased identity relates uniform and length-weighted trajectory distributions:

E∗[M]=E[M]+Cov⁡(N,M)E[N].\mathbb{E}^{*}[M]=\mathbb{E}[M]+\frac{\operatorname{Cov}(N,M)}{\mathbb{E}[N]}.

Here, E\mathbb{E} denotes expectation under the episode distribution, E∗\mathbb{E}^{*} denotes expectation under its length-weighted counterpart, and MM is the trajectory-mean surrogate.

Second, with the rollout batch, advantages, and attribution weights held fixed, Proposition 2 gives the exact gradient difference:

gflat−gTMN=Cov⁡B(N,g)Nˉ.g_{\mathrm{flat}}-g_{\mathrm{TMN}}=\frac{\operatorname{Cov}_B(N,g)}{\bar N}.

Here, gig_i is the gradient of trajectory ii’s token-averaged surrogate. This identity isolates the length-dependent weighting removed by TMN. When all trajectories have equal valid policy-token counts, the two reducers coincide. The connection to evaluation concerns the uniform aggregation structure; improvements in task return are supported by the controlled training comparisons below.

Q2.2: What does TMN contribute beyond applying existing sequence-mean reduction to complete trajectories?

TMN applies sequence-mean reduction to complete multi-turn interactions. When a sequence is already defined as a complete trajectory, the reduction forms coincide. We analyze this aggregation choice under the length imbalance observed in agent training, characterize its update effects through Proposition 2, and integrate it with BFA.

The two components have distinct roles: TMN sets each complete trajectory’s outer coefficient, while BFA redistributes its internal coefficients using feedback. The token-proportional reference allocation retained by TMN,

bi,t=ni,tNi,b_{i,t}=\frac{n_{i,t}}{N_i},

is the KL-regularization reference for BFA in Proposition 1(ii).

1. Length imbalance and weight concentration.

Across four GRPO training seeds, we collect 600 batches and 72,000 trajectories per environment. Under global token averaging, the weight-based effective sample size is:

ESS=(∑iNi)2∑iNi2.\mathrm{ESS}=\frac{\left(\sum_i N_i\right)^2}{\sum_i N_i^2}.

We compute ESS/B within each batch, then average within and across seeds. TMN’s uniform outer coefficients yield ESS/B = 1.

Environment / modelFailed/successful mean token-length ratioMean ESS/BTMN gain over GRPO
ALFWorld / Qwen2.5-1.5B2.0130.6416+9.5 pp
WebShop / Qwen2.5-1.5B1.7160.6788+6.9 pp
SearchQA / Qwen2.5-3B1.3750.7701+1.4 pp

The gains use ALFWorld/WebShop success and SearchQA average exact match from Table 3. Greater length imbalance and outer-weight concentration accompany larger TMN gains across these settings.

2. Controlled normalization comparisons.

Using Qwen2.5-1.5B, GRPO, and two supplementary training seeds, we compare reducers with equal-budget validation-based learning-rate selection:

ReductionALFWorld successWebShop success+BFA ALFWorld success+BFA WebShop success
Global token mean74.0%57.8%84.7%67.4%
Constant denominator80.5%63.1%84.3%67.4%
Turn mean81.7%63.5%——
TMN82.8%65.1%86.0%69.0%

Turn mean also averages trajectories uniformly but gives equal mass to decisions within each trajectory. TMN achieves higher success rates in both environments, supporting its token-proportional internal allocation. Combined with BFA, TMN also achieves the highest success rates among the tested reducers.

Reviewer_acQuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BATON, a framework inspired by hindsight credit assignment that reweights the policy losses of GRPO and GiGPO at two levels: within and across trajectories. Starting from the original GRPO/GiGPO loss, Bayesian Feedback Attribution (BFA) uses the frozen rollout policy to compute the likelihood of the observed environment feedback under both the executed action and an alternative action sampled from the same history. Based on their relative likelihoods, BFA computes a posterior lift, defined as the posterior-to-prior ratio of the executed action, and normalizes it within each trajectory to obtain decision-level weights for the policy loss. This preserves the sign of the original advantage. Trajectory Mass Normalization (TMN) first averages the loss over tokens within each complete trajectory and then averages across trajectories with equal weight. This is designed to address the disproportionate contribution of longer trajectories under global token averaging. The paper evaluates BFA and TMN in combination with GRPO and GiGPO on ALFWorld, WebShop, and search-augmented QA, reporting strong performance and improvements from both individual components and their combination.

Critical strengths
  • A simple and interesting approach based on Bayes' rule and hindsight credit assignment. The method provides a straightforward way to convert the likelihood of observed feedback under different candidate actions into posterior lifts, which are then used as decision-level weights while conserving the total coefficient mass within each trajectory. I find it particularly interesting that the method reuses the frozen policy's token likelihoods without training a separate feedback prediction model.
  • Performance improvements across different base algorithms and environments. The method achieves strong performance when integrated with both GRPO and GiGPO, across multiple model sizes on ALFWorld, WebShop, and search-augmented QA.
Critical weaknesses

1. Further validation of BFA's attribution signal is needed.

BFA uses the relative likelihood of the observed feedback to reweight the original RL updates. However, even task-irrelevant actions can produce distinctive observations, so it is important to understand how well a high posterior lift reflects an action's actual contribution to task success. While Section 3.3 and Appendices D, E, and G provide relevant analyses, the following two issues deserve further investigation.

  • Stability of counterfactual sampling: Since only one alternative response is sampled, the resulting lift may depend heavily on which alternative is selected. As training progresses and the policy becomes more concentrated on particular actions, the candidates may also become increasingly similar, potentially reducing BFA's ability to distinguish among decisions. An analysis of candidate diversity and lift distributions at different training stages would be helpful.

  • Sensitivity to reasoning variations: BFA uses both the reasoning segment and the executable command when scoring feedback. Even when the executable command is identical, small differences in reasoning may lead to different likelihoods and posterior lifts. It would be useful to distinguish the effects of actual differences in environment actions from those of variations in textual expression.

 

2. The scope of the contributions and the key analyses could be presented more clearly.

  • I find BFA to be the most interesting contribution of this paper. However, the mathematical transition from Section 3.2 to Section 3.3 is somewhat condensed. A clearer explanation of how the equal-mass prior leads to the posterior lift, and how this is converted into the training weights in Equation (9), would improve readability.

  • TMN is empirically effective, but its methodological novelty appears relatively limited. The authors could better clarify its specific contribution compared with existing normalization methods. Furthermore, the observation that failed trajectories tend to be longer may partly reflect environment termination conditions, such as ALFWorld's maximum of 50 steps. Explaining under what conditions trajectory-uniform averaging is particularly beneficial would strengthen the motivation.

  • The emphasis of the experimental analysis could also be improved. The reduction in execution length reported in Section 4.5 is useful, but Appendix G provides more direct insight into how BFA works. In particular, highlighting the effects of reweighting positive- and negative-advantage trajectories in the main text would help clarify how BFA interacts with the original advantage signals.

Questions for authors

1. How stable is BFA's attribution signal?

BFA uses only one sampled counterfactual action. How sensitive are the resulting posterior lifts to this sampling choice? As training progresses and the policy becomes more concentrated, do candidate diversity and the differences between decision weights decrease? An analysis across training stages would help assess the reliability of the attribution signal.

2. How does BFA handle variations in reasoning for the same executable action?

Since feedback scoring includes both reasoning and executable commands, two responses with identical commands but different reasoning may receive different posterior lifts. How frequently does this occur, and how much does it affect attribution? A comparison with command-only scoring would help clarify what BFA captures.

3. Under what conditions is TMN most beneficial?

Failed trajectories may be longer partly because of environment-specific termination rules. How much does TMN's benefit depend on this length imbalance? An analysis across different trajectory lengths, success outcomes, or termination conditions would help clarify its general applicability.

5 comments
AuthorsOpenReview ↗

Response to Reviewer acQu(1/3)

Thank you for raising these questions about counterfactual-sampling stability and candidate convergence. We have clarified the candidate-set construction in Section 3.3 (p. 4), added the multi-candidate interpretation in Appendix A.1 (p. 15), and included the cross-stage resampling diagnostic in Appendix E, Table 13 (p. 20). The main findings are summarized in Section 4.6 (p. 9). The corresponding changes are highlighted in blue; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.

W1/Q1: Stability of single-counterfactual attribution and candidate convergence during training.

Theoretically, single-counterfactual BFA is Bayesian attribution over a finite candidate set. The executed action and one independently sampled alternative form an equal-prior candidate set. After observing feedback, the executed action’s posterior-to-prior lift is:

e=2f0f0+f1=2σ(s0−s1).e=\frac{2f_0}{f_0+f_1}=2\sigma(s_0-s_1).

Here, f0f_0 and f1f_1 are the frozen policy’s likelihoods of the same feedback under the two candidate actions. This is the exact posterior-to-prior ratio within that candidate set; sampling variability comes from the choice of alternative.

The construction naturally extends to KK independently sampled alternatives:

eK=(K+1)f0∑j=0Kfj.e_K=\frac{(K+1)f_0}{\sum_{j=0}^{K}f_j}.

As the number of alternatives increases, the candidate-average feedback likelihood converges to its expectation under the frozen policy. A single alternative thus provides a local candidate comparison, while additional alternatives more fully cover the policy’s possible actions. When lifts throughout a trajectory approach the same value, the normalized weights approach one, recovering the original token-proportional allocation. This characterizes BFA’s behavior when feedback does not distinguish the candidates.

Empirically, we reuse the diagnostic trajectories from two seeds at the early, middle, and final checkpoints in Appendix E. Holding the policy, histories, executed actions, and feedback fixed, we independently sample five alternatives per decision and compute five separate BFA allocations, each normalized over the complete trajectory.

We compute pairwise weight Spearman correlations and allocation total variation (TV) distances across the five draws, averaging within each trajectory, then within each seed, and finally across seeds. These measure weight-ranking consistency and changes in allocation shares, respectively. The final column reproduces the original attribution diagnostic from Table 12.

Training stageSame-command rateMedian abs(log lift), different commandsWeight Spearman ↑Allocation TV ↓Conditional conflict: Uniform → BFA
Early17.8%0.3020.7620.11232.8% → 20.5%
Middle22.4%0.3250.8080.10126.8% → 13.6%
Final25.1%0.3380.8140.08721.5% → 10.2%

As training progresses, the same-command rate increases, while lifts among different-command pairs retain discrimination. Resampling weight correlations improve and allocation TV distances decrease. Across training stages, resampling agreement improves, and BFA reduces attribution conflict at every checkpoint.

AuthorsOpenReview ↗

Response to Reviewer acQu(2/3)

Thank you for suggesting a comparison that separates reasoning variations from executable-action differences. We have added the candidate-pair analysis and command-only scoring comparison in Appendix J (pp. 23–24), Tables 19 and 20 (p. 24), with the main finding summarized in Section 4.6 (p. 9). The corresponding changes are highlighted in blue. All references below refer to the revised manuscript.

W1/Q2: How do reasoning variations affect BFA when the executable command is unchanged?

BFA conditions feedback scoring on the complete generated action, so reasoning variations can change feedback likelihoods even when the executable command is identical. We reuse the final-checkpoint diagnostic trajectories and five independent candidate draws from Appendix E to quantify this effect and compare full-action with command-only scoring.

1. Frequency and attribution impact of reasoning variations.

Candidate-pair typeShare of all candidate pairsMedian abs(log lift)Share of total reallocated mass
Identical complete action text8.4%0.0005.6%
Same command, different reasoning16.7%0.07312.4%
Different commands74.9%0.33882.0%

Reallocated mass is measured by:

bi,t∣ωi,t−1∣,b_{i,t}\left|\omega_{i,t}-1\right|,

with each category’s share normalized by the total reallocated mass across all decisions. Identical action texts yield a raw lift of one; their final weights can differ from one because of trajectory-level normalization.

Same-command, different-reasoning pairs account for 16.7% of all pairs and 12.4% of total reallocated mass. Their median abs(log lift) is 0.073, compared with 0.338 for different-command pairs. Reasoning variations therefore have a measurable attribution effect, while different-command pairs account for most of the reallocated mass.

2. Command-only conditioning for feedback scoring.

We hold histories, candidate commands, observed feedback, and reference-advantage labels fixed. We remove only the reasoning segments from the current executed and alternative actions, then recompute the likelihoods of the same feedback and the complete-trajectory normalized weights. The following results aggregate the five independent draws:

Scoring conditionWeight Spearman ↑Allocation TV distance ↓Conditional conflictResolved mass
Uniform——21.50%79.00%
Full action: reasoning + command1.0000.00010.48%80.42%
Command only0.9230.05811.26%80.14%

Weight Spearman correlations and allocation TV distances are measured relative to full-action scoring on the same candidate draws. Conditional conflict and resolved mass follow the definitions in Appendix E.

Command-only scoring retains most of the conflict reduction, lowering conditional conflict from 21.50% to 11.26%, with weight rankings closely matching full-action scoring (Spearman 0.923). Full-action scoring further lowers conflict to 10.48%. These results quantify the contribution of reasoning text and show that the attribution improvement largely persists when scoring conditions only on the executable command.

AuthorsOpenReview ↗

Response to Reviewer acQu(3/3)

Thank you for suggesting clearer derivations, contribution positioning, and experimental emphasis. We have clarified the BFA derivation in Section 3.3 (p. 4) and TMN’s motivation and relationship to existing normalization methods in Sections 3.4 (pp. 5–6) and 5 (p. 10). The new length-imbalance and termination analyses appear in Appendix C, Tables 8 and 9 (pp. 17–18). We also summarize the normalization comparisons in Section 4.6, Table 5 (p. 10) and bring the sign-stratified ablation findings into Section 4.4 (p. 8). The corresponding reviewer-specific changes are highlighted in blue; shared changes addressing multiple reviewers may appear in other colors. All references below refer to the revised manuscript.

W2/Q3: Clarifying BFA’s derivation, TMN’s applicability, and the key analyses.

1. Clarifying the BFA derivation.

Sections 3.2–3.3 now explain how the equal-mass candidate prior leads to the posterior-to-prior lift and how this lift becomes the trajectory-normalized weight in Equation (9). Specifically, dividing the executed candidate’s posterior by its prior mass yields the lift; multiplying each original decision share by its lift and renormalizing preserves the trajectory’s total coefficient mass. Proposition 1 characterizes the resulting allocation as the solution to a KL-regularized objective. Appendix A.1 additionally provides the multi-candidate interpretation discussed in our response to Q1.

2. TMN’s contribution, applicable settings, and the role of termination rules.

TMN applies sequence-mean reduction to complete multi-turn trajectories. Our contribution is to characterize its update effects through the length–gradient covariance identity, integrate it with BFA’s within-trajectory allocation, and evaluate this combination through controlled comparisons.

Step limits can contribute to longer failed trajectories. TMN controls how these length differences enter the objective: each trajectory’s outer aggregation coefficient no longer increases with its token count. Proposition 2 characterizes the resulting update difference; when valid policy-token counts are equal across trajectories, TMN and global token averaging coincide.

Our training statistics and component ablations support TMN’s usefulness in settings with substantial length imbalance:

Environment / modelFailed/successful mean token-length ratioMean ESS/BTMN gain over GRPO
ALFWorld / Qwen2.5-1.5B2.0130.6416+9.5 pp
WebShop / Qwen2.5-1.5B1.7160.6788+6.9 pp
SearchQA / Qwen2.5-3B1.3750.7701+1.4 pp

These statistics cover four seeds and 72,000 training trajectories per environment. ESS/B is computed within each training batch, averaged within each seed, and then across seeds. Performance gains come from Table 3 and refer to ALFWorld/WebShop success and SearchQA average exact match. Across these settings, greater length imbalance and outer-weight concentration accompany larger TMN gains.

We also examine trajectory lengths beyond step-cap termination. We evaluated 97 tasks with five runs each in τ³-bench’s banking_knowledge domain, using GLM 5.2 as the user model and GPT-4.1 as the judge:

MetricSuccessfulFailed
Trajectories132353
Mean agent decisions18.9526.59
Mean output tokens5,94810,794
Step-cap terminations02
Mean output tokens, excluding step-cap terminations5,94810,677

After excluding the two step-cap terminations, failed trajectories still use 1.80× as many output tokens as successful trajectories. This shows that length imbalance persists among trajectories that terminate without reaching the step cap.

For direct training evidence, Table 5 summarizes controlled normalization comparisons with termination settings and rollout budgets held fixed. After equal-budget validation-based learning-rate selection, TMN improves ALFWorld/WebShop success over constant-denominator reduction from 80.5%/63.1% to 82.8%/65.1% without BFA, and from 84.3%/67.4% to 86.0%/69.0% with BFA. Full results are reported in Appendix C, Table 7.

3. Highlighting BFA’s interaction with advantage signs.

We have brought the key findings of Appendix G’s sign-stratified ablation into Section 4.4, retaining the full four-seed results in Appendix G, Table 16. On ALFWorld, applying BFA only to positive- or negative-advantage trajectories achieves 84.4% and 83.5% success, respectively, compared with 82.4% for uniform allocation, 85.6% for full BFA, and 81.1% for inverse-lift weighting (Anti-BFA).

These results show contributions from both sign groups and the importance of allocation direction. Section 4.6 also summarizes Appendix E’s rollout-based attribution analysis, and we have streamlined Section 4.5 to give greater emphasis to these mechanism-focused analyses.

Comments on the paper

2 comments

Dear Reviewers

The author-reviewer discussion period will close soon. The authors put much efforts for their rebuttal. Please read and respond to authors if you have not done yet.

Thank you.

AC

AuthorsOpenReview ↗

Summary of Discussion Status and Revisions

We thank the PC, SAC, AC and reviewers for the continued discussion. We briefly summarize the current status of our submission.

Reviewer responses.

  • Reviewer acQu: noted that the new analyses largely addressed their concerns on BFA and maintained the rating of 3. Their remaining reservation concerns the novelty and applicability of TMN, which we will clarify in the revision.
  • Reviewer heBL: indicated that their concerns were almost addressed and that they would raise their score.
  • Reviewers uGye and SVnf: we posted detailed responses on 13 Nov and would welcome any further feedback from them.

Main additions in the revised manuscript.

  • Validity of the BFA signal: a return-alignment diagnostic (Sec. 4.6, Table 4), an analysis of highly predictable but return-disadvantageous actions (App. K), lexical-perturbation and action–feedback matching tests (App. L), and candidate-resampling stability and command-only scoring (App. E, J).
  • TMN: a clarified derivation and relationship to sequence-mean reduction (Sec. 3.4, 5), training length-imbalance statistics (App. C, Table 8), controlled normalization comparisons (Table 5, App. C Table 7), and a trajectory-length analysis beyond step-cap termination.
  • Seed stability: a complete ablation on two supplementary seeds (App. F, Table 15), where BATON outperforms both single components in 29 of 32 seed-level comparisons.

Please feel free to contact us if there are any further questions.

Finally, we are deeply grateful to the AC and all the reviewers for taking the time to engage so carefully with our work. We were especially touched that the AC took the initiative to remind the reviewers to follow up during the discussion period. Thank you very much.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.