acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Early-Turn Credit Failure in Agentic RL via Rollout-Free Reattribution

Abstract

Long-horizon capabilities are essential for LLM agents to solve complex tasks that require iterative tool use and multi-step interaction. Reinforcement learning, the predominant training paradigm for such agents, typically provides no intermediate rewards and reveals success only through a terminal reward. Learning from this sparse feedback requires turn-level credit assignment to estimate how each action contributes to the outcome and therefore determine its learning signal. In this work, we show theoretically and empirically that current critic-free methods suffer from **early-turn credit failure**, yielding unreliable credit in early turns. At these turns, Monte Carlo estimates are dominated by sampling noise, while final-outcome surrogate methods suffer from the weak connection between early actions and the outcome. Consequently, crucial early actions that shape the subsequent trajectory receive weak learning signals, limiting effective training on long-horizon tasks. To address this, we propose FOLIA, a rollout-free plug-in method that combines local influence with terminal hindsight for early-turn credit estimation. Specifically, FOLIA measures each action's local influence by counterfactually ablating the observed environment response, anchors this influence to the final outcome through a terminal hindsight score, and fuses the two signals with a confidence gate to reattribute credit across turns. Extensive experiments across diverse long-horizon agentic tasks show that FOLIA consistently improves existing RL algorithms through better credit assignment with negligible computational overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic Training, Credit Assignment, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mRzLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies unreliable early-turn credit assignment in agentic reinforcement learning with sparse terminal rewards. Under a sequential-hardness assumption, it derives an exponential necessary rollout-budget scaling for maintaining the signal-to-noise ratio of independent Monte Carlo prefix-value differences. FOLIA then reweights existing turn advantages using the likelihood effect of ablating a tool response, together with a success-conditioned terminal-hindsight gate. The method uses teacher-forced scoring rather than additional environment rollouts. Experiments apply the plug-in to several critic-free RL algorithms with Qwen2.5 models on search QA and WebShop, with diagnostic comparisons against a higher-budget Monte Carlo reference.

Critical strengths
  1. The diagnosis of early-turn credit failure is useful. Proposition 3.2 makes the dependence on rare successful continuations explicit, and the prefix-resampling experiments connect that analysis to training behavior. The result is specific to the stated sequential-hardness assumption and independent rollout estimator, but within that setting the argument is clear.

  2. The improvements are reasonably consistent across base algorithms and model sizes. On 3B WebShop, success rises from 0.510 to 0.567 with GRPO and from 0.573 to 0.648 with GiGPO (Table 1). The measured iteration-time overhead is only 3.5–5.2% in Appendix A.2. A method that obtains these gains without more environment interaction would be useful in practice.

Critical weaknesses
  1. My main concern is the gap between the credit-assignment interpretation and the update in Section 4.3. The new advantage is the base advantage times a positive clipped weight: its sign cannot change. Hindsight only increases salience on successful trajectories when eta is positive. For GRPO, a harmful turn in a successful trajectory retains a positive surrogate advantage whenever the base group-relative advantage is positive, although its weight may decrease. This statement concerns the surrogate signal, not a guaranteed change in action probability after optimization. This does not rule out a useful reweighting method, but it limits what can be claimed about recovering the signed advantage defined in Section 2. Appendix A.6 acknowledges missing-signal cases; the wrong-sign case also needs to be addressed.

  2. The credit-error diagnostics need a more explicit account of calibration and reference uncertainty. Section 3.3 states that surrogate estimates are normalized to [−1, 1], and Appendix A.4 divides signed MC credits by their sum across turns for the case study. These descriptions do not specify a common transformation for every quantity in Figures 2 and 5, including the treatment of a near-zero or negative normalization denominator. The K=256 reference also has sampling error. Consequently, it is difficult to separate improved credit estimation from changes in scale. Table 2 compares the full method with hindsight-only, but omits local-influence-only; this is the most direct missing control for the fusion. The main QA results also need seed counts and uncertainty.

Questions for authors
  1. Is FOLIA intended to recover signed turn credit or to redistribute the magnitude of an existing signal? Please explain the directional claims in light of the positive multiplier. A breakdown of Figure 5 by agreement between the base-advantage sign and the MC-credit sign would help resolve this.

  2. Section 3.3 and Appendix A.4 give different levels of detail about normalization. Please provide the exact transformations used for all estimates and references in Figures 2 and 5, including denominator handling, and quantify uncertainty in the K=256 reference. Section 5.3 specifies 200 examples; how were these selected?

  3. Do the authors have a local-influence-only result under the same training settings? This would be more informative about the need for fusion than the current hindsight-only comparison.

6 comments
AuthorsOpenReview ↗

Response for Reviewer mRzL

We appreciate Reviewer mRzL for the detailed and insightful comments, and for recognizing the value of our problem formulation and empirical study. Below, we respond to each concern in turn.

W1: Sign-preserving update and the wrong-sign case

We thank the reviewer for this note on our setting. Incorrect directions could be a practical issue when FOLIA cannot correct them. However, we want to note that such cases (a turn with an incorrect direction but a large effect) are infrequent in practical training and that FOLIA can still be beneficial even when it cannot directly correct these signs.

To illustrate this, we examine FOLIA’s effect on incorrectly signed credit using an intermediate checkpoint of Qwen2.5-3B-Ins trained with GRPO on WebShop (training step 90 of 150), following the setting in Section 5.1 and using K=256K=256 to estimate reference credit. First, we find that turns with both an incorrect sign and a large credit magnitude are uncommon: only 13% of early turns (defined as the first half of each trajectory) have a significantly incorrect sign, and just 4% have both an incorrect sign and a large credit magnitude. Early turns with incorrect signs account for just 8% of a trajectory’s reference credit, compared with 27% for early turns with correct signs. Moreover, FOLIA reduces estimation error on incorrectly signed turns by moving the assigned credit toward zero. Among all incorrectly signed turns, FOLIA reduces the assigned credit magnitude for 81% of them, while their credit estimation error decreases from 0.334 to 0.302. FOLIA therefore reduces the weight these turns receive in the policy update, even without directly correcting their signs, which may help explain its training benefits.

Furthermore, we want to note that the influence score serves as an effective and cheap surrogate for the value. We find that high-influence turns, defined as the top quartile of influence scores, carry more than twice the average reference credit share per turn of low-influence turns (18.8% versus 9.3%). We further examine trajectories in which the turn most heavily weighted by FOLIA is unhelpful according to the MC reference. Within this subset, the selected turn’s mean credit error remains nearly unchanged (0.2086 → 0.2089), while total trajectory credit error decreases by 7.0%. These findings support the practical usefulness of influence as a proxy and of FOLIA’s redistribution mechanism, which combines influence with hindsight.

W1: Addressing missing signals

We thank the reviewer for highlighting cases where the learning signal is missing. We clarify that FOLIA is orthogonal to strategies for handling all-fail groups and can complement them. To illustrate this, we compare vanilla DAPO [1] with DAPO + FOLIA. DAPO uses dynamic sampling to filter out all-fail and all-success groups and fill the batch with mixed-outcome groups, while FOLIA further refines turn-level credit within the retained trajectories. Detailed results on WebShop using Qwen2.5-3B-Ins are presented below:

MethodSuccess rateTask score
DAPO + FOLIA0.760 ± 0.0190.885 ± 0.013
Vanilla DAPO0.721 ± 0.0470.870 ± 0.014

These results show that FOLIA’s turn-level credit redistribution is orthogonal to handling all-negative groups. The additional gains from combining FOLIA with DAPO support its complementarity with dynamic sampling.

Reference

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025

W2 && Q2: Transformation, scale, and reliable MC estimation

We thank the reviewer for suggesting that we provide more detail on the transformation and credit-error diagnostics. We address each point below.

(Transformation and scale) We thank the reviewer for this careful review. Figures 2 and 5 and App. A.4 use one uniform transformation. For each trajectory, the turn credits a1,…,aTa_1,\dots,a_T, from the MC reference (Δ^t=V^(ct)−V^(ct−1)\hat\Delta_t=\hat V(c_t)-\hat V(c_{t-1})) or from any method (like the information gain defined in IGPO), are mapped to signed shares st=at/∑j∣aj∣∈[−1,1]s_t=a_t/\sum_j|a_j|\in[-1,1]. The credit error of turn tt is ∣stest−stref∣|s_t^{\text{est}}-s_t^{\text{ref}}| averaged over trajectories; Figure 5 averages turns 1 and 2. This removes one positive scale per trajectory and keeps the sign and relative size of each turn's credit, as defined in Section 2. The denominator is a sum of absolute values, so it is never negative; "the sum of credits" in App. A.4 was imprecise and is corrected. It is zero only when all credits are zero, which holds for 16 of the 200 reference trajectories; these are excluded, and no other threshold is applied.

(Reference uncertainty) We thank the reviewer for raising this concern. The K=256K=256 reference still contains sampling noise. To assess its stability, we increase the sampling budget to K=1024K=1024 on 30 WebShop examples. The table reports the mean absolute difference between the credit estimates at each sampling budget and those obtained with K=1024K=1024.

Continuations KK8163264128256512
Overall0.1390.1330.1170.0880.0760.0510.047
Early turns0.1830.1630.1240.1100.0890.0570.053

Doubling the rollout budget from K=256K=256 to K=512K=512 reduces the discrepancy from the K=1024K=1024 reference by only 0.004, both overall (0.051 → 0.047) and at early turns (0.057 → 0.053). This trend toward stability, with only a marginal improvement despite doubling the sampling cost, supports using K=256K=256 as a practical approximate reference. By comparison, the early-turn discrepancy at K=8K=8 is approximately 3.2 times that at K=256K=256 (0.183 vs. 0.057), reinforcing the substantially lower reliability of early-turn estimates under small rollout budgets.

W2 && Q3: Local-influence-only ablation

We thank the reviewer for this suggestion. In the setting used for Table 2 of the original manuscript (Qwen2.5-3B-Ins on WebShop), we additionally evaluate local influence alone and two alternative fusion strategies, defined as follows:

  1. Local influence only: Uses only the local counterfactual influence score.

  2. Hindsight only: Uses only the hindsight signal.

  3. Ungated fusion (hard gate): Sets κt≡1\kappa_t \equiv 1, so the fusion score becomes uthard=δ^t[1+s(τ) 1[ηt>0]].u^{\text{hard}}_t = \hat{\delta}_t \left[ 1 + s(\tau)\,\mathbb{1}[\eta_t > 0] \right].

  4. Linear reweighting: Keeps the full model’s fusion score utu_t and changes only the reweighting step. The exponential weight mt=exp⁡(ut)/∑jpj0exp⁡(uj)m_t = \exp(u_t) / \sum_j p_j^0 \exp(u_j) is replaced by the proportional weight mtlin=ut∑jpj0ujm^{\text{lin}}_t = \frac{u_t}{\sum_j p_j^0 u_j} using the same token-weighted prior pj0p_j^0 as in the full model.

We report the mean success rate ± standard deviation over three runs.

VariantGiGPOGRPO
Baseline0.573 ± 0.0330.510 ± 0.028
Hindsight only0.563 ± 0.0350.546 ± 0.033
Local influence only0.582 ± 0.0250.530 ± 0.014
Linear reweighting0.568 ± 0.0230.501 ± 0.017
Hard gate0.596 ± 0.0520.540 ± 0.039
Full FOLIA0.648 ± 0.0390.567 ± 0.009

From these results, we find that local influence alone performs slightly better than the baselines but worse than full FOLIA under both algorithms. Hindsight alone also performs worse than full FOLIA, so neither signal alone recovers the gains from their fusion. Both alternative fusion strategies also perform worse than full FOLIA. All these results reinforce the necessity of our design.

Q1: Signed credit recovery vs. magnitude redistribution

We thank the reviewer for this suggestion, which helps clarify FOLIA’s role. FOLIA redistributes the magnitudes of existing advantages through positive multipliers while preserving their signs. Although it does not correct sign errors in the base credit, it can, in practice, reduce credit error both when the base advantage and the MC reference agree in sign and when they disagree.

Following the reviewer’s suggestion, we conduct an additional analysis on WebShop by separating cases where the base advantage and the MC reference agree or disagree in sign. We still use the Qwen2.5-3B-Ins GRPO checkpoint at training step 90, on 100 validation trajectories, and a K=256K=256 MC reference. We define early turns as the first half of each trajectory. We then group these turns by sign agreement between the base advantage and the MC reference:

Early turnsCredit error, base → FOLIA
Signs agree with MC reference0.142 → 0.131
Signs disagree with MC reference0.239 → 0.219

FOLIA reduces error in both sign groups. When the signs disagree, the improvement comes from reducing the magnitude of incorrectly signed credit toward zero without correcting its sign. This provides a practical way to suppress incorrect training signals. We have revised our Introduction to make this clear (line 69).

We sincerely thank Reviewer mRzL for their suggestions for improving our work and have incorporated them into our revision. We would be happy to address any remaining concerns or questions.

Reviewer_uFdbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the early turn credit assignment under the agentic RL context and propose FOLIA, a plug-in credit reweighs mechanism on existing advantages by using local response-ablation scores and terminal hindsight. Starting from SNR empirical and theoretical analysis, this paper derives the bound for MC credit estimator and empirically reveals the SNR collapse problem in the early turn of training. The empirical studies on various group-based RL methods and QA/search-based benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths

1, This paper addresses an important agentic RL training issue and combines theoretical motivation with empirical credit-estimation diagnostics.
2, The theoretical analysis appears to be sound under mild assumption and provides some insight on the regime where MC credit estimation becomes expensive with horizon.
3, The empirical studies support authors' statement in FOLIA's effectiveness.

Critical weaknesses

1, The exponential rollout requirement, though intuitive, applies to the specified independent-prefix MC estimator under an assumption implying exponentially rare success.
2, Ablating a tool response measures its influence on the next generated action, rather than the value of the preceding action. Teacher-forcing the final answer is also different from marginalizing over future interactions. The likelihood and Bayes identities are valid, but they do not establish accuracy of these practical proxies.
3, FOLIA's hindsight mechanism seems only boosts early turn credit assignment on successful trajectories.
4, It would be interesting to discuss how FOLIA's mechanism will influence the credit assignment to critical action/turns appear in the later step, which is also problematic for group-based methods under long horizon setting due to variance in trajectory.
5, Current empirical studies limit to QA and search based benchmark, it would be more interesting to see FOLIA performance under more complicated tool-use/call tasks.
6, It would be great to include additional ablation on local-only, randomized -weight, and confidence-gate ablations.

Questions for authors

1, What is the best way to understand constant \lambda in Assumption 3.1?
2, Results showing in Fig a & b are based on sample size of 8? And Turn index here literally means first turn, second turn, etc.?
3, Why the estimation of ablated context conditioned policy distribution don't need additional sampling here? What is the detailed entropy approximation scheme here?
4, How is the terminal info G approximated by y here? y is only answer for the current turn right? It would be great to detail a bit in this part.
5, How is the early-turn credit-estimation error calculated in Fig 5?

6 comments
AuthorsOpenReview ↗

Response for Reviewer uFdb

We sincerely thank Reviewer uFdb for their thoughtful review and acknowledgment of our motivation, methodology, and analysis. We provide detailed responses to each question below.

W1: Scope of the exponential rollout bound

We thank the reviewer for this question on the basic settings of our analysis. However, we would like to point out that, following previous work [1–5], our theoretical analysis adopts the conventional independent-prefix MC estimator for turn-level value estimation, which is an established formulation to characterize the reliability of turn-level credit estimates. Moreover, Assumption 3.1 requires only a nonvanishing average log-difficulty across the necessary bottlenecks of an agentic task. This is motivated by long-horizon tasks such as multi-hop search, where bridge-entity identification and evidence retrieval form successive necessary stages, each subject to failure. Similar sequential-reliability formulations also appear in prior work [6–8]; for example, [6] assumes independent steps with constant accuracy pp, yielding success probability pHp^H over HH steps. We have clarified this scope in the revised manuscript in Sec. 3.1.

Reference

[1] VinePPO: Refining Credit Assignment in RL Training of LLMs, ICML 2025

[2]Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning,ICLR 2025

[3]RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold, NeurIPS 2024

[4]Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models,NeurIPS 2025

[5]Know What You Don’t Know: Uncertainty Calibration of Process Reward Models, NeurIPS 2025

[6]The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs, ICLR 2026

[7]Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoning, ICML 2025

[8]Faith and Fate: Limits of Transformers on Compositionality, NeurIPS 2023

W2: Accuracy of the practical proxies

We agree that likelihood and Bayes identities motivate these proxies but do not establish their accuracy as value estimates. However, the influence score serves as an inexpensive yet effective surrogate, and prior work [1, 2] has also demonstrated the practicality of this teacher-forcing strategy. To assess their practical usefulness, we evaluate FOLIA’s credit allocation against an MC reference using the step-90 checkpoint of Qwen2.5-3B-Ins trained with GRPO on WebShop. We follow the exact protocol described in Section 5.2 and analyze 100 samples from the evaluation set. We find that high-influence turns, defined as the top quartile of influence scores, carry more than twice the average reference credit share per turn of low-influence turns (18.8% versus 9.3%). We further examine trajectories in which the turn most heavily weighted by FOLIA is unhelpful according to the MC reference. Within this subset, the selected turn’s mean credit error remains nearly unchanged (0.2086 → 0.2089), while total trajectory credit error decreases by 7.0%. These findings support the practical usefulness of influence as a proxy and of FOLIA’s redistribution mechanism, which combines influence with hindsight.

Reference

[1] Hindsight Credit Assignment for Long-Horizon LLM Agents, ICML 2026

[2] Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents, ICLR 2026

W3: Hindsight effect

We thank the reviewer for pointing this out. We apply hindsight only to successful trajectories, where we find our proxy more reliable. This choice aligns with prior work [1-3] that uses verified successful responses or outcomes to construct hindsight supervision. For unsuccessful trajectories, FOLIA instead uses local influence alone to guide turn-level credit assignment. To illustrate its effect in both settings, we break down early-turn credit error by whether hindsight is activated:

ConditionEarly-turn credit error, base → FOLIATrajectories with reduced error
Successful (hindsight activated)0.203 → 0.18979%
Unsuccessful (hindsight not activated)0.156 → 0.15272%

Although the error reduction is larger on successful trajectories, FOLIA also reduces error in 72% of unsuccessful trajectories, where hindsight is not activated. These results support FOLIA’s design of combining local influence with hindsight on successful trajectories and using local influence alone otherwise.

Reference

[1]Hindsight Credit Assignment, NeurIPS 2019

[2]Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning, AISTATS 2021

[3]Hindsight Credit Assignment for Long-Horizon LLM Agents, ICML 2026

W4: Credit for late critical turns

We thank the reviewer for suggesting an analysis of late turns, which also suffer from unreliable credit estimation, although to a lesser extent than early turns. Using the intermediate Qwen2.5-3B-Ins checkpoint described above, we find that FOLIA reduces the mean estimation error for late turns from 0.183 to 0.174, compared with a reduction from 0.198 to 0.180 for early turns. This trend aligns with our intuition and motivation: early turns exhibit larger estimation errors and benefit more from FOLIA, while FOLIA also improves credit estimation at late turns.

W5: Additional results on ALFWorld

We extend our experiment to the ALFWorld dataset, which requires more than 30 interactions with the environment to complete a task on average. We use Qwen2.5-1.5B-Ins as the base model and GiGPO as the training algorithm. The table below reports validation success rates (%), with means and standard deviations across 3 runs for each method.

MethodPickLookCleanHeatCoolPick2All
Vanilla GiGPO91.05 ± 3.8960.25 ± 12.0489.23 ± 1.9194.56 ± 9.4290.63 ± 1.9376.10 ± 6.4785.50 ± 1.59
GiGPO + FOLIA94.60 ± 1.5174.60 ± 8.8587.07 ± 3.4396.27 ± 6.4791.07 ± 2.8983.67 ± 3.3088.80 ± 0.30

These results show that FOLIA continues to provide improvements on more complex tasks requiring longer interaction sequences, further supporting the general applicability of our method.

W6: Additional ablations

We thank the reviewer for suggesting further ablations of each component of our method. We extend our ablation study using Qwen2.5-3B-Ins on WebShop under the exact settings reported in the main paper. The full model computes the fusion score ut=δ^t[1+s(τ) κt 1[ηt>0]],u_t = \hat{\delta}_t \left[ 1 + s(\tau)\,\kappa_t\,\mathbb{1}[\eta_t > 0] \right], where δ^t\hat{\delta}_t is the normalized counterfactual influence of turn tt and κt=tanh⁡(a∣ηt∣/2)\kappa_t = \tanh(a \lvert \eta_t \rvert / 2) is the confidence gate. It then computes the weights using the non-linear mapping mt:=exp⁡(ut)∑jpj0exp⁡(uj).m_t:=\frac{\exp(u_t)}{\sum_j p_j^0\exp(u_j)}. We consider the following settings, where linear reweighting serves as a stronger baseline than randomized weighting:

  1. Local influence only: Uses only the local counterfactual influence score.

  2. Hindsight only: Uses only the hindsight signal.

  3. Ungated fusion (hard gate): Sets κt≡1\kappa_t \equiv 1, so the fusion score becomes uthard=δ^t[1+s(τ) 1[ηt>0]].u^{\text{hard}}_t = \hat{\delta}_t \left[ 1 + s(\tau)\,\mathbb{1}[\eta_t > 0] \right].

  4. Linear reweighting: Keeps the full model’s fusion score utu_t and changes only the reweighting step. The exponential weight mt=exp⁡(ut)/∑jpj0exp⁡(uj)m_t = \exp(u_t) / \sum_j p_j^0 \exp(u_j) is replaced by the proportional weight mtlin=ut∑jpj0uj,m^{\text{lin}}_t = \frac{u_t}{\sum_j p_j^0 u_j}, using the same token-weighted prior pj0p_j^0 as in the full model.

We report the mean success rate ± standard deviation over three runs.

VariantGiGPOGRPO
Baseline0.573 ± 0.0330.510 ± 0.028
Hindsight only0.563 ± 0.0350.546 ± 0.033
Local influence only0.582 ± 0.0250.530 ± 0.014
Linear reweighting0.568 ± 0.0230.501 ± 0.017
Hard gate0.596 ± 0.0520.540 ± 0.039
Full FOLIA0.648 ± 0.0390.567 ± 0.009

Full FOLIA achieves the highest mean success rate under both algorithms, confirming the importance of combining local influence and hindsight signals. Linear reweighting underperforms both baselines due to its sensitivity to near-zero scores, whereas non-linear weighting of FOLIA provides more stable redistribution. The hard gate performs better than linear reweighting but remains inferior to FOLIA, as it applies the same maximal hindsight boost regardless of lift strength. In contrast, FOLIA's graded confidence gate adjusts the boost based on hindsight strength, avoiding abrupt weight changes. These results support FOLIA's fusion design and its effectiveness on long-horizon tasks.

Q1: Meaning of λ\lambda in Assumption 3.1

Assumption 3.1 aims to capture how difficulty accumulates across necessary stages. The constant λ>0\lambda>0 lower-bounds their average log-difficulty under a fixed policy. Since success requires passing every remaining bottleneck, the probability chain rule gives

\leq \prod_{j=1}^{h(c)}q_j(c) = \exp\left(-\sum_{j=1}^{h(c)}[-\log q_j(c)]\right) \leq e^{-\lambda h(c)}.$$ Here, $q_j(c)$ is the success probability of bottleneck $j$ conditional on passing the preceding ones. The last inequality follows from Assumption 3.1, which requires $\sum_j[-\log q_j(c)]\geq\lambda h(c)$ for $h(c)\geq h_0$. Thus, the success-probability upper bound decreases exponentially as more bottlenecks remain. The assumption allows heterogeneous and dependent bottlenecks in the task-completion process. ### Q2: Sample size and turn index Yes, we aim to demonstrate this effect under practical training conditions, so we compare results obtained with a group size of 8 against those obtained with (K=256), using the latter as a large-budget reference. Here, the turn index denotes a turn’s position within a trajectory: Turn 1 is the first turn, Turn 2 is the second, and so on. We define the first two turns as early turns, corresponding to the first half of a typical trajectory, because answering a multi-hop search question in this task requires approximately four turns on average. ### Q3: No extra sampling We thank the reviewer for this question. We reuse the next turn $r_{t+1}$ from the original rollout and teacher-force its tokens under the ablated context. A forward pass provides their log-probabilities through log-softmax; factual log-probabilities are reused from the base algorithm. So in this case, no additional generation or environment interaction is required. We do not explicitly estimate entropy. The KL interpretation follows from $\mathbb{E}_{r\sim\pi_t}[\log(\pi_t(r)/\pi_t^{\emptyset}(r))]=D_{\mathrm{KL}}(\pi_t\\|\pi_t^{\emptyset})$. The length-weighted score $|r_{t+1}|\delta_t$ is a single-sample proxy computed from the existing factual rollout, rather than an exact full-distribution KL evaluation. ### Q4: Approximating terminal information $G$ We thank the reviewer for pointing out this ambiguity. Here, $y$ is the final answer of the entire trajectory; the current turn’s output is $r_t$. We choose $G=y$ as a terminal anchor for assessing intermediate turns. For WebShop, which has no textual final answer, $G$ is instead the realized final observation—the raw product page viewed when the agent issues its terminal action. After completing the rollout, we teacher-force the same target $G$ under each prefix and compute $\eta_t=[\log\pi_{\theta_{\mathrm{old}}}(G\mid c_t)-\log\pi_{\theta_{\mathrm{old}}}(G\mid c_{t-1})]/|G|$. The approximation concerns the likelihood of $G$: its direct teacher-forced likelihood serves as a tractable proxy for the outcome likelihood marginalized over future interactions, requiring no additional rollouts. ### Q5: Computing the early-turn credit error After training each vanilla RL method to step 40 out of 100, we use its frozen policy to generate trajectories for 200 evaluation examples and estimate each prefix’s value using (K=256) continuation rollouts. The MC reference credit is the difference between consecutive prefix-value estimates: $\widehat{\Delta}_t^{\mathrm{MC}}=\widehat{V}(c_t)-\widehat{V}(c_{t-1})$. We compare both the original advantages and their FOLIA-redistributed counterparts with this reference, reporting the mean absolute error between normalized credits over the first two turns in Fig. 5. Both comparisons use the same trajectories, fixed policy, and MC reference, isolating the immediate effect of FOLIA’s credit redistribution. We appreciate Reviewer uFdb’s valuable feedback, which has guided our revisions and helped strengthen the paper. We have revised the manuscript to clarify the ambiguous points raised. We would be happy to address any remaining questions or concerns.
Reviewer_mS7GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates early-turn credit assignment in long-horizon LLM agent reinforcement learning. The authors argue that, with only terminal rewards, Monte Carlo-based credit estimates suffer from substantial sampling noise at early turns, while outcome-based surrogate methods struggle to capture the contributions of early actions to the final outcome.

To address this issue, the authors propose FOLIA, a rollout-free credit reattribution method. It estimates the local influence of each turn by ablating the tool response and measuring the resulting change in next-turn policy behavior. This signal is then combined with terminal hindsight through a confidence-gated mechanism to redistribute turn-level advantages. Experiments on search-based QA benchmarks and WebShop show that FOLIA improves several RL baselines with relatively low computational overhead.

Critical strengths
  1. The paper addresses a meaningful problem with systematic analysis.

    The paper highlights the often-overlooked issue of early-turn credit assignment in long-horizon agentic RL. The theoretical analysis and empirical diagnostics illustrate why credit estimates become unreliable under limited sampling budgets, providing a clear motivation for the proposed method.

  2. The method is broadly applicable and supported by encouraging experimental results.

    FOLIA does not require additional environment interactions and can be integrated into existing RL algorithms. The authors evaluate it with GRPO, IGPO, HCAPO, and GiGPO across different model scales and tasks. The results show generally consistent performance improvements with relatively small computational overhead, demonstrating its practical value.

Critical weaknesses
  1. There remains a gap between FOLIA's credit redistribution mechanism and accurate turn-level credit estimation. (Sections 3 and 4.3)

    The paper defines turn-level credit as the change in expected success probability, Δt=pt−pt−1\Delta_t = p_t - p_{t-1}, which can be either positive or negative. However, FOLIA essentially rescales the original advantages using non-negative weights, without changing their signs. Therefore, when the original advantage has an incorrect direction, FOLIA cannot directly correct it. For example, an early action in a successful trajectory may actually reduce the probability of eventual success. If the base algorithm assigns it a positive advantage, FOLIA can only adjust its magnitude rather than correct its sign.

    Moreover, the theoretical analysis emphasizes the difficulty of obtaining informative signals when successful trajectories are rare. However, FOLIA still relies on the advantages provided by the base algorithm. When all trajectories in a sampled group fail, reweighting alone cannot generate new learning signals, as also acknowledged in Appendix A.6.

    Thus, FOLIA seems more appropriately characterized as a heuristic advantage reweighting method rather than an estimator that directly recovers accurate turn-level credit. The paper should better clarify this distinction and explain to what extent the proposed mechanism actually addresses the early-turn credit failure identified in the theoretical analysis.

  2. The validation of the proposed credit signals and their fusion mechanism is not sufficiently convincing. (Sections 4.1 and 5.3–5.4)

    FOLIA measures local influence through changes in next-turn action likelihood when the tool response is removed. However, this signal primarily reflects how sensitive the policy is to environmental feedback, rather than how much the corresponding action contributes to eventual success. For example, irrelevant or misleading tool feedback may substantially change subsequent policy behavior and therefore receive a high influence score. Although the authors introduce a hindsight signal to guide the redistribution, it remains unclear whether this fusion can reliably distinguish beneficial from unhelpful behavioral changes.

    The current ablation study mainly compares the hindsight-only variant with the full FOLIA method. It lacks comparisons with a local-influence-only variant, a version without confidence gating, and simpler advantage reweighting strategies. As a result, it is difficult to determine whether the improvements come from the proposed complementary signals or from non-uniform advantage reweighting in general.

    Additionally, the paper uses MC estimates obtained from 256 continuations as reference credits. However, these estimates may still contain sampling errors, particularly in the early-turn, low-success-probability settings emphasized by the paper. The analysis also lacks a systematic comparison across different trajectory lengths, making it difficult to assess whether FOLIA becomes more effective as the decision horizon increases.

Questions for authors
  1. Credit estimation accuracy: FOLIA adjusts the magnitude of the base advantages but cannot change their signs. How does the method address cases where the original advantage has the opposite sign from the true turn-level credit? Also, can FOLIA still provide useful learning signals when all sampled trajectories fail?

  2. Local influence and credit contribution: Why should the policy change caused by ablating a tool response serve as a reliable proxy for the corresponding action's credit magnitude? How does FOLIA avoid over-assigning credit to feedback that substantially changes subsequent behavior but does not actually help complete the task?

  3. Ablation and experimental validation: Could the authors further clarify the individual contributions of local influence and hindsight? Is it possible to compare local-influence-only, ungated fusion, and simpler reweighting strategies using existing results? Additionally, are there analyses of credit estimation errors across different trajectory lengths to better support the effectiveness of FOLIA on long-horizon tasks?

4 comments
AuthorsOpenReview ↗

Response for Reviewer mS7G

We thank Reviewer mS7G for acknowledging our motivation and analysis, as well as for the constructive suggestions, which have led to more detailed analyses that strengthen the paper. We address each point below.

W1 & Q1: Concern about correcting the direction

We thank the reviewer for raising this valid concern. The original advantage could have an incorrect sign, which FOLIA cannot directly correct. However, our analysis suggests that such cases are relatively infrequent in training, and FOLIA can still mitigate their error by reducing the magnitude of incorrectly signed credit.

To examine this, we analyze Qwen2.5-3B-Ins on WebShop at an intermediate GRPO checkpoint (training step 90 of 150), following the setting in Section 5.1 and using K=256K=256 to estimate reference credit. We find that only 13% of early turns (defined as the first half of each trajectory) have a significantly incorrect sign, and just 4% have both an incorrect sign and a large credit magnitude. Early turns with incorrect signs account for only 8% of a trajectory’s reference credit, compared with 27% for early turns with correct signs. Moreover, FOLIA reduces estimation error even when the signs are incorrect by moving the assigned credit toward zero. Among all incorrectly signed turns, FOLIA reduces the assigned credit magnitude for 81% of them, while their average credit estimation error decreases from 0.334 to 0.302. These results suggest that FOLIA can limit the contribution of incorrectly signed credit to the policy update, which may help explain its training benefits despite preserving advantage signs.

To further address sign errors, FOLIA can also be combined with methods such as VinePPO [1], which estimate signed advantages through continuation sampling. However, obtaining more reliable estimates of this kind requires more samples, incurring additional rollouts and, in agentic settings, environment interactions. The added cost of obtaining more precise base credit estimates may make the combination less attractive under limited rollout budgets.

[1] VinePPO: Refining Credit Assignment in RL Training of LLMs. ICML 2025.

W1: Reliance on base algorithm

We thank the reviewer for this comment. We would like to clarify that FOLIA places few requirements on the base RL algorithm: it operates on existing advantage estimates and can be readily integrated into widely used methods such as GRPO, IGPO, and GiGPO across diverse tasks. Our experiments demonstrate consistent improvements across the evaluated algorithms, supporting its broad applicability.

The plug-in design addresses a shared challenge in existing RL algorithms: unreliable early-turn credit assignment. FOLIA refines the allocation of each algorithm’s advantage estimates across turns to improve early-turn learning signals while retaining the underlying training framework. Its broad compatibility is therefore a practical advantage: the same redistribution mechanism can benefit algorithms with different credit assignment strategies, allowing improvements in credit allocation to complement advances in the base algorithms.

W1 && Q1: Addressing the all-negative case

We clarify that FOLIA is orthogonal to strategies for handling all-fail groups and can complement them. To illustrate this, we compare vanilla DAPO [1] with DAPO + FOLIA. DAPO uses dynamic sampling to filter out all-fail and all-success groups and fill the batch with mixed-outcome groups, while FOLIA further refines turn-level credit within the retained trajectories. Detailed results on WebShop using Qwen2.5-3B-Ins are presented below:

MethodSuccess rateTask score
DAPO + FOLIA0.760 ± 0.0190.885 ± 0.013
Vanilla DAPO0.721 ± 0.0470.870 ± 0.014

These results show that FOLIA’s turn-level credit redistribution is orthogonal to handling all-negative groups. The additional gains from combining FOLIA with DAPO support its complementarity with dynamic sampling.

Reference

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025

W2 && Q2: Proxy validity, MC reference and length effect

We thank the reviewer for raising these questions about our experimental setup. Below, we clarify why the influence score is a useful proxy, how reliable the MC estimates are at our chosen sample size, and whether FOLIA remains effective as the decision horizon increases:

(Effect on unhelpful changes) We agree that the influence score conceptually measures the policy’s sensitivity to feedback, so a turn can have a large effect on the next action without benefiting the final result. However, we find in practice that the influence score is an inexpensive yet effective surrogate for each turn’s contribution, especially when combined with the hindsight signal. To examine FOLIA’s effect, we use an intermediate checkpoint (training step 90) of Qwen2.5-3B-Ins trained with GRPO on WebShop and compare the assigned credit with the reference credit.

First, we find that turns identified as important by FOLIA tend to have high reference credit: among turns whose credit increases by more than 25% under FOLIA, 68% have significantly positive reference credit, while only 12% have significantly negative reference credit. The latter cases, where credit increases despite significantly negative reference credit, account for only 2% of all turns. Second, we find that higher local influence itself corresponds to greater reference credit. Turns with high influence (top quartile) account for an average of 18.8% of their trajectory’s reference credit per turn, more than twice the share for turns with low influence (9.3%). These findings support local influence as a useful surrogate for a turn’s contribution to eventual success.

(Reliable reference) We thank the reviewer for raising this concern. The K=256K=256 reference still contains sampling noise. To assess its stability, we increase the sampling budget to K=1024K=1024 on 30 WebShop examples. The table reports the mean absolute difference between the credit estimates at each sampling budget and those obtained with K=1024K=1024.

Continuations KK8163264128256512
Overall0.1390.1330.1170.0880.0760.0510.047
Early turns0.1830.1630.1240.1100.0890.0570.053

Doubling the rollout budget from K=256K=256 to K=512K=512 reduces the discrepancy from the K=1024K=1024 reference by only 0.004, both overall (0.051 → 0.047) and at early turns (0.057 → 0.053). This trend toward stability, with only a marginal improvement despite doubling the sampling cost, supports using K=256K=256 as a practical approximate reference. By comparison, the early-turn discrepancy at K=8K=8 is approximately 3.2 times that at K=256K=256 (0.183 vs. 0.057), reinforcing the substantially lower reliability of early-turn estimates under small rollout budgets.

(Change with increasing length) We include an additional analysis on WebShop using the setting established in W1 and report credit-error reductions grouped by trajectory length. We report per-turn averages; because credit is distributed across fewer turns in shorter trajectories, their average per-turn credit shares are larger.

Trajectory length (turns)Credit error, GRPO → FOLIARelative change
≤ 40.309 → 0.277−10.5%
5–60.205 → 0.189−8.0%
7–90.156 → 0.145−7.4%
≥ 100.108 → 0.100−7.6%

FOLIA reduces credit error across all trajectory-length groups. For trajectories with at least 10 turns, credit error decreases by 7.6%, a reduction comparable to those observed for shorter trajectories. These results support FOLIA’s effectiveness over longer horizons.

We further extend our experiment to the ALFWorld dataset, which requires more than 30 interactions with the environment to complete a task on average. We use Qwen2.5-1.5B-Ins as the base model and GiGPO as the training algorithm. The table below reports validation success rates (%) on different tasks, with means and standard deviations across 3 runs for each method.

MethodPickLookCleanHeatCoolPick2All
Vanilla GiGPO91.05 ± 3.8960.25 ± 12.0489.23 ± 1.9194.56 ± 9.4290.63 ± 1.9376.10 ± 6.4785.50 ± 1.59
GiGPO + FOLIA94.60 ± 1.5174.60 ± 8.8587.07 ± 3.4396.27 ± 6.4791.07 ± 2.8983.67 ± 3.3088.80 ± 0.30

These results show that FOLIA continues to improve performance on tasks involving longer interactions, further supporting its general applicability when longer trajectories are required.

W2 && Q3: More Ablation Studies

We thank the reviewer for this suggestion and further ablate each component of our method. To isolate the effect of each signal, we use either the counterfactual influence or the hindsight signal alone. For the fusion step, we evaluate two variants: an ungated fusion (hard gate) and a simpler linear reweighting. In detail, the full model computes the fusion score ut=δ^t[1+s(τ) κt 1[ηt>0]],u_t = \hat{\delta}_t \left[ 1 + s(\tau)\,\kappa_t\,\mathbb{1}[\eta_t > 0] \right], where δ^t\hat{\delta}_t is the normalized counterfactual influence of turn tt and κt=tanh⁡(a∣ηt∣/2)\kappa_t = \tanh(a \lvert \eta_t \rvert / 2) is the confidence gate. It then computes the weights using the non-linear mapping mt:=exp⁡(ut)∑jpj0exp⁡(uj).m_t:=\frac{\exp(u_t)}{\sum_j p_j^0\exp(u_j)}. We consider the following settings:

  1. Local influence only: Uses only the local counterfactual influence score.

  2. Hindsight only: Uses only the hindsight signal.

  3. Ungated fusion (hard gate): Sets κt≡1\kappa_t \equiv 1, so the fusion score becomes uthard=δ^t[1+s(τ) 1[ηt>0]].u^{\text{hard}}_t = \hat{\delta}_t \left[ 1 + s(\tau)\,\mathbb{1}[\eta_t > 0] \right].

  4. Linear reweighting: Keeps the full model’s fusion score utu_t and changes only the reweighting step. The exponential weight mt=exp⁡(ut)/∑jpj0exp⁡(uj)m_t = \exp(u_t) / \sum_j p_j^0 \exp(u_j) is replaced by the proportional weight mtlin=ut∑jpj0uj,m^{\text{lin}}_t = \frac{u_t}{\sum_j p_j^0 u_j}, using the same pj0p_j^0 as in the full FOLIA.

We extend the ablation study using Qwen2.5-3B-Ins on WebShop under the same settings as in the paper, reporting the mean success rate ± standard deviation over three runs.

VariantGiGPOGRPO
Baseline0.573 ± 0.0330.510 ± 0.028
Hindsight only0.563 ± 0.0350.546 ± 0.033
Local influence only0.582 ± 0.0250.530 ± 0.014
Linear reweighting0.568 ± 0.0230.501 ± 0.017
Hard gate0.596 ± 0.0520.540 ± 0.039
Full FOLIA0.648 ± 0.0390.567 ± 0.009

Full FOLIA achieves the highest mean success rate under both algorithms. Neither signal alone recovers the full gains. This reconfirms the necessity of including both signals. Among the two alternative fusion strategies, linear reweighting falls below both baselines. This is because near-zero scores can make its relative weights sensitive to small perturbations, while FOLIA’s exponential weighting preserves a positive baseline before normalization, limiting unnecessary redistribution when all scores are weak. The hard gate performs better than linear reweighting but still falls short of FOLIA. It retains both signals but sets κt≡1\kappa_t\equiv1, applying the same maximal hindsight multiplier to every positive lift, regardless of its magnitude. FOLIA instead uses κt=tanh⁡(a∣ηt∣/2)\kappa_t=\tanh(a|\eta_t|/2) to scale the boost with lift strength. This gives weak lifts smaller boosts and limits large ones, avoiding abrupt weight changes near ηt=0\eta_t=0 and helping explain FOLIA’s better performance.

The analysis of credit estimation errors across different trajectory lengths is provided in the (Change with increasing length) section of W2. Together, these results support combining complementary local and hindsight signals with a graded confidence gate and normalized exponential reweighting to achieve the full performance gains, and demonstrate that FOLIA is an effective strategy for long-horizon tasks.

Q2: Local influence as a proxy

We thank the reviewer for this question about the reliability of the influence score. The policy shift is the earliest observable footprint of an action. We use it as a proxy for how much a turn matters. Our WebShop analysis in the (Effect on unhelpful changes) part of W2 && Q2: Proxy validity indicates that it is a reliable surrogate in most cases.

Furthermore, to directly examine how FOLIA avoids over-assigning credit, we select the most heavily weighted turn in each WebShop validation trajectory at checkpoint 90, using the same settings as before. The MC reference identifies this turn as unhelpful in 42% of these trajectories. Within this subset, its mean credit error remains nearly unchanged (0.2086 → 0.2089), while their total trajectory credit error decreases by 7.0%. In this way, we find that FOLIA avoids over-assigning credit to feedback that substantially changes subsequent behavior but does not actually help complete the task.

We are grateful to Reviewer mS7G for the constructive suggestions that helped improve our work. We have incorporated these suggestions into the revision and would be glad to address any remaining questions or concerns.

AuthorsOpenReview ↗

Friendly Reminder

Dear Reviewer mS7G,

Thank you for your constructive feedback, which has helped improve our paper. With one day remaining in the discussion period, we would be grateful for your feedback on whether our rebuttal addresses your concerns. We would be happy to clarify any remaining points.

Thank you for your time.

Best regards,

The Authors

Comments on the paper

2 comments

Dear Reviewers,

We are now halfway through the discussion phase. The authors have provided their responses to your reviews. If their responses have not adequately addressed your concerns, please take this opportunity to raise any remaining questions or concerns.

I encourage you to engage with the authors as soon as possible to ensure a productive discussion.

Many thanks,
AC

Dear Reviewers mS7G and mRzL,

The authors have provided detailed responses to your concerns, including additional experimental results. However, we have not yet received your follow-up feedback.

As we are at the end of the discussion period, please review the authors' responses and briefly indicate whether your concerns have been adequately addressed. Your feedback is important for the final assessment.

Thank you for your time and cooperation.

Best regards,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.