- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies unreliable early-turn credit assignment in agentic reinforcement learning with sparse terminal rewards. Under a sequential-hardness assumption, it derives an exponential necessary rollout-budget scaling for maintaining the signal-to-noise ratio of independent Monte Carlo prefix-value differences. FOLIA then reweights existing turn advantages using the likelihood effect of ablating a tool response, together with a success-conditioned terminal-hindsight gate. The method uses teacher-forced scoring rather than additional environment rollouts. Experiments apply the plug-in to several critic-free RL algorithms with Qwen2.5 models on search QA and WebShop, with diagnostic comparisons against a higher-budget Monte Carlo reference.
Critical strengths
-
The diagnosis of early-turn credit failure is useful. Proposition 3.2 makes the dependence on rare successful continuations explicit, and the prefix-resampling experiments connect that analysis to training behavior. The result is specific to the stated sequential-hardness assumption and independent rollout estimator, but within that setting the argument is clear.
-
The improvements are reasonably consistent across base algorithms and model sizes. On 3B WebShop, success rises from 0.510 to 0.567 with GRPO and from 0.573 to 0.648 with GiGPO (Table 1). The measured iteration-time overhead is only 3.5–5.2% in Appendix A.2. A method that obtains these gains without more environment interaction would be useful in practice.
Critical weaknesses
-
My main concern is the gap between the credit-assignment interpretation and the update in Section 4.3. The new advantage is the base advantage times a positive clipped weight: its sign cannot change. Hindsight only increases salience on successful trajectories when eta is positive. For GRPO, a harmful turn in a successful trajectory retains a positive surrogate advantage whenever the base group-relative advantage is positive, although its weight may decrease. This statement concerns the surrogate signal, not a guaranteed change in action probability after optimization. This does not rule out a useful reweighting method, but it limits what can be claimed about recovering the signed advantage defined in Section 2. Appendix A.6 acknowledges missing-signal cases; the wrong-sign case also needs to be addressed.
-
The credit-error diagnostics need a more explicit account of calibration and reference uncertainty. Section 3.3 states that surrogate estimates are normalized to [−1, 1], and Appendix A.4 divides signed MC credits by their sum across turns for the case study. These descriptions do not specify a common transformation for every quantity in Figures 2 and 5, including the treatment of a near-zero or negative normalization denominator. The K=256 reference also has sampling error. Consequently, it is difficult to separate improved credit estimation from changes in scale. Table 2 compares the full method with hindsight-only, but omits local-influence-only; this is the most direct missing control for the fusion. The main QA results also need seed counts and uncertainty.
Questions for authors
-
Is FOLIA intended to recover signed turn credit or to redistribute the magnitude of an existing signal? Please explain the directional claims in light of the positive multiplier. A breakdown of Figure 5 by agreement between the base-advantage sign and the MC-credit sign would help resolve this.
-
Section 3.3 and Appendix A.4 give different levels of detail about normalization. Please provide the exact transformations used for all estimates and references in Figures 2 and 5, including denominator handling, and quantify uncertainty in the K=256 reference. Section 5.3 specifies 200 examples; how were these selected?
-
Do the authors have a local-influence-only result under the same training settings? This would be more informative about the need for fusion than the current hindsight-only comparison.
6 comments
W2 && Q2: Transformation, scale, and reliable MC estimation
We thank the reviewer for suggesting that we provide more detail on the transformation and credit-error diagnostics. We address each point below.
(Transformation and scale) We thank the reviewer for this careful review. Figures 2 and 5 and App. A.4 use one uniform transformation. For each trajectory, the turn credits , from the MC reference () or from any method (like the information gain defined in IGPO), are mapped to signed shares . The credit error of turn is averaged over trajectories; Figure 5 averages turns 1 and 2. This removes one positive scale per trajectory and keeps the sign and relative size of each turn's credit, as defined in Section 2. The denominator is a sum of absolute values, so it is never negative; "the sum of credits" in App. A.4 was imprecise and is corrected. It is zero only when all credits are zero, which holds for 16 of the 200 reference trajectories; these are excluded, and no other threshold is applied.
(Reference uncertainty) We thank the reviewer for raising this concern. The reference still contains sampling noise. To assess its stability, we increase the sampling budget to on 30 WebShop examples. The table reports the mean absolute difference between the credit estimates at each sampling budget and those obtained with .
| Continuations | 8 | 16 | 32 | 64 | 128 | 256 | 512 |
|---|---|---|---|---|---|---|---|
| Overall | 0.139 | 0.133 | 0.117 | 0.088 | 0.076 | 0.051 | 0.047 |
| Early turns | 0.183 | 0.163 | 0.124 | 0.110 | 0.089 | 0.057 | 0.053 |
Doubling the rollout budget from to reduces the discrepancy from the reference by only 0.004, both overall (0.051 → 0.047) and at early turns (0.057 → 0.053). This trend toward stability, with only a marginal improvement despite doubling the sampling cost, supports using as a practical approximate reference. By comparison, the early-turn discrepancy at is approximately 3.2 times that at (0.183 vs. 0.057), reinforcing the substantially lower reliability of early-turn estimates under small rollout budgets.
W2 && Q3: Local-influence-only ablation
We thank the reviewer for this suggestion. In the setting used for Table 2 of the original manuscript (Qwen2.5-3B-Ins on WebShop), we additionally evaluate local influence alone and two alternative fusion strategies, defined as follows:
-
Local influence only: Uses only the local counterfactual influence score.
-
Hindsight only: Uses only the hindsight signal.
-
Ungated fusion (hard gate): Sets , so the fusion score becomes
-
Linear reweighting: Keeps the full model’s fusion score and changes only the reweighting step. The exponential weight is replaced by the proportional weight using the same token-weighted prior as in the full model.
We report the mean success rate ± standard deviation over three runs.
| Variant | GiGPO | GRPO |
|---|---|---|
| Baseline | 0.573 ± 0.033 | 0.510 ± 0.028 |
| Hindsight only | 0.563 ± 0.035 | 0.546 ± 0.033 |
| Local influence only | 0.582 ± 0.025 | 0.530 ± 0.014 |
| Linear reweighting | 0.568 ± 0.023 | 0.501 ± 0.017 |
| Hard gate | 0.596 ± 0.052 | 0.540 ± 0.039 |
| Full FOLIA | 0.648 ± 0.039 | 0.567 ± 0.009 |
From these results, we find that local influence alone performs slightly better than the baselines but worse than full FOLIA under both algorithms. Hindsight alone also performs worse than full FOLIA, so neither signal alone recovers the gains from their fusion. Both alternative fusion strategies also perform worse than full FOLIA. All these results reinforce the necessity of our design.
Q1: Signed credit recovery vs. magnitude redistribution
We thank the reviewer for this suggestion, which helps clarify FOLIA’s role. FOLIA redistributes the magnitudes of existing advantages through positive multipliers while preserving their signs. Although it does not correct sign errors in the base credit, it can, in practice, reduce credit error both when the base advantage and the MC reference agree in sign and when they disagree.
Following the reviewer’s suggestion, we conduct an additional analysis on WebShop by separating cases where the base advantage and the MC reference agree or disagree in sign. We still use the Qwen2.5-3B-Ins GRPO checkpoint at training step 90, on 100 validation trajectories, and a MC reference. We define early turns as the first half of each trajectory. We then group these turns by sign agreement between the base advantage and the MC reference:
| Early turns | Credit error, base → FOLIA |
|---|---|
| Signs agree with MC reference | 0.142 → 0.131 |
| Signs disagree with MC reference | 0.239 → 0.219 |
FOLIA reduces error in both sign groups. When the signs disagree, the improvement comes from reducing the magnitude of incorrectly signed credit toward zero without correcting its sign. This provides a practical way to suppress incorrect training signals. We have revised our Introduction to make this clear (line 69).
We sincerely thank Reviewer mRzL for their suggestions for improving our work and have incorporated them into our revision. We would be happy to address any remaining concerns or questions.
Response for Reviewer mRzL
We appreciate Reviewer mRzL for the detailed and insightful comments, and for recognizing the value of our problem formulation and empirical study. Below, we respond to each concern in turn.
W1: Sign-preserving update and the wrong-sign case
We thank the reviewer for this note on our setting. Incorrect directions could be a practical issue when FOLIA cannot correct them. However, we want to note that such cases (a turn with an incorrect direction but a large effect) are infrequent in practical training and that FOLIA can still be beneficial even when it cannot directly correct these signs.
To illustrate this, we examine FOLIA’s effect on incorrectly signed credit using an intermediate checkpoint of Qwen2.5-3B-Ins trained with GRPO on WebShop (training step 90 of 150), following the setting in Section 5.1 and using K=256 to estimate reference credit. First, we find that turns with both an incorrect sign and a large credit magnitude are uncommon: only 13% of early turns (defined as the first half of each trajectory) have a significantly incorrect sign, and just 4% have both an incorrect sign and a large credit magnitude. Early turns with incorrect signs account for just 8% of a trajectory’s reference credit, compared with 27% for early turns with correct signs. Moreover, FOLIA reduces estimation error on incorrectly signed turns by moving the assigned credit toward zero. Among all incorrectly signed turns, FOLIA reduces the assigned credit magnitude for 81% of them, while their credit estimation error decreases from 0.334 to 0.302. FOLIA therefore reduces the weight these turns receive in the policy update, even without directly correcting their signs, which may help explain its training benefits.
Furthermore, we want to note that the influence score serves as an effective and cheap surrogate for the value. We find that high-influence turns, defined as the top quartile of influence scores, carry more than twice the average reference credit share per turn of low-influence turns (18.8% versus 9.3%). We further examine trajectories in which the turn most heavily weighted by FOLIA is unhelpful according to the MC reference. Within this subset, the selected turn’s mean credit error remains nearly unchanged (0.2086 → 0.2089), while total trajectory credit error decreases by 7.0%. These findings support the practical usefulness of influence as a proxy and of FOLIA’s redistribution mechanism, which combines influence with hindsight.
W1: Addressing missing signals
We thank the reviewer for highlighting cases where the learning signal is missing. We clarify that FOLIA is orthogonal to strategies for handling all-fail groups and can complement them. To illustrate this, we compare vanilla DAPO [1] with DAPO + FOLIA. DAPO uses dynamic sampling to filter out all-fail and all-success groups and fill the batch with mixed-outcome groups, while FOLIA further refines turn-level credit within the retained trajectories. Detailed results on WebShop using Qwen2.5-3B-Ins are presented below:
These results show that FOLIA’s turn-level credit redistribution is orthogonal to handling all-negative groups. The additional gains from combining FOLIA with DAPO support its complementarity with dynamic sampling.
Reference
[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale, NeurIPS 2025