acceptodds
Under review as a conference paper at ICLR 2027

My Fault: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning (RL) has emerged as a powerful approach for training large language model agents on multi-step tasks, yet reliance on terminal outcome rewards creates two credit-assignment problems, particularly in long-horizon tasks. First, same-outcome rollout groups provide no learning signal from terminal rewards. Second, terminal rewards provide only trajectory-wide feedback, making it difficult to identify which decisions caused a failure. Recent work supplements terminal rewards with finer-grained information from trajectory analysis, such as natural-language reflections on intermediate decisions and errors. However, natural-language diagnoses are difficult to use directly for credit assignment: their error claims may be unreliable, and they do not quantify how much each error should affect learning. We propose Self-Diagnosis-guided Terminal Credit Redistribution (FAULT), which turns diagnosed errors into explicit step-level credit anchored by terminal outcomes. FAULT checks diagnostic evidence and learns relative error costs from task outcomes. During training, the policy and self-diagnoser co-evolve, while error costs are updated online from recent outcomes. On ALFWorld, FAULT recovers learning signals from same-outcome groups, reaching 95% signal coverage versus 41% for GRPO and 72% for GiGPO, while better localizing credit to specific error steps. Across two model scales, FAULT delivers strong improvements on the long-horizon ALFWorld and WebShop tasks while remaining competitive on short-horizon Search-based QA.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_wU58OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FAULT, a self-diagnosis-guided credit assignment method that converts LLM-identified errors into step-level advantages by calibrating error costs against terminal outcomes, thereby improving agentic reinforcement learning on long-horizon tasks.

Critical strengths
  1. The paper presents a methodologically interesting approach that explicitly converts LLM-based retrospective error diagnosis into step-level advantages. The proposed conditional regression further calibrates error costs against observed terminal outcomes.

  2. The paper provides detailed methodological descriptions and implementation settings, making the approach relatively easy to reproduce.

Critical weaknesses
  1. The proposed verification mechanism primarily checks whether the cited evidence exists in the trajectory, rather than whether the diagnosed error is semantically correct or causally responsible for the failure.

  2. Although diagnosis tokens do not directly receive policy gradients, the diagnoser shares parameters with the actor. Policy updates may therefore indirectly affect diagnostic quality and introduce instability.

Questions for authors
  1. How robust is FAULT to incorrect or incomplete error diagnoses? Have the authors evaluated how diagnosis errors affect credit assignment and downstream performance?

  2. How does diagnostic accuracy evolve during RL training? Is there evidence that the shared actor–diagnoser architecture maintains or improves diagnostic quality over time?

  3. How much additional benefit does outcome-calibrated error pricing provide compared with directly assigning uniform penalties to LLM-diagnosed error steps?

Reviewer_vyf4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FAULT, a credit-assignment framework that converts structured self-diagnoses into step-level penalties for agentic reinforcement learning. A teacher provides cold-start diagnoses and an error taxonomy. The policy then diagnoses its own trajectories, while a statistical estimator fits relative error prices from terminal outcomes. A bounded penalty budget is allocated to diagnosed steps and incorporated into two advantage channels for policy optimization. Experiments cover ALFWorld, WebShop, and Search-based QA, with additional analyses of signal coverage, localization, and diagnostic quality.

Critical strengths
  1. Separating error localization from error valuation is a useful design. Estimating relative error prices from observed outcomes reduces reliance on an LLM's subjective severity judgments.

  2. The additive diagnostic penalties can provide differentiated learning signals in zero-reward failure groups, addressing a limitation of purely outcome-based group comparisons.

  3. The paper includes extensive analyses beyond final performance and provides controls sharing the diagnosis-SFT initialization. The ALFWorld improvement over these controls is promising.

  4. The authors transparently acknowledge several limitations, including single-run results, differences in reward settings, and imperfect component ablations.

Critical weaknesses
  1. The paper lacks a direct comparison with a uniform error pricing scheme; in the ablation study where the online pricing mechanism is removed, the penalty budget cap is also eliminated. If the authors could provide robust experimental results based on uniform pricing or fixed initial pricing, while keeping all other settings constant, I might consider raising my score.

  2. In the ablation study concerning the fixed diagnostician, the penalty schedule is also altered. If a rigorous comparison between a fixed diagnostician and a dynamically updated diagnostician were provided, and demonstrating that the latter yields consistent improvements in downstream performance while maintaining diagnostic quality, I might increase my score.

  3. When a task requires acquiring an apple, an action such as pick up apple (verifiable via reference matching) might still be incorrectly labeled as a wrong object error. The evaluation process must account for both the task objective and the current state; relying solely on human evaluator consensus or centralized penalties is insufficient to ensure accurate credit assignment.

  4. In the WebShop task, different methods employ varying training reward functions, and FAULT’s strict success rate is lower than that of SEED. On the 4B QA task, FAULT also underperforms compared to the shared-initialization baseline. Comparative experiments should utilize a consistent reward function and the same diagnosis-SFT checkpoints, ensuring that all other training configurations remain identical. Furthermore, the results should report the mean performance and variance across multiple independent training runs (using different random seeds).

  5. Components such as first-error boosts, penalty schedules, and budget caps involve manually selected parameters, yet the paper lacks a comprehensive sensitivity analysis. Moreover, each benchmark requires a corresponding taxonomy, diagnostic dataset, and price calibration. The authors should clarify the rationale behind their parameter choices and specify which components would need to be reconstructed or recalibrated when transferring the approach to a new domain.

Questions for authors
  1. Does outcome-fitted pricing outperform uniform fixed error prices when diagnosis, localization, penalty magnitude, scheduling, and budget caps are controlled? Can online pricing be ablated without also removing the cap?

  2. Does updating the shared diagnoser improve performance over a frozen diagnosis-SFT model under an identical training recipe? How do their diagnostic accuracy and downstream task performance compare?

  3. Can the authors provide a targeted assessment of whether accepted quotations actually support the diagnosed errors, beyond verifying that the quotations exist? For example, if the task requires picking up an apple and the quoted action is “pick up the apple,” a diagnosis of “wrong object” could pass the quotation-matching check despite being incorrect. An audit considering the task goal and the state at the cited step would help establish the semantic correctness of accepted diagnoses.

  4. The WebShop comparisons involve differences in training rewards and initialization: FAULT uses graded task-score rewards and a diagnosis-SFT checkpoint, whereas some baselines use binary rewards or start from the raw backbone. Can the authors compare FAULT with GRPO and GiGPO using the same reward function and the same diagnosis-SFT checkpoint, while keeping the remaining training settings matched? This would clarify how much improvement comes from FAULT's credit-assignment mechanism rather than reward design or initialization. Additionally, can the authors report results from multiple independent training seeds, including means and standard deviations, to assess whether the gains are consistent across runs?

  5. How were the first-error multipliers and penalty-strength settings selected, and how sensitive are the conclusions to these choices? Which components must be reconstructed or recalibrated for a new task domain?

1 comment

Dear Reviewer, thank you very much for the time you spent reviewing our paper and for your valuable comments. Below is our response, which we have divided into five parts corresponding to the five weaknesses and questions you raised. All revisions mentioned in the response below will be incorporated into the revised version of the paper.

W1, Q1

W1: The benefit of learned pricing is not cleanly isolated.

Q1: Does outcome-fitted pricing outperform uniform fixed error prices when diagnosis, localization, penalty magnitude, scheduling, and budget caps are controlled? Can online pricing be ablated without also removing the cap?

Summary: With the penalty amount and all other settings unchanged, uniform prices solve only 110/134 tasks (82.1%), below all three FAULT seeds (120–123, mean 90.8%), showing that outcome-fitted prices clearly outperform uniform prices; the cap truncates only 0.27% of trajectories, and removing it increases the total penalty by only 0.7%, so its effect is small.

Experiments with uniform fixed error prices

We agree that the original paper did not isolate the effect of pricing. The additional results are as follows (ALFWorld, Qwen3-4B, 134 unseen tasks):

Method

All verified errors have the same price. To keep the penalty amount unchanged, each group of 8 trajectories directly uses the set of penalties computed by FAULT's pricing rule on the same trajectories and diagnoses, and then reassigns them according to their scores under uniform prices: the trajectory with the highest score receives the largest penalty, and so on. As a result, the mean penalty, the within-group spread and the number of trajectories that hit the cap are all the same as in FAULT. The diagnoses, step localization, first-error boost, cosine λ, cap, code and diag-SFT initialization are also unchanged.

Results:

SettingPick (24)Look (18)Clean (31)Heat (23)Cool (21)Pick2 (17)Total (134)
FAULT seed 1 (submitted version)221829201815122 (91.0%)
FAULT seed 2231828201915123 (91.8%)
FAULT seed 3221829191814120 (89.6%)
FAULT mean22.31828.719.718.314.7121.7 (90.8%)
Uniform prices181825191713110 (82.1%)

Conclusions:

  • Uniform prices still have a considerably lower success rate than the original method, even with three FAULT seeds.
  • Revision: we add uniform prices and the three FAULT seeds to D.6 and Table 4, attribute the gain from pricing to outcome-fitted prices, and no longer state that frozen prices are worse than the full method.

The cap has little effect

In FAULT's run, the cap truncated only 56/20,480 trajectories (0.27%), all of them successful, which is 0.64% of the penalized successful trajectories (Appendix D.2). Recomputing on the same trajectories with the cap removed increases the total penalty by only 0.7%.

Reviewer_1Ar3OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a credit assignment method: the model first self-diagnoses which steps in a trajectory went wrong (diagnoses must be verifiable against the trajectory text), then uses the final outcome to "price" each error type—errors that more often lead to failure get a larger cost—and finally redistributes the terminal credit across steps: diagnosed error steps are penalized more, so even same-outcome groups regain a learning signal.

How pricing works (§2.3 / Appendix B.3): ① Counting. Each trajectory's errors are converted to per-class error rates (verified class-k count divided by step count) and matched to that trajectory's success or failure. ② Removing task difficulty. Hard tasks fail more often, so trajectories cannot be compared across tasks; comparisons are made only within the same task—given the same number of successes, do the successful rollouts have lower error rates? This within-task comparison (conditional logit) automatically cancels task difficulty; all-success and all-fail groups are not used. ③ From coefficients to prices. The regression yields a cost coefficient per error class; only classes where more errors predict worse outcomes (negative coefficients) become prices, normalized so the most expensive class has price 1 and harmless errors get price 0; graded rewards (WebShop) instead use ridge regression. ④ Online updates. Every 10 updates the prices are refit on the last 50 updates of trajectories (smoothed as 0.7 old / 0.3 new, taking effect from the next batch), so prices co-evolve with the policy.

How prices are used: each trajectory is scored as the price-weighted sum of its error rates (clipped to 0–1), scaled by a penalty strength and capped to form a bounded penalty budget; the budget is then split across steps in proportion to the prices of the verified errors cited there—the more heavily priced the diagnosed errors, the larger the step penalty.

Results: 95% usable signal coverage on ALFWorld (GRPO 41%, GiGPO 72%), decisive-step MRR 0.494, leading on long-horizon tasks (ALFWorld, WebShop) at both scales, and parity with SEED on Search-based QA.

Critical strengths

Clean problem decomposition with a self-consistent mechanism: credit assignment is split into "where the error is" (localization via verified diagnosis) and "how costly it is" (valuation via terminal-outcome regression), enabling learning signals even within same-outcome groups—directly addressing a fundamental defect of GRPO-style methods.

Critical weaknesses

The "errors are enumerable and clusterable" assumption is questionable in open scenarios: the frozen taxonomy is induced by a strong teacher, yet the cold-start off-table rate is 0.31 (target 0.10) and decisive-step localization is 0.48 (target 0.60); the more open the task, the coarser the granularity (SearchQA degrades to L1 indicators). Long-tailed errors and within-class semantic drift may distort pricing and step-level credit, capping the method's ceiling.

Questions for authors

The Table 4 variants are not single-factor controls (e.g., fixed judge also uses constant λ) and come from different run eras, so gains cannot be attributed to specific components, and the strong claim that "every component matters" is unsupported. Could you provide truly single-factor ablations—removing online pricing, the budget cap, the shared diagnoser, or the first-error boost one at a time—instead of the current multi-component "recorded variants"?

Comments on the paper

3 comments

Thank you for taking my concerns seriously and revising the paper accordingly. I have carefully reviewed the newly added ablation studies, the evidence regarding diagnostic accuracy, and the comparative experiments on WebShop. The rationale for parameter selection is also much more clearly articulated. These improvements are excellent.

Your response and the supplementary experiments have effectively addressed my concerns. I have decided to raise my score to 3 and am now inclined to recommend acceptance.

AuthorsOpenReview ↗

Thank You for Your Reconsideration and Constructive Feedback

Thank you very much for carefully reviewing our response and the revised paper, and for raising your score to 3. This is a great encouragement to us. Your comments were very specific and constructive. Following your suggestions, we added the single-factor ablations with uniform prices and a frozen diagnoser, a manual semantic audit of the accepted diagnoses (your apple example served directly as the audit criterion), and multi-seed comparisons on WebShop under the same reward and initialization, and we explained the rationale for the parameter choices more clearly. These revisions have made the conclusions of the paper more rigorous and more convincing. We will keep all of these results and explanations in the final version of the paper. Thank you again for the time and effort you have devoted to our paper, and for your recognition of our work.

AuthorsOpenReview ↗

Rebuttal Summary: Reviewer Feedback, New Experiments and Revisions

Dear AC, SAC, PC and Reviewers,

As the discussion period comes to a close, our rebuttal is complete. We sincerely thank the three reviewers for their careful and constructive reviews, the AC, SAC and PC for their time and effort, and our collaborators for their hard work on the additional experiments and the revision. The current scores are 4, 3 and 3, all on the acceptance side.

Strengths noted by the reviewers. These points summarize the reviewers' assessments, not our own claims. FAULT cleanly splits credit assignment into where an error is (localized by verified diagnosis) and how costly it is (priced from observed outcomes), turning LLM retrospective diagnosis into step-level advantages with a self-consistent mechanism (R1, R2, R3). Fitting prices to outcomes reduces reliance on an LLM's subjective judgment of error severity (R2, R3). Diagnostic penalties give differentiated signals even within same-outcome groups, addressing a key limitation of GRPO-style methods (R1, R2). The reviewers also appreciated the extensive analyses and shared-initialization controls (R2), the detailed and reproducible method description (R3), and the candid discussion of limitations (R2).

R1(1Ar3) gave a score of 4, the highest score and the strongest recognition of our work; we are deeply grateful.

  • (1) The assumption that errors are enumerable and clusterable may not hold in open settings. We agree this is a current limitation: each dataset needs its own taxonomy, though the same cold-start procedure builds it automatically. We price only frequent errors, so long-tail errors are at worst missed rather than wrongly penalized. During RL, the off-table rate also falls from 0.32 to 0.16. The revision adds a Limitations appendix.

  • (2) The original ablations were not single-factor. The new ones strictly change one factor at a time, with the same code and diag-SFT initialization; FAULT was run with 3 seeds (mean 90.8%). Freezing the diagnoser, removing the first-error boost and using uniform prices give 86.6%, 85.1% and 82.1%, all below all three FAULT seeds. The cap truncates only 0.27% of trajectories, and removing it raises the total penalty by only 0.7%. We removed the claim that every component matters.

R2 (vyf4) raised the score from 2 to 3 after reviewing our response and the revised paper, noting that the new ablations, the evidence on diagnostic accuracy, the WebShop comparisons and the parameter rationale addressed the concerns; we sincerely thank the reviewer.

  • (1) Pricing: with the same penalty amount and all other settings unchanged, uniform prices are 8.7 points below FAULT.

  • (2) Diagnoser updates during training: a matched frozen diagnoser reaches 86.6%, below all three FAULT seeds. Over 17 RL checkpoints, diagnostic quality does not decline (L1 error-slot recall 0.66 → 0.71).

  • (3) Semantic correctness: we manually audited 40 randomly sampled accepted diagnoses against the task goal and the history up to the cited step; 39 cite an error that actually occurred, 38 have the correct class and 39 the correct step. When penalty positions are shuffled so that they no longer follow the diagnoses, success falls below the level without diagnostic penalty on both ALFWorld (77.6% vs. 82.1%) and WebShop (59.9% vs. 67.6%).

  • (4) Matched comparisons: on WebShop, with the same reward and diag-SFT initialization and 3 seeds each, FAULT reaches 71.6 ± 0.6 strict success, 4.0 points above GiGPO and 8.1 above GRPO. On ALFWorld, FAULT's 3-seed mean of 90.8 ± 1.1 is 8.7 points above the matched baseline without diagnostic penalty. We keep the results where FAULT trails SEED on WebShop and the controls on 4B Search-QA, and mark the short-trajectory explanation as a hypothesis.

  • (5) Parameters: we explained the rationale for the first-error boost, the λ schedule and the cap; removing the boost lowers success to 85.1%. For a new domain, only the cold start is rerun, automatically with the same procedure.

R3(wU58) gave a score of 3 and, after reading our response, kept it and leaned toward acceptance; we are equally grateful.

  • (1) On semantic correctness and robustness to diagnosis errors, the audit and shuffling results above also apply. Missed errors simply fall back to outcome-based credit, and how strongly an error is associated with failure is captured by its outcome-fitted price.

  • (2) On the shared diagnoser: diagnostic quality does not decline, parse and gate failures stay near 0, the share of entries passing the evidence checks rises from 89.0% to 92.0%, and the frozen diagnoser also performs worse.

  • (3) Outcome-calibrated pricing outperforms uniform penalties by 8.7 points.

All additions are marked in red in the revised paper, mainly in §3.3.3 (Table 2) and Appendices B.4, C.3, D.5 and H.

Once again, we sincerely thank the reviewers, the AC, SAC and PC, and our collaborators for their help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.