- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes FAULT, a self-diagnosis-guided credit assignment method that converts LLM-identified errors into step-level advantages by calibrating error costs against terminal outcomes, thereby improving agentic reinforcement learning on long-horizon tasks.
Critical strengths
-
The paper presents a methodologically interesting approach that explicitly converts LLM-based retrospective error diagnosis into step-level advantages. The proposed conditional regression further calibrates error costs against observed terminal outcomes.
-
The paper provides detailed methodological descriptions and implementation settings, making the approach relatively easy to reproduce.
Critical weaknesses
-
The proposed verification mechanism primarily checks whether the cited evidence exists in the trajectory, rather than whether the diagnosed error is semantically correct or causally responsible for the failure.
-
Although diagnosis tokens do not directly receive policy gradients, the diagnoser shares parameters with the actor. Policy updates may therefore indirectly affect diagnostic quality and introduce instability.
Questions for authors
-
How robust is FAULT to incorrect or incomplete error diagnoses? Have the authors evaluated how diagnosis errors affect credit assignment and downstream performance?
-
How does diagnostic accuracy evolve during RL training? Is there evidence that the shared actor–diagnoser architecture maintains or improves diagnostic quality over time?
-
How much additional benefit does outcome-calibrated error pricing provide compared with directly assigning uniform penalties to LLM-diagnosed error steps?
W1, Q1
Dear Reviewer, thank you very much for the time you spent reviewing our paper and for your valuable comments. Below is our response, which we have divided into five parts corresponding to the five weaknesses and questions you raised. All revisions mentioned in the response below will be incorporated into the revised version of the paper.
W1, Q1
Summary: With the penalty amount and all other settings unchanged, uniform prices solve only 110/134 tasks (82.1%), below all three FAULT seeds (120–123, mean 90.8%), showing that outcome-fitted prices clearly outperform uniform prices; the cap truncates only 0.27% of trajectories, and removing it increases the total penalty by only 0.7%, so its effect is small.
Experiments with uniform fixed error prices
We agree that the original paper did not isolate the effect of pricing. The additional results are as follows (ALFWorld, Qwen3-4B, 134 unseen tasks):
Method
All verified errors have the same price. To keep the penalty amount unchanged, each group of 8 trajectories directly uses the set of penalties computed by FAULT's pricing rule on the same trajectories and diagnoses, and then reassigns them according to their scores under uniform prices: the trajectory with the highest score receives the largest penalty, and so on. As a result, the mean penalty, the within-group spread and the number of trajectories that hit the cap are all the same as in FAULT. The diagnoses, step localization, first-error boost, cosine λ, cap, code and diag-SFT initialization are also unchanged.
Results:
Conclusions:
The cap has little effect
In FAULT's run, the cap truncated only 56/20,480 trajectories (0.27%), all of them successful, which is 0.64% of the penalized successful trajectories (Appendix D.2). Recomputing on the same trajectories with the cap removed increases the total penalty by only 0.7%.