- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a step-level reward shaping algorithm that enhances reward signals in Agentic-RL training process. The algorithm propagates the reward back to previous key tokens, using attention-derived dependencies across interaction steps. The paper examines its algorithm with Qwen2.5-7B-Instruct on ALFWorld, WebShop and ScienceWorld, reporting statistically detectable performance improvement in ALFWorld and ScienceWorld environments.
Critical strengths
-
The paper presents a reward shaping method without requiring an independent reward model or extra rollouts, therefore the method is at a rather low cost. Marking the significance of this work.
-
The paper explained algorithm details in the text. Making substantial efforts on reproducibility.
-
The paper experimented on different environments and setups. It also includes ablation studies on the result. Making the work more solid.
Critical weaknesses
-
Since your method is more about relevance, rather than causal contribution, I believe it would be more appropriate if you called it reward shaping rather than credit assignment like that you established between texts.
-
Although your method is more about distributing constructed reward over previous steps/tokens, there are still concepts or tricks that calls for more ablation studies or explanations. For example, as you mentioned in the paper, recency correction could be a problem for LLMs, but what exactly did it affect or how are you inspired to adopt it?
Questions for authors
-
Why would you want to call it credit assignment? Is there enough motivation in your experiment to call it so?
-
Are there more ablation studies you did not mention in the main text or supplementary materials?
-
Are the results single-run or calculated average results? What are the range and variance?