acceptodds
Under review as a conference paper at ICLR 2027

AttnSR: Attention Step Reward for Long-Horizon Agentic RL

Abstract

Long-horizon LLM agents make multi-step decisions, yet terminal-reward-only reinforcement learning provides little direct supervision for individual steps. We introduce AttnSR, a step-level reward augmentation method that derives reward allocations from policy attention extracted during training replay, without a separately trained large reward model or additional environment rollouts. AttnSR traces backward through the policy model attention graph to estimate the relevance of earlier steps to the final response. The terminal step, measured relative to a running baseline, determines whether these rewards reinforce or penalize the selected steps. In a controlled Qwen2.5-7B-Instruct study, AttnSR consistently outperforms baseline across all evaluated checkpoints on ALFWorld, raising final held-out success from 91.2% to 97.1%. The step-reward ablation experiment shows that the benefit depends on how intermediate rewards are assigned, rather than simply on their presence. AttnSR also improves ScienceWorld at every checkpoint while cutting interaction by 29.9%. No statistically detectable performance improvement is observed on WebShop. These results support policy attention as a practical prior for step-level reward allocation, with benefits that may depend on task complexity and horizon.

Primary area:
reinforcement learning
Keywords:
Credit Assignment, Agentic RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8Dp5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a step-level reward shaping algorithm that enhances reward signals in Agentic-RL training process. The algorithm propagates the reward back to previous key tokens, using attention-derived dependencies across interaction steps. The paper examines its algorithm with Qwen2.5-7B-Instruct on ALFWorld, WebShop and ScienceWorld, reporting statistically detectable performance improvement in ALFWorld and ScienceWorld environments.

Critical strengths
  1. The paper presents a reward shaping method without requiring an independent reward model or extra rollouts, therefore the method is at a rather low cost. Marking the significance of this work.

  2. The paper explained algorithm details in the text. Making substantial efforts on reproducibility‌.

  3. The paper experimented on different environments and setups. It also includes ablation studies on the result. Making the work more solid.

Critical weaknesses
  1. Since your method is more about relevance, rather than causal contribution, I believe it would be more appropriate if you called it reward shaping rather than credit assignment like that you established between texts.

  2. Although your method is more about distributing constructed reward over previous steps/tokens, there are still concepts or tricks that calls for more ablation studies or explanations. For example, as you mentioned in the paper, recency correction could be a problem for LLMs, but what exactly did it affect or how are you inspired to adopt it?

Questions for authors
  1. Why would you want to call it credit assignment? Is there enough motivation in your experiment to call it so?

  2. Are there more ablation studies you did not mention in the main text or supplementary materials?

  3. Are the results single-run or calculated average results? What are the range and variance?

Reviewer_bGbMOpenReview ↗
Rating
2(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes AttnSR, a method for assigning intermediate rewards to LLM agents on long-horizon tasks, using only attention obtained from the policy model itself. It constructs a dependency graph over actions and observations using policy attention, then traces backward from the final step to estimate the relevance of earlier steps. These estimates are combined with confidence gating and outcome centering to produce intermediate rewards. Experiments show a substantial improvement on ALFWorld, but smaller or mixed results on ScienceWorld and WebShop. With its effectiveness shown only on a single benchmark and limited validation of the attention-based credit allocation, I encourage the authors to substantially strengthen the next version of this manuscript by incorporating the convincing evidence.

Critical strengths
  1. The method offers a practical way to assign intermediate credit using the already available policy model, requiring neither a trained reward model nor additional rollouts. The performance improvement on ALFWorld is also promising.
Critical weaknesses
  1. The contribution and rationale of attention-based reward allocation are not sufficiently discussed. While the authors treat attention as a heuristic signal rather than evidence of causal contribution, the experiments do not establish whether it provides better credit allocation than other alternatives. The positional control (in Table 2) changes both allocation and confidence gating, making the contribution of attention difficult to isolate. The authors assess its effectiveness only via the final performance, without directly validating whether attention-derived allocations identify useful steps or explaining why they improve credit assignment.

  2. The provided gains are limited to ALFWorld. ScienceWorld shows only marginal gains, while WebShop’s score decreases despite increased interactions. Given that performance gains are confined to a single benchmark, stronger evidence is needed to show AttnSR’s effectiveness across diverse agentic tasks.

Questions for authors
  1. Does the improvement persist when attention-based allocation is replaced with uniform or random allocation while preserving confidence gating and each trajectory’s total shaping budget? This comparison would help distinguish the benefit of attention-based allocation from that of the surrounding reward-shaping scheme.

  2. Can the authors provide examples showing whether highly rewarded steps correspond to actions necessary for task completion? This would help clarify what attention-based allocation captures beyond the final performance gains.

Reviewer_5shuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to do credit assignment in long-horizon agentic tasks. The authors propose a method leveraging attention score to estimate the importance of each turn and using it in credit assignment. Empirically, it has improvement in ALFWorld, Webshop, and ScienceWorld over baseline GRPO and PPO.

Critical strengths
  • The paper is easy to follow. The method section is clear and well structured.
  • The proposed method doesn't need extra rollout/model training compared with PPO, and it has great performance improvement in ALFWorld.
Critical weaknesses
  • The improvements reported in Table 1 are largely concentrated on ALFWorld, while the gains on ScienceWorld and WebShop are limited. Moreover, several recent studies have already reported comparable or even higher success rates on ALFWorld (e.g. https://arxiv.org/abs/2602.16165).
  • The reason or intuition behind the design is not clear. The author simply introduce the method without telling any reason for this design. At the same time, authors didn't show too much ablation result to convince the necessity of this design.
Questions for authors
  1. The WebShop performance reported table 1 looks weird. Compared with recent papers (e.g. https://arxiv.org/abs/2505.10978, https://arxiv.org/abs/2602.16165), the baseline and the proposed method has significant higher point (~90), could authors explain what is the difference in evaluation protocal? Or what is the reason for that?
  2. The paper need more ablation result. The proposed method has a lot of component, attention allocation, onfidence gating and the design of equation (3), could the author argue the necessity of this design by providing ablation result?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.