- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ActFocus, a token-weighting method for multi-turn agent reinforcement learning. Motivated by the imbalance between long reasoning traces and short action spans, it downweights reasoning tokens and uses frozen-reference energy to prioritize action tokens within PPO and GRPO. Experiments with Qwen2.5-Instruct models across Sokoban, FrozenLake, Sudoku, and WebShop show improvements over uniform weighting in all reported PPO settings and most reported GRPO settings. The paper also analyzes training dynamics and provides ablations on reasoning weights, energy modulation strength, and alternative weighting signals.
Critical strengths
-
ActFocus introduces a simple token-weighting mechanism that integrates with both PPO and GRPO without changing their advantage estimators. This makes the contribution practically useful and straightforward to incorporate into existing agent-training pipelines.
-
The PPO results show consistent improvements across four environments and two model scales per environment, with means and standard deviations reported over five random seeds. This breadth and repeated evaluation provide substantive evidence that the benefits extend beyond a single task or configuration.
Critical weaknesses
-
The main experiments only compared ActFocus with PPO and GRPO, which use uniform token weights. These results show that ActFocus improves the selected baseline, but are insufficient to judge its performance relative to existing credit assignment methods. It is recommended to include representative methods such as GiGPO [1] for comparison under the same model and training budget.
-
Increasing not only increases the weight of high-energy action tokens, but also increases the total weight of the action part relative to the reasoning segment. Therefore, it is unclear whether the performance improvement comes from selective weighting of important action tokens or simply from increasing the overall weight on action tokens. It is recommended to add a control: assigning equal weights to all action tokens while keeping the total weight of the action part consistent with ActFocus, and to extend the comparison to settings beyond WebShop 3B.
-
ActFocus improved the performance of PPO on FrozenLake, but did not improve the performance of GRPO in the same environment. It is recommended to discuss the possible reasons for this difference, and whether the weighting parameters need to be adjusted in GRPO or environments with randomness.
-
Each point in Figure 2 summarizes multiple rollouts under the same prompt. This correlation suggests that prompts with higher average action energy tend to correspond to greater reward variance across rollouts, but this does not establish that tokens with higher energy within the same trajectory contribute more to the final result. It is recommended to clearly distinguish between these two levels of conclusions, using this result as a design motivation for the weighted scheme, rather than as direct evidence of token-level importance.
[1] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS, 2025.
Questions for authors
Please see weakness.
7 comments
Response to Reviewer W5xb (2)
W3: “ActFocus improved the performance of PPO on FrozenLake, but did not improve ... GRPO. ... discuss possible reasons ... and whether weighting parameters need to be adjusted.”
We thank the reviewer for this question. We believe this difference is closely related to the stochastic nature of FrozenLake and its interaction with the underlying RL optimizer. Unlike deterministic environments such as Sokoban, FrozenLake introduces stochastic slipping, where the same action can lead to different next states and rewards. As also discussed in RAGEN [3], this can create a mismatch between action quality and observed outcomes.
This stochasticity presents a particular challenge for fine-grained credit assignment. In GRPO, group-relative advantages are estimated from observed trajectory outcomes, which may be affected by environmental randomness rather than solely by action quality. Consequently, energy-guided weighting may amplify noisy advantage signals without necessarily improving policy learning.
Regarding parameter selection, the optimal balance between reasoning retention () and energy-guided action weighting () may depend on environment stochasticity and the RL optimizer. More conservative weighting or improved advantage estimation could potentially help in highly stochastic settings. Importantly, as emphasized in our submitted paper, we deliberately use the same hyperparameters (, ) across all environments and optimization algorithms, without task-specific tuning, to demonstrate the robustness and general applicability of ACTFOCUS. Despite this unified setting, ACTFOCUS consistently improves PPO performance across all four environments and multiple model scales. We will clarify the particular challenges of stochastic environments in the revised paper.
W4: “Each point in Figure 2 summarizes multiple rollouts under the same prompt. ... this does not establish that tokens with higher energy within the same trajectory contribute more to the final result.”
We thank the reviewer for the question. We would like to clarify that Figures 1 and 2 are intended to motivate the Action Bottleneck, rather than establish the causal contribution of individual tokens within a trajectory. Figure 1 illustrates the structural imbalance between reasoning and action tokens and the corresponding allocation of training weights. Figure 2 provides the detailed group-level analysis behind Figure 1, showing that action-token energy is substantially more correlated with reward variability than reasoning-token energy.
Importantly, the distinction raised by the reviewer is already explicitly acknowledged in the submitted paper. Appendix B.1 (following Eq. 37) states that the group-level energy–reward correlation does not establish token-gradient alignment. Appendix D.5 (Table 10) further evaluates energy-guided weighting through a direct intervention: randomly permuting the same action-token weights reduces peak WebShop strict success from 25.4% to 3.1%, while preserving the weight distribution and total action weight. We will make this distinction clearer in the revised paper. Thank you for bringing the question to our attention.
References
[1] Tong Wei et al., GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Training. ICCV 2025. https://openaccess.thecvf.com/content/ICCV2025/html/Wei_GTR_Guided_Thought_Reinforcement_Prevents_Thought_Collapse_in_RL-based_VLM_ICCV_2025_paper.html
[2] Shenzhi Wang et al., Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/a797c2d2e0c1fdabf4d1ab8cd0b465c6-Abstract-Conference.html
[3] Zihan Wang et al., RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073. https://arxiv.org/abs/2504.20073
[4] Lang Feng et al., Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025. https://arxiv.org/abs/2505.10978
[5] Weitang Liu et al., Energy-based Out-of-distribution Detection. NeurIPS 2020. https://proceedings.neurips.cc/paper_files/paper/2020/hash/f5496252609c43eb8a3d147ab9b9c006-Abstract.html
Thank you for the additional experiments and explanations. I apologize for overlooking the GiGPO comparison and weight-permutation control already included in the submission. The new matched-action-weight experiments substantially address my concern about the source of the gains. I also appreciate the discussion of the FrozenLake regression under GRPO. However, this regression leaves me less convinced of the method’s reliability across optimizers, and the suggested mitigation through more conservative weighting remains untested. Although the response has alleviated several concerns, my overall assessment remains a weak reject, and I will retain my original rating.
Response to Reviewer W5xb (1)
We thank the reviewer for recognizing the simplicity and applicability of ACTFOCUS to both PPO and GRPO and for the positive assessment of our evaluation. We agree that the sources of improvement and the FrozenLake exception deserve additional scrutiny. We address the overlapping concerns below.
W1: Comparison with other credit-assignment methods
Thank you for raising the question and taking the time to review our paper. We would like to kindly point out that Appendix D.2, Table 6 of the paper already reports a five-seed GiGPO comparison using its official implementation, with the same model, environment, horizon, training budget, and evaluation setting. The final-step results for the larger scales are:
These results show improvements in three of four displayed environments. GiGPO modifies step-level credit via its advantage construction, whereas ACTFOCUS redistributes a given advantage across response tokens; the approaches address different granularity and can in principle be combined. We will move the GiGPO evidence into a more prominent position in the main paper. In addition, we implemented the scaling-only adaptation of GTR and the global high-entropy top-20% rule in two PPO environments (see the new table in our response to WP3V W1/Q1).
We thank the reviewer for raising this question. We would first like to clarify the distinct roles of α and β in ACTFOCUS. As defined in Sec. 4, α controls the retained training weight of reasoning tokens, while β controls the strength of energy-guided action weighting. Importantly, the effect of β on both aggregate action emphasis and within-action allocation is already explicitly characterized in the submitted Appendix B.1 (Propositions 2–3).
Specifically, Proposition 2 analyzes how β changes the total action-weight coefficient, while Proposition 3 decomposes its effect into aggregate action weighting and energy-dependent within-action differentiation. Eq. (35) further provides a principled matched constant-action-weight control to isolate the latter effect. Following this analysis, we assign each action token the constant weight 1+βuˉ, where uˉ denotes the mean energy-derived sigmoid score across action tokens. This preserves exactly the same total action weight, reasoning weight, and loss normalizer for a fixed batch, while eliminating token-specific energy variation.
We further implement this requested control with five independent runs on WebShop 3B PPO (100 steps) and Sokoban 3B PPO (200 steps), extending the comparison beyond WebShop:
Meanwhile, we would like to point out that the submitted paper already includes a direct weight-position intervention (Appendix D.5, Table 10). By randomly permuting the same energy-derived weights across action tokens, we preserve their distribution and total weight while breaking the energy-token correspondence. On WebShop 3B, peak strict success decreases from 25.4% (energy-paired) to 3.1% (randomly paired).
Together, the existing theoretical decomposition, submitted random-assignment experiment, and additional matched-constant control provide evidence for the importance of energy-guided within-action prioritization beyond merely increasing the aggregate action weight. We will make this distinction and the corresponding experimental evidence clearer in the revised paper.