acceptodds
Under review as a conference paper at ICLR 2027

Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy

Abstract

Agentic reinforcement learning trains large language models using multi-turn trajectories that interleave long reasoning traces with short environment-facing actions. Common policy-gradient methods, such as PPO and GRPO, treat each token in a trajectory equally, leading to uniform credit assignment. In this paper, we critically demonstrate that such uniform credit assignment largely misallocates token-level training signals. From an energy-based modeling perspective, we show that token-level training signals, quantified by their correlations with reward variance of different rollouts sampled from a given prompt, concentrate sharply on action tokens rather than reasoning tokens, even though action tokens account for only a small fraction of the trajectory. We refer to this phenomenon as the Action Bottleneck. Motivated by this observation, we propose an embarrassingly simple token reweighting approach, ActFocus, that downweights gradients on reasoning tokens, along with an additional energy-based redistribution mechanism that further increases the weights on action tokens with higher uncertainty. Across four environments and different model sizes, ActFocus improves all reported PPO final-step results and four of five reported GRPO final-step results, yielding final-step gains of up to 21.5 and 63.7 percentage points, respectively, with negligible additional runtime or memory cost.

Primary area:
reinforcement learning
Keywords:
LLMs, Agentic Reinforcement Learning, Energy, Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W5xbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ActFocus, a token-weighting method for multi-turn agent reinforcement learning. Motivated by the imbalance between long reasoning traces and short action spans, it downweights reasoning tokens and uses frozen-reference energy to prioritize action tokens within PPO and GRPO. Experiments with Qwen2.5-Instruct models across Sokoban, FrozenLake, Sudoku, and WebShop show improvements over uniform weighting in all reported PPO settings and most reported GRPO settings. The paper also analyzes training dynamics and provides ablations on reasoning weights, energy modulation strength, and alternative weighting signals.

Critical strengths
  1. ActFocus introduces a simple token-weighting mechanism that integrates with both PPO and GRPO without changing their advantage estimators. This makes the contribution practically useful and straightforward to incorporate into existing agent-training pipelines.

  2. The PPO results show consistent improvements across four environments and two model scales per environment, with means and standard deviations reported over five random seeds. This breadth and repeated evaluation provide substantive evidence that the benefits extend beyond a single task or configuration.

Critical weaknesses
  1. The main experiments only compared ActFocus with PPO and GRPO, which use uniform token weights. These results show that ActFocus improves the selected baseline, but are insufficient to judge its performance relative to existing credit assignment methods. It is recommended to include representative methods such as GiGPO [1] for comparison under the same model and training budget.

  2. Increasing β\beta not only increases the weight of high-energy action tokens, but also increases the total weight of the action part relative to the reasoning segment. Therefore, it is unclear whether the performance improvement comes from selective weighting of important action tokens or simply from increasing the overall weight on action tokens. It is recommended to add a control: assigning equal weights to all action tokens while keeping the total weight of the action part consistent with ActFocus, and to extend the comparison to settings beyond WebShop 3B.

  3. ActFocus improved the performance of PPO on FrozenLake, but did not improve the performance of GRPO in the same environment. It is recommended to discuss the possible reasons for this difference, and whether the weighting parameters need to be adjusted in GRPO or environments with randomness.

  4. Each point in Figure 2 summarizes multiple rollouts under the same prompt. This correlation suggests that prompts with higher average action energy tend to correspond to greater reward variance across rollouts, but this does not establish that tokens with higher energy within the same trajectory contribute more to the final result. It is recommended to clearly distinguish between these two levels of conclusions, using this result as a design motivation for the weighted scheme, rather than as direct evidence of token-level importance.

[1] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS, 2025.

Questions for authors

Please see weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer W5xb (1)

We thank the reviewer for recognizing the simplicity and applicability of ACTFOCUS to both PPO and GRPO and for the positive assessment of our evaluation. We agree that the sources of improvement and the FrozenLake exception deserve additional scrutiny. We address the overlapping concerns below.

W1: Comparison with other credit-assignment methods

W1: “The main experiments only compared ActFocus with PPO and GRPO, which use uniform token weights. ... It is recommended to include representative methods such as GiGPO.”

Thank you for raising the question and taking the time to review our paper. We would like to kindly point out that Appendix D.2, Table 6 of the paper already reports a five-seed GiGPO comparison using its official implementation, with the same model, environment, horizon, training budget, and evaluation setting. The final-step results for the larger scales are:

EnvironmentGiGPO final (%)ACTFOCUS final (%)
Sokoban 3B15.5 ± 4.423.2 ± 5.9
FrozenLake 3B19.8 ± 7.46.8 ± 3.1
Sudoku 1.5B68.3 ± 10.479.5 ± 2.7
WebShop 3B, strict success43.2 ± 2.951.6 ± 6.1

These results show improvements in three of four displayed environments. GiGPO modifies step-level credit via its advantage construction, whereas ACTFOCUS redistributes a given advantage across response tokens; the approaches address different granularity and can in principle be combined. We will move the GiGPO evidence into a more prominent position in the main paper. In addition, we implemented the scaling-only adaptation of GTR and the global high-entropy top-20% rule in two PPO environments (see the new table in our response to WP3V W1/Q1).

W2: “Increasing β not only increases the weight of high-energy action tokens, but also increases the total weight of the action part relative to the reasoning segment. ... add a control assigning equal weights to all action tokens while keeping the total action weight consistent, and extend the comparison beyond WebShop 3B.”

We thank the reviewer for raising this question. We would first like to clarify the distinct roles of α\alpha and β\beta in ACTFOCUS. As defined in Sec. 4, α\alpha controls the retained training weight of reasoning tokens, while β\beta controls the strength of energy-guided action weighting. Importantly, the effect of β\beta on both aggregate action emphasis and within-action allocation is already explicitly characterized in the submitted Appendix B.1 (Propositions 2–3).

Specifically, Proposition 2 analyzes how β\beta changes the total action-weight coefficient, while Proposition 3 decomposes its effect into aggregate action weighting and energy-dependent within-action differentiation. Eq. (35) further provides a principled matched constant-action-weight control to isolate the latter effect. Following this analysis, we assign each action token the constant weight 1+βuˉ1+\beta\bar{u}, where uˉ\bar{u} denotes the mean energy-derived sigmoid score across action tokens. This preserves exactly the same total action weight, reasoning weight, and loss normalizer for a fixed batch, while eliminating token-specific energy variation.

We further implement this requested control with five independent runs on WebShop 3B PPO (100 steps) and Sokoban 3B PPO (200 steps), extending the comparison beyond WebShop:

ConfigurationMatched constant, final (%)ACTFOCUS, final (%)
WebShop 3B PPO strict18.1 ± 5.536.7 ± 3.8
Sokoban 3B PPO28.7 ± 6.537.5 ± 1.9

Meanwhile, we would like to point out that the submitted paper already includes a direct weight-position intervention (Appendix D.5, Table 10). By randomly permuting the same energy-derived weights across action tokens, we preserve their distribution and total weight while breaking the energy-token correspondence. On WebShop 3B, peak strict success decreases from 25.4% (energy-paired) to 3.1% (randomly paired).

Together, the existing theoretical decomposition, submitted random-assignment experiment, and additional matched-constant control provide evidence for the importance of energy-guided within-action prioritization beyond merely increasing the aggregate action weight. We will make this distinction and the corresponding experimental evidence clearer in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer W5xb (2)

W3: “ActFocus improved the performance of PPO on FrozenLake, but did not improve ... GRPO. ... discuss possible reasons ... and whether weighting parameters need to be adjusted.”

We thank the reviewer for this question. We believe this difference is closely related to the stochastic nature of FrozenLake and its interaction with the underlying RL optimizer. Unlike deterministic environments such as Sokoban, FrozenLake introduces stochastic slipping, where the same action can lead to different next states and rewards. As also discussed in RAGEN [3], this can create a mismatch between action quality and observed outcomes.

This stochasticity presents a particular challenge for fine-grained credit assignment. In GRPO, group-relative advantages are estimated from observed trajectory outcomes, which may be affected by environmental randomness rather than solely by action quality. Consequently, energy-guided weighting may amplify noisy advantage signals without necessarily improving policy learning.

Regarding parameter selection, the optimal balance between reasoning retention (α\alpha) and energy-guided action weighting (β\beta) may depend on environment stochasticity and the RL optimizer. More conservative weighting or improved advantage estimation could potentially help in highly stochastic settings. Importantly, as emphasized in our submitted paper, we deliberately use the same hyperparameters (α=0.1\alpha=0.1, β=0.5\beta=0.5) across all environments and optimization algorithms, without task-specific tuning, to demonstrate the robustness and general applicability of ACTFOCUS. Despite this unified setting, ACTFOCUS consistently improves PPO performance across all four environments and multiple model scales. We will clarify the particular challenges of stochastic environments in the revised paper.

W4: “Each point in Figure 2 summarizes multiple rollouts under the same prompt. ... this does not establish that tokens with higher energy within the same trajectory contribute more to the final result.”

We thank the reviewer for the question. We would like to clarify that Figures 1 and 2 are intended to motivate the Action Bottleneck, rather than establish the causal contribution of individual tokens within a trajectory. Figure 1 illustrates the structural imbalance between reasoning and action tokens and the corresponding allocation of training weights. Figure 2 provides the detailed group-level analysis behind Figure 1, showing that action-token energy is substantially more correlated with reward variability than reasoning-token energy.

Importantly, the distinction raised by the reviewer is already explicitly acknowledged in the submitted paper. Appendix B.1 (following Eq. 37) states that the group-level energy–reward correlation does not establish token-gradient alignment. Appendix D.5 (Table 10) further evaluates energy-guided weighting through a direct intervention: randomly permuting the same action-token weights reduces peak WebShop strict success from 25.4% to 3.1%, while preserving the weight distribution and total action weight. We will make this distinction clearer in the revised paper. Thank you for bringing the question to our attention.


References

[1] Tong Wei et al., GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Training. ICCV 2025. https://openaccess.thecvf.com/content/ICCV2025/html/Wei_GTR_Guided_Thought_Reinforcement_Prevents_Thought_Collapse_in_RL-based_VLM_ICCV_2025_paper.html

[2] Shenzhi Wang et al., Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/a797c2d2e0c1fdabf4d1ab8cd0b465c6-Abstract-Conference.html

[3] Zihan Wang et al., RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073. https://arxiv.org/abs/2504.20073

[4] Lang Feng et al., Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025. https://arxiv.org/abs/2505.10978

[5] Weitang Liu et al., Energy-based Out-of-distribution Detection. NeurIPS 2020. https://proceedings.neurips.cc/paper_files/paper/2020/hash/f5496252609c43eb8a3d147ab9b9c006-Abstract.html

Thank you for the additional experiments and explanations. I apologize for overlooking the GiGPO comparison and weight-permutation control already included in the submission. The new matched-action-weight experiments substantially address my concern about the source of the gains. I also appreciate the discussion of the FrozenLake regression under GRPO. However, this regression leaves me less convinced of the method’s reliability across optimizers, and the suggested mitigation through more conservative weighting remains untested. Although the response has alleviated several concerns, my overall assessment remains a weak reject, and I will retain my original rating.

Reviewer_PXmoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ACTFOCUS, which improves PPO and GRPO by reducing the weight of reasoning tokens and adjusting the weight of action tokens using frozen-reference energy. Experiments across four environments demonstrate that this method enhances final-step performance in most settings.

Critical strengths
  1. The method modifies only token weighting, making it easy to integrate with existing PPO and GRPO implementations.
  2. The main experiments report five-seed results; under a unified configuration, all PPO settings show improvements.
  3. The appendix provides gradient analysis, a comparison with GiGPO, and a baseline using random weights.
Critical weaknesses
  1. Energy changes with a uniform shift in logits, whereas the predictive distribution remains invariant; thus, it cannot be directly equated with uncertainty.
  2. The group-level correlation observed in Sokoban 3B is insufficient to demonstrate a lack of effective training signals for reasoning tokens.
  3. Since β simultaneously alters both the total action weight and the internal allocation, existing control experiments have not adequately disentangled their respective contributions to the primary reward.
  4. With only 32 evaluation prompts per symbolic environment, the evidence for generalization is limited.
  5. The claim regarding negligible overhead lacks supporting measurements for runtime and memory usage.
Questions for authors

NaN

5 comments
AuthorsOpenReview ↗

Response to Reviewer PXmo (1)

We thank the reviewer for taking the time to review our paper. Below we clarify the scope of the energy interpretation and directly address the experimental concerns.

W1: Interpretation of free energy

W1: “Energy changes with a uniform shift in logits, whereas the predictive distribution remains invariant; thus, it cannot be directly equated with uncertainty.”

We thank the reviewer for raising this point. We agree that Energy shouldn't be directly equated with uncertainty. We respectfully clarify that shift sensitivity is a known property of free energy, rather than a flaw in our formulation. In fact, retaining information from unnormalized logits that softmax-based measures discard is precisely one motivation for energy-based scoring, as established in [1]. Importantly, ACTFOCUS computes energy from a frozen reference model, rather than the evolving policy, providing a consistent scoring function throughout RL training. The energy scores are further batch-normalized before being used for action-token weighting.

We do not claim that energy is a calibrated or shift-invariant measure of predictive uncertainty. Instead, we investigate its effectiveness as a reference-derived credit assignment signal. We already provide a controlled comparison in Section 5.6, where frozen-reference energy achieves 36.7% strict success versus 30.9% for frozen-reference entropy under the identical weighting architecture. Thus, the reviewer's observation distinguishes energy from probability-based uncertainty measures, but does not invalidate its use as an empirically effective token-weighting signal.

[1] Energy-based Out-of-distribution Detection, NeurIPS 2020, arXiv:2010.03759. https://arxiv.org/abs/2010.03759

W2: “The group-level correlation observed in Sokoban 3B is insufficient to demonstrate a lack of effective training signals for reasoning tokens.”

We thank the reviewer for raising the point. We respectfully clarify that the reviewer’s concern does not reflect the intended claim of our analysis. ACTFOCUS does not claim that reasoning tokens lack useful training signals. Reasoning tokens shape action selection, while action tokens directly determine environment transitions. Our goal is to address the structural imbalance caused by long reasoning spans dominating uniform token-level optimization, while explicitly retaining positive reasoning credit. We kindly refer the reviewer to Sec. 5.4 (Fig. 6a) of the submitted paper, where we already investigate the contribution of reasoning tokens through a controlled sweep of the reasoning-retention coefficient α\alpha. In particular, completely removing reasoning credit leads to training collapse on WebShop, demonstrating that reasoning tokens remain important for effective learning.

Figure 1 and Figure 2 serve as the empirical motivation for the Action Bottleneck, showing that action-token energy exhibits a stronger association with reward variability than reasoning-token energy. To further support this observation, we extend the diagnostic to WebShop 3B, where action energy achieves a Spearman correlation of +0.28, compared with −0.02 for reasoning and −0.08 for the full response.

Importantly, the effectiveness of token-level prioritization is independently validated through controlled interventions, including the submitted random-assignment experiment (Appendix D.5, Table 10) and our additional matched constant-action-weight experiments (see W3). We will clarify the distinction between the structural diagnostic and token-level weighting in the revised paper.

W3: “Since β simultaneously alters both the total action weight and the internal allocation, existing control experiments have not adequately disentangled their respective contributions.”

We thank the reviewer for raising this point. This coupling is already explicitly analyzed in the submitted Appendix B.1 (Propositions 2–3 and Eq. 35). Our theoretical analysis further provides a precise way to disentangle these two effects through a matched constant-action-weight control. Specifically, we fix α=0.1\alpha=0.1 and β=0.5\beta=0.5, and replace each energy-dependent action-token weight with the batch-mean action weight. This preserves exactly the same total reasoning weight, total action weight, and loss normalization for any fixed batch, while removing energy-dependent differentiation among action tokens.

We evaluate the setting using five random seeds on WebShop 3B PPO (100 steps) and Sokoban 3B PPO (200 steps):

ConfigurationMatched constant, final (%)ACTFOCUS, final (%)
WebShop 3B PPO strict18.1 ± 5.536.7 ± 3.8
Sokoban 3B PPO28.7 ± 6.537.5 ± 1.9

Furthermore, the submitted paper already includes a direct weight-position intervention (Appendix D.5, Table 10). By randomly permuting the same energy-derived weights across action-token positions, we preserve their distribution and total weight while breaking the energy-token correspondence. On WebShop 3B, peak strict success decreases from 25.4% (energy-paired) to 3.1% (randomly paired). Together, the theoretical decomposition and controlled weight-assignment experiments support the importance of energy-guided within-action prioritization, beyond merely increasing the aggregate action weight.

W4: Evaluation size and generalization

W4: “With only 32 evaluation prompts per symbolic environment, the evidence for generalization is limited.”

We thank the reviewer for raising this concern. We agree that more evaluation prompts can better test the generalization ability of the model. However, we would like to clarify that our evaluation uses 32 distinct prompts, each evaluated with 16 stochastic rollouts, totaling 512 evaluation trajectories per environment, rather than only 32 evaluations. WebShop additionally uses 256 distinct evaluation prompts, and all main results are reported as mean ± standard deviation over five independent training seeds.

Our evaluation setting is consistent with established practice in agentic RL. The official RAGEN configuration [1] adopts exactly the same 32 × 16 = 512 rollouts. Other closely related works also use fixed-size evaluation sets: SDAR [2] and MCGPO [3] evaluate on 128 validation tasks, while recent credit-assignment methods such as CRBC [4] and EMPO [5] use 128 validation trajectories or episodes per evaluation.

Furthermore, our experiments cover four environments, multiple model scales, and both PPO and GRPO, with five-seed statistics for the main comparisons. Together, these results provide substantial empirical evidence for the effectiveness and reproducibility of ACTFOCUS across the studied settings.

References:

[1] RAGEN official evaluation configuration: https://github.com/mll-lab-nu/RAGEN/blob/main/config/eval.yaml

[2] Self-Distilled Agentic Reinforcement Learning, arXiv:2605.15155

[3] Matched-Context Group Policy Optimization for LLM Agent Training, arXiv:2610.13755

[4] Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning, arXiv:2609.35082

[5] One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents, arXiv:2610.29427

W5: “The claim regarding negligible overhead lacks supporting measurements for runtime and memory usage.”

We thank the reviewer for raising this concern. We agree that we lack support for the claim. Theoretically, ACTFOCUS introduces no additional trainable parameters, model components, or backward passes. Token energy is obtained from the frozen reference model already used for KL regularization, requiring only a log-sum-exp operation and lightweight token-wise weighting. Thus, ACTFOCUS preserves the computational complexity of the underlying PPO/GRPO framework. To support the claim, we instrumented WebShop 3B PPO for 30 steps on four A100 GPUs and report non-evaluation steps 2–30:

MetricUniformACTFOCUS
End-to-end step time171.5 ± 11.2 s202.3 ± 14.7 s
Rollout generation134.6 s158.3 s
Reference forward2.90 s6.70 s
Peak actor memory, allocated29.3 GB30.0 GB

The additional energy-related computation and token-weight construction require only approximately 3.8 seconds per step, with merely 0.7 GB additional actor memory. These measurements confirm that ACTFOCUS introduces negligible additional method-specific runtime and memory overhead, without modifying the underlying PPO/GRPO training framework. Thank you for pointing out this to make our paper better.

Dear Reviewer PXmo,

Thank you for your review.

We have added all the experiments and discussions that you have asked for in your rebuttal. The discussion phase will end soon and we are hoping to hear your thoughts.

Regards,

Authors

Reviewer_WP3VOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the imbalance between long reasoning traces and short action spans in multi-turn agentic RL. It introduces ACTFOCUS to downweight reasoning tokens and prioritize action tokens using frozen-reference energy. The method retains the original PPO and GRPO advantage estimators. Experiments across four environments show gains and improved training stability in several GRPO settings.

Critical strengths
  1. The motivation is reasonable. Long reasoning traces and short action spans play different roles in agent trajectories. The paper investigates this imbalance and tests its weighting approach across four environments.
  2. The empirical results show practical benefits. Five-seed PPO experiments show gains across the tested configurations. Several GRPO settings also show reduced late-stage degradation.
Critical weaknesses
  1. The baseline comparisons are insufficient. GTR [1] already balances thought and action contributions in PPO. Beyond the 80/20 Rule [2] selectively updates high-entropy tokens. These methods differ from ACTFOCUS, but provide relevant comparison points. The paper includes GiGPO and several ablations. However, matched comparisons with the closest weighting methods are still missing. This limits the evidence for its incremental contribution.
  2. The energy mechanism and its generality need stronger evidence. Raw-logit energy is not inherently a measure of uncertainty. The reported group-level correlation does not establish token-level importance. The β\beta ablation also changes total action weight, so it cannot isolate energy-based prioritization. Most training experiments use small models and compact tasks. Table 7 shows reasoning-token fractions above 90%, while the 7B training evaluation covers only WebShop. It remains unclear whether the gains extend to less extreme token imbalance and more complex tasks.

References:

[1] GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Training. ICCV 2025.

[2] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. NeurIPS 2025.

Questions for authors
  1. How does ACTFOCUS compare with adapted GTR [1] and high-entropy token selection [2]? Please provide matched-budget comparisons to clarify its incremental benefit.

  2. Figure 2 shows a group-level correlation on Sokoban 3B. How does this justify weighting individual action tokens by energy? Please provide token-level evidence and repeat the diagnostic on other environments.

  3. What insight do the propositions in Appendix B.1 provide beyond the direct effects of reweighting? Please explain how they inform the method’s design. A multi-seed comparison with the matched constant-action-weight control would help isolate the benefit of energy prioritization.

  4. What explains the lower peak and final performance on FrozenLake 3B under GRPO? Please separate reasoning downweighting from energy modulation in an ablation to identify the source of degradation.

  5. Beyond the acknowledged limitations, do you have results on larger models or more complex tasks beyond the 7B WebShop experiment? A controlled comparison across reasoning/action token proportions would also clarify whether the gains depend on extreme token imbalance.

9 comments
AuthorsOpenReview ↗

Response to Reviewer WP3V (1)

We thank the reviewer for recognizing the motivation, the empirical improvements in our experiments, and the training-stability findings. The reviewer raises related concerns about incremental novelty, what the energy diagnostic actually establishes, and generalization. We group the overlapping weaknesses (W) and questions (Q) into five responses.

W1: “The baseline comparisons are insufficient. GTR already balances thought and action contributions in PPO. Beyond the 80/20 Rule selectively updates high-entropy tokens. ... matched comparisons with the closest weighting methods are still missing.”

Q1: “How does ACTFOCUS compare with adapted GTR and high-entropy token selection? Please provide matched-budget comparisons to clarify its incremental benefit.”

We thank the reviewer for suggesting these comparisons. We implemented the transferable part of each method in our PPO pipeline, with the same tasks, step budgets, and five seeds:

MethodWebShop 3B final (%)Sokoban 3B final (%)
Adapted GTR20.6 ± 6.927.7 ± 5.2
Adapted 80/206.4 ± 4.122.6 ± 3.0
Uniform PPO22.7 ± 0.226.2 ± 2.8
ACTFOCUS36.7 ± 3.837.5 ± 1.9

However, GTR addresses a substantially different research problem from ACTFOCUS. While ACTFOCUS and Beyond the 80/20 Rule share a high-level observation that useful policy-gradient updates may concentrate on a small subset of tokens, they address fundamentally different credit-assignment problems.

On GTR. We respectfully clarify that the full GTR framework is not a directly comparable baseline. GTR addresses thought collapse in VLM agent training, built on RL4VLM-style PPO with an external VLM corrector, supervised thought cloning, and DAgger. In contrast, ACTFOCUS addresses the Action Bottleneck in multi-turn language-agent RL, using veRL without additional process supervision. Moreover, ACTFOCUS is a general token-level weighting mechanism applicable to both PPO and GRPO, whereas GTR is developed and evaluated with PPO. Although both involve reasoning and action scaling, their objectives and supervision requirements are fundamentally different. Nevertheless, we implement GTR's transferable reasoning/action scaling component in our setting to provide a relevant comparison.

On Beyond the 80/20 Rule. Both methods recognize that useful gradients may concentrate on a minority of tokens, but address different credit-assignment problems. Beyond 80/20 selects high-entropy tokens across single-turn reasoning traces, whereas ACTFOCUS explicitly reallocates credit toward environment-facing actions in multi-turn RL and uses frozen-reference energy to prioritize tokens within action spans. Our submission already compares energy against entropy (Sec. 5.6, Fig. 6c): energy achieves 36.7% vs. 30.9% strict success on WebShop 3B under identical weighting architectures.

To further understand the limitations of global entropy-based token selection in multi-turn agent RL, we analyze which tokens are selected by the adapted 80/20 method. On WebShop, only 3–5% of the selected high-entropy tokens are action tokens, meaning that 95–97% of the selected tokens are predominantly reasoning tokens. Moreover, only 17–18% of all action tokens are selected, leaving over 80% of action tokens outside the high-entropy subset. A similar pattern emerges on Sokoban, where action tokens account for only 4–19% of selected tokens (median 7%), and merely 15–33% of all action tokens are selected.

These findings highlight a key distinction: selecting high-entropy tokens globally does not necessarily prioritize environment-facing actions. Instead, the selected tokens remain dominated by lengthy reasoning spans, while most action tokens are overlooked. ACTFOCUS directly addresses this Action Bottleneck by explicitly emphasizing action tokens and using reference energy to differentiate their training weights, while retaining positive reasoning credit. We thank the reviewers for giving us the opportunity to dive deep into potential related works.

AuthorsOpenReview ↗

Response to Reviewer WP3V (2)

W2: “Raw-logit energy is not inherently a measure of uncertainty. The reported group-level correlation does not establish token-level importance.”

Q2: “Figure 2 shows a group-level correlation on Sokoban 3B. How does this justify weighting individual action tokens by energy? Please provide token-level evidence and repeat the diagnostic on other environments.”

We thank the reviewer for raising these questions regarding the motivation and design of ACTFOCUS. We agree on both points and have narrowed our claims. Regarding the energy signal, we respectfully clarify that frozen-reference free energy is an unnormalized logit-derived score rather than a calibrated probability-based uncertainty measure. Its sensitivity to common logit shifts is a known property of energy-based scoring, which can retain logit-scale information removed by softmax normalization [1]. This property is already explicitly acknowledged in the submitted Appendix B.1 (following Eq. 38). Moreover, Sec. 5.6 (Fig. 6c) already empirically compares energy against alternative weighting signals under the same framework, where energy achieves 36.7% strict success versus 30.9% for frozen-reference entropy on WebShop 3B.

Second, we would like to clarify that our claim is that token energy provides a useful signal for prioritizing action-token updates, rather than a ground-truth measure of individual-token causal contributions to the final reward. Figure 2 provides empirical motivation through group-level energy–reward correlations, while the effectiveness of energy-guided token weighting is independently evaluated through controlled interventions. This distinction is already explicitly acknowledged in the submitted paper. Appendix B.1 (following Eq. 37) clarifies that group-level correlations alone do not establish token-gradient alignment.

Furthermore, the submitted Appendix D.5 (Table 10) already provides direct token-level weighting intervention evidence. By randomly permuting energy-derived weights across action-token positions, we preserve their distribution and total weight while breaking the energy-token correspondence. On WebShop 3B, peak strict success decreases from 25.4% (energy-aligned) to 3.1% (random assignment). This directly evaluates whether assigning energy-derived weights to their corresponding action tokens matters for training, beyond aggregate action reweighting.

To further examine the generality of our empirical motivation, we extend the group-level diagnostic beyond Sokoban using Uniform GRPO rollouts ( Same settings as the submitted version). Representative Spearman correlations are:

EnvironmentAction EnergyReasoning EnergyFull-Response Energy
FrozenLake 3B+0.30−0.15−0.14
WebShop 3B+0.28−0.02−0.08

These representative results show a stronger positive association for action energy than for reasoning or full-response energy, extending the original observation beyond Sokoban.

References:

[1] Energy-based Out-of-distribution Detection, NeurIPS 2020, arXiv:2010.03759. https://arxiv.org/abs/2010.03759

AuthorsOpenReview ↗

Response to Reviewer WP3V (3)

W2: “The β ablation also changes total action weight, so it cannot isolate energy-based prioritization.”

Q3: “What insight do the propositions in Appendix B.1 provide beyond the direct effects of reweighting? ... A multi-seed comparison with the matched constant-action-weight control would help isolate the benefit of energy prioritization.”

Thank the reviewer for the comment. We would like to point out that we already separate the two effects mathematically. The parameter α controls reasoning-token retention, while β differentiates action-token weights by energy and also increases their aggregate weight. Proposition 2 explicitly quantifies this coupling. More importantly, Proposition 3 and Eq. (35) identify the appropriate control: give each action token the batch-mean energy-derived action weight, preserving total reasoning weight, total action weight, and the loss normalizer. The only fixed-batch gradient difference is the within-action score–gradient covariance term:

∇θLα,β−∇θLconst=βNaZCa\nabla_\theta L_{\alpha,\beta}-\nabla_\theta L_{\mathrm{const}}=\frac{\beta N_a}{Z}C_a Following this derivation and reviewer's suggestion, we implement the matched-constant control on WebShop 3B PPO and Sokoban 3B PPO with 5 random seeds:

ConfigurationMatched constant: finalACTFOCUS: final
WebShop 3B PPO, strict18.1 ± 5.536.7 ± 3.8
Sokoban 3B PPO28.7 ± 6.537.5 ± 1.9

The matched-constant baseline achieves lower final success than the ACTFOCUS results on both environments.These results suggest that increasing the aggregate action weight alone is insufficient to reproduce the benefits of energy-guided weighting, and that differentiating weights among action tokens can play an important role.

It is worth mentioning that the submitted Appendix D.5 (Table 10) provides a direct weight-position intervention: permuting the same energy-derived action weights preserves their distribution and total weight while breaking energy–token correspondence, reducing WebShop 3B peak strict success from 25.4% to 3.1%. Together, these experiments support the importance of not only how much training weight is assigned to action tokens, but also where that weight is allocated according to token energy.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Recap

Dear Area Chairs and Reviewers,

We want to thank all the reviewers for their thorough reviews. We are pleased that the reviewers found the motivation reasonable (WP3V), the method simple and easy to integrate into both PPO and GRPO (PXmo, W5xb), and the five-seed PPO gains consistent across environments and model scales (WP3V, PXmo, W5xb), with reduced late-stage degradation under GRPO (WP3V).

The major questions center on comparison with the closest weighting methods (WP3V, W5xb), disentangling energy-based allocation from total action weight (WP3V, PXmo, W5xb), the scope of the energy interpretation and the diagnostic (WP3V, PXmo, W5xb), model scale and evaluation size (WP3V, PXmo), and computational overhead (PXmo). Our individual responses address these questions in detail; here, we summarize the new results.

  1. Additional baselines. On WebShop 3B / Sokoban 3B (PPO), ActFocus reaches 36.7 / 37.5%, vs. 20.6 / 27.7% for adapted GTR and 6.4 / 22.6% for top-20% entropy selection. Entropy selection mostly picks reasoning tokens and skips most action tokens (82–83% on WebShop). (WP3V Q1, W5xb W1)
  2. Where the action weight goes matters, not just how much. A matched control with exactly ActFocus's total reasoning and action weights, but no energy differentiation (Eq. 35), reaches 28.7% on Sokoban 3B and 18.1% on WebShop 3B, vs. 37.5% and 36.7% for ActFocus. (WP3V Q3, PXmo W3, W5xb W2)
  3. Diagnostic beyond Sokoban. Action-token energy again has the strongest association with group reward variance on FrozenLake (+0.30) and WebShop (+0.28), vs. −0.15 and −0.02 for reasoning-token energy. (WP3V Q2, PXmo W2, W5xb W4)
  4. 14B. With the default recipe and no 14B-specific tuning, ActFocus improves WebShop 14B strict success under PPO (52.0 vs. 23.0%) and GRPO (59.8 vs. 49.6%). Action tokens remain 9.4% of generated tokens on WebShop. (WP3V Q5)
  5. FrozenLake under GRPO. A component ablation shows the drop is not caused by energy alone; we trace it to stochastic transitions under critic-free advantages. (WP3V Q4, W5xb W3)
  6. Overhead. On WebShop 3B PPO, ActFocus's own computation costs at most 3.8 s per step and 0.7 GB of actor memory. (PXmo W5)

We also want to highlight that several concerns were already addressed in the submitted paper, and are reiterated with pointers in our individual responses, including: the GiGPO comparison under matched settings (Appendix D.2; W5xb); the energy-vs-entropy comparison (Sec. 5.6; WP3V, PXmo); the analytical decomposition of total vs. within-action weighting (Appendix B.1; WP3V, PXmo, W5xb); and the reasoning-retention ablation (Sec. 5.4; PXmo). We flag these here so they aren't read as gaps, and we will make them more visible in the revised version.

We sincerely thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.