acceptodds
Withdrawn from ICLR 2027

DisTAR: Disentangled Token Aggregation for Reasoning and Action in LLM Agent Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement Learning (RL) has been pivotal in empowering autonomous agents to solve long-horizon, multi-turn tasks. However, most existing methods directly inherit standard token aggregation paradigms in single-turn reasoning, thereby overlooking the fundamental divergence between reasoning and action: reasoning requires thorough exploration within a single turn, whereas action demands concise and precise sequences to prevent error propagation. In this paper, we first identify and formalize this “aggregation dilemma" in agentic RL: applying token-mean to action tokens inadvertently assigns higher gradient weights to longer, redundant action chains, while enforcing sequence-mean on reasoning tokens hinders the model's Chain-of-Thought (CoT) exploration. To resolve this conflict, we propose DisTAR (Disentangled Token Aggregation for Reasoning and Action), a novel framework that decouples the token aggregation paradigm. Specifically, DisTAR enforces a token-mean constraint on reasoning tokens to encourage sufficient reasoning before action, while executing a mathematically equivalent sequence-mean reweighting mechanism on action tokens to optimize for concise action chains. Furthermore, at the trajectory level, an inverse-step penalty is integrated into the advantage estimation to proactively suppress redundant paths. Finally, we introduce a globally normalized surrogate loss to maintain stable training convergence under this heterogeneous token aggregation strategy. Extensive experiments on WebShop, AlfWorld, and SearchQA across multiple foundation models (e.g., Qwen-2.5, Qwen-3) demonstrate that DisTAR significantly boosts agent performance. Notably, on WebShop with Qwen-3-4B, DisTAR enhances the success rate of the baseline GRPO from 68.6% to 82.8%, while consistently inducing shorter, more precise action chains alongside richer CoT. Code is available at https://anonymous.4open.science/r/DisTAR-A387.

Primary area:
reinforcement learning
Keywords:
Agent, Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GFQNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In LLM RL training, there are two common ways to aggregate token contributions: token-mean and sequence-mean. The authors claim that token-mean prefers longer trajectories and vice versa. They propose to use token-mean on reasoning parts for better exploration, and sequence-mean on action parts for efficient actions. They also propose inverse-step advantage penalty and globally normalized surrogate loss to facilitate training.

Critical strengths
  1. Good motivation. The Figure 2 effectively demonstrate the difference between token-mean and sequence-mean.
Critical weaknesses
  1. The authors say that "Token-mean favors richer CoT exploration at the cost of behavioral redundancy" and "applies a token-mean constraint on reasoning tokens to foster sufficient exploration". Longer reasoning parts in experiments does not only mean sufficient exploration, but also repetition, unnecessary deliberation. A direct comparison between token-mean and sequence-mean that shows better exploration would be helpful.
  2. Section 4.1 claims that "This decoupled optimization introduces additional backward passes and memory overhead". The method only changes the coefficient, nothing about gradient. It does not require additional gradient computation. The contribution is overstated.
  3. Inverse-step advantage penalty is actually a heuristic trick, it does not deserve a section. Furthermore, for negative advantage trajectories, inverse-step advantage penalty actually makes smaller negative updates. This trick needs further justification.
Questions for authors
  1. Equation 9-14 use trajectory level advantage, section 5.1 says "GRPO with the step-level advantage", and Algorithm 1 uses a value function to compute baseline, instead of GRPO style. Could you please clarify which one you use?
  2. Similar to the issue that longer reasoning does not directly imply better exploration, how could you say that shorter action tokens could mean efficient actions? Would smaller number of function calls be a better representation of efficient action?
Reviewer_j6qBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies token aggregation in GRPO-style RL training for multi-turn LLM agents. The authors argue that seq-mean aggregation discourages long reasoning, while token-mean aggregation gives more weight to trajectories with longer action chains, which they call the ''aggregation dilemma''. To address it, DisTAR uses token-mean for reasoning tokens and sequence-mean (implemented by reweighting) for action tokens, divides the advantage by the number of action steps, and normalizes the loss by the total token weight. Experiments on ALFWorld, WebShop and search-augmented QA with Qwen2.5 and Qwen3 models show consistent improvements over GRPO, GiGPO and T2PO, together with longer reasoning and shorter action chains.

Critical strengths

The idea is simple and practical. The paper points out that reasoning tokens and action tokens may need different aggregation, which most agentic RL methods do not consider. The method only reweights the existing objective without introducing new hyperparameters, so it is easy to adopt. The improvements are also consistent across three types of benchmarks and four backbones.

Critical weaknesses
  • The analysis mainly considers successful trajectories. The aggregation dilemma is illustrated with successful trajectories (Fig. 1), and the discussion in Sec. 4 also mainly considers this case. It is less clear how the action reweighting and the inverse-step penalty affect failed trajectories. In my understanding, both of them reduce the penalty on actions in long failed trajectories. This seems to be in some tension with the goal of discouraging redundant actions, although the good results suggest it may not be a problem in practice. Since this is part of the core motivation, I think discussing this case would make the paper more convincing.

  • The roles of the components could be explained more clearly. For example, Sec. 4.3 motivates the globally normalized loss by the heterogeneous token weights. However, the action reweighting only redistributes weight among trajectories and does not change the total token weight in a group. So this normalization seems to mainly compensate for the scaling introduced by the inverse-step penalty. A clearer explanation of what each component does would help readers understand where the improvement comes from.

Questions for authors
  • Could the authors discuss how DisTAR treats failed trajectories? After the reweighting and the inverse-step penalty, the penalty on each action token seems to become smaller for a long failed trajectory. Is this intended, or is it compensated in practice, for example by the step-level advantages of GiGPO? Some simple statistics on failed trajectories (e.g., their length or the rate of repeated actions) would be helpful if available.

  • How are the ±\pm values in Table 1 computed? Some values, such as 98.8$\pm$4.2 for Pick with Qwen3-4B, look larger than what three runs with a mean close to 100% would allow.

Reviewer_cWAeOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose DisTAR (Disentangled Token Aggregation for Reasoning and Action), a novel framework that decouples the token aggregation paradig. The authors argue that applying a single aggregation rule to both reasoning and action tokens creates an "aggregation dilemma". DisTAR combines three components: sequence-mean aggregation and weighting for actions token, an inverse-step penalty of advantages, and global normalization of the policy surrogate by the total effective token weight. The method achieves superior performance on ALFWorld, WebShop, and search-augmented QA using Qwen2.5-1.5B/3B/7B and Qwen3-4B models.

Critical strengths
  1. Reasoning and action tokens have different functional roles, and it is a useful design to explicitly distinguishing their aggregation in agentic RL. DisTAR requires no new learned module for aggregation and is compatible with an existing step-level advantage estimator. The pilot experiment in Section 3.3 provides concrete behavioral motivation.

  2. The experimental results demonstrate notable improvements over GiGPO and T2PO, particularly on WebShop with Qwen2.5-1.5B. Appendix A.3 provides a comprehensive ablation study covering DTA-only, IAP-only, and pairwise configurations. These results provide empirical support for heterogeneous token weighting as a promising direction for agentic reinforcement learning.

Critical weaknesses
  1. The optimization objective is inconsistently specified. The final optimization objective requires clarification. Eq. 14 and Algorithm 1 appear to reverse the standard PPO loss sign if the expression is minimized; the authors should specify the actual update convention. Moreover, separately defined reasoning and action losses can share a single forward pass and be summed before one backward pass. Thus, the claimed computational advantage should be validated against this straightforward implementation.

  2. The gains are not clearly attributable to more efficient reasoning. (1) The inverse-step term is not a general action-cost penalty: it weakens negative updates for long failed trajectories. (2) Figure 3 does not show that shorter trajectories cause higher accuracy, since length may reflect task difficulty or failure-induced retries. Although Appendix A.5 notes that some tasks require long trajectories, this limitation is not tested experimentally.

Questions for authors
  1. Is Eq. 14 maximized or minimized in the released implementation? Please provide the exact implemented PPO loss. Algorithm 1 clarifies that inverse-step scaling is applied once, but does not resolve the sign or advantage-estimation inconsistencies.

  2. Can you compare against directly summing the reasoning token-mean loss and action sequence-mean loss using shared log-probabilities and one backward pass?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.