- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to train LLM coding agents that resolve SWE tasks with fewer tokens, measuring the trade-off with TRS. Motivated by two observations (successful trajectories for the same task vary widely in token usage, and low-entropy turns are associated with unproductive behaviors), it proposes HERO, a GRPO-based method. Hierarchical policy optimization adds an efficiency advantage only once the group's resolution rate exceeds a threshold, and clips advantages so that resolved rollouts are never negative and unresolved ones never positive. The efficiency advantage combines a trajectory-level penalty on token usage above the median of same-outcome rollouts with a turn-level penalty on the group's 20% lowest-entropy turns. Trained on 640 tasks and evaluated with Qwen3.5-4B/9B/35B-A3B on SWE-bench Verified and Multilingual (Claude Code scaffold), HERO raises the average resolution rate from 37.4% to 42.2% (GRPO: 39.9%), using fewer tokens than GRPO and roughly the same as the base models.
Critical strengths
-
The paper addresses an efficiency problem, the token cost of coding agents, and identifies two failure modes: standard GRPO substantially inflates token usage, while a naive efficiency penalty without hierarchical policy optimization pushes resolution below the base model. HERO avoids both, and the ablations indicate that each component contributes. This matters because it offers a working recipe in a setting where naive length penalties fail.
-
The gains over GRPO are consistent: HERO achieves higher resolution with fewer tokens across all model sizes and both benchmarks. On the 4B model it also gives a better trade-off than existing token-efficient RL and inference-time methods, while preserving prefix-cache reuse and transferring to another scaffold.
Critical weaknesses
-
The efficiency claims go beyond the evidence. The reported token savings are relative to GRPO, which itself inflates token usage. Compared with the base models, HERO's token usage is only slightly lower on average, and for the 4B model it is unchanged on swebench verified and higher on swebench multilingual. The claimed superiority over state-of-the-art agents therefore holds only for the 35B-A3B model, whose base model already leads. This matters because the evidence supports a narrower claim, larger resolve rate gains than GRPO at roughly base-level cost, rather than "doing more with fewer tokens" as the title and abstract suggest.
-
Novelty is moderate, and the new parts are not strongly supported. HERO's core principles, that resolved trajectories always score above unresolved ones and that token usage is judged relative to other correct trajectories for the same task, already appear in Training Efficient (Arora & Zanette), which the paper itself uses as a baseline. HERO's main additions are turning on the efficiency term only when the group's resolution rate is high, also comparing token usage among unresolved trajectories, and penalizing low-entropy turns. However, these additions are tested only on the 4B model, removing each one changes resolution by only a few points, and no variance across seeds is reported.
Questions for authors
-
What are the standard deviations over the three seeds in Tables 1–3, and were , , the entropy percentile, and the 35B-specific values chosen on SWE-bench Verified or on held-out data?
-
Please also report the learning rate, number of RL steps, KL and clipping settings, and training compute.
-
A repeated tool call mostly copies earlier context, so low entropy may result from repetition rather than serve as an independent signal. How often are penalized turns actually productive, for example running tests or applying the final edit?
7 comments
Q3: A repeated tool call mostly copies earlier context, so low entropy may result from repetition rather than serve as an independent signal. How often are penalized turns actually productive, for example running tests or applying the final edit?
A3: Thanks for your comments. In the following, we will answer your questions one by one.
Low entropy beyond repetition. We argue that low entropy is associated with more unproductive behaviors, not just repeated tool calls. In the manuscript, our observation study in Fig. 1(b) examines repeated tool calls, excessive exploration, and erroneous commands.
To further address your concern, we conduct additional analyses on unproductive behaviors. For your convenience, we provide the results below.
Table D: Unproductive behaviors among the lowest-entropy 20% of turns on SWE-bench Verified. Behavior categories may overlap.
| Model | Any Issue (%) | Repeated Tool Calls (%) | Excessive Exploration (%) | Erroneous Commands (%) |
|---|---|---|---|---|
| Qwen3.5-4B | 98.8 | 90.9 | 70.2 | 9.2 |
| Qwen3.5-9B | 96.3 | 92.8 | 80.8 | 7.8 |
| Qwen3.5-35B-A3B | 71.2 | 58.4 | 30.2 | 6.1 |
From the results, low-entropy turns also exhibit excessive exploration and erroneous commands, supporting their association with more unproductive behaviors.
Impact on productive operations. To address your concern, we carry out more experiments to investigate how often are penalized turns actually unproductive. For your convenience, we provide the results below.
Table E: Frequency and effectiveness of low-entropy penalization during training.
| Statistic | Proportion (%) |
|---|---|
| Penalized turns among all turns | 5.8 |
| Penalized turns among turns in trajectories with efficiency optimization activated | 22.5 |
| Penalized low-entropy turns exhibiting necessary operations | 21.8 |
| Penalized turns exhibiting unproductive behaviors | 78.2 |
From the results, one could draw the following conclusions. On the one hand, with the efficiency gating, only a small proportion of turns receive low-entropy penalties, limiting the scope of potential suppression of necessary operations. On the other hand, most penalized low-entropy turns exhibit unproductive behaviors, while a minority may involve necessary editing or verification operations, supporting the effectiveness of low-entropy selection.
Q4: The efficiency claims go beyond the evidence. The reported token savings are relative to GRPO, which itself inflates token usage. Compared with the base models, HERO's token usage is only slightly lower on average, and for the 4B model it is unchanged on swebench verified and higher on swebench multilingual. The claimed superiority over state-of-the-art agents therefore holds only for the 35B-A3B model, whose base model already leads. This matters because the evidence supports a narrower claim, larger resolve rate gains than GRPO at roughly base-level cost, rather than "doing more with fewer tokens" as the title and abstract suggest.
A4: Thanks for your comments. We argue that the trade-off between task resolution and token efficiency could be adjusted through hyperparameters, e.g., in Eq. (5). To further address your concern, we increase from 0.3 to 0.5 in the Qwen3.5-4B experiments.
Table F: Performance with increased efficiency weight using Qwen3.5-4B.
| Benchmark | Method | Resolved Rate (%) | Tokens (M) |
|---|---|---|---|
| Verified | Base | 32.8 | 4.5 |
| Verified | HERO () | 39.0 | 3.7 |
| Multilingual | Base | 20.0 | 4.0 |
| Multilingual | HERO () | 24.0 | 3.8 |
The results show that HERO achieves higher resolution rates and lower token consumption than the base model.
To balance token efficiency and resolution performance, we adopt the default parameter setting described in the manuscript. Under this setting, we believe that HERO achieves competitive token usage compared with the base models across various model scales, avoiding the token inflation introduced by GRPO. For your convenience, we report the average results across the three model scales below.
Table G: Average performance across Qwen3.5-4B, 9B, and 35B-A3B.
| Benchmark | Method | Resolved Rate (%) | Tokens (M) |
|---|---|---|---|
| Verified | Base | 45.0 | 4.0 |
| Verified | GRPO | 47.9 | 6.0 |
| Verified | HERO | 49.9 | 3.6 |
| Multilingual | Base | 29.9 | 3.8 |
| Multilingual | GRPO | 32.0 | 6.3 |
| Multilingual | HERO | 34.4 | 3.8 |
From the results, compared with base model and GRPO, HERO improves resolution performance and achieves competitive token consumption.
Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.
A1: Thanks for your comments. In the following, we will answer your questions one by one.
A1-1: Variability across three seeds.
To address your concerns, we provide the mean and sample standard deviation across three runs for the main results in Tables 1–3. For your convenience, we present the results below.
Table A: Performance on SWE-bench Verified and SWE-bench Multilingual across three runs (Tables 1–2). Results are reported as Mean ± Std.
Table B: Ablation study on SWE-bench Verified using Qwen3.5-4B across three runs (Table 3).
From the results, one could observe that the main experiments and ablation studies show generally consistent trends across runs, supporting the effectiveness of HERO and its components.
A1-2: Hyperparameter selection.
For a fair evaluation, we selected λ and τ based on the behavior on the 640 training tasks, without using evaluation set and held-out data. Specifically, we examine how different values balance task resolution and token efficiency during training. As for the entropy percentile, we adopt a 20% selection fraction inspired by [A], applying it to low-entropy turns in HERO. To verify the reasonableness of our parameter settings, we have conducted sensitivity analyses in Figs. 6, 8, and 9, which show that HERO maintains favorable performance within a certain range of these parameters.
A2: Thanks for your comments. For fair comparisons across methods, we use the same optimization settings across all three model sizes. For your convenience, we provide the corresponding settings below.
Table C: Training settings across all three model sizes.
For training compute, experiments on Qwen3.5-4B and Qwen3.5-9B each use 8 GPUs on one node, with 4 GPUs for training and 4 for rollouts. Experiments on Qwen3.5-35B-A3B use 64 GPUs across 8 nodes, shared between training and rollouts. In the revised manuscript, we have added further training and optimization details to Appendix A.2.