acceptodds
Under review as a conference paper at ICLR 2027

Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents

Abstract

Recently, coding agents have emerged as a dominant paradigm for real-world software engineering (SWE) scenarios, which solve complex tasks through multi-turn interactions with development environments. However, frequent interactions with environments would inevitably introduce substantial token overhead, leading to high usage costs and latency. Although recent studies have explored reducing token usage by context manipulation and interaction limits at inference time, these approaches focus on improving token efficiency while overlooking the risk of discarding task-relevant information, thus struggling to balance the trade-off between resolution rate and token efficiency. In this paper, we study a more general paradigm without suffering from the limitation, i.e., training token-efficient coding agents with promising resolution performance, which is a highly-practical yet less-explored problem. To this end, we reveal two core observations in SWE scenarios: i) Efficiency Variation: successful resolution could be achieved with fewer tokens; ii) Entropy Correlation: unproductive behaviors are associated with turn-level entropy. Motivated by observations, we propose a novel reinforcement learning framework, dubbed HERO. Specifically, HERO prioritizes task resolution over token efficiency during policy optimization and encourages efficient reasoning patterns at both trajectory and turn levels. Extensive experiments on SWE-bench Verified and SWE-bench Multilingual demonstrate that HERO achieves a favorable trade-off between resolution rate and token efficiency compared with state-of-the-art coding agents and reinforcement learning methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding Agents, Large Language Model, Efficient Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_NoT6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to train LLM coding agents that resolve SWE tasks with fewer tokens, measuring the trade-off with TRS. Motivated by two observations (successful trajectories for the same task vary widely in token usage, and low-entropy turns are associated with unproductive behaviors), it proposes HERO, a GRPO-based method. Hierarchical policy optimization adds an efficiency advantage only once the group's resolution rate exceeds a threshold, and clips advantages so that resolved rollouts are never negative and unresolved ones never positive. The efficiency advantage combines a trajectory-level penalty on token usage above the median of same-outcome rollouts with a turn-level penalty on the group's 20% lowest-entropy turns. Trained on 640 tasks and evaluated with Qwen3.5-4B/9B/35B-A3B on SWE-bench Verified and Multilingual (Claude Code scaffold), HERO raises the average resolution rate from 37.4% to 42.2% (GRPO: 39.9%), using fewer tokens than GRPO and roughly the same as the base models.

Critical strengths
  1. The paper addresses an efficiency problem, the token cost of coding agents, and identifies two failure modes: standard GRPO substantially inflates token usage, while a naive efficiency penalty without hierarchical policy optimization pushes resolution below the base model. HERO avoids both, and the ablations indicate that each component contributes. This matters because it offers a working recipe in a setting where naive length penalties fail.

  2. The gains over GRPO are consistent: HERO achieves higher resolution with fewer tokens across all model sizes and both benchmarks. On the 4B model it also gives a better trade-off than existing token-efficient RL and inference-time methods, while preserving prefix-cache reuse and transferring to another scaffold.

Critical weaknesses
  1. The efficiency claims go beyond the evidence. The reported token savings are relative to GRPO, which itself inflates token usage. Compared with the base models, HERO's token usage is only slightly lower on average, and for the 4B model it is unchanged on swebench verified and higher on swebench multilingual. The claimed superiority over state-of-the-art agents therefore holds only for the 35B-A3B model, whose base model already leads. This matters because the evidence supports a narrower claim, larger resolve rate gains than GRPO at roughly base-level cost, rather than "doing more with fewer tokens" as the title and abstract suggest.

  2. Novelty is moderate, and the new parts are not strongly supported. HERO's core principles, that resolved trajectories always score above unresolved ones and that token usage is judged relative to other correct trajectories for the same task, already appear in Training Efficient (Arora & Zanette), which the paper itself uses as a baseline. HERO's main additions are turning on the efficiency term only when the group's resolution rate is high, also comparing token usage among unresolved trajectories, and penalizing low-entropy turns. However, these additions are tested only on the 4B model, removing each one changes resolution by only a few points, and no variance across seeds is reported.

Questions for authors
  1. What are the standard deviations over the three seeds in Tables 1–3, and were λ\lambda, τ\tau, the entropy percentile, and the 35B-specific values chosen on SWE-bench Verified or on held-out data?

  2. Please also report the learning rate, number of RL steps, KL and clipping settings, and training compute.

  3. A repeated tool call mostly copies earlier context, so low entropy may result from repetition rather than serve as an independent signal. How often are penalized turns actually productive, for example running tests or applying the final edit?

7 comments

Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.

Q1: What are the standard deviations over the three seeds in Tables 1–3, and were λ\lambda, τ\tau, the entropy percentile, and the 35B-specific values chosen on SWE-bench Verified or on held-out data?

A1: Thanks for your comments. In the following, we will answer your questions one by one.

A1-1: Variability across three seeds.

To address your concerns, we provide the mean and sample standard deviation across three runs for the main results in Tables 1–3. For your convenience, we present the results below.

Table A: Performance on SWE-bench Verified and SWE-bench Multilingual across three runs (Tables 1–2). Results are reported as Mean ± Std.

SWE-bench VerifiedSWE-bench VerifiedSWE-bench MultilingualSWE-bench Multilingual
ModelMethodResolved Rate (%)Tokens (M)Resolved Rate (%)Tokens (M)
GLM-4.7-Flash-30BBase42.6 ± 1.24.6 ± 0.425.3 ± 1.34.0 ± 0.3
KAT-Dev-32BBase48.4 ± 1.02.4 ± 0.327.6 ± 1.72.3 ± 0.4
Qwen3.5-4BGRPO37.4 ± 1.16.8 ± 0.322.0 ± 1.57.6 ± 0.3
Qwen3.5-4BHERO40.0 ± 0.94.5 ± 0.324.3 ± 0.74.5 ± 0.2
Qwen3.5-9BGRPO47.0 ± 0.95.9 ± 0.331.0 ± 1.96.0 ± 0.3
Qwen3.5-9BHERO49.2 ± 0.73.5 ± 0.234.0 ± 0.74.1 ± 0.2
Qwen3.5-35B-A3BGRPO59.2 ± 0.95.2 ± 0.343.0 ± 1.75.4 ± 0.3
Qwen3.5-35B-A3BHERO60.6 ± 1.12.7 ± 0.245.0 ± 0.52.9 ± 0.2

Table B: Ablation study on SWE-bench Verified using Qwen3.5-4B across three runs (Table 3).

MethodResolved Rate (%)Tokens (M)
GRPO37.4 ± 1.16.8 ± 0.3
w/o HPO29.2 ± 1.82.1 ± 0.3
w/o Coarse39.0 ± 0.86.0 ± 0.2
w/o Fine37.8 ± 1.04.9 ± 0.3
HERO40.0 ± 0.94.5 ± 0.3

From the results, one could observe that the main experiments and ablation studies show generally consistent trends across runs, supporting the effectiveness of HERO and its components.

A1-2: Hyperparameter selection.

For a fair evaluation, we selected λ\lambda and τ\tau based on the behavior on the 640 training tasks, without using evaluation set and held-out data. Specifically, we examine how different values balance task resolution and token efficiency during training. As for the entropy percentile, we adopt a 20% selection fraction inspired by [A], applying it to low-entropy turns in HERO. To verify the reasonableness of our parameter settings, we have conducted sensitivity analyses in Figs. 6, 8, and 9, which show that HERO maintains favorable performance within a certain range of these parameters.

Q2: Please also report the learning rate, number of RL steps, KL and clipping settings, and training compute.

A2: Thanks for your comments. For fair comparisons across methods, we use the same optimization settings across all three model sizes. For your convenience, we provide the corresponding settings below.

Table C: Training settings across all three model sizes.

SettingValue
Learning rate1×10−61\times10^{-6}
Training tasks640
Tasks per rollout batch8
Rollout group size16
RL steps80
KL coefficient0.001
PPO clippingϵ=0.2\epsilon=0.2

For training compute, experiments on Qwen3.5-4B and Qwen3.5-9B each use 8 GPUs on one node, with 4 GPUs for training and 4 for rollouts. Experiments on Qwen3.5-35B-A3B use 64 GPUs across 8 nodes, shared between training and rollouts. In the revised manuscript, we have added further training and optimization details to Appendix A.2.

Q3: A repeated tool call mostly copies earlier context, so low entropy may result from repetition rather than serve as an independent signal. How often are penalized turns actually productive, for example running tests or applying the final edit?

A3: Thanks for your comments. In the following, we will answer your questions one by one.

Low entropy beyond repetition. We argue that low entropy is associated with more unproductive behaviors, not just repeated tool calls. In the manuscript, our observation study in Fig. 1(b) examines repeated tool calls, excessive exploration, and erroneous commands.

To further address your concern, we conduct additional analyses on unproductive behaviors. For your convenience, we provide the results below.

Table D: Unproductive behaviors among the lowest-entropy 20% of turns on SWE-bench Verified. Behavior categories may overlap.

ModelAny Issue (%)Repeated Tool Calls (%)Excessive Exploration (%)Erroneous Commands (%)
Qwen3.5-4B98.890.970.29.2
Qwen3.5-9B96.392.880.87.8
Qwen3.5-35B-A3B71.258.430.26.1

From the results, low-entropy turns also exhibit excessive exploration and erroneous commands, supporting their association with more unproductive behaviors.

Impact on productive operations. To address your concern, we carry out more experiments to investigate how often are penalized turns actually unproductive. For your convenience, we provide the results below.

Table E: Frequency and effectiveness of low-entropy penalization during training.

StatisticProportion (%)
Penalized turns among all turns5.8
Penalized turns among turns in trajectories with efficiency optimization activated22.5
Penalized low-entropy turns exhibiting necessary operations21.8
Penalized turns exhibiting unproductive behaviors78.2

From the results, one could draw the following conclusions. On the one hand, with the efficiency gating, only a small proportion of turns receive low-entropy penalties, limiting the scope of potential suppression of necessary operations. On the other hand, most penalized low-entropy turns exhibit unproductive behaviors, while a minority may involve necessary editing or verification operations, supporting the effectiveness of low-entropy selection.

Q4: The efficiency claims go beyond the evidence. The reported token savings are relative to GRPO, which itself inflates token usage. Compared with the base models, HERO's token usage is only slightly lower on average, and for the 4B model it is unchanged on swebench verified and higher on swebench multilingual. The claimed superiority over state-of-the-art agents therefore holds only for the 35B-A3B model, whose base model already leads. This matters because the evidence supports a narrower claim, larger resolve rate gains than GRPO at roughly base-level cost, rather than "doing more with fewer tokens" as the title and abstract suggest.

A4: Thanks for your comments. We argue that the trade-off between task resolution and token efficiency could be adjusted through hyperparameters, e.g., λ\lambda in Eq. (5). To further address your concern, we increase λ\lambda from 0.3 to 0.5 in the Qwen3.5-4B experiments.

Table F: Performance with increased efficiency weight using Qwen3.5-4B.

BenchmarkMethodResolved Rate (%)Tokens (M)
VerifiedBase32.84.5
VerifiedHERO (λ=0.5\lambda=0.5)39.03.7
MultilingualBase20.04.0
MultilingualHERO (λ=0.5\lambda=0.5)24.03.8

The results show that HERO achieves higher resolution rates and lower token consumption than the base model.

To balance token efficiency and resolution performance, we adopt the default parameter setting described in the manuscript. Under this setting, we believe that HERO achieves competitive token usage compared with the base models across various model scales, avoiding the token inflation introduced by GRPO. For your convenience, we report the average results across the three model scales below.

Table G: Average performance across Qwen3.5-4B, 9B, and 35B-A3B.

BenchmarkMethodResolved Rate (%)Tokens (M)
VerifiedBase45.04.0
VerifiedGRPO47.96.0
VerifiedHERO49.93.6
MultilingualBase29.93.8
MultilingualGRPO32.06.3
MultilingualHERO34.43.8

From the results, compared with base model and GRPO, HERO improves resolution performance and achieves competitive token consumption.

Reviewer_4EZxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HERO, a reinforcement learning method for improving the token efficiency of coding agents. HERO combines success-dependent efficiency gating and resolution-first advantage clipping with trajectory-level cost rewards and turn-level entropy penalties. Experiments on two SWE-bench benchmarks and three model sizes report higher resolution rates and approximately 39.8% lower average token usage than GRPO.

Critical strengths

The paper addresses a practical problem: improving coding agents without substantially increasing interaction costs. HERO improves both resolution rate and token usage over GRPO across all six main evaluation settings.

The success-prioritized design is well motivated. Removing hierarchical policy optimization substantially reduces token usage but also harms resolution performance, supporting the need to protect task success during efficiency optimization.

Critical weaknesses
  1. The GRPO baseline and advantage-processing pipeline are insufficiently specified. Reward normalization, handling of uniform-outcome groups, and loss weighting across different trajectory lengths can materially affect both learning and token usage. Without these details, it is difficult to attribute the reported gains confidently.
  2. The paper does not clearly isolate the benefits over simple, well-tuned cost penalties. Comparisons with existing efficiency methods are useful, but a direct success-minus-cost baseline and a separately normalized success/cost baseline would better establish the need for HERO’s specific design. The “w/o HPO” ablation does not directly answer this question.
Questions for authors
  1. Please provide the complete reward-to-loss pipeline for GRPO and HERO. How are advantages normalized, how are all-success and all-failure groups handled, and is there any further normalization after HERO combines its advantage terms? How are tokens, turns, and trajectories weighted in the loss?

  2. Are the training budgets, optimization settings, and checkpoint selection procedures matched? Please clarify any differences beyond the proposed method.

  3. How does HERO compare with a tuned reward ri=yi−αcir_i=y_i-\alpha c_i, and with separately normalized success and cost advantages? For uniform-outcome groups, group normalization cancels a positive (α\alpha), ignoring the numerical stabilizer; for mixed-outcome groups, it generally does not. Please clarify how the implementation handles this distinction.

6 comments

Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.

Q1: Please provide the complete reward-to-loss pipeline for GRPO and HERO. How are advantages normalized, how are all-success and all-failure groups handled, and is there any further normalization after HERO combines its advantage terms? How are tokens, turns, and trajectories weighted in the loss?

A1: Thanks for your suggestions. We acknowledge that some details and edge cases were not clarified, i.e., clipping strategy in Eq. (A.6), and we have revised them in the revised manuscript. In the following, we will answer your questions one by one.

Reward-to-loss pipeline for GRPO.

We adopt standard GRPO [A] as the baseline and the task-resolution component in HERO.

(1) Task reward and advantage. A generated patch receives yi=1y_i=1 if it passes all task-specific tests, and yi=0y_i=0 otherwise. Accordingly, the resolution advantage is

\tag{A.1}$$ where $\bar y$ and $\sigma_y$ denote the mean and standard deviation of the binary resolution rewards within the rollout group, respectively. **Resolution rewards are standardized within each rollout group.** (2) Policy loss. The resolution advantage is assigned to every generated token in the trajectory, and the loss of each token is derived as $$\ell_i(\theta)=- \min\left( \rho(\theta)A_i^{\mathrm{res}}, \operatorname{clip}\left(\rho(\theta),1-\epsilon,1+\epsilon\right) A_i^{\mathrm{res}} \right), \tag{A.2}$$ where $\rho(\theta)$ denotes the current-to-rollout policy probability ratio for the corresponding token, and $\epsilon$ is the clipping threshold. Token indices are omitted for simplicity. **Reward-to-loss pipeline for HERO.** (1) Resolution advantage. HERO uses the same task reward and advantage in GRPO, as defined in Eq. (A.1). (2) Efficiency advantage. Following Eqs. (7)–(9) in the manuscript, the efficiency advantage combines coarse-grained trajectory credit and fine-grained turn credit, $$A_{i,j}^{\mathrm{eff}}=A_i^{\mathrm{coarse}}+A_{i,j}^{\mathrm{fine}}. \tag{A.3}$$ The coarse-grained advantage is computed as $$\begin{aligned} A_i^{\mathrm{coarse}}&=\frac{u_i-\bar u}{\sigma_u+\delta}, \qquad u_i=1-\tanh\left(\max\left(\frac{c_i}{B^{y_i}}-1,0\right)\right),\\ B^{y_i}&=\operatorname{Median}\{c_k\mid k\in G,\ y_k=y_i\}, \end{aligned} \tag{A.4}$$ where $\bar u$ and $\sigma_u$ denote the mean and standard deviation over trajectories with valid efficiency rewards in the rollout group. **Note that these rewards are standardized jointly across eligible trajectories, rather than separately within each outcome subset.** The fine-grained advantage is computed as $$A_{i,j}^{\mathrm{fine}}= \begin{cases} -\tanh\left(1-\frac{h_{i,j}}{q}\right), & h_{i,j}<q,\\ 0, & h_{i,j}\ge q, \end{cases} \tag{A.5}$$ where $q$ is the 20th percentile of turn entropy within the rollout group. **The fine-grained advantage is not standardized.** (3) Advantage combination. Following Eqs. (4)–(6) in the manuscript, the final advantage $\widehat A_{i,j}$ is computed as $$\widehat A_{i,j}= \begin{cases} \max(A_{i,j},0), & 0<p<1,\\ y_i=1,\\\\ \min(A_{i,j},0), & 0<p<1,\\ y_i=0,\\\\ A_{i,j}, & p\in\\{0,1\\}, \end{cases} \qquad A_{i,j}=A_i^{\mathrm{res}}+wA_{i,j}^{\mathrm{eff}}. \tag{A.6}$$ where $$w=\lambda\operatorname{clip}\left(\frac{p-\tau}{1-\tau},0,1\right), \qquad p=\frac{1}{|G|}\sum_{i\in G}y_i. \tag{A.7}$$ **No further normalization is applied after advantage combination.** (4) Policy loss. The turn-level advantage is assigned to every generated token within the corresponding turn, and the loss of each token is derived as $$\ell_{i,j}(\theta)=- \min\left( \rho(\theta)\widehat A_{i,j}, \operatorname{clip}\left(\rho(\theta),1-\epsilon,1+\epsilon\right) \widehat A_{i,j} \right). \tag{A.8}$$ **All-success and all-failure groups.** For both groups, $A_i^{\mathrm{res}}=0$ in GRPO and HERO. For all-success groups, HERO learns from efficiency advantages with $w=\lambda$ and the resolution-first clipping is not applied, as shown in Eq. (A.6). For all-failure groups, neither method receives a resolution- or efficiency-based policy gradient signal.

Q2: Are the training budgets, optimization settings, and checkpoint selection procedures matched? Please clarify any differences beyond the proposed method.

A2: Thanks for your comments. In the manuscript, we have provided some of the implementation details you mentioned in Section 4.1 and Appendix A.2.

To address your concerns, we provide further clarification below. Note that, HERO and the baselines use the same settings you mentioned for fair comparisons, ensuring that the observed improvements are attributable to the proposed method. Specifically,

(1) Training budgets. All methods use the same 640-task training set and are trained for 80 steps, with 8 tasks per rollout batch and 16 rollouts per task.

(2) Optimization settings. All methods adopt Adam with a learning rate of 10−610^{-6}, coefficients (β1,β2)=(0.9,0.98)(\beta_1,\beta_2)=(0.9,0.98), and weight decay of 0.1.

(3) Checkpoint selection. We evaluate all methods using their checkpoints from the final training iteration, i.e., step 80.

In the revised manuscript, we have added these implementation details to Appendix A.2.

Q3: How does HERO compare with a tuned reward ri=yi−αcir_i=y_i-\alpha c_i, and with separately normalized success and cost advantages? For uniform-outcome groups, group normalization cancels a positive (α\alpha), ignoring the numerical stabilizer; for mixed-outcome groups, it generally does not. Please clarify how the implementation handles this distinction.

A3: Thanks for your comments. In the following, we will answer your question one by one.

Comparison with simple baselines. To address your concerns, we conduct more experiments with two simple baselines you mentioned based on GRPO: i) Success-minus-cost reward, which uses ri=yi−αcir_i=y_i-\alpha c_i as the reward, where cic_i denotes cumulative input and output token usage in millions;
ii) Separately normalized success and cost advantages, which separately standardizes yiy_i and −ci-c_i within each rollout group, then combines them as Ai=Aires+αAicostA_i=A_i^{\mathrm{res}}+\alpha A_i^{\mathrm{cost}}. Specifically, for the uniform-outcome problem you point out, α\alpha cancels in the success-minus-cost baseline (ignoring the numerical stabilizer), but remains effective in the separately normalized baseline. While for the mixed-outcome groups, α\alpha still controls the reward trade-off, making the success-minus-cost baseline a meaningful comparison. For your convenience, we provide the corresponding results below. Note that, for reasonable comparisons, we tune the parameter and set α∈{0.1,0.3}\alpha\in\{0.1,0.3\}.

Table A: Comparisons with simple cost-aware baselines using Qwen3.5-4B on SWE-bench Verified.

Methodα\alphaResolved Rate (%)Tokens (M)TRS
Untrained—32.84.514.0
GRPO—37.46.813.4
Success-minus-cost0.133.64.814.0
Success-minus-cost0.327.23.512.8
Separate normalization0.135.05.513.7
Separate normalization0.333.24.114.7
HERO—40.04.517.1

From the results, HERO achieves higher resolution rates and TRS than the baselines you mentioned, which verifies the effectiveness of HERO. Intuitively, for these baselines, efficiency signals may override resolution signals and severely degrade resolution performance, supporting the need for HPO and resolution-first clipping. Besides, even with a smaller α\alpha, these baselines struggle to balance resolution and efficiency, as they impose efficiency pressure regardless of LLM's resolution ability.

[A] Zhihong Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.

Dear Reviewer 4EZx,

As the discussion deadline approaches, we would like to know if our response has addressed your concerns. We have provided additional experiments and detailed clarifications to address your questions. If you have any further concerns or suggestions for the paper or our rebuttal, please let us know.

If your concerns have been satisfactorily addressed, we would be grateful if you would consider revising your score accordingly. Thank you again for the time and effort you dedicated to reviewing this work.

Reviewer_L5mbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Coding agents typically require frequent interactions with development environments, resulting in substantial token consumption and inference latency. Motivated by two observations—that successful trajectories exhibit considerable variation in token usage and that low-entropy turns are more likely to involve unproductive behaviors—this paper proposes HERO, a reinforcement learning framework for improving both task resolution and token efficiency. Building upon GRPO, HERO incorporates trajectory-level token efficiency and turn-level entropy signals into advantage estimation. It further introduces resolution-prioritized policy optimization to balance task-solving capability and token efficiency. Experiments on SWE-bench Verified and SWE-bench Multilingual demonstrate improvements in both resolution rate and token consumption.

Critical strengths
  1. The paper addresses an important and practical problem, as repeated agent-environment interactions introduce substantial computational overhead during both training and inference. Improving the token efficiency of coding agents is a valuable research direction.
  2. The experiments report both resolution rates and token consumption across multiple benchmarks and model scales, providing empirical evidence that HERO achieves a favorable trade-off between task-solving performance and token efficiency.
  3. The paper is generally well organized and easy to follow. In particular, Figure 2 provides an intuitive overview of the proposed framework.
Critical weaknesses
  1. Potentially misleading use of hierarchical reinforcement learning. Hierarchical reinforcement learning is a well-established research area that typically involves temporally abstract actions or hierarchical policies. However, the proposed method primarily modifies GRPO through resolution-prioritized advantage optimization and multi-granularity efficiency credit assignment, without explicitly introducing hierarchical policy learning. Therefore, the term "hierarchical reinforcement learning" may not accurately reflect the technical nature of the proposed method.
  2. Insufficient justification for the complexity of the proposed design. Although the two motivating observations are intuitive, the resulting framework incorporates several mechanisms, including efficiency gating, resolution-first clipping, outcome-calibrated trajectory advantages, and entropy-based turn-level penalties. While the paper provides component-wise ablations, it remains unclear whether such a combination is necessary to achieve the reported improvements. More direct comparisons with simpler alternatives would help establish the necessity of the proposed design choices.
Questions for authors
  1. Under what experimental settings were the two key observations in Figure 1 obtained? Do these patterns consistently hold across different model scales, training stages, and task settings? In particular, does low turn-level entropy remain a reliable indicator of unproductive behavior across different stages of agent-environment interaction?
  2. Could simpler alternatives, such as GRPO with a straightforward token-length penalty or a success-conditioned efficiency reward, achieve comparable performance? Although the existing ablations demonstrate the contributions of individual components, further comparisons with simpler methods would help clarify whether the additional complexity is justified.
5 comments

Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.

Q1: Under what experimental settings were the two key observations in Figure 1 obtained? Do these patterns consistently hold across different model scales, training stages, and task settings? In particular, does low turn-level entropy remain a reliable indicator of unproductive behavior across different stages of agent-environment interaction?

A1: Thanks for your comments. In Figure 1(b), the observation of ''Efficiency Variation'' is obtained using Qwen3.5-4B, Qwen3.5-9B, and Qwen3.5-35B-A3B on SWE-bench Verified, while the observation of ''Entropy Correlation'' is obtained using Qwen3.5-4B on the same benchmark.

To address your concerns, we carry out additional observation experiments across model scales, training stages, and task settings. Specifically, we examine Qwen3.5-9B and Qwen3.5-35B-A3B on SWE-bench Verified and SWE-bench Multilingual, and Qwen3.5-4B at GRPO training steps 0, 40, and 80. For your convenience, we provide the results below.

Table A: Efficiency variation across model scales, training stages, and task settings. P20 and P80 denote token-usage percentiles.

Task SettingModelMean Tokens (M)P20 Tokens (M)P80 Tokens (M)
VerifiedQwen3.5-9B4.50.59.1
VerifiedQwen3.5-35B-A3B2.90.54.4
MultilingualQwen3.5-9B4.30.79.0
MultilingualQwen3.5-35B-A3B3.20.55.0
VerifiedQwen3.5-4B (Step 0)4.50.211.1
VerifiedQwen3.5-4B + GRPO (Step 40)6.00.611.8
VerifiedQwen3.5-4B + GRPO (Step 80)6.80.813.2

Table B: Unproductive behaviors among the lowest-entropy 20% of turns across model scales, training stages, and task settings. “Any Issue” denotes the union of the three behaviors.

Task SettingModelAny Issue (%)Repeated Tool Calls (%)Excessive Exploration (%)Erroneous Commands (%)
VerifiedQwen3.5-9B96.392.880.87.8
VerifiedQwen3.5-35B-A3B71.258.430.26.1
MultilingualQwen3.5-9B95.193.981.74.1
MultilingualQwen3.5-35B-A3B70.160.241.410.5
VerifiedQwen3.5-4B (Step 0)98.890.970.29.2
VerifiedQwen3.5-4B + GRPO (Step 40)98.290.670.49.0
VerifiedQwen3.5-4B + GRPO (Step 80)98.090.069.89.1

From the results, one could conclude that the two core observations in Figure 1 hold across the evaluated model scales, training stages and task settings, further supporting our motivation.

Q2: Could simpler alternatives, such as GRPO with a straightforward token-length penalty or a success-conditioned efficiency reward, achieve comparable performance? Although the existing ablations demonstrate the contributions of individual components, further comparisons with simpler methods would help clarify whether the additional complexity is justified.

A2: Thanks for your comments. It is worth noting that we have compared baselines with token-length penalty and success-conditioned efficiency reward, as shown in Table 6 (Appendix B.1). Specifically, Training Efficient [A] adopts adaptive threshold to penalize excessively long trajectories, while LASER [B], OTC-GRPO [C], and SlimSearcher [D] use success-conditioned efficiency rewards to encourage shorter trajectories. For your convenience, we provide the corresponding results below.

Table 6: Comparisons with token-efficient RL methods on SWE-bench Verified and SWE-bench Multilingual.

SWE-bench VerifiedSWE-bench VerifiedSWE-bench VerifiedSWE-bench MultilingualSWE-bench MultilingualSWE-bench Multilingual
MethodResolved Rate (%)Tokens (M)TRSResolved Rate (%)Tokens (M)TRS
Base32.84.514.020.04.08.9
GRPO37.46.813.422.07.67.5
Training Efficient37.46.413.721.36.87.6
LASER33.23.415.821.74.89.0
OTC-GRPO32.44.014.517.03.28.3
SlimSearcher37.66.313.921.06.47.7
Ours40.04.517.124.34.510.4

From the results, the proposed HERO achieves a better trade-off than other baselines.

To further address your concerns, we conduct additional experiments on GRPO with the following strategies: i) Length penalty, which penalizes trajectories exceeding a specified length threshold; ii) Success-conditioned efficiency reward, which applies the length penalty only to successful trajectories. In the implementation, let Lmax⁡L_{\max} denote the maximum generation length and ℓi\ell_i its actual generation length. For length penalty, we define

ri=yi−max⁡(ℓi−b,0)Lmax⁡.r_i=y_i-\frac{\max(\ell_i-b,0)}{L_{\max}}.

For success-conditioned efficiency reward, we define

ri=yi(1−max⁡(ℓi−b,0)Lmax⁡).r_i=y_i\left(1-\frac{\max(\ell_i-b,0)}{L_{\max}}\right).

For your convenience, we provide the corresponding results below.

Table C: Comparisons with length-aware GRPO variants using Qwen3.5-4B on SWE-bench Verified.

MethodLength ThresholdResolved Rate (%)Tokens (M)
GRPO-37.46.8
Length PenaltyLmax⁡/4L_{\max}/432.04.7
Length PenaltyLmax⁡/2L_{\max}/235.85.3
Length Penalty3Lmax⁡/43L_{\max}/436.46.4
Success-conditioned Reward-37.05.5
HERO-40.04.5

From the results, one could observe that HERO achieves higher resolution rates and better token efficiency than the two strategies you mentioned, mainly contributing to resolution-prioritized policy optimization and multi-granularity credit assignment.

Q3: Potentially misleading use of hierarchical reinforcement learning. Hierarchical reinforcement learning is a well-established research area that typically involves temporally abstract actions or hierarchical policies. However, the proposed method primarily modifies GRPO through resolution-prioritized advantage optimization and multi-granularity efficiency credit assignment, without explicitly introducing hierarchical policy learning. Therefore, the term "hierarchical reinforcement learning" may not accurately reflect the technical nature of the proposed method.

A3: Thanks for your suggestions. We acknowledge that our use of ''hierarchical'' differs from conventional HRL involving temporally abstract actions or hierarchical policies. In this paper, ''hierarchical'' refers to the prioritization of task resolution over token efficiency, which establishes a hierarchy of optimization priorities. We will clarify the meaning of ''hierarchical'' in the revised manuscript to avoid confusion with conventional HRL.

[A] Daman Arora and Andrea Zanette. Training Language Models to Reason Efficiently.

[B] Wei Liu et al. Learn to Reason Efficiently with Adaptive Length-Based Reward Shaping.

[C] Hongru Wang et al. OTC: Optimal Tool Calls via Reinforcement Learning.

[D] Zequn Xie et al. SlimSearcher: Training Efficiency-Aware Web Agents via Adaptive Reward Gating.

Reviewer_jpLGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains coding agents to spend fewer tokens, arguing that inference-time pruning trades accuracy for cost while training can internalize efficiency instead. Two observations motivate the method: successful trajectories for the same task vary widely in cost, and low-entropy turns coincide with unproductive behavior. HERO therefore combines a resolution-prioritized group-relative advantage with an efficiency term that is gated on the group's resolution rate, clipped so resolved rollouts never rank below unresolved ones, and credited at both trajectory and turn level. Across two SWE benchmarks and three backbones, it reports higher resolution with fewer tokens than GRPO.

Critical strengths

1.Treating the turn, rather than the token, as the unit of credit is the right abstraction for multi-turn agents, and using turn entropy as a free, rollout-time signal for where to apply that credit is a genuinely practical idea. 2.The resolution-first machinery is well designed: gating efficiency on demonstrated capability and clipping advantages by outcome both target the real failure mode, where an efficiency signal quietly teaches the agent to stop trying, and the ablations show each piece carries weight. 3.The empirical scope is broader than typical for this kind of systems paper, covering scaffold transfer, token breakdowns, prefix-cache reuse and relative inference cost, and comparisons against both inference-time and token-efficient RL methods; the appendix result that HERO still benefits from inference-time pruning is a fair and unusually honest experiment.

Critical weaknesses

1.The savings exist only relative to GRPO, whose token inflation is a known artifact of group-relative training; against the untrained model token totals are unchanged or higher, and they fall because turns are fewer, not shorter. No length-penalized GRPO baseline isolates the contribution. 2.The composite metric does most of the ranking but is unnormalized: its token term is arbitrarily scaled and measures absolute tokens rather than cost, contradicting the paper's own cost analysis, and no resolution-versus-cost Pareto view is given. 3.Both motivating observations are unquantified, and the entropy penalty always removes a fixed fraction of turns, including necessary edit and test actions, while the capability gate leaves the most expensive unsolved tasks untouched.

Questions for authors

1.How does HERO compare with GRPO under a standard length penalty or length-normalized advantage? If that baseline closes most of the gap, the turn-level entropy machinery is not doing the work the paper attributes to it. 2.Where exactly do the saved tokens come from — fewer turns, shorter patches, shorter reasoning, or fewer failed commands? A breakdown by turn type, and token usage per turn and per resolved task relative to the untrained model, would settle whether the agent became more efficient or merely avoided becoming less efficient. 3.What fraction of penalized low-entropy turns are necessary edit or verification actions, and what happens if the capability gate is replaced by a per-task difficulty signal so that unsolved tasks also receive efficiency pressure?

5 comments

Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.

Q1: How does HERO compare with GRPO under a standard length penalty or length-normalized advantage? If that baseline closes most of the gap, the turn-level entropy machinery is not doing the work the paper attributes to it.

A1: Thanks for your comments. It is worth noting that we have compared length-based reward baselines in Table 6 (Appendix B.1). Specifically, Training Efficient [A], LASER [B] and SlimSearcher [C] are length penalty methods, which penalize excessively long successful trajectories with self-adaptive or predefined threshold. For your convenience, we provide the corresponding results below.

Table A: Comparisons with token-efficient RL methods on SWE-bench Verified and SWE-bench Multilingual (selected results from Table 6).

SWE-bench VerifiedSWE-bench VerifiedSWE-bench VerifiedSWE-bench MultilingualSWE-bench MultilingualSWE-bench Multilingual
MethodResolved Rate (%)Tokens (M)TRSResolved Rate (%)Tokens (M)TRS
Base32.84.514.020.04.08.9
GRPO37.46.813.422.07.67.5
Training Efficient37.46.413.721.36.87.6
LASER33.23.415.821.74.89.0
SlimSearcher37.66.313.921.06.47.7
Ours40.04.517.124.34.510.4

To further address your concerns, we conduct additional experiments on GRPO with two length-aware strategies:
i) Length penalty, which applies penalty to trajectories exceeding a specified length threshold; ii) Length-normalized advantage, which scales down the advantage magnitude of longer trajectories. In the implementation, let Lmax⁡L_{\max} denote the maximum generation length and ℓi\ell_i its actual generation length.

For length penalty, we define the task reward as

ri=yi−max⁡(ℓi−b,0)Lmax⁡.r_i=y_i-\frac{\max(\ell_i-b,0)}{L_{\max}}.

For length-normalized advantage, we use

A~i=AiresLmax⁡−ℓiLmax⁡.\widetilde A_i=A_i^{\mathrm{res}}\frac{L_{\max}-\ell_i}{L_{\max}}.

For your convenience, we provide the corresponding results below.

Table B: Comparisons with length-aware GRPO variants using Qwen3.5-4B on SWE-bench Verified.

MethodLength ThresholdResolved Rate (%)Tokens (M)
GRPO-37.46.8
Length PenaltyLmax⁡/4L_{\max}/432.04.7
Length PenaltyLmax⁡/2L_{\max}/235.85.3
Length Penalty3Lmax⁡/43L_{\max}/436.46.4
Length-Normalized Advantage-36.66.3
HERO-40.04.5

From the results, the length penalty and length-normalized advantage baselines improve token efficiency at the cost of resolution performance, while the proposed HERO achieves better token efficiency with higher task resolution.

Q2: Where exactly do the saved tokens come from — fewer turns, shorter patches, shorter reasoning, or fewer failed commands? A breakdown by turn type, and token usage per turn and per resolved task relative to the untrained model, would settle whether the agent became more efficient or merely avoided becoming less efficient.

A2: Thanks for your comments. We have conducted analysis of interaction turns in Table 10 (Appendix B.5). To further address your concerns, we provide more comprehensive analyses of the factors you mentioned. Specifically, we analyze token usage, turns, and generated tokens per turn for both all tasks and resolved tasks in Table C, and the types of different unproductive turn in Table D. For your convenience, we provide the corresponding results below.

Table C: Token usage and interaction turns on SWE-bench Verified.

All TasksAll TasksAll TasksResolved TasksResolved TasksResolved Tasks
MethodTokens (M)TurnsGenerated Tokens/TurnTokens (M)TurnsGenerated Tokens/Turn
Qwen3.5-4B4.568.7248.72.444.5249.3
+ GRPO6.894.8263.54.872.3263.1
+ HERO4.565.6250.62.344.0246.5
Qwen3.5-9B4.570.0236.22.949.0244.9
+ GRPO5.986.8300.13.757.5311.1
+ HERO3.558.3230.11.739.4240.1

Table D: Unproductive turn types on SWE-bench Verified.

MethodAny Issue (%)Repeated Tool Calls (%)Excessive Exploration (%)Erroneous Commands (%)
Qwen3.5-4B47.027.325.517.2
+ GRPO42.821.415.819.0
+ HERO26.813.210.612.8
Qwen3.5-9B34.521.722.115.3
+ GRPO40.631.920.214.2
+ HERO17.05.32.210.1

From Table C, for all tasks and resolved tasks, one could observe that token savings are primarily associated with fewer turns. More importantly, the results in Table D show that HERO significantly reduces turns exhibiting unproductive behaviors and thus achieves better efficiency.

Q3: What fraction of penalized low-entropy turns are necessary edit or verification actions? and what happens if the capability gate is replaced by a per-task difficulty signal so that unsolved tasks also receive efficiency pressure?

A3: Thanks for your comments. In the following, we will answer your question one by one.

A3-1: Fraction of penalizing necessary operations. To answer your question, we carry out additional experiments on low-entropy penalization. Specifically, we report the proportion of penalized turns during training and these turns exhibiting unproductive behaviors. For your convenience, we provide the corresponding results below.

Table E: Frequency and effectiveness of low-entropy penalization during training.

StatisticProportion (%)
Penalized turns among all turns5.8
Penalized turns among all turns in trajectories with efficiency optimization activated22.5
Penalized low-entropy turns exhibiting necessary editing or verification operations13.2
Penalized low-entropy turns exhibiting unproductive behaviors78.2
Penalized low-entropy turns exhibiting other exploration operations8.6

From the results, one could draw the following conclusions. On the one hand, with the efficiency gating, only a small proportion of turns receive low-entropy penalties, limiting the scope of potential suppression of necessary operations. On the other hand, most penalized low-entropy turns exhibit unproductive behaviors, while a minority may involve necessary editing or verification operations, supporting the effectiveness of low-entropy selection. Taken together, we believe that the negative impact on necessary operations is acceptable.

A3-2: Replacing the capability gate with a per-task difficulty signal. We answer this question from both theoretical and empirical perspectives.

Theoretical perspective. For tasks where all sampled trajectories fail, we believe that imposing efficiency pressure may hinder the improvement of problem-solving capabilities. In other words, hard tasks typically require more exploration and reasoning to reach a correct solution, thus simply reducing token usage for these hard tasks would harm resolution performance.

Empirical perspective. To address your concerns, we manually categorize the 640 training tasks by difficulty to construct a per-task difficulty signal. Specifically, we classify 247 tasks as hard, 147 as medium, and 246 as easy, and set the efficiency weight ww in Eq. (4) to 0.1, 0.3, and 0.5, respectively.

Table F: Analytic study with per-task difficulty weighting using Qwen3.5-4B on SWE-bench Verified.

MethodResolved Rate (%)Tokens (M)TRS
Per-task Difficulty Weighting36.04.215.8
Ours40.04.517.1

The results show that per-task difficulty weighting reduces token usage at the cost of lower resolution performance, while our method achieves a better TRS.

Reviewer_C5NeOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

HERO combines success-prioritized advantage shaping with trajectory-level token rewards and turn-level entropy penalties to train efficient coding agents. Across two SWE-bench benchmarks, it improves resolution rates and reduces token usage relative to GRPO.

Critical strengths
  1. Practical objective. Reducing the token cost of multi-turn coding agents while preserving task success is an important practical problem.

  2. Consistent gains. HERO outperforms GRPO across three model sizes and two benchmarks, with additional comparisons against efficiency-oriented RL and inference-time methods.

  3. Useful ablations. Component ablations and comparisons with random and high-entropy penalties help assess the proposed designs.

Critical weaknesses
  1. Limited novelty. HERO combines familiar mechanisms: efficiency gating, advantage clipping, length reward calibration, and entropy-based turn selection. Its methodological contribution is largely this combination within GRPO and its application to coding agents.

  2. Unclear mechanism. Low entropy is only a proxy for unproductive behavior. The experiments do not distinguish improved identification of wasted reasoning from a general regularization effect of changing reinforcement strength.

  3. Incomplete reporting. Missing variability estimates make the modest resolution-rate gains difficult to assess. Training data selection, overlap checks, and compute matching also need clearer documentation.

Questions for authors
  1. Does the low-entropy penalty identify unproductive turns, or mainly act as regularization? How often does it penalize necessary operations?

  2. Are the gains stable across seeds and against a tuned GRPO baseline with matched training compute?

  3. How were the 640 training tasks sampled and checked for overlap with the evaluation sets?

4 comments

Thanks for your insightful comments and suggestions. We will answer your questions one by one in the following.

Q1: Does the low-entropy penalty identify unproductive turns, or mainly act as regularization? How often does it penalize necessary operations?

A1: Thanks for your comments. In the following, we will answer your questions one by one.

A1-1: The effectiveness of low-entropy penalty.

As far as we understand, your concern is whether low-entropy penalization acts as regularization by weakening reinforcement for certain turns, rather than identifying and discouraging unproductive turns. To verify this, we have evaluated token usage under different turn-level penalty strategies, as reported in Table 14 (Appendix B.10). Specifically, we compare low-entropy penalization with two alternative strategies under the same penalty strength: i) Random, which randomly selects 20% of turns for penalization; ii) High Entropy, which selects the 20% of turns with the highest entropy for penalization.

For your convenience, we provide the corresponding results below.

Table 14: Comparison of turn-level penalty strategies on SWE-bench Verified.

MethodResolved Rate (%)Tokens (M)TRS
Random36.85.414.5
High Entropy36.06.013.6
Low Entropy (HERO)40.04.517.1

From the results, penalizing low-entropy turns achieves higher resolution rates with lower token usage than random or high-entropy penalization, suggesting that the gains stem from learning more efficient patterns rather than merely a regularization effect.

To further address your concern, we conduct additional experiments about the unproductive behaviors under various penalty strategies. Specifically, we report the proportions of turns exhibiting unproductive behaviors, i.e., repeated tool calls, excessive exploration, and erroneous commands. For your convenience, we provide the corresponding results below.

Table A: Unproductive behaviors under different penalty strategies on SWE-bench Verified.

MethodAny Issue (%)Repeated Tool Calls (%)Excessive Exploration (%)Erroneous Commands (%)
Base (No Training)47.027.325.517.2
Random42.823.017.219.0
High Entropy46.626.924.417.5
Low Entropy (HERO)26.813.210.612.8

The results show that the proposed low-entropy penalty significantly reduces unproductive behaviors,futher supporting the suppression of inefficiency beyond a regularization effect.

A1-2: Frequency of penalizing necessary operations.

To answer your question, we conduct additional experiments to investigate the frequency of penalizing necessary operations. Specifically, we report the proportion of penalized turns during training and whether these turns exhibiting unproductive behaviors.

Table B: Frequency and effectiveness of low-entropy penalization during training.

StatisticProportion (%)
Penalized turns among all turns5.8
Penalized turns among all turns in trajectories with efficiency optimization activated22.5
Penalized low-entropy turns exhibiting necessary operations21.8
Penalized low-entropy turns exhibiting unproductive behaviors78.2

From the results, one could draw the following conclusions. On the one hand, with the efficiency gating, only a small proportion of turns receive low-entropy penalties, limiting the scope of potential suppression of necessary operations. On the other hand, most penalized low-entropy turns exhibit unproductive behaviors, while a minority may involve necessary operations, supporting the effectiveness of low-entropy selection.

Q2: Are the gains stable across seeds and against a tuned GRPO baseline with matched training compute?

A2: Thanks for your comments. In the following, we will answer your question one by one.

Stable improvement. As mentioned in Section 4.1, we repeat training three times with different random seeds and report the mean results. For your convenience, we provide the results of training with Qwen3.5-4B.

Table C: Performance of GRPO and HERO with Qwen3.5-4B across three training seeds.

GRPOGRPOHEROHERO
Training SeedResolved Rate (%)Tokens (M)Resolved Rate (%)Tokens (M)
136.26.541.04.7
238.47.039.44.2
337.66.939.64.6
Mean ± Std37.4 ± 1.16.8 ± 0.340.0 ± 0.94.5 ± 0.3

From the results, one could observe that HERO consistently outperforms GRPO in various seeds.

Fair comparison with GRPO. As described in Section 4.1 and Appendix A.2, all experimental settings of HERO are consistent with those of the baselines, including GRPO. Specifically, we follow the widely used training configurations in slime [A] and tune the hyperparameters for GRPO. After that, we train HERO using the same hyperparameters, as task-resolution advantage is computed using GRPO estimator. As a result, such a process not only ensures fair comparisons with GRPO, but also provides a well-tuned GRPO baseline.

Q3: Training data selection, overlap checks, and compute matching also need clearer documentation. How were the 640 training tasks sampled and checked for overlap with the evaluation sets?

A3: Thanks for your comments. In Appendix A.1, we have provided the language distribution of the 640 training tasks. Specifically, to improve multilingual coding capabilities, we sample tasks covering seven major programming language from three datasets, i.e., 352 tasks from SWE-Gym, 238 from Multi-SWE-bench, and 50 from SWE-rebench. In particular, the training set contains 352 Python tasks, 38 Java tasks, and 50 tasks each in C/C++, JavaScript/TypeScript, PHP, Go, and Rust.

To further address your concerns, we carry out more experiments on task-level overlap between training and evaluation sets. For your convenience, we provide the corresponding results below.

Table D: Task-level overlap between the training and evaluation sets.

DatasetSWE-bench VerifiedSWE-bench Multilingual
Training Set (640 tasks)00

The results confirm that no training tasks overlap with either evaluation set. In the revised manuscript, we have clarified the overlap between training and evaluation sets.

Q4: Limited novelty. HERO combines familiar mechanisms: efficiency gating, advantage clipping, length reward calibration, and entropy-based turn selection. Its methodological contribution is largely this combination within GRPO and its application to coding agents.

A4: Thanks for your comments. We argue that it is quite difficult to claim that any technique is entirely novel in the modern RL area. However, we would like to highlight that the mechanisms you mentioned are carefully tailored to the novel research problem and observations, rather than simply adopted in their vanilla forms. In the following, we clarify our novelty for the method designs from two main perspectives.

Resolution-prioritized policy optimization. Existing token-efficient RL methods focus on reducing token usage, struggling to achieve a good trade-off between token efficiency and task resolution. To address this, the proposed efficiency gating adaptively activates and scales efficiency optimization based on the task-solving capability. To further preserve resolution priority, we propose a novel resolution-first clipping strategy. Different from existing clipping methods that clip the probability ratio to stabilize policy optimization, our strategy aims to prioritize task resolution by ensuring the advantages for successful trajectories are no lower than those for unsuccessful ones.

Multi-granularity efficiency credit assignment. To facilitate token efficiency, we introduce a novel credit assignment mechanism from coarse-grained and fine-grained perspectives. From the coarse-grained perspective, we consider a characteristic of long-horizon SWE tasks, i.e., unresolved trajectories typically consume more tokens than resolved ones, thus avoiding conflating token efficiency with resolution performance. From the fine-grained perspective, we propose to adaptively penalize low-entropy turns, motivated by our novel observation that low-entropy turns are more prone to reasoning issues.

[A] THUDM. slime: An LLM training framework for RL scaling.

Dear Reviewer C5Ne,

As the discussion deadline approaches, we would like to know if our response has addressed your concerns. We have provided additional experiments and detailed clarifications to address your questions. If you have any further concerns or suggestions for the paper or our rebuttal, please let us know.

If your concerns have been satisfactorily addressed, we would be grateful if you would consider revising your score accordingly. Thank you again for the time and effort you dedicated to reviewing this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.