acceptodds
Under review as a conference paper at ICLR 2027

Turn-Level Critic Guidance for Training and Inference in Terminal Agents

Abstract

Terminal agents solve tasks by reasoning, executing commands, and adapting to feedback from the environment. They can make mistakes and recover within a single rollout, yet Group Relative Policy Optimization (GRPO) assigns the same advantage to every action in that rollout. Finer-grained methods typically derive advantages from a learned value function, or rely on matched or restorable states, known answers, or judges. We introduce GRPO with Turn-Level Critic Guidance (TCG-GRPO), which uses a co-trained critic to guide policy learning and action selection. The critic predicts eventual task success from the interaction history, and the change in its score across each turn provides turn-level credit that augments the GRPO advantage. We also adopt an auxiliary objective that trains the policy to predict environment observations. The resulting Qwen3-8B policy achieves 7.3% pass@1 on Terminal-Bench 2.1 and 72.8% on held-out Endless Terminals tasks, compared with 2.9% and 54.3% for GRPO. During inference, we reuse the co-trained critic for turn-level best-of-N action selection without additional training. With four candidates per turn, this achieves 79.2% pass@1 on Endless Terminals, compared with 71.6% for standard best-of-N with log-probability ranking.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic reinforcement learning, large language model, terminal-based agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hdz4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TCG-GRPO for terminal agents. A critic trained on final task outcomes predicts success from partial interaction histories, and changes in its logits provide turn-level corrections to GRPO advantages. The method also adopts observation prediction and reuses the critic for action selection at inference, without additional selector training or environment snapshots.

Critical strengths
  1. The method addresses a concrete limitation of trajectory-level credit assignment while retaining GRPO's group-relative signal. Learning the critic from final outcomes avoids requiring additional process annotations.
  2. Experiments show improvements over GRPO and ECHO in both the 8B and 14B settings. Reusing the critic for inference-time action selection is also practically appealing and outperforms log-probability ranking at matched candidate counts.
Critical weaknesses
  1. The evidence that critic score changes reliably represent individual actions' contributions remains limited. Predicting eventual success does not by itself validate the particular weighted, clipped logit differences used here: changes after an observation may reflect newly revealed information rather than progress caused by the action. Section 4.4 provides useful descriptive evidence, but it relies on LLM annotations of 80 trajectories, and only 5 of 11 annotated setbacks receive negative credit.
  2. The empirical scope is narrow. All experiments use terminal/bash environments and two dense Qwen3 models. Although this scope is explicitly acknowledged, it limits the broader relevance of the findings. The mechanism is expressed through generic action–observation histories, yet its dependence on terminal-specific feedback and its applicability to other agent settings remain unclear.
  3. Presentation and reporting need more careful revision. Table 3 would be easier to interpret with separate columns for point estimates and 95% confidence intervals, alongside uncertainty for the difference from the shuffled baseline. There are also concrete omissions and inconsistencies: Appendix F.1 leaves the human-audit result as "[k] of the 20 labels"; Appendix A.1 announces three ordering constraints but lists only two; and Appendix C describes four evaluations of the 8B base model, whereas Section 4.1 and Table 2 report one. Appendix B also omits the offline critic warmup's data size and training duration. These issues make the experimental protocol harder to verify and reproduce.
Questions for authors
  1. Could you test shuffled turn corrections during policy training, or compare critic differences with action-value estimates obtained from repeated continuations at matched states, to validate the claimed local credit signal more directly?
  2. Can you assess another agent setting or model family, and clarify which properties of terminal feedback are necessary for the method to work?
3 comments

We thank the reviewer for their detailed review and address their comments and questions below. We hope that the reviewer will consider raising their score in light of our response.

  1. The evidence that critic score changes reliably represent individual actions' contributions remains limited. Predicting eventual success does not by itself validate the particular weighted, clipped logit differences used here: changes after an observation may reflect newly revealed information rather than progress caused by the action. / Could you test shuffled turn corrections during policy training … to validate the claimed local credit signal more directly?

Thank you for the suggestions. We agree that predicting eventual success does not by itself validate our credit design, so our main evidence is its effect on task success. In turn-level best-of-N sampling (Section 4.5), action candidates share the same history and are scored before any gets executed, so the critic ranks them by the action part alone. Executing the critic’s top choice raises pass@1 from 72.8% to 79.2% at 8B (Best-of-N with log-probability ranking got only 71.6%) and from 73.0% to 80.6% at 14B, showing that the action score can help pick better actions. In training, adding the turn-level credit raises pass@1 by 8.0 points at 8B and 14.6 at 14B compared to ECHO.

To test whether this gain comes from giving credit to the right turns, we followed the reviewer's suggestion and resumed training from the step-140 checkpoint of 8B TCG-GRPO with each rollout's turn credits randomly shuffled across its turns, which keeps the credit values but breaks their link to specific turns (see Figure 13 in new Appendix F.4, page 28). The shuffled run reaches 65.1% pass@1 at step 280, compared with 72.8% for TCG-GRPO and 64.8% for ECHO. With credits shuffled, the model's final performance is similar to ECHO, indicating that the gain depends on which turns receive the credit.

We also agree that an observation can reveal information the action did not cause; this is why the action is scored separately before the command runs. Revealed bad news can still lower the observation part (new Appendix F.3), so we do not claim that each credit exactly measures the action's contribution, only that it is a useful turn-level signal, which the evaluation results support. We now state this explicitly in Section 3.1 (Line 154-155).

  1. The empirical scope is narrow. All experiments use terminal/bash environments and two dense Qwen3 models … its dependence on terminal-specific feedback and its applicability to other agent settings remain unclear. / Can you assess another agent setting or model family, and clarify which properties of terminal feedback are necessary for the method to work?

We agree that a broader evaluation would strengthen the paper and acknowledged this limitation in Section 6. We focused on terminal agents because they are among the most widely used and actively benchmarked LLM agents, and a bash interface already covers diverse work. For example, the Tmax dataset spans nine categories, including security, data processing, file operations, software engineering, and data science. Within this setting, the gains hold at both model sizes. Unfortunately, our compute is limited to a single H100 node, and we are unable to add another environment or model family during the rebuttal period.

Regarding which properties of terminal feedback are needed, we'd like to clarify that the method does not rely on terminal-specific features. It requires only multi-turn episodes split into actions and observations, a final outcome reward, and several rollouts per task, without gold answers or restorable states. It benefits from useful observations, such as error messages, file listings, and test output. Webpages and game states often share this property, so we expect the method to apply to these settings as well, and we leave their evaluation to future work.

  1. Presentation and reporting need more careful revision. …

Thank you for the suggestions. We have addressed each point in the revised PDF (changes in red). For Table 3, point estimates and 95% confidence intervals are now in separate columns, and we add the difference from the shuffled baseline with its own 95% CI from the same task-level resampling. Both difference CIs exclude zero (+12 points [4, 20] and +14 points [4, 23]). We’ve fixed typos and inconsistency in the Appendix (see line 777, 1025, and line 1299-1301). We also add offline critic warmup training details in Appendix B (line 899-902).

Thanks for your response. I appreciate that the additional results have strengthened the clarity of this paper. However, I still have the following concerns. (1) I still suspect that, whether the critic can accurately estimate the value of an action or step. Though this paper shows a performance gain, this gain can come from many other perspectives. Maybe the critic can partially represent the step value, it is hard to say they give exact value. For instance, I don't think it is convincing to say that, if step A has a critic score of 0.8 and step B has a critic score of 0.9, A must be better than B. In general, the whole framework is so similar to the existing process reward models (PRMs), which have been shown to be less effective. (2) The scope of this paper is still too narrow. Agentic tasks vary a lot in practice, and in my opinion, the ability to handle diverse tasks is crucial for agent research. The current terminal setting is not sufficient. Given the concerns above, I tend to maintain my score.

Thank you for your reply. We agree that the critic may not give accurate estimation, and Section 3.1 states each turn's credit as a noisy estimate. It only needs to be more informative than GRPO's uniform credit, which also rewards mistakes in successful rollouts. The shuffled-credit control you suggested confirms this: with each rollout's credits randomly reassigned across its turns, pass@1 falls from 72.8% to 65.1%. At inference, ranking candidate actions by critic score raises pass@1 by 6.4 points.

On PRMs, the problems usually reported (e.g., in the DeepSeek-R1 report) are that steps are hard to define, step labels are costly, and a fixed reward model gets exploited during RL. In our setting, turns are natural steps, the critic is trained only on final outcomes, and it is co-trained with the policy rather than fixed.

On scope, terminal agents are a standard testbed in recent work (e.g., Terminal-Bench, Endless Terminals, ECHO), and our training tasks span diverse categories. We agree that other agent domains are a natural next step.

Reviewer_bdgcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies reinforcement learning for multi-turn terminal agents, where a model reasons, executes shell commands, and adapts to environment feedback, with only a binary outcome reward at the end of each rollout. The authors observe that GRPO assigns the same trajectory-level advantage to every turn, which gives poor credit assignment in long interactions. They propose TCG-GRPO, which co-trains a critic that predicts task-success probability from the interaction history. The critic’s change in logit across each turn, split into an action part and an observation part, is weighted and clipped to give a turn-level credit that is added, with a mixing coefficient, to the GRPO advantage of that turn’s action tokens (Eq. 3-4).

The method also includes ECHO’s auxiliary loss for predicting environment observations, and optionally reuses the frozen critic at inference for turn-level best-of-N action selection. Experiments use Qwen3-8B and Qwen3-14B. On held-out tasks from the training distribution (Endless Terminals for 8B, TMAX-8000 for 14B), TCG-GRPO improves pass@1 over GRPO and ECHO, for example 72.8 vs. 54.3 (GRPO) and 64.8 (ECHO) at 8B and 73.1 vs. 61.1 and 58.5 at 14B. On Terminal-Bench 2.1, the 8B model reaches 7.3% pass@1 vs. 2.9% (GRPO) and 5.5% (ECHO), with smaller and less clear-cut gains at 14B.

Critical strengths

The main strength is:

The in-distribution improvement is large and consistent against the like-for-like baseline. Since TCG-GRPO includes ECHO's observation-prediction loss, ECHO is the matched comparison, and TCG-GRPO beats it at both scales on held-out tasks (Table 1, 256 rollouts per step for all methods): 72.8 vs. 64.8 pass@1 at 8B and 73.1 vs. 58.5 at 14B, with pass@8 also higher (88.8 vs. 85.7; 94.0 vs. 87.0). The advantage is not an end-point artifact: in the training curves (Fig. 2), TCG-GRPO stays above all baselines from roughly step 140 at 8B and step 100 at 14B, and it reaches the baselines' final performance in noticeably fewer steps. The problem it targets, credit assignment under sparse outcome rewards in long terminal interactions, is well motivated.

Critical weaknesses

The main weaknesses are:

  1. Out-of-distribution transfer is marginal. The only evidence beyond the training distribution is Terminal-Bench 2.1 (89 tasks), and it is weak. The abstract compares 7.3% against GRPO (2.9%), but TCG-GRPO also includes ECHO’s observation-prediction loss; the like-for-like baseline is ECHO (5.51%), so the 8B gap is about 1.8 points, fewer than two tasks’ worth. At 14B, pass@1 is 12.96±0.79 vs. 11.76±1.01 for GRPO (a gap of about one standard deviation), and pass@5 for TCG-GRPO (22.85±1.72) is identical to GRPO’s, while ECHO is higher (23.22). The 8B base model is evaluated only once, and the reported standard deviations come from repeated evaluations of fixed checkpoints, so they exclude training-seed variance. The large gains in Table 1 and Figure 2 are on held-out tasks from the training distribution (Endless Terminals / TMAX-8000). These show in-distribution improvement, but the claim of more broadly capable terminal agents is not supported.

  2. Co-training the critic on its own on-policy rollouts raises leakage, exploitation and exploration concerns that are not addressed. Notation (Sec. 3.1): sks_k is the history before turn kk, uk=sk⊕aku_k = s_k \oplus a_k the history after the action, sk+1=uk⊕oks_{k+1} = u_k \oplus o_k after the observation. The critic Φ\Phi outputs a logit and is trained with binary cross-entropy against the final outcome R∈{0,1}R \in \{0,1\}. The token advantage is $ A_{i,t} = A_i^{\mathrm{GRPO}} + \lambda,\tilde\delta_{i,k(t)}, \qquad \tilde\delta_k = \operatorname{clip}\Big(w_a,[\Phi(u_k)-\Phi(s_k)] + w_o,[\Phi(s_{k+1})-\Phi(u_k)],\ \pm c\Big). $ These conditions may happen:

  • Possible label leakage. The critic regresses on RR for histories from the same rollouts whose advantages it then shapes. If Φ\Phi is evaluated on a trajectory τ\tau from its own training batch, a high-capacity LLM critic can fit R(τ)R(\tau) directly, i.e. Φ(sk)=logit⁡Vπ(sk)+ϵk(R)\Phi(s_k) = \operatorname{logit} V^{\pi}(s_k) + \epsilon_k(R) with an error term that depends on the label. Then δ~k\tilde\delta_k carries the trajectory's own outcome and duplicates AiGRPOA_i^{\mathrm{GRPO}} instead of adding independent turn-level information. Standard PPO/GAE computes advantages with the pre-update critic [Schulman et al., 2017; Schulman et al., 2016]; the paper should state that this ordering (or a lagged or cross-fitted critic) is used.

  • Exploitation through a feedback loop. With wa=wo=1w_a = w_o = 1 and no clipping, ∑k=0K−1δ~k=Φ(sK)−Φ(s0)\sum_{k=0}^{K-1}\tilde\delta_k = \Phi(s_K) - \Phi(s_0), so the policy is rewarded for reaching histories where Φ\Phi is high, whether or not success is actually more likely. Because Φ\Phi is refit on the policy's own samples, its errors are reinforced rather than corrected, the same over-optimization dynamic documented for learned reward models [Gao et al., 2023]. Clipping bounds per-turn credit but does not remove this incentive.

  • Degenerate groups. If all GG rollouts of a task share the same outcome, then AiGRPO=0A_i^{\mathrm{GRPO}} = 0 and Ai,t=λ δ~i,k(t)A_{i,t} = \lambda\,\tilde\delta_{i,k(t)}, so the entire update is driven by the critic. For all-failure groups the critic has only seen R=0R = 0 for that task, so the signal comes from cross-task generalization with no ground truth. Prior work finds learned value networks to be unreliable credit estimators for LLMs [Kazemnejad et al., 2024], and the paper gives no evidence that δ~k\tilde\delta_k tracks true turn quality.

  • Exploration. The critic only observes outcomes of actions the policy takes with non-negligible probability, so credit for novel actions is extrapolation. At inference, selecting ak⋆=arg⁡max⁡j≤Nσ(Φ(sk⊕ak(j)))a_k^\star = \arg\max_{j \le N}\sigma\big(\Phi(s_k \oplus a_k^{(j)})\big) suffers from the winner's curse: E[max⁡jp^j]≥max⁡jE[p^j]\mathbb{E}[\max_j \hat p_j] \ge \max_j \mathbb{E}[\hat p_j], so scores of selected actions are optimistically biased, and the selection further concentrates the visited states. Entropy collapse is a known failure mode of RL for LLMs [Cui et al., 2025], yet the paper reports no policy entropy or action diversity.

  1. Novelty is incremental and the nearest baselines are missing. Eq. (3)–(4) add a clipped one-step critic difference to the trajectory-level GRPO advantage. Prior work on multi-turn LLM agents already provides turn- or step-level credit through different routes: an off-policy hierarchical critic (ArCHer, [Zhou et al., 2024]), a critic trained with privileged training-time information (SWEET-RL, [Zhou et al., 2025]), and critic-free step-level groups built from repeated environment states (GiGPO, [Feng et al., 2025]). The paper compares only against GRPO, ECHO and (at 8B only) SAO, so it cannot be determined whether the gain is specific to the proposed δ~k\tilde\delta_k or comes from any turn-level credit signal. Please compare against at least one such method, or explain why it is inapplicable (e.g., GiGPO relies on repeated environment states).

  2. Little agent- or domain-specific insight, and the motivation is not validated. The method is a generic outcome-trained critic over text histories; apart from the turn segmentation and the no-snapshot selection procedure, it does not use terminal structure (exit codes, filesystem state, partial test progress, error recovery), and the formulas would be unchanged for any multi-turn task, yet it is evaluated only on terminal tasks. The motivating claim that agents “make mistakes and recover within a single rollout” is not supported by case studies of δ~k\tilde\delta_k at error and recovery turns, by the correlation of δ~k\tilde\delta_k with Monte Carlo value differences, or by gains as a function of trajectory length.

Questions for authors

R1. Out-of-distribution gain on Terminal-Bench 2.1 is small (W1)

  • Training-seed variance. Report mean ± std over at least 3 training runs per method and scale; the current standard deviations come from repeated evaluations of fixed checkpoints and exclude training variance.
  • Paired significance on the 89 tasks. Report the pass@1 difference between TCG-GRPO and ECHO with a paired bootstrap 95% CI over tasks, plus a per-task table of which tasks move from 0 to non-zero success. The like-for-like gap at 8B is only about 1.8 points (7.30 vs. 5.51), under two tasks' worth.
  • Table 2 consistency. The 14B pass@5 of TCG-GRPO (22.85±1.7222.85\pm1.72) is identical to GRPO's; please confirm this is correct.

R2. Co-training leakage and exploration (W2)

Leakage

  • Cross-fitted critic. Compute δ~i,k\tilde\delta_{i,k} for each rollout with a critic that was not trained on that rollout (split each batch into folds, or use a lagged critic, i.e. Φ\Phi with the parameters it had mm policy steps earlier), and state whether advantages are computed with the critic before or after it is updated on the same batch. If the gain over ECHO survives, leakage is not the source.
  • Label-permutation control. Train the critic on outcome labels permuted across all rollouts in the batch (permuting only within a task would leave all-fail and all-success groups unchanged), and compute δ~i,k\tilde\delta_{i,k} from this critic. The gain over ECHO should then disappear (or reverse); if it does not, the improvement is not coming from outcome-informed turn-level credit.

Exploration

  • Entropy and diversity. Plot action-token policy entropy over training for all methods (entropy collapse is a known failure mode of RL for LLMs [Cui et al., 2025]), together with the number of distinct command sequences per task.
  • Larger sampling budgets. Sample n≥32n \ge 32 attempts per task and report pass@kk for kk up to 32 at fixed training steps, using the unbiased estimator pass@k^=Etasks[1−(n−ck)/(nk)],\widehat{\mathrm{pass@}k} = \mathbb{E}_{\text{tasks}}\Big[1 - \binom{n-c}{k}\Big/\binom{n}{k}\Big], where cc is the number of successful attempts of a task. This tests whether the gain reflects sharpening of already-solvable tasks or a wider solved set.

R3. Prior process-level credit assignment and missing baselines (W3&W4)

  • Nearest baseline. Compare under matched rollouts per step against at least one turn/step-level method: GiGPO [Feng et al., 2025], an ArCHer-style critic [Zhou et al., 2024], or a SWEET-RL-style critic [Zhou et al., 2025]. If a method is inapplicable, justify it with data: e.g., for GiGPO, report the fraction of terminal-task steps whose state recurs across rollouts of the same task, since its step-level groups rely on repeated environment states.
  • Either broaden or narrow the claim. Evaluate the same recipe on at least one non-terminal multi-turn agent benchmark (e.g., ALFWorld or WebShop, used by GiGPO), or restrict the title and abstract to terminal agents.
  • Show what is terminal-specific. If the method is meant to exploit terminal structure, show it: per-turn δ~i,k\tilde\delta_{i,k} at error and recovery turns (expected negative at a turn that produces a non-zero exit code, positive at the turn that fixes it), and gains as a function of trajectory length. This directly tests the motivating claim that agents "make mistakes and recover within a single rollout".

References

  • Schulman, Wolski, Dhariwal, Radford, Klimov, 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347.
  • Schulman, Moritz, Levine, Jordan, Abbeel, 2016. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016, arXiv:1506.02438.
  • Cui et al., 2025. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617.
  • Zhou, Zanette, Pan, Levine, Kumar, 2024. ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. ICML 2024, arXiv:2402.19446.
  • Zhou, Jiang, Tian, Weston, Levine, Sukhbaatar, Li, 2025. SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks. arXiv:2503.15478.
  • Feng, Xue, Liu, An, 2025. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025, arXiv:2505.10978.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part I)

We thank the reviewer for their review and address their comments and questions below. We hope that the reviewer will consider raising their score in light of our response. We first note several statements in the review that do not match the paper:

  • Pre-update critic (W2). The review states that the leakage concern is "not addressed" and asks us to state that credit is computed before the critic update. Appendix A.1 already states: "The critic scores each batch before its own update, so credit always comes from the pre-update critic."
  • Baselines (W3). The review states that we compare only against GRPO, ECHO, and SAO, with SAO at 8B only. SAO is compared at both 8B (Section 4.2) and 14B (Appendix C). We also compare against Turn-PPO, a turn-level credit assignment method (Appendix C), and PaW, which co-trains the policy with a world model (Appendix E).
  • Case studies (W4). The review states that the motivation that agents "make mistakes and recover within a single rollout" is not supported by case studies of credit at error and recovery turns. Section 4.4 presents such a case study, with credit −0.469 at the mistake turn and +0.845 at the repair turn.
  • Claims (W1). The review states that "the claim of more broadly capable terminal agents is not supported." The paper does not make this claim. It reports held-out and Terminal-Bench 2.1 results and describes the 14B transfer results as weaker evidence (Section 4.2).
  • Scope (R3). The review asks us to either evaluate a non-terminal benchmark or "restrict the title and abstract to terminal agents." Both are already restricted: the title ends with "Terminal Agents," and the abstract is framed around terminal agents.

1. Out-of-distribution transfer (W1, R1). We agree that the Terminal-Bench 2.1 gains are modest. The benchmark is hard even for much larger model (with the same harness, gpt-oss-120b gets 26.2%). Against the strongest baseline at each scale, TCG-GRPO improves pass@1 from 5.51% to 7.30% at 8B (ECHO) and from 11.76% to 12.96% at 14B (GRPO); the introduction and Table 2 report both comparisons. We trained 8B TCG-GRPO with an additional seed, and two jobs (seed=1,42) reached nearly identical held-out pass@1 (73.0% and 72.8%). The identical 14B pass@5 values are correct: across the three repeats, GRPO solves 20, 22, and 19 tasks and TCG-GRPO solves 19, 20, and 22 (see Appendix C).

2. Co-training the critic (W2, R2).

  • Exploitation. Gao et al. (2023) optimize against a reward model that is frozen during policy training. Our critic is refit at every step on the policy's newest rollouts with verifier outcomes, so if the policy reaches high-scoring histories that fail, the critic is corrected on exactly those histories; Gao et al. recommend such online retraining. The true outcome also stays in the GRPO advantage. Empirically, our held-out pass@1 shows no rise-then-fall pattern and ends at or near its peak at both scales (Figure 2).
  • Degenerate groups. We view this as an advantage of the method. When all rollouts of a task fail, GRPO provides no learning signal, while the turn-level credit, learned from verifier outcomes across all tasks, can still reward progress within those failures. In failed rollouts, progress turns receive positive credit 14 points more often than under a within-trajectory shuffle (95% CI [4, 23]; Table 3), and shuffling the credits across turns during training lowers pass@1 from 72.8% to 65.1%, close to ECHO's 64.8% (Appendix F.4).
  • Exploration. The critic always scores actions sampled from the current policy, the distribution it is trained on. The winner's curse would bias the critic's scores of selected actions, but verified task success rises and then converges as N grows (76.7%, 79.2%, and 79.0% at N = 2, 4, and 8 for the 8B model). TCG-GRPO also raises pass@8 (88.8% vs. 80.6% for GRPO and 85.7% for ECHO at 8B; 94.0% vs. 87.0% at 14B), so it widens the set of solved tasks rather than only sharpening already-solvable ones.
AuthorsOpenReview ↗

Rebuttal by Authors (Part II)

3. Novelty and nearest baselines (W3, R3). The critic-based baselines SAO and Turn-PPO both underperform TCG-GRPO (55.2% and 47.4% vs. 72.8% at 8B; 53.5% vs. 65.4% for SAO at 300 steps at 14B), so the gain does not come from turn-level credit in general. GiGPO builds step-level groups only from states that recur across rollouts. In terminal tasks, command outputs are long and depend on earlier commands, so we expect such recurrences to be rare, leaving most groups with a single step and reducing GiGPO largely to GRPO. SWEET-RL trains its critic with privileged training-time information, which our outcome-only setting does not assume, and ArCHer is an off-policy hierarchical actor-critic. Within the on-policy regime, SAO and Turn-PPO are the closest critic-based baselines. TCG-GRPO differs from these methods in keeping GRPO's group-relative anchor, scoring actions and observations separately, and reusing the same critic for snapshot-free action selection.

4. Domain-specific insight and motivation (W4, R3). The method is deliberately general: instead of hand-engineered signals such as exit codes, the critic reads the raw terminal output, which enters through the observation part of the credit. The snapshot-free design reflects a terminal constraint, since container states are costly to snapshot and restore. Beyond the case studies above, Table 3 and the shuffled-credit control support the motivation more systematically. A Monte Carlo comparison would require many continuations from each intermediate state, which exceeded our compute during the rebuttal period. Instead, turn-level best-of-N tests whether the critic’s scores reflect how good each action is: at each turn, the critic scores several candidate actions from the same history, and executing the highest-scored one raises verified pass@1 from 72.8% to 79.2% (log-probability ranking: 71.6%), which would not happen if the scores did not reflect action quality.

Reviewer_bdgcOpenReview ↗

Response to author

Thank you for the response and for clarifying my other questions.

At 8B, TCG-GRPO improves pass@1 over ECHO by 1.79 percentage points on Terminal-Bench 2.1. Without the requested paired confidence interval and per-task results, it remains unclear how reliable this gain is and which tasks account for it. The additional results on the in-distribution held-out set do not resolve this uncertainty about out-of-distribution performance.

Given the limited evidence for a reliable improvement on Terminal-Bench 2.1, I am maintaining my original score.

Thank you for following up. We've added a paired bootstrap CI and the per-task comparison to the revised PDF (new Appendix C.2 and Table 8).

For each 8B checkpoint, we pooled its four evaluation runs (20 attempts on each of the 89 tasks). TCG-GRPO succeeds on 130 of the 1,780 attempts (7.30%) and ECHO on 98 (5.51%), a gap of 1.8%. A paired bootstrap over tasks (100k resamples) gives a 95% CI of [0.34%, 3.43%] for this gap, which excludes zero. Against GRPO, the gap is 4.4% (95% CI [2.19%, 7.08%]).

As for which tasks drive the gain, neither model solves 68 of the 89 tasks. Among the remaining 21, TCG-GRPO does better on 12, worse on 6, and ties on 3. Five tasks below account for most (27 of the 32) extra successes:

TaskECHOTCG-GRPO
path-tracing-reverse4/2010/20
hf-model-inference11/2017/20
fix-ocaml-gc1/206/20
modernize-scientific-stack15/2020/20
portfolio-optimization4/209/20

Most of TCG-GRPO's gain comes from solving the same tasks more reliably. The two models solve nearly the same set of tasks (two are solved only by TCG-GRPO and three only by ECHO), and TCG-GRPO succeeds more often on most of the shared ones.

Reviewer_MRugOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TCG-GRPO, a framework for improving credit assignment in multi-turn terminal agents. The authors argue that standard GRPO assigns the same trajectory-level advantage to every action, even when a trajectory contains mistakes and subsequent corrections. To address this limitation, the method trains a critic to predict eventual task success from partial interaction histories, using only final trajectory outcomes as supervision. Changes in the critic's predictions after each action and environment observation are then used to construct turn-level advantage corrections, which are combined with the original GRPO advantages. The method also adopts an auxiliary observation-prediction objective from ECHO. During inference, the trained critic is reused to select promising actions from multiple candidates. Experiments on Endless Terminals, Tmax, and Terminal-Bench 2.1 show improvements over standard GRPO and several related baselines.

Critical strengths

The authors identify an important limitation of trajectory-level credit assignment in multi-turn agents, where successful trajectories may contain harmful actions and unsuccessful trajectories may still contain useful progress.

Combining a trajectory-level GRPO advantage with critic-based turn-level corrections is simple to understand and implement within existing RL frameworks. The idea of using the same critic for both policy optimization and inference-time action selection is also useful, since it avoids training a separate selection model.

The method achieves substantial improvements over GRPO and ECHO on held-out terminal tasks across two model scales. The authors also evaluate transfer to Terminal-Bench 2.1, investigate the observation-prediction objective, and provide qualitative examples of mistakes and subsequent corrections. I appreciate that the paper attempts to evaluate whether the learned critic can distinguish progress from setbacks within individual trajectories.

Critical weaknesses

My main concern is that the critic is trained using the final binary outcome of each trajectory, with the same outcome label assigned to every intermediate prefix. Although this is a reasonable way to train a value predictor, accurate prediction of eventual success does not necessarily imply accurate attribution of credit to individual actions. For example, a diagnostic command may reveal an unfavorable environment condition, causing the critic's predicted success probability to decrease, even though the command itself was useful and necessary for subsequent progress. Similarly, an intermediate action may have delayed benefits that are not immediately reflected in the critic's predictions. Therefore, changes in predicted success may partly reflect newly acquired information or correlations with outcomes rather than the actual contribution of the action. I think this distinction is important because accurate turn-level credit assignment is the central motivation and the story of the paper.

Section 4.4 evaluates critic scores against LLM-annotated progress and setback labels. However, only 60% of progress turns in failed trajectories receive positive credit, and only 5 of 11 annotated setbacks receive negative credit. These results suggest that the critic has some ability to recognize "intermediate progress", but its reliability in identifying harmful actions is less convincing.

TCG-GRPO requires training an additional critic with the same backbone size as the policy, as well as evaluating partial trajectories to construct turn-level advantages. Although the authors report that critic scoring and updates account for a relatively small fraction of training-step time, the main comparisons match rollout counts rather than total training computation. A clearer compute-matched evaluation would be useful.

I think the paper would benefit from a more systematic evaluation of turn-level credit quality, especially for diagnostic actions and actions with delayed benefits. An ablation separating action-based and observation-based score changes would also help clarify which component provides useful supervision. However currently me as a reviewer, still not sure that how to design a experiment to establish this part well, so I would leave this part alone and not treat it as a strict "weakness". I would really appreciate that if the author could provide more interesting experiments or analysis to prove this part.

Questions for authors

see the weakness part above

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the careful review and for recognizing the contributions of our paper. We are glad that the reviewer finds that our paper identifies "an important limitation of trajectory-level credit assignment in multi-turn agents," and that the method is "simple to understand and implement within existing RL frameworks," and "achieves substantial improvements over GRPO and ECHO on held-out terminal tasks across two model scales." Revisions in the updated PDF are marked in red, and we address the reviewer's comments below.

  1. Changes in predicted success may partly reflect newly acquired information or correlations with outcomes rather than the actual contribution of the action. / … a more systematic evaluation of turn-level credit quality, especially for diagnostic actions and actions with delayed benefits. An ablation separating action-based and observation-based score changes would also help …

Thank you for the suggestions. Because the critic predicts the final outcome from every prefix, an action whose benefit comes later, such as creating a file that a later step needs, can already receive positive credit when it is taken. We also score each action before the command runs, so what a diagnostic reveals affects only the observation part of the credit. We'd like to note that finding a problem is not always penalized: the problem existed before the diagnostic, and finding it lets the agent fix it. In a new example we add in Appendix F.2, inspecting CHANGELOG.md reveals a misplaced release entry, and the critic's predicted success rises from 0.102 before the action to 0.148 after it and 0.202 after the observation.

Following the reviewer's suggestion, we add an analysis of diagnostic and preparatory turns (new Appendix F.3). Useful preparation receives positive credit more often than redundant preparation (51% vs. 37% at step 280; 69% vs. 24% at step 140), though the critic does not foresee which preparation will pay off later. As the reviewer suggested, revealed bad news can lower a useful diagnostic's credit: diagnostics that reveal a problem receive positive credit in about 40% of cases. On both kinds of useful turns, the action part is positive more often than the observation part (48% vs. 34% for diagnostics and 64% vs. 47% for preparation at step 280).

We also added a training ablation (new Appendix F.4). From the step-140 checkpoint of 8B TCG-GRPO, we continued training with each rollout's turn credits randomly shuffled across its turns, which keeps the credit values but breaks their link to specific turns. The shuffled run reaches 65.1% pass@1 at step 280, compared with 72.8% for TCG-GRPO and 64.8% for ECHO, so the gain depends on which turns receive the credit.

  1. Only 60% of progress turns in failed trajectories receive positive credit, and only 5 of 11 annotated setbacks receive negative credit … its reliability in identifying harmful actions is less convincing.

We agree that the critic does not perfectly recognize intermediate progress or harmful actions, but its credit is still informative. In both successful and failed rollouts, progress turns receive positive credit 12–19 points more often than when the credits are shuffled among each trajectory's annotated turns (Sec. 4.4). This holds at step 280 (Table 3; both 95% CIs of the difference exclude zero) and at an earlier checkpoint that we add (Appendix F.1):

CheckpointSuccessful rolloutsFailed rollouts
Step 14062% (shuffled: 43%)47% (shuffled: 31%)
Step 28075% (shuffled: 63%)60% (shuffled: 46%)

For harmful actions, we agree that the evidence is limited. The step-280 samples have only 11 setbacks, too few to draw a conclusion. Since the critic is trained on the policy's own rollouts, it sees fewer harmful actions as the policy improves, which may make them harder to recognize. Consistent with this, the step-140 critic gives negative credit to all 5 setbacks in its sample, although this sample is also small. We'd like to note that the credit does not need to be perfect: it is added to GRPO's advantage, which gives all turns of a rollout the same credit, so it only needs to be more informative than uniform credit, and the gains over ECHO show that it is.

  1. The main comparisons match rollout counts rather than total training computation. A clearer compute-matched evaluation would be useful.

Thank you for the suggestion. We've added a compute-matched comparison of the 8B models (new Appendix C.1), counting online training FLOPs, including the critic's. TCG-GRPO uses about 2× GRPO's FLOPs over 280 steps, so it trails early in training. It overtakes SAO after about 5e+19 FLOPs and reaches GRPO's and ECHO's final pass@1 with about 1.3× and 1.1× their total FLOPs. In wall-clock time, the overhead is small: rollout generation dominates each step, and critic scoring and updates take under 10% of step time (Sec. 4.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.