- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes TCG-GRPO for terminal agents. A critic trained on final task outcomes predicts success from partial interaction histories, and changes in its logits provide turn-level corrections to GRPO advantages. The method also adopts observation prediction and reuses the critic for action selection at inference, without additional selector training or environment snapshots.
Critical strengths
- The method addresses a concrete limitation of trajectory-level credit assignment while retaining GRPO's group-relative signal. Learning the critic from final outcomes avoids requiring additional process annotations.
- Experiments show improvements over GRPO and ECHO in both the 8B and 14B settings. Reusing the critic for inference-time action selection is also practically appealing and outperforms log-probability ranking at matched candidate counts.
Critical weaknesses
- The evidence that critic score changes reliably represent individual actions' contributions remains limited. Predicting eventual success does not by itself validate the particular weighted, clipped logit differences used here: changes after an observation may reflect newly revealed information rather than progress caused by the action. Section 4.4 provides useful descriptive evidence, but it relies on LLM annotations of 80 trajectories, and only 5 of 11 annotated setbacks receive negative credit.
- The empirical scope is narrow. All experiments use terminal/bash environments and two dense Qwen3 models. Although this scope is explicitly acknowledged, it limits the broader relevance of the findings. The mechanism is expressed through generic action–observation histories, yet its dependence on terminal-specific feedback and its applicability to other agent settings remain unclear.
- Presentation and reporting need more careful revision. Table 3 would be easier to interpret with separate columns for point estimates and 95% confidence intervals, alongside uncertainty for the difference from the shuffled baseline. There are also concrete omissions and inconsistencies: Appendix F.1 leaves the human-audit result as "[k] of the 20 labels"; Appendix A.1 announces three ordering constraints but lists only two; and Appendix C describes four evaluations of the 8B base model, whereas Section 4.1 and Table 2 report one. Appendix B also omits the offline critic warmup's data size and training duration. These issues make the experimental protocol harder to verify and reproduce.
Questions for authors
- Could you test shuffled turn corrections during policy training, or compare critic differences with action-value estimates obtained from repeated continuations at matched states, to validate the claimed local credit signal more directly?
- Can you assess another agent setting or model family, and clarify which properties of terminal feedback are necessary for the method to work?
3 comments
Thanks for your response. I appreciate that the additional results have strengthened the clarity of this paper. However, I still have the following concerns. (1) I still suspect that, whether the critic can accurately estimate the value of an action or step. Though this paper shows a performance gain, this gain can come from many other perspectives. Maybe the critic can partially represent the step value, it is hard to say they give exact value. For instance, I don't think it is convincing to say that, if step A has a critic score of 0.8 and step B has a critic score of 0.9, A must be better than B. In general, the whole framework is so similar to the existing process reward models (PRMs), which have been shown to be less effective. (2) The scope of this paper is still too narrow. Agentic tasks vary a lot in practice, and in my opinion, the ability to handle diverse tasks is crucial for agent research. The current terminal setting is not sufficient. Given the concerns above, I tend to maintain my score.
Thank you for your reply. We agree that the critic may not give accurate estimation, and Section 3.1 states each turn's credit as a noisy estimate. It only needs to be more informative than GRPO's uniform credit, which also rewards mistakes in successful rollouts. The shuffled-credit control you suggested confirms this: with each rollout's credits randomly reassigned across its turns, pass@1 falls from 72.8% to 65.1%. At inference, ranking candidate actions by critic score raises pass@1 by 6.4 points.
On PRMs, the problems usually reported (e.g., in the DeepSeek-R1 report) are that steps are hard to define, step labels are costly, and a fixed reward model gets exploited during RL. In our setting, turns are natural steps, the critic is trained only on final outcomes, and it is co-trained with the policy rather than fixed.
On scope, terminal agents are a standard testbed in recent work (e.g., Terminal-Bench, Endless Terminals, ECHO), and our training tasks span diverse categories. We agree that other agent domains are a natural next step.
We thank the reviewer for their detailed review and address their comments and questions below. We hope that the reviewer will consider raising their score in light of our response.
Thank you for the suggestions. We agree that predicting eventual success does not by itself validate our credit design, so our main evidence is its effect on task success. In turn-level best-of-N sampling (Section 4.5), action candidates share the same history and are scored before any gets executed, so the critic ranks them by the action part alone. Executing the critic’s top choice raises pass@1 from 72.8% to 79.2% at 8B (Best-of-N with log-probability ranking got only 71.6%) and from 73.0% to 80.6% at 14B, showing that the action score can help pick better actions. In training, adding the turn-level credit raises pass@1 by 8.0 points at 8B and 14.6 at 14B compared to ECHO.
To test whether this gain comes from giving credit to the right turns, we followed the reviewer's suggestion and resumed training from the step-140 checkpoint of 8B TCG-GRPO with each rollout's turn credits randomly shuffled across its turns, which keeps the credit values but breaks their link to specific turns (see Figure 13 in new Appendix F.4, page 28). The shuffled run reaches 65.1% pass@1 at step 280, compared with 72.8% for TCG-GRPO and 64.8% for ECHO. With credits shuffled, the model's final performance is similar to ECHO, indicating that the gain depends on which turns receive the credit.
We also agree that an observation can reveal information the action did not cause; this is why the action is scored separately before the command runs. Revealed bad news can still lower the observation part (new Appendix F.3), so we do not claim that each credit exactly measures the action's contribution, only that it is a useful turn-level signal, which the evaluation results support. We now state this explicitly in Section 3.1 (Line 154-155).
We agree that a broader evaluation would strengthen the paper and acknowledged this limitation in Section 6. We focused on terminal agents because they are among the most widely used and actively benchmarked LLM agents, and a bash interface already covers diverse work. For example, the Tmax dataset spans nine categories, including security, data processing, file operations, software engineering, and data science. Within this setting, the gains hold at both model sizes. Unfortunately, our compute is limited to a single H100 node, and we are unable to add another environment or model family during the rebuttal period.
Regarding which properties of terminal feedback are needed, we'd like to clarify that the method does not rely on terminal-specific features. It requires only multi-turn episodes split into actions and observations, a final outcome reward, and several rollouts per task, without gold answers or restorable states. It benefits from useful observations, such as error messages, file listings, and test output. Webpages and game states often share this property, so we expect the method to apply to these settings as well, and we leave their evaluation to future work.
Thank you for the suggestions. We have addressed each point in the revised PDF (changes in red). For Table 3, point estimates and 95% confidence intervals are now in separate columns, and we add the difference from the shuffled baseline with its own 95% CI from the same task-level resampling. Both difference CIs exclude zero (+12 points [4, 20] and +14 points [4, 23]). We’ve fixed typos and inconsistency in the Appendix (see line 777, 1025, and line 1299-1301). We also add offline critic warmup training details in Appendix B (line 899-902).