acceptodds
Under review as a conference paper at ICLR 2027

CAGPO: Critic-Anchored Group Policy Optimization for Multi-Turn LLM Agent Reinforcement Learning

Abstract

Reinforcement learning for LLM agents on long-horizon, multi-turn tasks requires assigning credit not just to whole trajectories but to the specific turns, or even tokens, that actually drive progress. GRPO-style group-relative methods estimate advantage without a critic, but comparing only at the trajectory level discards this fine-grained signal: the policy has no way to tell which actions within a long trajectory mattered. Critic-based methods can recover this signal through value estimation, but jointly training a critic with the policy in agentic settings tends to be unstable and prone to collapse. We propose CAGPO (Critic-Anchored Group PPO), which anchors long-horizon value estimation with a critic while keeping group-relative comparison as the source of fine-grained credit. CAGPO splits the advantage into a macro component, computed via temporal-difference learning along the executed trajectory at either turn or token granularity, and a micro component that runs group-relative comparison among extra candidate responses sampled, but never executed, at selected high-entropy positions. Across several long-horizon agentic benchmarks, CAGPO outperforms both critic-free and hierarchical group-relative baselines, with the largest gains on tasks that require sustained, multi-step completion, surpassing the current state of the art.

Primary area:
reinforcement learning
Keywords:
Multi-Turn LLM Agent, Agentic LLM, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tsjNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a new approach that combines token-level GAE on executed trajectories with a micro branch. For this method, at high-entropy turns the policy samples extra candidate responses, a shared critic scores them without environment execution, and group-normalized scores give candidate advantages used in the actor update. The authors also evaluate this method using multiple benchmarks.

Critical strengths

The underlying research problem is practically useful. In many real settings, it cannot replay an episode, and this work may be able to provide some insights into this.

Critical weaknesses
  1. The main contribution of the micro mechanism is not supported. At a certain step, zero-advantage candidate rows already lift WebShop success, but the critic adds only a very little. It also shows a near-zero Spearman correlation, and the majority of groups map to one action.
  2. The motivating setting is never tested. The pitch is non-replayable or costly environments, yet every experiment uses resettable simulators. The method needs a trained critic plus extra generation on many candidate rows per step, but there is no compute-matched or wall-clock comparison with GiGPO-style baselines.
Questions for authors

It might be helpful for discussing why the progress loss adds little.

Reviewer_RzhnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CAGPO extends PPO training for multi-turn LLM agents by combining token-level GAE on executed trajectories with critic-based comparisons of alternative responses that are not executed in the environment. At selected high-entropy turns, the policy samples alternatives from the same context. A shared critic scores these candidates, and the scores are normalized within each group to provide additional policy advantages. Both real and candidate responses train the actor, while only real trajectories train the critic. An optional critic loss encourages adjacent-turn value changes to follow the direction of changes in return targets. Experiments on ALFWorld, WebShop, and ScienceWorld report gains from candidate augmentation in most tested configurations at matched real-interaction budgets, with score-source controls and an offline ranking probe.

Critical strengths
  1. The separation between real experience and unexecuted alternatives is useful. Sections 4.2–4.4 specify how candidates enter the actor objective while remaining excluded from critic targets. This provides a concrete way to obtain local policy supervision when collecting comparable executed alternatives is expensive. The paper also acknowledges the dependence on critic estimates and the extra model computation, which makes the contribution’s scope clear.

  2. The within-paper experiments provide evidence that candidate augmentation can improve learning. Table 3 reports WebShop peak-success gains of 10.9 and 15.6 percentage points for the token- and turn-level configurations at identical environment interaction. Figure 5 shows gains across training, and Table 6 reports improvements in both score and success in a single evaluation per configuration. The zero/random-score controls and Appendix D’s offline probe make the proposed mechanism open to closer examination.

Critical weaknesses
  1. The evidence does not yet isolate the value of critic-based local credit assignment. In Figure 3, retaining candidate rows with zero advantage already raises step-300 success from 68.0% to 82.0%; learned scores reach 85.2%. Under Equation (6), zero candidate advantages remove their PPO gradient but can still change the weighting of real-token gradients through mixed-batch averaging. Thus the large micro-on/off gain cannot be attributed entirely to informative candidate scores. Appendix D further reports within-group score/return correlations of 0.035, −0.020, and −0.073, with all four candidates mapping to the same environment action in 65–72% of groups. The small continuation-sampling budget limits this diagnostic, so it does not establish that the critic is useless. It nevertheless leaves the central credit-assignment interpretation weakly supported. The default progress loss also needs reconciliation with the stated return targets, as detailed in Question 2.

  2. The headline results are not sufficient to establish the magnitude of the improvement over prior methods. Section 5.1 explicitly reports single-seed runs, repeated checkpoint selection on the evaluation sets, and published baselines with different protocols. The ScienceWorld comparison is especially difficult to interpret: Section 5.2 reports 62.0% for the extended checkpoint at the reference 20-turn limit, while the highlighted comparison is 78.6% versus ScalingInter’s reported 57.0%. Table 2 places family-level results from an 80.5% extended-checkpoint evaluation beside the 78.6% overall peak, and does not specify the corresponding evaluation horizons. These disclosures are welcome, but they leave uncertainty about how much of the reported advantage comes from the algorithm, the horizon, or checkpoint selection. Together with the unresolved attribution above, this keeps the evidence below the threshold for acceptance.

Questions for authors
  1. How many actor optimizer steps and valid tokens contribute in the micro-off and zero-advantage controls, and where is the KL loss applied? Would the learned-score advantage persist if real-token loss weight, optimizer steps, and KL exposure were matched? For the existing offline probe, what is the ranking result on groups with distinct actions, with uncertainty from continuation sampling? This would help distinguish useful action comparison from changes to the training objective induced by extra rows.

  2. With γ = λ = 1 in Appendix C and the terminal bootstrap in Equation (3), the unnormalized target telescopes to the remaining reward sum. If WebShop uses only the purchase reward described in Appendix B, adjacent turn-end targets are equal and Equation (5) contributes zero. What produces nonzero target differences in the token-level runs of Table 4? Please specify any intermediate rewards or target-processing details and the fraction of active progress-loss pairs.

  3. Please identify the checkpoint, evaluation turn limit, and aggregation rule for each CAGPO entry in Table 2, including the 78.6% overall value. Is there an independently selected checkpoint evaluated at the reference 20-turn limit on a set not used for selection? If further validation is needed, a matched-horizon comparison with repeated seeds would be more consequential than adding another benchmark.

Reviewer_7RHwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CAGPO for multi-turn LLM agent reinforcement learning. It combines temporal credit assignment on executed trajectories through GAE with group-relative advantages computed from critic-scored, unexecuted candidate responses. A shared critic is trained using real-trajectory return targets and an auxiliary sign-hinge loss on adjacent-turn value changes. Experiments on ALFWorld, WebShop, and ScienceWorld report strong performance, with component studies examining candidate supervision and progress regularization.

Critical strengths

1. The framework addresses a consequential practical problem: obtaining additional policy supervision when environment interactions are expensive or difficult to replay. Combining temporal credit with local candidate comparisons is a plausible approach to using collected experience more effectively.

2. The reported benchmark performance is promising. The micro-layer on/off experiments and candidate-score controls provide useful initial evidence that critic-guided candidate supervision can improve learning at a fixed environment interaction budget.

Critical weaknesses
  1. The novelty relative to closely related prior work is insufficiently established. Android Coach (Gan et al., ACL 2026) already trains a critic on executed online trajectories, evaluates multiple unexecuted candidate actions at a visited state, and uses group-relative advantages for policy updates without additional environment execution. This substantially overlaps with the central motivation and mechanism of CAGPO, yet the submission does not discuss or compare against it. CAGPO has potentially meaningful distinctions, including its explicit macro/micro combination, candidate-score normalization, entropy-based selection, and progress regularization. However, the paper needs to explain which distinctions constitute its contribution and demonstrate their advantages through a direct comparison under matched interaction and computational budgets.
  1. The experiments do not isolate several central design choices or establish statistical stability. Retaining real-trajectory macro supervision in the actor update is reasonable, but the existing micro-off experiment only establishes the benefit of adding candidate supervision to the real-trajectory branch. It does not establish the benefit of retaining the real-trajectory actor loss once candidate supervision is available. A complementary macro-off/micro-on control is needed, while retaining the same real rollouts for critic training and candidate-prefix construction. A controlled sweep that increases the relative contribution of real-trajectory actor supervision would further clarify this design. The existing candidate-weight, candidate-count, and turn-selection sweeps provide partial sensitivity analysis but do not isolate this question.
  1. The experiments do not justify preferring direction-only supervision. Section 4.3 explains how the sign-hinge progress loss operates, and Table 4 reports progress-loss on/off results. However, these experiments do not justify preferring direction-only supervision over the difference-matching objective described in Appendix A. A direct comparison is particularly relevant because the reported progress-loss gains are small or absent in several configurations.
  1. Finally, Section 5.1 explicitly states that the runs use a single seed. Single-seed learning curves and peak-checkpoint results cannot establish robustness across seeds or substantiate general claims about reduced collapse. Repeated runs with uncertainty estimates are needed, especially for the smaller ablation gains.
Questions for authors
  1. What is the precise methodological contribution beyond Android Coach? Can you provide a direct comparison using a common backbone, benchmark, and matched interaction and computational budgets?

  2. Does the real-trajectory macro actor loss improve performance over candidate-only actor supervision when both variants retain identical real data for critic training? How does performance change as the relative contribution of real-trajectory supervision increases?

  3. Why should the sign-hinge formulation be preferred to matching adjacent-turn value differences? Can you directly compare these objectives, including sensitivity to the progress-loss weight?

  4. Do the main improvements and claimed stability benefits persist across multiple independent training seeds? Please report uncertainty estimates using a consistent checkpoint-selection protocol.

1 comment

Relationship to Android Coach

CAGPO integrates cross-turn GAE on executed responses with group-normalized candidate advantages in one actor update. Android Coach shares the use of a critic trained on executed trajectories to supervise unexecuted alternatives at visited contexts; we will cite it and explain this relationship. The same critic supplies token values for temporal estimation and response-end values for candidate comparison. Candidates are allocated to selected high-entropy turns; critic targets come exclusively from executed trajectories.

The normalization difference can be stated precisely. For the same candidate scores, Android Coach's ACLOO advantage is

AiACLOO=GG−1(qi−qˉ),A_i^{\mathrm{ACLOO}}=\frac{G}{G-1}(q_i-\bar q),

whereas CAGPO uses

Aimicro=qi−qˉσq+ϵ.A_i^{\mathrm{micro}}=\frac{q_i-\bar q}{\sigma_q+\epsilon}.

Both preserve the same within-group ordering. Standardization changes the relative contribution of groups with different score spreads. Android Coach also uses PRM supervision and critic pretraining in its full method; CAGPO uses real-return regression with an optional directional regularizer. We will revise the related-work discussion and contribution statement around these specific differences. A matched empirical comparison with Android Coach is absent from the current evaluation.

Role of the real-trajectory actor loss

The macro actor loss directly uses the terminal reward already obtained from executed trajectories. Removing it would discard this observed-reward policy signal while retaining supervision through critic-predicted candidate scores. Our design supplements terminal-reward learning with candidate comparisons; the relevant ablation therefore removes the candidate branch, as in Table 3.

Directional versus difference-matching supervision

Sign-hinge imposes a weaker constraint than squared difference matching. It penalizes an opposite-signed predicted change while leaving its magnitude to the base value-regression loss. Difference matching additionally fits the magnitude of the adjacent-target difference. When the target difference is zero, sign-hinge contributes zero, whereas difference matching still encourages equal adjacent predictions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.