acceptodds
Withdrawn from ICLR 2027

CRGPO: Context-guided graph refinement for group-based agent reinforcement learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Group-based reinforcement learning (RL) has improved the performance of large language models (LLMs) on agentic tasks. Despite these gains, it remains hampered by sparse and delayed rewards in long-horizon settings, where feedback often arrives only after dozens of interactions. Recent frameworks have shifted from trajectory-level to step-level training to enable finer-grained policy updates. Yet observation-based credit assignment still aggregates visits with the same local observation, overlooking contextual information from preceding interactions that is relevant to state values. Consequently, visits with different prospects of task success can fall into the same comparison group, distorting the credit assigned to their actions. To address this limitation, we propose CRGPO a context-guided graph refinement method for agentic RL. CRGPO combines separately sampled linear trajectories into a state-transition graph by merging visits with the identical local observations, then refines its nodes through increasingly long context windows. At the first level that distinguishes two contexts, it attributes their estimated value difference to earlier actions at a shared decision branch. Policy updates combine these signals with suffix advantages from the final refined graph and episode-level feedback. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B and Qwen2.5-7B show that CRGPOachieves the highest overall success rates among the challenge benchmarks.

Primary area:
reinforcement learning
Keywords:
LLM Post Training, Agentic Reinforcement Learning, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SpWJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CRGPO refines a state-transition graph using increasingly long interaction contexts and then combines a contextual advantage with a suffix advantage for step-level credit assignment in group-based RL. The experiments on ALFWorld and WebShop are promising, and the method does improve over baselines. However, I don’t think the current evidence fully backs the main claims.

Critical strengths

Value aliasing is a genuine weakness of observation-based step-level grouping, especially in multi-turn agent tasks. CRGPO offers a reasonable compromise between short-context samples and historical distinction by refining graph nodes and using first-separation comparisons. I see this as a step forward for credit assignment in group-based step-level RL.

Critical weaknesses
  1. The branch comparison needs an identical full prefix. With eight rollouts at temperature 1.0 that will mostly be the first one or two steps. Contextual credit therefore concentrates on early actions.

  2. Novelty is a recombination. The context-length hierarchy is HGPO's, graph node values and value-change advantages are G2PO's, and suffix GAE on graph values appears in other recent graph-credit work. The contextual advantage is new, but Figure 3 shows EP+ctx and EP+suf giving similar gains, so the paper has not shown the new term matters more than a known one.

  3. The K sweep is too narrow to support the claim that context refinement is the main contribution. The hyperparameter K drives the refined graph, first-separation credit, and first-arrival comparison. However, the main text only reports K=2 and K=4, making it difficult to determine whether the gain comes from one step of history or from longer contexts.

  4. The post-selection bias of ActxA_{\mathrm{ctx}} is not discussed. It uses the mean of first-arrival visits at a child node to estimate expected return after branch uu. This comparison conditions on both rollouts reaching oo, yet the probability of reaching oo can depend on actions taken after uu. If pi(o∣u)≠pj(o∣u)p_i(o|u)\neq p_j(o|u), the child-node mean difference mixes policy quality with arrival propensity. The method section should acknowledge this and clarify when the approximation is still be acceptable.

  5. ActxA_{\mathrm{ctx}} and AsufA_{\mathrm{suf}} may be partially redundant, and the ablation cannot separate complementarity from duplication. I’m also not convinced the ablation separates complementarity from redundancy. When the branching action is itself a rollout step, the two signals overlap.

Questions for authors

See above.

Reviewer_KnjHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CRGPO to improve credit assignment in long-horizon agent tasks where identical observations can hide differences in task progress. It constructs a graph from rollouts of the same task and refines nodes using increasingly long observation–action histories. For trajectories that diverge at an action and later reach the same observation, CRGPO compares branch-conditioned first-arrival group values at the first context level that separates them, then assigns the differences to the earlier branching actions. These contextual advantages are combined with suffix advantages based on the final refined nodes and episode-level GRPO advantages, without a learned critic. Experiments cover ALFWorld and WebShop using Qwen2.5-1.5B and Qwen2.5-7B.

Critical strengths
  1. The motivation is well grounded in the tasks studied. Whether an object has already been cleaned or heated can affect future success without changing the current observation. Grouping such visits solely by observation can therefore mix states with different values.
  2. Assigning later value differences to earlier branching actions is a useful part of the design. It gives those actions a training signal when their consequences become apparent only after several subsequent interactions.
  3. Table 2 directly tests the effect of context refinement while keeping EP+suf and the other training settings fixed. Success rates increase from 75.6% to 82.5% on ALFWorld and from 64.5% to 69.9% on WebShop, supporting the use of refined groups for suffix credit assignment.
  4. The method does not require training a critic. Its graph refinement and advantage computation are also inexpensive in the measured setting: together, they account for about 0.17% of the summed duration of the six training phases reported for Qwen2.5-1.5B on ALFWorld in Figure 4.
Critical weaknesses
  1. Several reported results do not agree. For the 1.5B model, Table 1 gives 79.4% success on WebShop, while Section 5.3 gives 74.4%. For 7B, the improvement over HGPO is 7.3 percentage points in the table but 5.4 in the text. The text and Appendix C.3 also refer to K=2 and K=4 comparisons in Table 1, although each model scale has only one CRGPO row with no K specified. These discrepancies affect the main performance claims. Please identify the correct values and provide the configurations and per-seed results for each context length.
  2. It is unclear how much the two main rules contribute beyond history-based grouping, which HGPO already uses according to the paper. Figure 3 removes entire advantage components, and Table 2 changes the grouping, but neither isolates the choice of comparison level or credit assignment location. How do first-separation comparisons perform against comparisons at the final separating level? Does assigning credit to earlier branching actions improve over assigning it to actions at the compared visits, with the remaining settings fixed?
  3. With only eight rollouts per task, finer grouping may leave too few samples for reliable value estimates. Appendix E shows that offline refinement of the same trajectories reduces the share of visits supported by multiple rollouts from 93.50% at K=0 to 53.52% at K=2. As the authors acknowledge, Proposition 1 concerns expected values and does not guarantee finite-sample accuracy. The analysis does not establish how this loss of support affects the training signal. Node-level rollout counts, nonzero advantage coverage, and value errors measured with independent samples would help assess this tradeoff.
  4. All matched-backbone RL experiments use Qwen2.5-1.5B or Qwen2.5-7B, limiting the evaluation to one model generation and two relatively small scales. Besides similarly sized successors such as Qwen3-1.7B and Qwen3-8B, it would be useful to test a larger, more recent open model, such as Qwen3.8-27B, officially released on August 14, 2026. Would CRGPO retain its gains with more capable policies, whose shared-observation and history-matching rates may differ? Comparing CRGPO with GRPO and HGPO on these additional backbones would help assess generalization across model generations and scales. On each backbone, the methods should use matched training, rollout, and inference budgets, observation inputs, history windows, and reasoning settings.
Questions for authors
  1. Which WebShop results and configurations are correct? Please reconcile the 1.5B success rates of 79.4% in Table 1 and 74.4% in Section 5.3, as well as the 7B improvement over HGPO reported as 5.4 percentage points in the text versus 7.3 points implied by the table. Please identify the K value for each reported result and provide the K=2 and K=4 results, including per-seed measurements and the configuration used for the ablations.
  2. What evidence isolates the benefits of first-separation comparisons and credit assignment to earlier branching actions? Do you have controlled comparisons that change only the comparison level or only the credit assignment location? The current component-removal and grouping ablations do not separately establish the contribution of these two rules.
  3. How much usable step-level training signal remains after context refinement? Please clarify the distribution of distinct rollouts per final node and the proportion of actions receiving nonzero contextual and suffix advantages. Is there direct evidence, such as value-estimation error measured with independent samples, that reduced aliasing outweighs the loss of sample support in the eight-rollout setting?
  4. What motivated restricting the matched-backbone RL experiments to Qwen2.5-1.5B and Qwen2.5-7B, and are there additional results on newer backbones? A comparison on a newer model would affect my assessment of generalization across model generations. Testing a larger model such as Qwen3.8-27B is a further suggestion for assessing scale; it is not a prerequisite for resolving the numerical discrepancies and mechanism questions above.
Reviewer_MStCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CRGPO approach for training LLM agents on multi-turn textual tasks. It groups rollout visits using observation-action histories, compares continuation values to assign credit to earlier branching actions, and combines this contextual signal with suffix and episode-level advantages. Experiments on ALFWorld and WebShop report improvements over existing baselines.

Critical strengths
  • The paper addresses temporal credit assignment under sparse task rewards, which is especially challenging for LLM agents with long reasoning and interaction trajectories.

  • The reported gains on ALFWorld and WebShop across two model sizes are promising, subject to resolving the reporting and baseline discrepancies discussed below.

Critical weaknesses

W1: The problem setup permits general goal-conditioned POMDPs, but the method relies on recurring, exactly matching observation–action contexts and on the last K interactions capturing information needed to distinguish situations with different future outcomes. The evaluation does not establish applicability beyond this restricted setting, particularly under stochastic transitions, variable observation wording, or dependencies extending beyond the context window. At minimum, the authors should clarify these assumptions in the problem setup and explicitly discuss the resulting limitations of the approach.

W2: ALFWorld baselines are substantially below their original reports (Qwen 1.5B/7B): G2PO achieves 87.5/87.6% versus 95.0/96.9%, and HGPO achieves 89.2/93.1% versus 92.77/95.44%. The authors should explain differences in splits and training/evaluation settings.

W3: The section 5.3 mentions "Full CRGPO reaches 95.4% and 74.4%.", which countradicts with the Table 1.

W4: The training and interaction budgets differ from the original G2PO setup: 160 versus 100 training iterations and 30 versus 15 maximum WebShop steps. Although Appendix C3 states that these budgets are shared across baselines.

W5: Although reporting only final-result tables seems common in this area, learning curves on in-distribution tasks for CRGPO and the baselines, with variability across seeds, would help assess sample efficiency, stability, and sensitivity to the training budget.

W6: No implementation is available, limiting reproducibility and verification of the baseline discrepancies above. Providing code and exact configurations for CRGPO and the reproduced baselines, together with raw per-task results for each training seed, would help verify the reported means and variability.

Minor:

  • Figure 2 contains blurry text and inconsistent fonts, making it difficult to read. A higher-resolution or vector version would improve clarity.
Questions for authors

Q1: What explains the lower G2PO and HGPO results compared with their original papers?

Q2: How do CRGPO and the baselines compare throughout training, including at 100 and 160 iterations?

Q3: Could you provide implementation details and raw per-task results for each seed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.