- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CRGPO refines a state-transition graph using increasingly long interaction contexts and then combines a contextual advantage with a suffix advantage for step-level credit assignment in group-based RL. The experiments on ALFWorld and WebShop are promising, and the method does improve over baselines. However, I don’t think the current evidence fully backs the main claims.
Critical strengths
Value aliasing is a genuine weakness of observation-based step-level grouping, especially in multi-turn agent tasks. CRGPO offers a reasonable compromise between short-context samples and historical distinction by refining graph nodes and using first-separation comparisons. I see this as a step forward for credit assignment in group-based step-level RL.
Critical weaknesses
-
The branch comparison needs an identical full prefix. With eight rollouts at temperature 1.0 that will mostly be the first one or two steps. Contextual credit therefore concentrates on early actions.
-
Novelty is a recombination. The context-length hierarchy is HGPO's, graph node values and value-change advantages are G2PO's, and suffix GAE on graph values appears in other recent graph-credit work. The contextual advantage is new, but Figure 3 shows EP+ctx and EP+suf giving similar gains, so the paper has not shown the new term matters more than a known one.
-
The K sweep is too narrow to support the claim that context refinement is the main contribution. The hyperparameter K drives the refined graph, first-separation credit, and first-arrival comparison. However, the main text only reports K=2 and K=4, making it difficult to determine whether the gain comes from one step of history or from longer contexts.
-
The post-selection bias of is not discussed. It uses the mean of first-arrival visits at a child node to estimate expected return after branch . This comparison conditions on both rollouts reaching , yet the probability of reaching can depend on actions taken after . If , the child-node mean difference mixes policy quality with arrival propensity. The method section should acknowledge this and clarify when the approximation is still be acceptable.
-
and may be partially redundant, and the ablation cannot separate complementarity from duplication. I’m also not convinced the ablation separates complementarity from redundancy. When the branching action is itself a rollout step, the two signals overlap.
Questions for authors
See above.