- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes Matched-Context Group Policy Optimization (MCGPO), which constructs local action preferences from existing rollouts and incorporates them into GraphGPO training through an auxiliary pairwise loss. Eight trajectories collected for the same task are divided into two folds of four: one provides candidate decisions, while the other supplies outcome evidence for assigning preferences.
-
Match: A comparison key is constructed from the pre-action state summary , observed rubric status , and a frontier computed from task rules. The method considers exact and then coarse matching, selecting the first rule under which the candidate decisions share a key, take different actions, and have evidence for both actions in the opposite fold.
-
Label: Differences in action-conditioned terminal success in the opposite fold determine the preference direction. An evidence weight is computed from the outcome contrast, action support, and matching resolution.
-
Learn: The policy is trained by adding an evidence-weighted pairwise loss to the existing GraphGPO loss, while retaining each response's original history.
Critical strengths
- MCGPO constructs auxiliary supervision from already collected rollouts without requiring a separate critic or additional continuations for preference construction.
- The paper provides three-seed factorial controls that separate the effects of opposite-fold evidence and evidence weighting, together with a comparison against a reward-shaping baseline using the same scorer information.
Critical weaknesses
1. The main text is difficult to follow because key definitions and the underlying training formulation are scattered across the appendices.
- GraphGPO [1] is both the underlying training method and the primary baseline. However, the main text introduces the auxiliary preference branch without sufficiently explaining GraphGPO's advantage construction and objective; the relevant equations appear in Appendix A.3. Explaining the step-level credit already provided by GraphGPO would help readers understand why the proposed preference branch is needed and what it adds. As written, understanding the relationship between the two training branches requires consulting the main text, the appendices, and the prior paper.
- Section 3.1 similarly introduces and immediately proceeds through candidate pairs, evidence sets, support, and matching-resolution selection. The actual fields in , the activation criterion for , the computation of , and the task-specific dependency templates are distributed across Appendices A.1 and D.1. These details determine which decisions are treated as comparable and what additional environment-specific information is required. Introducing the key definitions and a concrete task example in the main text would substantially improve readability.
- Figures 1 and 2 do not adequately address these explanatory gaps. Figure 1 depicts the relationship between state, progress, and frontier abstractly, without showing how these quantities are computed or used to distinguish decisions in an actual task. The support and outcome examples in Figure 2 are also disconnected from concrete states and actions, making them difficult to follow on a first reading. In addition, the terminology differs between Match–Label–Learn in the text and Match–Verify–Learn in the figure. Consistent terminology and a worked example covering matching, preference construction, and learning would help clarify the method.
2. The methodological novelty and added value are not sufficiently convincing relative to the environment-specific design effort.
- The concern that a current-state summary may conflate different stages of task progress is reasonable. However, the paper does not adequately explain which limitations of GraphGPO are addressed by this particular combination of exact/coarse selection, fold separation, evidence weighting, and a pairwise objective. Equations (1)–(7) specify the computations, while the justification for why these choices produce appropriate comparisons and useful preferences remains limited.
- Matching requires environment-specific state fields, rubric extractors, and dependency templates. In WebShop, pre-purchase goal-scorer information is used to construct both the rubric and the coarse state representation. The paper provides limited guidance on the information required to apply this design elsewhere, the extent of manual specification, and which components can be reused. Although these requirements are acknowledged in the limitations, the current evaluation leaves the method's applicability and environment-specific design cost difficult to assess.
- The State-only/Rubric-only, shuffled-rubric, opposite-fold, and weighting controls support some design choices. However, the rationale for individual rules, the information added by , and the justification for exact-first selection remain unclear. In particular, Table 5 includes a remaining-budget bin only in the ALFWorld coarse key. If the two keys contain different kinds of information, the paper should explain why exact matching is preferred and how this choice affects comparison quality.
- The gains in the main Qwen2.5 experiments are modest. Mean success improves over matched GraphGPO by 0.47–1.46 percentage points, and the paired confidence intervals in Appendix Table 10 include zero in all four settings. The graded score on Qwen2.5-1.5B/WebShop decreases from 88.79 to 86.83. The larger Qwen3 gains and additional controls are encouraging, but I remain unconvinced that the demonstrated benefits sufficiently justify adopting the full procedure and constructing the required environment-specific representations and rules.
- The comparison conditions in Table 1 also need to be distinguished more clearly. MCGPO and GraphGPO are compared under the same 250-update protocol, whereas the GiGPO [2] and GRPO [3] entries use results reported under the 150-iteration setup of Feng et al. (2025). Please make the differences in training budgets and evaluation protocols explicit when presenting rerun results alongside previously published results.
Reference
[1] Xin Cheng, Shuo He, Lang Feng, HaiYang Xu, Ming Yan, Lei Feng, and Bo An. Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning. arXiv preprint arXiv:2605.26684, 2026.
[2] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. In Advances in Neural Information Processing Systems, 2025.
[3] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Questions for authors
- What is the relationship between the exact and coarse keys, and why is exact matching preferred?
- Table 5 includes a remaining-budget bin only in the ALFWorld coarse key. Please clarify whether the actual exact key also contains information that distinguishes the remaining horizon. Is the coarse key a coarsening of the exact key, or a separate matching rule that uses additional information?
- How are the environment-specific rubrics and rules selected, and what additional role does play in matching?
-
Appendix D.1 specifies deterministic extractors and task-specific dependency templates. Please distinguish which state fields, rubric channels, prerequisite relations, and discretization thresholds are provided directly by the environment and which must be separately specified when applying the method. What principles guide these choices?
-
In particular, Eq. (8) makes a deterministic function of when the vocabulary and task-rule graph are fixed. Under these conditions, how does adding to change the matching results or preference construction?
-
To help assess applicability beyond the evaluated benchmarks, please also provide a conceptual example for one additional environment. What information would be used to construct , which rules would need to be newly specified, and how could the method be applied when that information is unavailable?
- What limitation of the existing GraphGPO credit does the proposed preference branch address?
- Using a concrete example with the same set of rollouts, please show the credit assigned by GraphGPO, the additional preference constructed by MCGPO, and the specific problem that this preference is intended to address. The relationship between the auxiliary objective and the existing task objective is currently difficult to understand from the main text.
Thank you for the detailed response and additional experiments. The new results address several of my concerns, particularly regarding the effectiveness of Joint training under matched supervision. I have one remaining question regarding the same-state action validation:
Exclusion criteria. What were the reasons for excluding 12 of the 200 source states, and were the exclusion criteria established before observing continuation outcomes? In particular, how were cases where only one candidate action was executable handled?
Manuscript revision. I also encourage the authors to incorporate the methodological clarifications and additional experimental results from the rebuttal into the revised manuscript, rather than leaving them only in the discussion. Please clearly highlight the changes and specify the corresponding sections, pages, or tables so that the revisions can be easily verified.