acceptodds
Under review as a conference paper at ICLR 2027

Matched-Context Group Policy Optimization for LLM Agent Training

15 views

Abstract

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context Group Policy Optimization (MCGPO), a procedure for constructing weighted action preferences from already collected trajectories. MCGPO defines comparison contexts with pre-action state summaries and observed rubric progress, annotated by task rules. Two-action support selects among ordered exact and coarse matching rules. Fixed folds separate candidate decisions from the terminal outcomes used to orient their preferences, and support-dependent evidence weights scale an auxiliary pairwise objective alongside shared GraphGPO task learning. MCGPO reuses rollout data and each response’s original history, preserving critic-free training. Across six environment–backbone settings with three training seeds each, MCGPO improves mean success over matched GraphGPO: by 0.47–1.46 percentage points on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, and by 3.96 and 5.16 points on WebShop with Qwen3-4B and 8B. The ALFWorld gains build on baseline success above 92%, where remaining headroom is limited. Integrated success improves in three of four Qwen2.5 settings. Three-seed factorial controls support opposite-fold evidence and evidence weighting; joint state–rubric matching achieves the highest mean in all four representation-control settings.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Long-horizon credit assignment, Rubric-guided learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dHKvOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Matched-Context Group Policy Optimization (MCGPO), which constructs local action preferences from existing rollouts and incorporates them into GraphGPO training through an auxiliary pairwise loss. Eight trajectories collected for the same task are divided into two folds of four: one provides candidate decisions, while the other supplies outcome evidence for assigning preferences.

  1. Match: A comparison key is constructed from the pre-action state summary hh, observed rubric status CC, and a frontier FF computed from task rules. The method considers exact and then coarse matching, selecting the first rule under which the candidate decisions share a key, take different actions, and have evidence for both actions in the opposite fold.

  2. Label: Differences in action-conditioned terminal success in the opposite fold determine the preference direction. An evidence weight is computed from the outcome contrast, action support, and matching resolution.

  3. Learn: The policy is trained by adding an evidence-weighted pairwise loss to the existing GraphGPO loss, while retaining each response's original history.

Critical strengths
  • MCGPO constructs auxiliary supervision from already collected rollouts without requiring a separate critic or additional continuations for preference construction.
  • The paper provides three-seed factorial controls that separate the effects of opposite-fold evidence and evidence weighting, together with a comparison against a reward-shaping baseline using the same scorer information.
Critical weaknesses

1. The main text is difficult to follow because key definitions and the underlying training formulation are scattered across the appendices.

  • GraphGPO [1] is both the underlying training method and the primary baseline. However, the main text introduces the auxiliary preference branch without sufficiently explaining GraphGPO's advantage construction and objective; the relevant equations appear in Appendix A.3. Explaining the step-level credit already provided by GraphGPO would help readers understand why the proposed preference branch is needed and what it adds. As written, understanding the relationship between the two training branches requires consulting the main text, the appendices, and the prior paper.

 

  • Section 3.1 similarly introduces g=(h,C,F)g=(h,C,F) and immediately proceeds through candidate pairs, evidence sets, support, and matching-resolution selection. The actual fields in hh, the activation criterion for CC, the computation of FF, and the task-specific dependency templates are distributed across Appendices A.1 and D.1. These details determine which decisions are treated as comparable and what additional environment-specific information is required. Introducing the key definitions and a concrete task example in the main text would substantially improve readability.

 

  • Figures 1 and 2 do not adequately address these explanatory gaps. Figure 1 depicts the relationship between state, progress, and frontier abstractly, without showing how these quantities are computed or used to distinguish decisions in an actual task. The support and outcome examples in Figure 2 are also disconnected from concrete states and actions, making them difficult to follow on a first reading. In addition, the terminology differs between Match–Label–Learn in the text and Match–Verify–Learn in the figure. Consistent terminology and a worked example covering matching, preference construction, and learning would help clarify the method.

 

2. The methodological novelty and added value are not sufficiently convincing relative to the environment-specific design effort.

  • The concern that a current-state summary may conflate different stages of task progress is reasonable. However, the paper does not adequately explain which limitations of GraphGPO are addressed by this particular combination of exact/coarse selection, fold separation, evidence weighting, and a pairwise objective. Equations (1)–(7) specify the computations, while the justification for why these choices produce appropriate comparisons and useful preferences remains limited.

 

  • Matching requires environment-specific state fields, rubric extractors, and dependency templates. In WebShop, pre-purchase goal-scorer information is used to construct both the rubric and the coarse state representation. The paper provides limited guidance on the information required to apply this design elsewhere, the extent of manual specification, and which components can be reused. Although these requirements are acknowledged in the limitations, the current evaluation leaves the method's applicability and environment-specific design cost difficult to assess.

 

  • The State-only/Rubric-only, shuffled-rubric, opposite-fold, and weighting controls support some design choices. However, the rationale for individual rules, the information added by FF, and the justification for exact-first selection remain unclear. In particular, Table 5 includes a remaining-budget bin only in the ALFWorld coarse key. If the two keys contain different kinds of information, the paper should explain why exact matching is preferred and how this choice affects comparison quality.

 

  • The gains in the main Qwen2.5 experiments are modest. Mean success improves over matched GraphGPO by 0.47–1.46 percentage points, and the paired confidence intervals in Appendix Table 10 include zero in all four settings. The graded score on Qwen2.5-1.5B/WebShop decreases from 88.79 to 86.83. The larger Qwen3 gains and additional controls are encouraging, but I remain unconvinced that the demonstrated benefits sufficiently justify adopting the full procedure and constructing the required environment-specific representations and rules.

 

  • The comparison conditions in Table 1 also need to be distinguished more clearly. MCGPO and GraphGPO are compared under the same 250-update protocol, whereas the GiGPO [2] and GRPO [3] entries use results reported under the 150-iteration setup of Feng et al. (2025). Please make the differences in training budgets and evaluation protocols explicit when presenting rerun results alongside previously published results.

Reference

[1] Xin Cheng, Shuo He, Lang Feng, HaiYang Xu, Ming Yan, Lei Feng, and Bo An. Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning. arXiv preprint arXiv:2605.26684, 2026.

[2] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. In Advances in Neural Information Processing Systems, 2025.

[3] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.

Questions for authors
  1. What is the relationship between the exact and coarse keys, and why is exact matching preferred?
  • Table 5 includes a remaining-budget bin only in the ALFWorld coarse key. Please clarify whether the actual exact key also contains information that distinguishes the remaining horizon. Is the coarse key a coarsening of the exact key, or a separate matching rule that uses additional information?

 

  1. How are the environment-specific rubrics and rules selected, and what additional role does FF play in matching?
  • Appendix D.1 specifies deterministic extractors and task-specific dependency templates. Please distinguish which state fields, rubric channels, prerequisite relations, and discretization thresholds are provided directly by the environment and which must be separately specified when applying the method. What principles guide these choices?

  • In particular, Eq. (8) makes FF a deterministic function of CC when the vocabulary and task-rule graph are fixed. Under these conditions, how does adding FF to (h,C)(h,C) change the matching results or preference construction?

  • To help assess applicability beyond the evaluated benchmarks, please also provide a conceptual example for one additional environment. What information would be used to construct h,C,Fh,C,F, which rules would need to be newly specified, and how could the method be applied when that information is unavailable?

 

  1. What limitation of the existing GraphGPO credit does the proposed preference branch address?
  • Using a concrete example with the same set of rollouts, please show the credit assigned by GraphGPO, the additional preference constructed by MCGPO, and the specific problem that this preference is intended to address. The relationship between the auxiliary objective and the existing task objective is currently difficult to understand from the main text.
Reviewer_Qp3UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MCGPO constructs auxiliary action preferences from existing agent rollouts. It matches decisions using state summaries and rubric progress, selects exact/coarse matches with support for both actions, and uses opposite-fold terminal outcomes to label and weight preferences. A pairwise loss supplements GraphGPO while retaining each response’s original history. Experiments on ALFWorld and WebShop report improved success, with factorial, representation, and scorer-matched shaping controls.

Critical strengths
  1. Practical gains using existing rollouts. MCGPO improves WebShop success over matched GraphGPO by 3.96/5.16 percentage points with Qwen3-4B/8B, without additional rollout collection or a learned critic (Table 3).
  2. Relevant controls. Factorial experiments support opposite-fold evidence and weighting. Against shaping with access to the same scorer information, gains remain 2.29/2.66 points (Tables 12–13), supporting the utility of the complete scheme.
Critical weaknesses
  1. Local preference validity remains unresolved. ALFWorld’s exact keys omit receptacle openness and remaining horizon (Table 5). Matched histories may therefore require different actions, such as opening a closed drawer versus placing an object into an open one. Opposite-fold labeling separates outcome sources but does not resolve this mismatch. This is a plausible failure mode whose frequency and effect remain unmeasured, leaving a central gap in the credit-assignment interpretation.
  2. Representation effects are not isolated. Table 14 changes matching keys together with retained visits, selected pairs, support, and weights. Because Equation (6) normalizes by microbatch pairing capacity, auxiliary optimization strength can also change. The results establish the utility of joint-key selection, but do not isolate improved comparability. Table 12’s weight-matched control addresses a separate comparison.
Questions for authors
  1. Clarification: Do the implemented exact keys include openness and remaining horizon? If not, how often do accepted pairs differ in action preconditions, and does this affect preference direction?
  2. Targeted validation: Does the Table 14 joint-key advantage persist when accepted-pair quantity and total auxiliary weight are matched?
Reviewer_e8q9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Matched-Context Group Policy Optimization (MCGPO), a method for constructing local action preferences from existing agent rollouts. It matches decisions using environment states and observed task progress, then uses terminal outcomes from separate rollout folds to determine preference directions. Support-dependent weights control the resulting pairwise supervision, which is combined with the GraphGPO task objective. Experiments on ALFWorld and WebShop show consistent mean success improvements across six model–environment settings, with larger gains on the Qwen3 backbones. Additional ablations examine context matching, opposite-fold evidence, and preference weighting.

Critical strengths
  1. Interesting motivation and a coherent method. The observation that identical state summaries may correspond to different stages of task progress is well motivated. Combining state and rubric information to construct local action preferences offers a useful extension to existing graph-based credit assignment methods.

  2. Careful experimental design. The matched GraphGPO comparisons and Qwen3 results are encouraging. I particularly appreciate the factorial ablations on opposite-fold evidence and evidence weighting, as well as the analysis of how preferences are constructed during training.

Critical weaknesses
  1. Although opposite-fold evidence avoids directly using candidate outcomes for labeling, terminal success also depends on subsequent actions. A higher outcome rate may therefore reflect differences in future behavior rather than the quality of the matched action itself.

  2. Figure 4 shows that 90.7% of WebShop preferences come from coarse matching, while 77.7% have only one supporting trajectory for the less frequent action. Evidence weighting helps address uncertainty, but it does not necessarily correct mismatched decision contexts.

  3. MCGPO requires predefined rubric channels and task rules, with WebShop additionally using structured information from the goal scorer. This makes its applicability to agent environments without such supervision less clear.

Questions for authors
  1. Opposite-fold outcomes provide separate evidence for labeling, but the compared trajectories can still have different continuations. How should the resulting preferences be interpreted as feedback on the local action rather than its subsequent trajectory?

  2. Most accepted WebShop preferences rely on coarse matching and limited action support. Does the evidence weight account for possible mismatches between the underlying decision states, or primarily for the amount of available outcome evidence?

  3. MCGPO relies on task-specific progress signals and matched state contexts, which may be difficult to obtain in more complex environments such as Terminal-Bench, where recurring states and explicit progress indicators are less readily available. How well would the method generalize to such settings without predefined matching rules or structured progress information?

Comments on the paper

2 comments

Thank you for the detailed response and additional experiments. The new results address several of my concerns, particularly regarding the effectiveness of Joint training under matched supervision. I have one remaining question regarding the same-state action validation:

Exclusion criteria. What were the reasons for excluding 12 of the 200 source states, and were the exclusion criteria established before observing continuation outcomes? In particular, how were cases where only one candidate action was executable handled?

Manuscript revision. I also encourage the authors to incorporate the methodological clarifications and additional experimental results from the rebuttal into the revised manuscript, rather than leaving them only in the discussion. Please clearly highlight the changes and specify the corresponding sections, pages, or tables so that the revisions can be easily verified.

Area_Chair_vFMYOpenReview ↗

Please engage with authors

Dear Reviewers,

The author-reviewer discussion will be closed on November 18 (AoE). If you haven’t yet replied to the authors’ latest response, please do so on OpenReview as soon as possible so they have time to follow up. A brief reply indicating which of your concerns have been addressed and what questions remain would be helpful. Thank you for your time and for engaging with the authors.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.