acceptodds
Under review as a conference paper at ICLR 2027

Contextual Credit Policy Optimization

12 views

Abstract

Group-based reinforcement learning enables large language model agents to learn from sparse outcome rewards in long-horizon tasks. Recent methods derive step-level credit by comparing observation-matched experiences across trajectories. However, identical observations can occur in different contexts, making some peers more informative for a given decision. Moreover, favorable terminal outcomes can obscure redundant actions or detours within successful trajectories. We propose Contextual Credit Policy Optimization (CCPO), which combines contextual peer weighting with history-conditioned return credit and realized future progress. CCPO combines frozen LLM representations and accumulated trajectory statistics to weight observation-matched peers from other trajectories by contextual similarity. Historical credit compares the realized return against the weighted peer baseline, while future credit measures the change in contextual outcome value over a short realized continuation. CCPO reuses collected rollouts without an auxiliary value critic or additional environment interactions; future information is used only during training. With Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, CCPO reaches up to 97.4 ± 0.5% success on ALFWorld and 79.4 ± 1.3% on WebShop; across variants, 7B gains over GRPO are 12.3–15.4 and 5.5–6.5 percentage points. Ablations support query-trajectory exclusion, combined historical and future credit, and structured context representation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Credit Assignment, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QwUmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes Contextual Credit Policy Optimization (CCPO), a critic-free step-level credit assignment method for reinforcement learning of long-horizon LLM agents. CCPO addresses the state-aliasing problem that arises when steps with identical textual observations but different histories are treated as equally informative peers.

The method constructs a context representation by combining hidden states from a frozen reference LLM with hand-designed trajectory statistics, including elapsed steps, observation novelty, and revisit indicators. It then uses context-dependent kernel weights to aggregate returns from observation-matched steps in other trajectories. CCPO combines two credit signals: (1) a historical return residual relative to a contextual peer baseline, and (2) a future-credit term based on the change in contextual outcome value over a short realized continuation.

Experiments with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct on ALFWorld and WebShop show improvements over reproduced GRPO, GiGPO, HGPO, and G2PO baselines. The paper also provides ablations on peer exclusion, contextual weighting, context summaries, and temporal windows, as well as analyses of action preferences and repeated observations.

Critical strengths
  1. Context-aware peer weighting addresses a limitation of observation-level grouping. Identical observations may correspond to different agent states due to differences in interaction history or task progress. Weighting matched peers by contextual similarity provides a more informative comparison than treating them equally.

  2. The paper provides reasonably broad empirical validation. CCPO is evaluated on two benchmarks and two model sizes, with consistent improvements over several reproduced baselines. The ablations also support key choices such as query-trajectory exclusion, contextual weighting, and structured context features.

Critical weaknesses
  1. The motivation and necessity of the future-credit component are not sufficiently established. Historical credit already incorporates future outcomes through the discounted return Y_{i,t}, while the future branch is also constructed from terminal outcomes. Moreover, the history-only H=4, F=0 variant already achieves 94.3%, close to 96.9% with the full H=2, F=2 setting, making the unique contribution of future credit unclear.

  2. The primary peer construction relies on exact observation matching, which may limit generalization. This assumption is convenient in ALFWorld and WebShop, where observations are structured and often repeated, but may be restrictive in more realistic environments where semantically similar states rarely share identical observations.

Questions for authors
  1. Why is V_{t+K_F} - V_t an appropriate step-level credit signal for the action at step t? The value change is measured over an entire K_F-step continuation and may be driven by subsequent actions within the window. Could the authors clarify the rationale behind attributing this change to the action at the beginning of the window?

  2. Given that historical credit already incorporates future outcomes through the discounted return Y_{i,t}, what distinct information does the future-credit branch provide? Both branches also use history-conditioned contextual peers, making their claimed complementarity less clear.

  3. Have the authors compared the proposed future credit with simpler alternatives derived from the same outcome signal? For example, direct episode-level/final-reward advantage, one-step value differences, or a TD-style residual would help determine whether the specific V_{t+K_F} - V_t formulation is necessary.

  4. Could future work explore semantic or learned peer matching beyond exact observation matching, particularly for environments with more diverse observations?

6 comments
AuthorsOpenReview ↗

Credit signal for the action at step t; separating the effects of historical and future credits

We thank the reviewer for these thoughtful questions, and we would like to respond to both Q1 and Q2 in the following response. The reviewer is right that the value change over a K_F-step window depends on every action in the window, not only on a_t. We do not claim that it isolates the effect of a_t. This design is motivated by the following rationales:

  1. Crediting a multi-step outcome to the first action is standard in policy gradient methods. GAE [1] is built from k-step advantage estimates, which credit a_t with the rewards and value change over the next k steps. Later actions in the window contribute and add variance to the estimate for a_t. Our signal is analogous with a short window.

  2. Because identical observations can conceal different task progress (Section 4.4), V_{t+K_F} − V_t therefore asks: two steps after a_t, is the agent in a context from which its peers succeeded more often than from the context before a_t? V_t is computed before a_t is generated, so it depends only on the state and serves as a baseline for a_t.

  3. We hypothesize that historical credit contributes differently than future credit, and our analyses separate the effects of the two channels (Section 4.4). For instance, Figure 4(b) plots historical, future and combined credit over training: the ratio of future to historical magnitude rises from 0.25 to 0.60 at 1.5B and from 0.32 to 1.01 at 7B between iterations 1–30 and 121–150, suggesting distinct and evolving contributions of the two channels during training. Additionally, Figure 5 shows that the future channel might favor exploratory transitions. The overall benefit of the designs of CCPO are partially illustrated in Figure 3, showing that one of the evaluation games CCPO is the only method that trains the agent to solve a task efficiently and effectively (succeeded in 7 steps vs. baselines that exhaust the 50 step budget without success).

We will revise the text to describe the signal as a short-horizon change in peer value following a_t, not as the isolated effect of a_t. Additional ablation experiments on the future credit method are discussed in the next response and will be included during revision. We again thank the reviewer for this insightful question and the opportunity to expand the discussion.

References:

[1] High-Dimensional Continuous Control Using Generalized Advantage Estimation, 2016.

AuthorsOpenReview ↗

Design of the future credit term

We thank the reviewer for this insightful question. We have the following comparisons, all with the same rollout and optimizer budget as the standard ablation experiments. The TD-style and final-reward variants are the two controls also requested by Reviewer cjmV. Our response to Reviewer cjmV (Q1) reports them by task type, together with a third, terminal-aligned variant.

MethodSetupALFWorldWebShop
CCPO (ours): Eq. (7)z(V_{t+2} − V_t)96.9 ± 0.875.0 ± 0.8
TD-style residual/discount-alignedz(γ^h · V_{t+h} − V_t); rewards are terminal, so this is the multi-step TD residual of the peer value91.9 ± 0.570.8 ± 3.0
Final-reward signal/terminal-proximityz(γ^(T−t) · R), no peer value89.3 ± 0.571.4 ± 3.9
No future termhistorical credit only (Table 2)91.4 ± 0.872.7 ± 2.8
Shrink future termasymmetric historical context and future context (Table 2)92.7 ± 0.9---
Episode-level advantage aloneGRPO (Table 1)75.3 ± 1.056.5 ± 2.9

The TD-style variant is lower than Eq. (7) on both benchmarks (by 5.0 points on ALFWorld and 4.2 on WebShop). A final-reward-only signal is also lower on both (by 7.6 and 3.6 points). Removing the future term lowers ALFWorld by 5.5 points; on WebShop it is marginally lower and within noise of the full method, potentially due to the shorter horizon of the WebShop task and lower initial success rate, both are discussed in the main text or supplementary sections of the paper. Shrinking the future term by asymmetrically varying the context window also lowers performance.

We hypothesize that the benefit of Eq. (7) comes from combining its two parts. The discount-aligned part, γ^h · V_u − V_t, measures whether the context reached after the window has a higher peer value than the one before it. The remaining part, (1 − γ^h) · V_u, is proportional to the value at the end of the window, so it is zero when the window ends in failure and largest when it ends in success. It therefore adds credit for arriving in high-value contexts, concentrated at the end of successful trajectories. The ablations/controls are consistent with this reading. In revision, we will add the ablations to Table 2 and clarify the claim about CCPO mechanism that a short-horizon change in peer value is likely a useful credit signal and is not a terminal-proximity signal. We again thank the reviewer for this important question.

AuthorsOpenReview ↗

Future work for semantic/learned peer matching

We thank the reviewer for this insightful suggestion. We agree, and the paper includes ideas related to this direction. Section 5 lists it as a limitation ("exact observation matching can limit peer availability") and names "semantic peer retrieval beyond exact observation matching" as future work. We would like to expand this discussion by including the new ablation results on peer matching methods. Interpretations of these results are summarized below and reported in our second response to Reviewer FSji. The experiments used the same ablation settings and the Qwen2.5-1.5B-Instruct backbone:

Peer matchingALFWorldWebShop
CCPO (ours)96.9 ± 0.875.0 ± 0.8
Task only92.4 ± 0.568.2 ± 3.2
No matching83.3 ± 1.2collapses
GRPO75.3 ± 1.056.5 ± 2.9

Based on the ablation results, we develop the following thoughts:

  • Structured grouping decides which steps are eligible peers, and semantic matching with frozen-LLM representations decides how much each eligible peer contributes. The two methods are complementary, as shown by our ablation studies: removing either component or loosening the gate (such as in the matching by task-only variant) reduces performance.

  • The choice of grouping method has a bias-variance trade-off. Matching based on exact observations may leave some steps with few or no peers (Introduction; Section 3.2). However, loosening the gate admits peers that faced different decision points, biasing the baseline's estimation of how good an action is. The no-matching ablation demonstrates this failure - the baseline may reflect task difficulty when peers are drawn across tasks, and ALFWorld drops to 83.3 while WebShop collapses.

Our ideas for future work (Section 5) include:

  • Adaptive semantic peer retrieval. When a step has too few exact matches (high diversity among observations), retrieve peers by context similarity, adaptively widening the peer set until a minimum number of peers is reached. This may be combined with memory mechanisms as well.
  • Uncertainty estimates for correlated visits. When variance among peers is low between correlated visits (such as repeated visits within a rollout), the baseline estimate may overstate this contribution. Uncertainty quantification can potentially help reduce such credit inflation when support is weak.
  • Open-ended environments. Observations may differ in details that are irrelevant for the decision (timestamps, page layout, long tool outputs). They can be normalized before matching. In our response to Reviewer FSji, we also discussed related works that study the effects of per-prompt normalization in agentic RL.

In revision, we will expand the future-work paragraph of Section 5 along these lines in either main text or the supplementary section. We again thank the reviewer for this insightful question and the opportunity to expand this discussion.

Reviewer_FSjiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CCPO, a critic-free policy optimization method for multi-turn LLM agents. The method aims to improve step-level credit assignment by constructing context-dependent baselines from previously collected trajectories and combining two credit signals: historical credit and future credit. Experiments on ALFWorld and WebShop using Qwen2.5-1.5b and Qwen2.5-7b demonstrate improvements over several RL baselines.

Critical strengths
  1. The critic-free design may improve the training efficiency.

  2. The paper perform useful ablation studies. The paper investigates several design choices, providing evidence that the proposed components are beneficial under the evaluated settings.

Critical weaknesses
  1. My primary concern is the reliability and scalability of contextual peer-based value estimation. In complex multi-turn environments, such as software engineering, open-web interaction, or multi-step deep research, observations and interaction histories can be highly diverse. Different trajectories may rarely encounter similar observations.

  2. The evaluation is restricted to ALFWorld and WebShop, which have relatively constrained environments compared with modern open-ended agent applications. In particular, ALFWorld contains structured household interaction tasks with recurring observations and a relatively limited action space. WebShop also operates within a constrained shopping environment with predefined interaction patterns. These characteristics may be especially favorable to CCPO, whose contextual value estimation relies on matching and comparing observations across trajectories.

Questions for authors

How does the method behave when observation diversity increases substantially? Does CCPO still provide a meaningful advantage over GRPO under such conditions?

4 comments
AuthorsOpenReview ↗

Experiments of CCPO on additional benchmark

We thank the reviewer for this important question, and we agree that the current evaluation is limited. We chose ALFWorld and WebShop because they are the benchmarks used by the step-level baselines we compare with (GiGPO, HGPO, G2PO), but both have recurring observations, and that favors observation-matched methods, CCPO included. Within the rebuttal period, we added two sets of experiments.

  1. Ablations of structured matching (not matched by observation nor task) and task-based matching only (group peers who share the same task). Results from these ablations are included in the second response.
  2. Using the Qwen2.5-7B-Instruct backbone, we trained and evaluated CCPO on a third benchmark, Search-R1 (multi-turn retrieval question answering; training on NQ and HotpotQA, evaluation on seven QA datasets), where observations are retrieved passages instead of states of a fixed environment [1]. CCPO reaches an average of 48.8 across the seven evaluation QA datasets, compared to published performance of GiGPO (47.2, [2]), Search-R1 (38.5, [1]), PPO (37.0, [3]) and GRPO (39.6, [3]) on the Qwen2.5-7B-Instruct backbone.
Qwen2.5-7B-InstructNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAverage
Search-R1-instruct (PPO, outcome only)39.361.039.737.041.414.636.838.5
PPO + format reward38.359.339.937.631.715.137.137.0
GRPO, outcome only42.962.342.738.634.616.240.039.6
GRPO + format reward39.360.939.736.734.414.738.737.8
GiGPO46.464.746.141.643.618.968.947.2
CCPO (ours)49.765.746.745.542.221.570.248.8

In revision, we will add both results to the paper and revise the limitations section to clarify the scope of the evaluation, such as the type of environments covered, observation recurrence, whether tasks are open-ended or retrieval-based. We again thank the reviewer for this important feedback.

References:

[1] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning, 2025.

[2] Group-in-Group Policy Optimization for LLM Agent Training, 2025.

[3] An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents, 2025.

AuthorsOpenReview ↗

Increased observation diversity

We thank the reviewer for this insightful question. When observations stop repeating (i.e., when a step has no peer with the same observation), CCPO does not drop it. It compares the step with all steps of the same task from the other rollouts, under the same contextual weights, so every step still receives credit (task-based matching) when observation diversity is high. We also evaluated CCPO on Search-R1, where observations are retrieved passages instead of states of a fixed environment; the results are in our previous response.

Additionally, we ablated peer matching in two stages while keeping the rest of the methods the same:

  • Task matching only: observation matching is disabled, and every step is compared with all same-task steps from the other trajectories under the same contextual weights. This is the testing scenario for an environment whose observations never repeat.
  • No matching: task identity is removed as well, and peers come from all tasks in the batch.

The experiments used the same ablation settings and the Qwen2.5-1.5B-Instruct backbone:

Peer matchingALFWorldWebShop
CCPO (ours)96.9 ± 0.875.0 ± 0.8
Task only92.4 ± 0.568.2 ± 3.2
No matching83.3 ± 1.2collapses
GRPO75.3 ± 1.056.5 ± 2.9

Without observation matching, performance worsens across benchmarks and settings, and WebShop collapses when peers are not matched by any priors. We hypothesize that without matching, the baseline function estimates the average performance of the entire batch and becomes insensitive to the context of each rollout/task. As a result, the advantage may reflect the difficulty of a task instead of the value of each action/step. WebShop suffers the most under this setting especially due to the shorter horizon of the task and lower success rate when training starts.

Performance also drops less on the task-only variant than the no-matching variant, suggesting that structured grouping is helpful. This agrees with the original results reported in GRPO [1], which estimates group-relative advantage within each prompt (similar to task-based matching). Both GiGPO and HGPO incorporate structured grouping in their methods as well. While CCPO is inspired by these works, recent publications also study the effects of per-prompt normalization in agentic RL [2,3]. We agree that the choice of grouping and normalization method/baseline function is a valuable research direction in agentic RL, and some of our ideas for future work discussed in the paper are related to this line, such as semantic peer retrieval beyond exact observation matching and uncertainty estimates for correlated visits.

In revision, we will append these ablation results and revise our introduction/related work sections to include discussion on grouping and normalization methods in agentic RL. We again thank the reviewer for this insightful question and the opportunity to expand on these important topics.

References:

[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024.

[2] REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization, 2025.

[3] Understanding R1-Zero-Like Training: A Critical Perspective, 2025.

I appreciate authors' response and additional experimental results. However, I am still not convinced the the proposed method can be extended to training for more realistic, complex, long-horizon task. So, I will maintain my current rating.

Thank you for considering the additional experiments. We agree that extending CCPO to longer, open-ended tasks such as software engineering remains an open question due to compute constraints, and we will make that limitation explicit in the revision.

Your original review raised a specific question about whether CCPO retains an advantage over GRPO when observations rarely repeat. To test how much CCPO depends on exact matches, we disabled observation matching at every training step. CCPO still exceeded GRPO by 17.1 percentage points on ALFWorld and 11.7 points on WebShop. We also trained CCPO on Search-R1, where observations are passages retrieved in response to the agent’s own queries. Across seven QA datasets, CCPO averaged 48.8, compared with the published GRPO result of 39.6 on the same 7B backbone.

Given that these results directly address the concern about reliance on repeated observations and extend the evaluation to a third setting, would you be willing to reconsider your overall recommendation in light of this evidence?

Reviewer_cjmVOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CCPO, a critic-free step-level credit assignment method for language-model agents trained with grouped rollouts. For each decision, it selects other trajectories with the same task and observation, weights those peers by a frozen model's prompt representation and simple trajectory-history statistics, and uses the weighted returns to form a historical residual. A second signal is the change in weighted outcome value over a short realized continuation. The combined signal is used in a PPO-style update. Experiments with Qwen2.5-1.5B and 7B on ALFWorld and WebShop compare CCPO with GRPO and several step-level alternatives; the paper also studies peer selection, context features, and the two credit channels.

Critical strengths
  1. The method addresses a real weakness of observation-matched credit assignment: the same observation can occur after very different histories. Query-specific peer weighting and exclusion of the query trajectory are coherent design choices (Section 3.2); Table 2 shows a particularly large deterioration when same-trajectory peers are reintroduced.
  2. The empirical package is broader than a single headline comparison. It covers two model sizes and two interactive environments, includes strong step-level baselines, and ablates peer weighting, context summaries, and historical/future credit (Tables 1-2). The results make the approach worth taking seriously even where the margin over the strongest baseline is narrow.
Critical weaknesses
  1. The paper interprets the future channel as short-horizon progress, but Eq. (7) does not discount-align the later endpoint with the current one, and Appendix C.3, Eq. (10), shows that the signal also contains a nonnegative time-dependent term. A successful rollout can therefore receive a positive difference simply as the terminal reward gets closer, even without a useful action. The ablations compare the future channel with removing it, but do not compare it with a discount-aligned difference or a simple remaining-steps/terminal-proximity control. This leaves the paper's main mechanistic explanation underdetermined.
  2. The decisive performance comparison is less clear than the GRPO gains suggest. In Table 1, the 7B ACC-only variant leads reproduced HGPO by 1.6 points on ALFWorld and 1.8 points on WebShop, while published G2PO/HGPO means are closer still and are not directly paired. At 1.5B on WebShop, ACC-only is below reproduced G2PO; the winning CCPO variant adds an episode-level advantage, which all WebShop ablations retain. With only three training seeds and no uncertainty analysis focused on these nearest comparisons, I cannot tell how much of the claimed gain is robustly attributable to the proposed credit signal.
Questions for authors
  1. Could you compare the proposed future credit with a discount-aligned value difference, and with a control using only remaining steps or terminal proximity, under the same rollout and optimizer budget? Equation (10) explicitly includes a time-dependent term; seeing whether CCPO retains its gain without that term would substantially change my view of the claimed progress mechanism.
  2. Could you report paired or otherwise directly comparable uncertainty estimates for CCPO versus the strongest reproduced step-level baselines on each model/benchmark, especially the 7B results? Please also clarify why the 1.5B WebShop ACC-only variant trails G2PO while ACC + episode advantage wins, and show whether the contextual component adds value when the episode channel is held fixed. These results would determine whether I view the empirical advantage as robust.
  3. What is the incremental training cost of frozen-model feature extraction, batchwise PCA, and peer-distance computation relative to GiGPO/HGPO/G2PO (wall-clock or GPU-hours with the same 19,200 collected trajectories)? This would clarify the practical meaning of the no-critic/no-extra-interaction claim.
4 comments
AuthorsOpenReview ↗

Additional ablations to analyze the design of the proposed future credit term

We thank the reviewer for this insightful question. We ran both controls, and a third variant that applies discount alignment only at the end of a trajectory. The code, rollout budget (150 iterations, 16 tasks × 8 rollouts per iteration) and optimizer settings replicate those of the main run.

  • Discount-aligned: z(γ^h · V_u − V_t) in every window.
  • Terminal-aligned: Eq. (7) in every window except those that reach the terminal step, where the discount-aligned form is used. This variant tests whether those terminal windows account for the gap between Eq. (7) and discount alignment.
  • Terminal proximity only: z(γ^(T−t) · R).

Results are reported in the same format as the ablation experiments in Table 2.

Future termALFWorld AllPickLookCleanHeatCoolPick2WebShop Succ.WebShop Score
CCPO (ours): z(V_u − V_t)96.9 ± 0.8100.0 ± 0.075.2 ± 4.5100.0 ± 0.093.1 ± 0.7100.0 ± 0.097.0 ± 2.675.0 ± 0.885.3 ± 1.6
Discount-aligned: z(γ^h · V_u − V_t)91.9 ± 0.593.9 ± 0.289.3 ± 0.693.5 ± 1.9100.0 ± 0.091.3 ± 0.483.8 ± 2.870.8 ± 3.084.3 ± 0.6
Terminal-aligned: discount-aligned only in windows that reach the terminal step89.3 ± 1.691.9 ± 1.975.2 ± 4.589.6 ± 1.591.4 ± 7.591.3 ± 0.488.2 ± 2.7------
Terminal proximity only: z(γ^(T−t) · R)89.3 ± 0.591.9 ± 1.981.9 ± 7.193.5 ± 1.993.1 ± 0.791.3 ± 0.479.4 ± 2.871.4 ± 3.985.4 ± 0.6

We interpret these results as follows:

  1. A signal that only measures terminal proximity is worse than the original implementation of CCPO using Eq. (7) on both benchmarks (ALFWorld 89.3 against 96.9; WebShop success 71.4 against 75.0). The future channel is therefore not reducible to the time-dependent term and contributes in the same direction across both benchmarks.
  2. The terminal-aligned variant locates the difference between Eq. (7) and discount alignment. Applying discount alignment only in the windows that reach the terminal step lowers ALFWorld from 96.9 to 89.3, which is also below the variant that is discount-aligned. The gap between Eq. (7) and discount alignment therefore likely arises from how the last windows of a trajectory are credited rather than proximity to termination alone.
  3. Eq. (7) is essential to the performance of CCPO. For instance, on AlfWorld, the discount-aligned variant of CCPO (91.9) is only marginally above GiGPO (89.8) and HGPO (90.6) and is 3.2 below G2PO (95.1), whereas full CCPO using Eq. (7) achieves 96.9.

We hypothesize that the benefit of Eq. (7) comes from combining its two parts. The discount-aligned part, γ^h · V_u − V_t, measures whether the context reached after the window has a higher peer value than the one before it. The remaining part, (1 − γ^h) · V_u, is proportional to the value at the end of the window, so it is zero when the window ends in failure and largest when it ends in success. It therefore adds credit for arriving in high-value contexts, concentrated at the end of successful trajectories. The three controls are consistent with this reading: the peer-value difference without the second part (discount-aligned), the second part removed where it is largest (terminal-aligned), and a completion signal without peer values (terminal proximity) all fall below Eq. (7). In revision, we will add the three ablations to Table 2 and clarify the claim about CCPO mechanism that a short-horizon change in peer value is likely a useful credit signal and is not a terminal-proximity signal. We again thank the reviewer for this important question.

AuthorsOpenReview ↗

Paired uncertainty tests against baselines

We thank the reviewer for this important question and the opportunity to conduct further analysis. We agree that the current reporting, mean ± s.d. is a limited comparisons across methods. On ALFWorld, we report the difference between methods over the three evaluations, with a 95% bootstrap confidence interval over tasks (5,000 resamples), the number of tasks only one arm solves, and an exact McNemar test on those tasks.

ComparisonCCPOBCCPO − B95% CISolved by CCPO only / B onlyp
ALFWorld 1.5B: CCPO − G²PO96.9 ± 0.895.1 ± 0.5+1.8 ± 1.2[−2.1, +5.9]6 / 40.75
ALFWorld 1.5B: CCPO − HGPO96.9 ± 0.890.6 ± 0.0+6.3 ± 0.8[+1.3, +11.5]11 / 40.12
ALFWorld 1.5B: CCPO − GiGPO96.9 ± 0.889.8 ± 0.8+7.0 ± 0.8[+2.3, +12.1]11 / 20.022
ALFWorld 7B: CCPO − G²PO97.4 ± 0.594.0 ± 0.5+3.4 ± 0.9+2.9 [−2.9, +8.6]4 / 10.38
ALFWorld 7B: CCPO − HGPO97.4 ± 0.595.8 ± 0.5+1.6 ± 0.8+1.4 [−3.6, +7.1]3 / 30.80
ALFWorld 7B: CCPO − GiGPO97.4 ± 0.592.2 ± 0.8+5.2 ± 0.9+5.7 [−0.7, +12.1]6 / 00.03

On ALFWorld 1.5B and 7B, CCPO is significantly better than GiGPO (p = 0.022; p = 0.03). For HGPO the interval excludes zero on 1.5B but does not reach statistical significance (p = 0.12). Differences in success are not significant of CCPO over G²PO or HGPO on this benchmark.

Therefore, in revision, we will avoid claiming statistical significance of CCPO's advantage in task success on ALFWorld, and explain the other strengths of CCPO with evidence. For example, CCPO solves unique tasks which the baseline method misses (column: solved by CCPO only / B only); on challenging games, agents trained with baseline methods could loop in circles while CCPO efficiently solves the task in fewer steps (Figure 3), achieving lower revisit rates (Table 4). We will include the comparison table above in the Supplementary section and make sure to clarify in the main text.

We also agree with the reviewer that the ALFWorld 7B results are especially limited for comparisons against baselines. At 7B, the LLM agent becomes much more capable and develops similar success rates by the end of the training across methods, despite CCPO achieving the highest (97.4 ± 0.5), it is not statistically significant. We will include this note in revision as well and state the full explanations in the Supplementary section.

We again thank the reviewer for providing this valuable feedback and the opportunity to conduct further analysis.

AuthorsOpenReview ↗

Performance of CCPO variants on WebShop

We thank the reviewer for raising this important question and the opportunity to discuss this result.

1. Why A_CC alone trails G²PO on WebShop 1.5B, and why adding A_EP changes this. G²PO combines an episode-level advantage with its step-level terms. CCPO with A_CC alone has no episode-level term. On WebShop with the 1.5B model, the step-level signal is scarce early in training, for two reasons:

  • Short trajectories. Over the same iterations a WebShop trajectory has 9.6 turns on average, and 5.54 at the end of training. On ALFWorld it has 47.1 and 8.65. Each rollout therefore gives about five times fewer steps to compare. In the first 50 iterations, 25% of the WebShop steps have no peer with the same observation and fall back to task-level peer comparisons, against 11% on ALFWorld. The short horizon also limits the method from fully leveraging the sliding context window, which requires 5 turns per window in the default setting.
  • Few successes. Over the first 30 iterations, only 10.1% of the training rollouts succeed (4.7% over the first 10), due to the limited capability of the 1.5B backbone. With 8 rollouts per task, many tasks have no successful rollout. The lack of variance among peers makes it difficult to extract learning signals with A_CC alone.

The variant includes an episode advantage which does not require matched peers: it gives every step of a successful trajectory the same signal, therefore it helps where the step-level signal is scarce, either due to the trajectories being short or initial model capability limiting success. As shown in the table below, the variant helps in the WebShop 1.5B case alone. On WebShop 7B, the difference between the two variants are within noise.

Effect of adding A_EP to A_CCA_CC + A_EPA_CCDifference
WebShop 1.5B75.0 ± 0.868.8 ± 3.4+6.3 ± 3.6
WebShop 7B78.4 ± 3.979.4 ± 1.6−1.0
ALFWorld 1.5B91.9 ± 0.596.9 ± 0.8−4.9 ± 1.2
ALFWorld 7B94.3 ± 0.597.4 ± 0.5−3.1 ± 0.8

Inspired by these observations, in Section 5, we proposed ideas including adaptive horizons, stage-dependent weighting, semantic peer retrieval beyond exact observation matching, and uncertainty estimates for correlated visits as future research directions. In revision, we will clarify in main text about our rationales and interpretations of the performance across variants on WebShop and ALFWorld. We will include the analysis above in the supplement.

2. Does the contextual component add value when the episode channel is held fixed? Yes. On WebShop 1.5B we kept A_EP on and changed one part of the contextual term at a time:

Variant (+ A_EP in every row)SuccessCCPO − ablation
CCPO (A_CC + A_EP)75.0 ± 0.8—
A_EP alone (GRPO*)56.5 ± 2.9+12.3–18.5
No observation matching (task-level peers only)68.2 ± 3.2+6.8 ± 3.9
Cosine-based context similarity in place of the distance kernel70.3 ± 2.7+4.7
Discount-aligned future credit in place of Eq. (7)70.8 ± 3.0+4.2 ± 3.7
Terminal-proximity future credit in place of Eq. (7)71.4 ± 3.9+3.6 ± 4.7
Historical credit only72.7 ± 2.8+2.3 ± 3.6
No context summary73.7 ± 3.5+1.3 ± 4.0
Future credit only74.0 ± 2.4+1.0 ± 3.2

Adding the contextual term to the episode advantage raises success by up to 18.5 points from 56.5 to 75.0. The full implementation with the contextual term also improves performance over other ablations of the contextual component, including alternative calculation of context similarity, various future credit functions, without context summary or altered context channels (historical or future credit only). In revision, we will include the detailed interpretations of the value of the contextual component and its interaction with the episode term.

We again thank the reviewer for raising these important questions and consider our analysis.

AuthorsOpenReview ↗

Incremental training cost of CCPO

We thank the review for raising this practical question. Compared to baseline methods, CCPO adds no environment interaction, no rollouts and no model forward passes. Its extra cost comes from CPU work in the advantage stage. We timed CCPO, GiGPO, HGPO and G²PO on 8×H200 nodes, using the same configuration as the main runs (Qwen2.5-1.5B, 16 tasks × 8 trajectories per iteration). Times are means over iterations 2–10 (iteration 1 is warm-up). These early iterations have the longest episodes (most number of turns), so per-turn computations cost the most there.

Per iterationALFWorldWebShop
GiGPO iteration time201.5 s86.7 s
HGPO iteration time201.3 s86.2 s
G²PO iteration time (its own codebase)205.9 s88.8 s
CCPO iteration time≈ 207.4 s≈ 88.0 s
Incremental cost of CCPO over the baselines+1.5 to +6.0 s (0.7–3.0%)−0.8 to +1.6 s (−0.9% to +1.8%)
  Feature extraction+0.4–0.6 s+0.1–0.3 s
  Advantage stage (all CCPO credit computation)+5.4–5.5 s+1.4 s
    batch-wise PCA3.0 s1.1 s
    peer distances≤ 0.01 s≤ 0.01 s
  • Feature extraction. φ is the frozen reference model's hidden state at the last prompt token. A hook captures it during the reference log-probability pass. The KL term already requires this pass, and GiGPO and HGPO run it too.
  • Batch-wise PCA. Before taking distances, CCPO centres φ over the batch and removes its top three principal directions. These directions mostly encode the shared style of the text, not the agent's state. φ keeps its full dimension. This takes one SVD per iteration on a single CPU thread.
  • Peer distances. CCPO computes distances only within matched buckets of about five peers.

In summary, CCPO may add up to 3.0% of training time on ALFWorld and up to 1.8% of training time on WebShop during the early iterations. Training time may shorten as the number of turns required to solve a task reduces in later iterations. During inference, on some challenging tasks, CCPO also enables the agent to succeed more efficiently in fewer turns, while other methods may demonstrate high revisit rates causing the agent to circle until budget exhausted (Figure 3, Table 4). We will include the training cost analysis in revision. We thank the reviewer for the opportunity to discuss this practical question.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General summary of responses

We thank Reviewers cjmV (R1), FSji (R2) and QwUm (R3) for their time and insightful questions. We appreciate the strengths you recognized:

  1. A real limitation of step-wise credit estimation: identical observations can follow different histories, and weighting matched peers by context gives a more informative comparison (R1, R3).
  2. Design choices supported by comprehensive and useful ablations (R1, R2, R3).
  3. Broad empirical evaluation over two model sizes, two interactive environments and strong step-level baselines, "the results make the approach worth taking seriously" (R1, R3).

We added the following experiments and results, detailed in the individual responses:

  • Three alternative future-credit terms (R1, R3): replacing Eq. (7) with a discount-aligned (TD-style) difference lowers success from 96.9 to 91.9 on ALFWorld and from 75.0 to 70.8 on WebShop. A terminal-proximity signal gives 89.3 and 71.4. Applying discount alignment only in the windows that reach the terminal step also gives 89.3 on ALFWorld. The future channel is therefore not reducible to its time-dependent term.
  • A third benchmark with diverse observations (R2): on Search-R1 (seven QA datasets; observations are retrieved passages), CCPO with Qwen2.5-7B-Instruct averages 48.8, against published results of 47.2 for GiGPO, 39.6 for GRPO and 38.5 for Search-R1.
  • Peer-matching ablations (R2, R3): with task-level peers only, CCPO reaches 92.4 on ALFWorld and 68.2 on WebShop, still above GRPO (75.3 and 56.5). Without any matching, ALFWorld drops to 83.3 and WebShop training collapses.
  • Training cost (R1): CCPO adds no environment interaction, rollouts or model forward passes. Per iteration during early training (the most costly phase due to longer rollouts), it adds at most 3.0% on ALFWorld and 1.8% on WebShop relative to GiGPO, HGPO and G²PO, while improving agent efficiency at inference on challenging tasks that other methods struggle to solve within turn budgets.
  • Paired uncertainty against the strongest step-level baselines (R1): bootstrap confidence intervals over tasks and exact McNemar tests on ALFWorld. CCPO is significantly above GiGPO at 1.5B and 7B (p = 0.022 and 0.03). Though its differences in task success from G²PO and HGPO are not statistically significant, analysis shows the unique strength of CCPO for solving tasks where other baseline methods fail. We will clarify and tighten relevant claims, and include all analysis results with detailed interpretations in the supplement.
  • Contextual term when episode channel is held fixed, and CCPO variants on WebShop (R1): at 1.5B, WebShop trajectories are short (9.6 turns early in training, against 47.1 on ALFWorld) and few rollouts succeed (10.1% over the first 30 iterations), so the step-level signal is scarce. The episode advantage adds 6.3 points in this setting but does not help on longer tasks or when model is capable to generate variable rollouts early in training. With the episode channel held fixed, the contextual term raises success from 56.5 (GRPO) to 75.0. We will include all analysis results with detailed explanations in revision.

We also expanded the discussion of the following points:

  • What Eq. (7) adds over a discount-aligned difference (R1, R3):
  • Distinct roles of historical and future credit (R3):
  • Steps without matching observations (R2):
  • Grouping and normalization in agentic RL (R2, R3):
  • Future directions such as semantic peer retrieval, uncertainty estimates for correlated visits, normalizing decision-irrelevant details of observations in open-ended environments, and adaptive horizons (R1, R2, R3):

In revision, we will add these experiments and analysis to the paper, qualify the claims of statistical significance, and expand the discussions of limitations and future work.

We hope these results address the reviewers' concerns. We again thank the reviewers for their time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.