- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes Contextual Credit Policy Optimization (CCPO), a critic-free step-level credit assignment method for reinforcement learning of long-horizon LLM agents. CCPO addresses the state-aliasing problem that arises when steps with identical textual observations but different histories are treated as equally informative peers.
The method constructs a context representation by combining hidden states from a frozen reference LLM with hand-designed trajectory statistics, including elapsed steps, observation novelty, and revisit indicators. It then uses context-dependent kernel weights to aggregate returns from observation-matched steps in other trajectories. CCPO combines two credit signals: (1) a historical return residual relative to a contextual peer baseline, and (2) a future-credit term based on the change in contextual outcome value over a short realized continuation.
Experiments with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct on ALFWorld and WebShop show improvements over reproduced GRPO, GiGPO, HGPO, and G2PO baselines. The paper also provides ablations on peer exclusion, contextual weighting, context summaries, and temporal windows, as well as analyses of action preferences and repeated observations.
Critical strengths
-
Context-aware peer weighting addresses a limitation of observation-level grouping. Identical observations may correspond to different agent states due to differences in interaction history or task progress. Weighting matched peers by contextual similarity provides a more informative comparison than treating them equally.
-
The paper provides reasonably broad empirical validation. CCPO is evaluated on two benchmarks and two model sizes, with consistent improvements over several reproduced baselines. The ablations also support key choices such as query-trajectory exclusion, contextual weighting, and structured context features.
Critical weaknesses
-
The motivation and necessity of the future-credit component are not sufficiently established. Historical credit already incorporates future outcomes through the discounted return Y_{i,t}, while the future branch is also constructed from terminal outcomes. Moreover, the history-only H=4, F=0 variant already achieves 94.3%, close to 96.9% with the full H=2, F=2 setting, making the unique contribution of future credit unclear.
-
The primary peer construction relies on exact observation matching, which may limit generalization. This assumption is convenient in ALFWorld and WebShop, where observations are structured and often repeated, but may be restrictive in more realistic environments where semantically similar states rarely share identical observations.
Questions for authors
-
Why is V_{t+K_F} - V_t an appropriate step-level credit signal for the action at step t? The value change is measured over an entire K_F-step continuation and may be driven by subsequent actions within the window. Could the authors clarify the rationale behind attributing this change to the action at the beginning of the window?
-
Given that historical credit already incorporates future outcomes through the discounted return Y_{i,t}, what distinct information does the future-credit branch provide? Both branches also use history-conditioned contextual peers, making their claimed complementarity less clear.
-
Have the authors compared the proposed future credit with simpler alternatives derived from the same outcome signal? For example, direct episode-level/final-reward advantage, one-step value differences, or a TD-style residual would help determine whether the specific V_{t+K_F} - V_t formulation is necessary.
-
Could future work explore semantic or learned peer matching beyond exact observation matching, particularly for environments with more diverse observations?
6 comments
Design of the future credit term
We thank the reviewer for this insightful question. We have the following comparisons, all with the same rollout and optimizer budget as the standard ablation experiments. The TD-style and final-reward variants are the two controls also requested by Reviewer cjmV. Our response to Reviewer cjmV (Q1) reports them by task type, together with a third, terminal-aligned variant.
| Method | Setup | ALFWorld | WebShop |
|---|---|---|---|
| CCPO (ours): Eq. (7) | z(V_{t+2} − V_t) | 96.9 ± 0.8 | 75.0 ± 0.8 |
| TD-style residual/discount-aligned | z(γ^h · V_{t+h} − V_t); rewards are terminal, so this is the multi-step TD residual of the peer value | 91.9 ± 0.5 | 70.8 ± 3.0 |
| Final-reward signal/terminal-proximity | z(γ^(T−t) · R), no peer value | 89.3 ± 0.5 | 71.4 ± 3.9 |
| No future term | historical credit only (Table 2) | 91.4 ± 0.8 | 72.7 ± 2.8 |
| Shrink future term | asymmetric historical context and future context (Table 2) | 92.7 ± 0.9 | --- |
| Episode-level advantage alone | GRPO (Table 1) | 75.3 ± 1.0 | 56.5 ± 2.9 |
The TD-style variant is lower than Eq. (7) on both benchmarks (by 5.0 points on ALFWorld and 4.2 on WebShop). A final-reward-only signal is also lower on both (by 7.6 and 3.6 points). Removing the future term lowers ALFWorld by 5.5 points; on WebShop it is marginally lower and within noise of the full method, potentially due to the shorter horizon of the WebShop task and lower initial success rate, both are discussed in the main text or supplementary sections of the paper. Shrinking the future term by asymmetrically varying the context window also lowers performance.
We hypothesize that the benefit of Eq. (7) comes from combining its two parts. The discount-aligned part, γ^h · V_u − V_t, measures whether the context reached after the window has a higher peer value than the one before it. The remaining part, (1 − γ^h) · V_u, is proportional to the value at the end of the window, so it is zero when the window ends in failure and largest when it ends in success. It therefore adds credit for arriving in high-value contexts, concentrated at the end of successful trajectories. The ablations/controls are consistent with this reading. In revision, we will add the ablations to Table 2 and clarify the claim about CCPO mechanism that a short-horizon change in peer value is likely a useful credit signal and is not a terminal-proximity signal. We again thank the reviewer for this important question.
Future work for semantic/learned peer matching
We thank the reviewer for this insightful suggestion. We agree, and the paper includes ideas related to this direction. Section 5 lists it as a limitation ("exact observation matching can limit peer availability") and names "semantic peer retrieval beyond exact observation matching" as future work. We would like to expand this discussion by including the new ablation results on peer matching methods. Interpretations of these results are summarized below and reported in our second response to Reviewer FSji. The experiments used the same ablation settings and the Qwen2.5-1.5B-Instruct backbone:
| Peer matching | ALFWorld | WebShop |
|---|---|---|
| CCPO (ours) | 96.9 ± 0.8 | 75.0 ± 0.8 |
| Task only | 92.4 ± 0.5 | 68.2 ± 3.2 |
| No matching | 83.3 ± 1.2 | collapses |
| GRPO | 75.3 ± 1.0 | 56.5 ± 2.9 |
Based on the ablation results, we develop the following thoughts:
-
Structured grouping decides which steps are eligible peers, and semantic matching with frozen-LLM representations decides how much each eligible peer contributes. The two methods are complementary, as shown by our ablation studies: removing either component or loosening the gate (such as in the matching by task-only variant) reduces performance.
-
The choice of grouping method has a bias-variance trade-off. Matching based on exact observations may leave some steps with few or no peers (Introduction; Section 3.2). However, loosening the gate admits peers that faced different decision points, biasing the baseline's estimation of how good an action is. The no-matching ablation demonstrates this failure - the baseline may reflect task difficulty when peers are drawn across tasks, and ALFWorld drops to 83.3 while WebShop collapses.
Our ideas for future work (Section 5) include:
- Adaptive semantic peer retrieval. When a step has too few exact matches (high diversity among observations), retrieve peers by context similarity, adaptively widening the peer set until a minimum number of peers is reached. This may be combined with memory mechanisms as well.
- Uncertainty estimates for correlated visits. When variance among peers is low between correlated visits (such as repeated visits within a rollout), the baseline estimate may overstate this contribution. Uncertainty quantification can potentially help reduce such credit inflation when support is weak.
- Open-ended environments. Observations may differ in details that are irrelevant for the decision (timestamps, page layout, long tool outputs). They can be normalized before matching. In our response to Reviewer FSji, we also discussed related works that study the effects of per-prompt normalization in agentic RL.
In revision, we will expand the future-work paragraph of Section 5 along these lines in either main text or the supplementary section. We again thank the reviewer for this insightful question and the opportunity to expand this discussion.
Credit signal for the action at step t; separating the effects of historical and future credits
We thank the reviewer for these thoughtful questions, and we would like to respond to both Q1 and Q2 in the following response. The reviewer is right that the value change over a K_F-step window depends on every action in the window, not only on a_t. We do not claim that it isolates the effect of a_t. This design is motivated by the following rationales:
Crediting a multi-step outcome to the first action is standard in policy gradient methods. GAE [1] is built from k-step advantage estimates, which credit a_t with the rewards and value change over the next k steps. Later actions in the window contribute and add variance to the estimate for a_t. Our signal is analogous with a short window.
Because identical observations can conceal different task progress (Section 4.4), V_{t+K_F} − V_t therefore asks: two steps after a_t, is the agent in a context from which its peers succeeded more often than from the context before a_t? V_t is computed before a_t is generated, so it depends only on the state and serves as a baseline for a_t.
We hypothesize that historical credit contributes differently than future credit, and our analyses separate the effects of the two channels (Section 4.4). For instance, Figure 4(b) plots historical, future and combined credit over training: the ratio of future to historical magnitude rises from 0.25 to 0.60 at 1.5B and from 0.32 to 1.01 at 7B between iterations 1–30 and 121–150, suggesting distinct and evolving contributions of the two channels during training. Additionally, Figure 5 shows that the future channel might favor exploratory transitions. The overall benefit of the designs of CCPO are partially illustrated in Figure 3, showing that one of the evaluation games CCPO is the only method that trains the agent to solve a task efficiently and effectively (succeeded in 7 steps vs. baselines that exhaust the 50 step budget without success).
We will revise the text to describe the signal as a short-horizon change in peer value following a_t, not as the isolated effect of a_t. Additional ablation experiments on the future credit method are discussed in the next response and will be included during revision. We again thank the reviewer for this insightful question and the opportunity to expand the discussion.
References:
[1] High-Dimensional Continuous Control Using Generalized Advantage Estimation, 2016.