- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a counterfactual-supervised credit assignment framework that uses environment-grounded Monte Carlo advantages to guide an LLM-based retrospective evaluator, whose predicted turn-level advantages are then used for critic-free policy optimization in language model agents.
Critical strengths
-
The paper addresses an important problem in agentic reinforcement learning: fine-grained credit assignment for long-horizon tasks.
-
The proposed framework uses environment-grounded counterfactual rollouts to supervise an LLM-based credit selector, providing a meaningful improvement over purely heuristic hindsight evaluation.
-
The paper provides relatively comprehensive implementation details and evaluates the method across multiple benchmarks and model scales, demonstrating promising empirical improvements.
Critical weaknesses
-
The proposed method mainly combines Monte Carlo advantage estimation, LLM-based retrospective evaluation, prompt optimization, and PPO-style policy updates. The individual components are largely established, and the primary contribution lies in their integration rather than a fundamentally new credit assignment mechanism.
-
The optimized selector uses three evolved prompts instead of a single prompt, introducing additional inference and aggregation effects. The experiments do not adequately isolate whether the improvements come from counterfactual supervision itself or from stronger prompt engineering and increased inference compute.
-
The gold advantage is explicitly policy-dependent, whereas the selector is frozen throughout actor training. It remains unclear whether a fixed retrospective evaluator can reliably estimate relative advantages as the actor policy changes, especially when the policy moves beyond the distribution represented in the supervision data.
-
Despite optimization, the selector achieves only 44.70% precision and 46.08% recall for negative-credit prediction. This raises concerns about the reliability of the resulting advantage estimates and the potential impact of incorrect credit assignment during training.
Questions for authors
-
Can the authors provide an ablation that isolates the contribution of counterfactual supervision from prompt optimization and multi-prompt aggregation, ideally under matched inference budgets?
-
How does the selector's credit prediction accuracy change as the actor policy evolves? Have the authors evaluated its calibration or sign agreement across different training checkpoints?
-
How sensitive are the results to the correctness–efficiency utility design? Would a success-only counterfactual advantage provide similar improvements?
-
Given the relatively low negative-credit precision, how sensitive is downstream RL performance to incorrect advantage signs, particularly for high-magnitude predictions?
4 comments
Additional outcome-only critic-free comparison
We have now completed an additional outcome-only, critic-free comparison using standard GRPO at both actor scales (seed 42). Each update uses eight groups of eight trajectories, matching Full's 64-trajectory batch.
On SWE-Pro, GRPO obtains 23.15% at 4B and 29.86% at 9B, compared with 30.41% and 36.16% for Full. Across the four benchmarks and two scales, Full has higher success rates in seven of eight settings; GRPO is higher on 9B LiveCodeBench (41.98% versus 41.22%). Full also reduces mean turns by 16.41–40.95% across SWE-Pro, Terminal-Bench, and BFCL.
This adds a comparison in which both methods use no learned value critic, complementing the equal-call Initial-repeat3 experiment in our earlier response. The complete results and configuration are provided in our response to Reviewer cGye.
The authors' response has addressed most of my concerns, and I will raise my rating to 3. However, I would still appreciate further clarification on the reliability of the frozen selector as the actor policy evolves, particularly given the policy-dependent nature of the target advantages.
Thank you for your positive reassessment and for highlighting this remaining question. We evaluated the 4B seed42 actor at updates 0, 64, and 128 on the same 32 tasks, sampling three turns per trajectory (96 turns per checkpoint). The selector remained fixed: DeepSeek v4.1 Flash with v5/v6/v8 and the original aggregation rule.
At each checkpoint, we recomputed MC targets using that checkpoint’s actor for alternative-action sampling and all continuations (, ). The frozen selector scored the corresponding full trajectories.
| Actor updates | High-impact turns | Weighted sign agreement | Negative precision | Negative recall |
|---|---|---|---|---|
| 0 | 34 | 74.92% | 47.17% | 44.64% |
| 64 | 31 | 74.08% | 46.81% | 46.81% |
| 128 | 29 | 70.45% | 53.49% | 46.94% |
High-impact turns satisfy ; sign agreement is weighted by .
From update 0 to 128, high-impact agreement decreases by 4.47 percentage points, while negative precision increases by 6.32 points and recall by 2.30 points. Our inspection identified beneficial high-impact actions receiving negative credit among the late-stage errors. These errors carry greater weight in the agreement metric, explaining why it can decline while negative precision improves.
The audit shows that the frozen selector maintains substantial agreement with current-policy targets across training, with some late-stage drift. We will include these results in the revision.
Thank you for the careful review and the questions on MC supervision and credit reliability.
Contribution and MC supervision.
MC supervision defines what the selector should predict. We measure the difference between an observed action and same-policy alternatives through repeated environment execution, then use these targets to develop the scoring configuration. The prompts and their aggregation are optimized against this supervision. The resulting evaluator provides turn-level advantages for critic-free training, making the expensive execution-based credit reusable without online counterfactual branching.
To examine whether additional evaluator calls explain the improvement, we added Initial-repeat3: the original Appendix F prompt called independently three times. It uses the same aggregation, actor initialization, and 128 actor updates as Full.
The MC-developed configuration retains a 4.11-point advantage after equalizing the call count. The three-prompt choice is itself an outcome of this development process: MC feedback guides both the scoring instructions and how they are combined.
Policy changes during training.
We address policy variation through mixed-policy supervision. The development corpus includes trajectories from 4B and 9B actors, both before and after RL, covering different capabilities and training stages. Each MC label is computed relative to its corresponding source policy.
The frozen configuration then scores each new training trajectory using its actions, subsequent tool feedback, and outcome. Its rules stay fixed, while the evidence it evaluates comes from the current actor. We reuse this configuration across both scales and all three training seeds.
The correctness–efficiency utility.
We examined what efficiency contributes to the labels by keeping the sampled actions and continuations fixed and recomputing their utility as success +1, failure −1. This comparison uses 50 trajectories, 544 turns, and 113,141 valid continuations.
The zero-credit share increases by 30.51 points (95% task-bootstrap interval: [22.35, 39.06]). When every continuation succeeds, the observed action and its alternatives all have success-only value Q=1, so their relative credit becomes zero.
For example, at
smbprotocol-264, turn 5, all continuations succeed, but the observed finish action takes 6.00 total turns versus 9.59 for alternatives. Its credit is +0.3698 under our utility and 0 under success-only utility. Assigning zero utility to inefficient successes preserves a learning signal for timely completion even when eventual success is saturated.Incorrect signs and their effect on learning.
Negative precision counts all predicted-negative turns equally. To understand the training effect, we also need to consider the coefficients attached to those signs. The reported 83.43% weighted sign agreement measures prediction quality on high-impact MC labels, where ∣AMC∣≥0.25. We added a probe to measure the actor’s sensitivity to the coefficients it actually receives.
For each of four 64-trajectory batches, we restore the same 4B step-64 actor and optimizer state and perform one update per branch. Full keeps the original credit; Flip-small reverses the smallest 20% of nonzero training coefficients; Flip-large reverses the largest 20%. Both interventions flip 1,820 turns in total while preserving coefficient magnitudes. Matching the starting model and batch lets us examine the immediate effect of changing the credit signs.
We then evaluate correctness–efficiency utility in the environment, using the same Pre-derived length reference. Below, ΔU=Uˉ−UˉPre.
Flipping small coefficients leaves the mean gain close to Full, while flipping the same number of large coefficients reduces the gain by 89.7%. This is why we assess credit quality through both sign agreement and the strength of the signal used for training.