acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Supervision for Credit Assignment in Language Model Agents

Abstract

We develop a shared turn-credit selector using counterfactual supervision from 4B and 9B policies before and after reinforcement learning. Successful coding trajectories can contain harmful actions, while failed ones can contain useful decisions. To distinguish these local roles, we restore the state before each action and repeatedly execute both the observed action and same-policy alternatives in the environment. Their difference in average correctness–efficiency utility defines a gold credit label. These labels guide development of a parameter-frozen retrospective selector, which supplies the actor's complete turn-level advantage without online counterfactual branching. Each score is shared by the turn's action tokens in a clipped policy objective; training uses no learned value critic, GAE, or additional outcome advantage. Across four evaluation sets and both actor scales, our method exceeds all four trained baselines. SWE-bench Pro gains over PPO are 4.52 and 4.79 percentage points at 4B and 9B, respectively. At 9B, it matches the Base model on LiveCodeBench while exceeding the other trained methods. Credit-removal experiments distinguish effective learning from simple trajectory compression. Matched-policy probes further support improved local action selection under a shared continuation policy.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, credit assignment, language model agents, counterfactual supervision, turn-level advantages, process supervision, coding agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cuHVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a counterfactual-supervised credit assignment framework that uses environment-grounded Monte Carlo advantages to guide an LLM-based retrospective evaluator, whose predicted turn-level advantages are then used for critic-free policy optimization in language model agents.

Critical strengths
  1. The paper addresses an important problem in agentic reinforcement learning: fine-grained credit assignment for long-horizon tasks.

  2. The proposed framework uses environment-grounded counterfactual rollouts to supervise an LLM-based credit selector, providing a meaningful improvement over purely heuristic hindsight evaluation.

  3. The paper provides relatively comprehensive implementation details and evaluates the method across multiple benchmarks and model scales, demonstrating promising empirical improvements.

Critical weaknesses
  1. The proposed method mainly combines Monte Carlo advantage estimation, LLM-based retrospective evaluation, prompt optimization, and PPO-style policy updates. The individual components are largely established, and the primary contribution lies in their integration rather than a fundamentally new credit assignment mechanism.

  2. The optimized selector uses three evolved prompts instead of a single prompt, introducing additional inference and aggregation effects. The experiments do not adequately isolate whether the improvements come from counterfactual supervision itself or from stronger prompt engineering and increased inference compute.

  3. The gold advantage is explicitly policy-dependent, whereas the selector is frozen throughout actor training. It remains unclear whether a fixed retrospective evaluator can reliably estimate relative advantages as the actor policy changes, especially when the policy moves beyond the distribution represented in the supervision data.

  4. Despite optimization, the selector achieves only 44.70% precision and 46.08% recall for negative-credit prediction. This raises concerns about the reliability of the resulting advantage estimates and the potential impact of incorrect credit assignment during training.

Questions for authors
  1. Can the authors provide an ablation that isolates the contribution of counterfactual supervision from prompt optimization and multi-prompt aggregation, ideally under matched inference budgets?

  2. How does the selector's credit prediction accuracy change as the actor policy evolves? Have the authors evaluated its calibration or sign agreement across different training checkpoints?

  3. How sensitive are the results to the correctness–efficiency utility design? Would a success-only counterfactual advantage provide similar improvements?

  4. Given the relatively low negative-credit precision, how sensitive is downstream RL performance to incorrect advantage signs, particularly for high-magnitude predictions?

4 comments

Thank you for the careful review and the questions on MC supervision and credit reliability.

Contribution and MC supervision.

MC supervision defines what the selector should predict. We measure the difference between an observed action and same-policy alternatives through repeated environment execution, then use these targets to develop the scoring configuration. The prompts and their aggregation are optimized against this supervision. The resulting evaluator provides turn-level advantages for critic-free training, making the expensive execution-based credit reusable without online counterfactual branching.

To examine whether additional evaluator calls explain the improvement, we added Initial-repeat3: the original Appendix F prompt called independently three times. It uses the same aggregation, actor initialization, and 128 actor updates as Full.

ConfigurationCalls per trajectorySWE-bench Pro
Initial-repeat33192/730 — 26.30%
Full3222/730 — 30.41%

The MC-developed configuration retains a 4.11-point advantage after equalizing the call count. The three-prompt choice is itself an outcome of this development process: MC feedback guides both the scoring instructions and how they are combined.

Policy changes during training.

We address policy variation through mixed-policy supervision. The development corpus includes trajectories from 4B and 9B actors, both before and after RL, covering different capabilities and training stages. Each MC label is computed relative to its corresponding source policy.

The frozen configuration then scores each new training trajectory using its actions, subsequent tool feedback, and outcome. Its rules stay fixed, while the evidence it evaluates comes from the current actor. We reuse this configuration across both scales and all three training seeds.

The correctness–efficiency utility.

We examined what efficiency contributes to the labels by keeping the sampled actions and continuations fixed and recomputing their utility as success +1+1, failure −1-1. This comparison uses 50 trajectories, 544 turns, and 113,141 valid continuations.

UtilityZero-credit turnsMean absolute credit
Correctness–efficiency77/544 — 14.15%0.1354
Success-only243/544 — 44.67%0.0744

The zero-credit share increases by 30.51 points (95% task-bootstrap interval: [22.35, 39.06]). When every continuation succeeds, the observed action and its alternatives all have success-only value Q=1Q=1, so their relative credit becomes zero.

For example, at smbprotocol-264, turn 5, all continuations succeed, but the observed finish action takes 6.00 total turns versus 9.59 for alternatives. Its credit is +0.3698 under our utility and 0 under success-only utility. Assigning zero utility to inefficient successes preserves a learning signal for timely completion even when eventual success is saturated.

Incorrect signs and their effect on learning.

Negative precision counts all predicted-negative turns equally. To understand the training effect, we also need to consider the coefficients attached to those signs. The reported 83.43% weighted sign agreement measures prediction quality on high-impact MC labels, where ∣AMC∣≥0.25|A^{\mathrm{MC}}|\geq 0.25. We added a probe to measure the actor’s sensitivity to the coefficients it actually receives.

For each of four 64-trajectory batches, we restore the same 4B step-64 actor and optimizer state and perform one update per branch. Full keeps the original credit; Flip-small reverses the smallest 20% of nonzero training coefficients; Flip-large reverses the largest 20%. Both interventions flip 1,820 turns in total while preserving coefficient magnitudes. Matching the starting model and batch lets us examine the immediate effect of changing the credit signs.

We then evaluate correctness–efficiency utility in the environment, using the same Pre-derived length reference. Below, ΔU=Uˉ−UˉPre\Delta U=\bar U-\bar U_{\mathrm{Pre}}.

ConfigurationMean utility Uˉ\bar UΔU\Delta U vs. Pre
Pre−0.6200660
Full−0.555921+0.064145
Flip-small−0.554276+0.065790
Flip-large−0.613487+0.006579

Flipping small coefficients leaves the mean gain close to Full, while flipping the same number of large coefficients reduces the gain by 89.7%. This is why we assess credit quality through both sign agreement and the strength of the signal used for training.

AuthorsOpenReview ↗

Additional outcome-only critic-free comparison

We have now completed an additional outcome-only, critic-free comparison using standard GRPO at both actor scales (seed 42). Each update uses eight groups of eight trajectories, matching Full's 64-trajectory batch.

On SWE-Pro, GRPO obtains 23.15% at 4B and 29.86% at 9B, compared with 30.41% and 36.16% for Full. Across the four benchmarks and two scales, Full has higher success rates in seven of eight settings; GRPO is higher on 9B LiveCodeBench (41.98% versus 41.22%). Full also reduces mean turns by 16.41–40.95% across SWE-Pro, Terminal-Bench, and BFCL.

This adds a comparison in which both methods use no learned value critic, complementing the equal-call Initial-repeat3 experiment in our earlier response. The complete results and configuration are provided in our response to Reviewer cGye.

The authors' response has addressed most of my concerns, and I will raise my rating to 3. However, I would still appreciate further clarification on the reliability of the frozen selector as the actor policy evolves, particularly given the policy-dependent nature of the target advantages.

Thank you for your positive reassessment and for highlighting this remaining question. We evaluated the 4B seed42 actor at updates 0, 64, and 128 on the same 32 tasks, sampling three turns per trajectory (96 turns per checkpoint). The selector remained fixed: DeepSeek v4.1 Flash with v5/v6/v8 and the original aggregation rule.

At each checkpoint, we recomputed MC targets using that checkpoint’s actor for alternative-action sampling and all continuations (K=12K=12, M=16M=16). The frozen selector scored the corresponding full trajectories.

Actor updatesHigh-impact turnsWeighted sign agreementNegative precisionNegative recall
03474.92%47.17%44.64%
643174.08%46.81%46.81%
1282970.45%53.49%46.94%

High-impact turns satisfy ∣AMC∣≥0.25|A_{\mathrm{MC}}|\geq0.25; sign agreement is weighted by ∣AMC∣|A_{\mathrm{MC}}|.

From update 0 to 128, high-impact agreement decreases by 4.47 percentage points, while negative precision increases by 6.32 points and recall by 2.30 points. Our inspection identified beneficial high-impact actions receiving negative credit among the late-stage errors. These errors carry greater weight in the agreement metric, explaining why it can decline while negative precision improves.

The audit shows that the frozen selector maintains substantial agreement with current-policy targets across training, with some late-stage drift. We will include these results in the revision.

Reviewer_cGyeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a Counterfactual Supervision framework to address the credit assignment challenge in long-horizon LLM agent tasks, where traditional RL methods often fail due to sparse rewards or miscalibrated value estimators. The proposed approach computes environment-grounded Monte Carlo gold credits by restoring states and sampling counterfactual actions, then distills these expensive measurements into a frozen, reusable retrospective selector via an external LLM to avoid online branching overhead. By replacing traditional value critics with this selector’s turn-level advantages within a PPO-style objective, the framework enables critic-free policy optimization that consistently outperforms strong baselines like PPO and CriticSearch across complex benchmarks, yielding improvements in both task success rates and trajectory efficiency.

Critical strengths
  1. Unlike previous heuristic rules or black-box reward models relying on language models' verbal critique, this paper constructs gold credits through real virtual machine state restoration and on-policy counterfactual sampling. By doing so, the authors provides clear definitions and empirical measurability.

  2. Although counterfactual branch search is precise, it is expensive. To address this, the authors transform expensive environment probing into reusable scoring rules by training a fixed retrospective selector. This makes subsequent reinforcement learning training free of online counterfactual branches.

  3. The proposed framework directly utilizes the selector's predicted advantages alongside PPO clipped updates. On the one hand, this successfully eliminates the value function estimation problem in long-horizon reinforcement learning. On the other hand, the training architecture is simplified while policy optimization stability is ensured.

  4. The paper not only conducts extensive evaluations across multiple mainstream code agent benchmarks, but also explores the rationality of negative credits and internal mechanisms through credit-removal ablations and behavioral audits.

Critical weaknesses
  1. Constructing the mixed-policy gold corpus requires restoring the environment and running massive counterfactual continuations for every single turn across numerous trajectories. While online branching is avoided during training, the computational overhead of offline annotation is massive, and the discussion on total compute cost is somewhat insufficient.

  2. The shared retrospective selector relies on a powerful external model for prompt evolution and scoring. The extent to which the selector's performance is bottlenecked by this base model requires further validation. Moroever, whether it can smoothly generalize to weaker or entirely different base models also requires deeper analysis and validation.

  3. Despite employing signed max-absolute aggregation and PPO clipping, residual errors may still exist in the credit labels predicted by the frozen selector. Ablation experiments show that removing certain negative credits exerts complex impacts on training dynamics. This indicates that potential side effects of incorrect sign assignments on policy optimization are not yet fully eliminated.

Questions for authors
  1. The shared retrospective selector was developed on a specific set of code agent tasks via prompt evolution. If this selector is directly applied to agent tasks in completely different domains (such as web interaction, multimodal embodied agents, etc.), what would its zero-shot generalization capability look like?

  2. What are the total costs of annotation, prompt development, and online scoring, and do the gains persist against outcome-only critic-free baselines and process-feedback baselines under matched computational budgets?

  3. During prompt evolution and multi-prompt aggregation of the selector, how destructive are sign prediction disagreements to actor policy training? Have the authors evaluated the potential of introducing uncertainty estimation to filter out low-confidence selector scores?

  4. How stable are the credit signs under increased continuation budgets, and would uncertainty-aware filtering or weighting improve downstream training?

4 comments

Thank you for the careful review and for highlighting the questions about cost, evaluator dependence, and credit reliability. We have added a GRPO baseline and further analyses addressing these points.

Cost and the outcome-only baseline.

Annotating the 300-trajectory corpus, including the main retries, required approximately 15,000 Ascend 910B accelerator-hours. Prompt development used 15 rounds. Both the corpus and the resulting scoring configuration are reusable across actor runs and training seeds.

Online training uses three external API calls per trajectory. At 128 updates and 64 trajectories per update, the planned scoring budget is 24,576 calls per run. These calls run on remote inference infrastructure without requiring additional local training GPUs. Annotation cost depends strongly on continuation length and rollout throughput, while this online call count is fixed by the training schedule.

To compare against an outcome-only, critic-free method, we trained GRPO at both 4B and 9B. Each update uses eight groups of eight trajectories, matching Full’s 64-trajectory batch. GRPO uses group-standardized terminal rewards; groups with identical outcomes are retained with zero advantage. The existing CriticSearch and CAPO results provide the process-feedback comparisons.

All results below use seed 42.

ScaleMethodSWE-ProTBBFCLLCB
4BGRPO23.1524.725.1331.30
4BFull30.4128.096.1335.11
9BGRPO29.8631.4618.3841.98
9BFull36.1638.2019.1341.22

Full has higher scores in seven of eight settings, including SWE-Pro gains of 7.26 points at 4B and 6.30 points at 9B. GRPO scores higher on 9B LCB by 0.76 points. On the three multi-turn benchmarks, Full also completes trajectories in fewer turns:

ScaleSWE-ProTerminal-BenchBFCL
4B: turn reduction vs. GRPO26.45%40.95%16.41%
9B: turn reduction vs. GRPO33.42%39.27%22.48%

This comparison addresses whether the gains persist against a method that also removes the learned critic: Full improves task performance in most settings while reducing the number of actions taken.

To match evaluator-call counts, Initial-repeat3 uses the original Appendix F prompt three times. With 128 actor updates for both configurations, it obtains 26.30% on 4B SWE-Pro, versus 30.41% for Full.

Evaluator dependence and transfer.

MC labels supply the target for developing the scoring configuration. The prompts and aggregation are developed against these execution labels, making expensive MC measurements reusable during online training. This lets us adapt the configuration to different evaluators, whose error patterns can differ substantially.

We evaluated alternative configurations on a common set of 172 trajectories and 2,083 turns. Both Qwen rows use thinking mode and three scoring calls; GPT uses one call, and Flash uses three.

Evaluator configurationHigh-impact weighted sign agreementNegative recallNegative precision
Qwen3.5-35B-A3B, initial62.73%20.41%49.65%
Qwen3.5-35B-A3B, adapted65.07%32.80%46.68%
GPT-6.1-sol, adapted71.00%50.44%56.08%
DeepSeek v4.1 Flash, Full77.96%45.34%43.80%

Qwen adaptation increases negative recall and high-impact agreement, with a decrease in precision. GPT has higher negative precision, while Flash achieves the highest high-impact agreement in this comparison. These results support adapting scoring instructions and aggregation to each evaluator.

The same principle guides transfer to another agent domain. For a web or simulated embodied environment with restorable states, we would construct MC targets using that domain’s outcomes and adapt the scoring rules to navigation or control actions. The reusable part is the execution-supervision procedure; the scoring instructions are developed against the relevant environment feedback.

Prompt disagreement and the effect of sign errors.

We examined disagreement to identify predictions that might benefit from confidence weighting. Among 2,259 audited turns, 393 (17.40%) contain both positive and negative prompt scores. When signed-maxabs selects the minority sign, 42/96 turns (43.75%) have the opposite sign to MC Gold, compared with 32.91% when all three prompts agree on a nonzero sign. Minority-sign selection therefore provides a concrete risk indicator.

We then tested how sign perturbations affect an actual policy update. For each of four 64-trajectory batches, all branches independently start from the same 4B step-64 actor and optimizer state. Each branch performs one update. Full keeps the original credit; Flip-small reverses the 20% of nonzero turn coefficients with the smallest absolute values; Flip-large reverses those with the largest absolute values. Each intervention flips 1,820 turns in total, preserving coefficient magnitudes, token masks, and the loss denominator.

The updated actors are evaluated with eight executions per task/state. A separate set of 48 Pre rollouts per task supplies the shared length reference. Utility is +1 for successful voluntary completion within this reference length, 0 for other successes, and −1 for checker failure. Here ΔU=Uˉ−UˉPre\Delta U = \bar{U} - \bar{U}_{\mathrm{Pre}}.

ConfigurationMean utilityDelta U vs. Pre
Pre−0.6200660
Full−0.555921+0.064145
Flip-small−0.554276+0.065790
Flip-large−0.613487+0.006579

Small-coefficient flips leave the mean gain close to Full, whereas large-coefficient flips reduce it by 89.7%. Together, the disagreement audit and update probe suggest targeting large, disputed predictions when designing uncertainty weighting.

Stability under additional MC continuations.

To check reproducibility of the labels, we selected 60 positions from 50 tasks, with 12 positions in each of five strata: strong positive, weak positive, zero, weak negative, and strong negative. Strong credit is defined by ∣AMC∣≥0.25|A^{\mathrm{MC}}|\geq 0.25 using the original labels.

We fixed the actions, source policies, and efficiency thresholds, then added 16 independent continuations for each observed action and each of its 12 alternatives: 12,480 new valid continuations.

Original credit groupPositionsSame sign with independent new 16Same sign after pooling to 32
Strong nonzero2424/2424/24
Weak nonzero2422/2423/24
Zero1212/1212/12
Total6058/6059/60

The independent replication retains 96.67% of signs, including all 24 strong-credit positions. Pooling the additional samples raises agreement with the original labels to 98.33%. In this audit, the sign changes occur in the weak-credit group.

Thank you for taking my feedback seriously and for incorporating a substantial amount of additional experimentation and analysis. I particularly appreciate the comparative experiments involving GRPO, as well as the further validation regarding the stability of the credit-sign. However, I remain concerned: when accounting for the computational costs of offline labeling, prompt development, and online scoring, and assuming a consistent end-to-end computational budget, does this method still offer an advantage over GRPO? Consequently, I am maintaining my original score (3).

AuthorsOpenReview ↗

Computational cost and reuse of execution supervision

Thank you for clarifying the remaining cost question. End-to-end accounting should include offline annotation, prompt development, and online scoring, including inference performed on external hardware.

Offline annotation consists of environment execution and model inference, without backpropagation or optimizer-state requirements. This allows it to run on separate inference resources. The approximately 15,000 Ascend 910B accelerator-hours reported above represent an upfront investment in execution supervision.

Our recorded training-success curves provide additional context for the optimization budget. Full reaches its peak training success around updates 96–128, whereas GRPO peaks around updates 128–162 and subsequently plateaus at a lower training success rate under the current configuration. These curves show Full reaching a higher training-success level earlier.

Once developed, the annotation corpus and scoring configuration can support successive actor-training runs across model scales and seeds. For repeated training on related tasks, annotation and prompt development are shared costs: their amortized contribution per run decreases as more runs reuse the same supervision. Rollout, policy updates, and online scoring remain recurring expenses. This reuse is central to our design: expensive environment measurements are converted into a scoring configuration that can be applied throughout subsequent training.

The resulting agents also offer an efficiency benefit during use. On independent evaluations, Full achieves higher success rates in seven of eight scale–benchmark settings. It uses 16.41–40.95% fewer turns than GRPO across the six multi-turn settings covering SWE-Pro, Terminal-Bench, and BFCL. Thus, the benefits include both stronger task performance and more concise execution.

Our main contribution is using environment-grounded counterfactual supervision to develop a reusable selector for critic-free, turn-level learning. We hope this discussion clarifies the cost structure and the role of reuse in practical applications. We appreciate your thoughtful feedback and would be happy to address any further questions.

Reviewer_QH16OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies turn-level credit allocation for LLM-encoded agents: successful trajectories may also contain harmful actions, and failure trajectories may also contain useful decisions. The method is divided into three steps: (1) Counterfactual golden annotation - restore the interaction state of each step, repeatedly execute the observed action and K alternative actions with the same frozen strategy, and define the round-level Monte Carlo credit based on the difference between the "correctness + efficient success" utility; (2) Shared backtracking selector - use DeepSeek v4.1 Flash to develop an offline scorer with frozen parameters, reusing the 4B/9B mixed strategy corpus (300 (3) Critic-free round-level learning - the selector prediction is used as a complete round advantage, and the actor is updated with a PPO-style clipping target, without using value network, GAE or additional result advantages. Verification: Two scales × four evaluation sets (SWE-bench Pro, Terminal-Bench 2.1, BFCL, LiveCodeBench) all exceed four training baselines.

Critical strengths

Complete and verifiable method loop: real execution provides a testable credit reference, successfully distilled into an offline selector that avoids online counterfactual branching at training time — a clean, credible path from environment measurement to a reusable scoring rule.

Critical weaknesses

Limited evaluator accuracy with unclear error propagation: negative precision of 44.7% and sign disagreements concentrated on small-magnitude labels, with no answer on how label noise affects training gains and no shuffled/random-label control to bound the required label quality.

Questions for authors

All main results in Table 1 come from single runs, the matched-policy probes in Table 4 are described as "two actor scales, one independent run each", and §7 Limitations itself concedes that single-run training leaves uncertainty about seed variability. As a result, we cannot tell whether the 9B gain of +0.13pt on BFCL (Ours 19.13 vs PPO/CAPO 19.00) and the LiveCodeBench tie with untrained Base (41.22 vs 41.22) reflect small but genuine improvements or mere training noise. Please provide means and variances over at least 3 seeds per scale across the four evaluation sets (or equivalent multi-run statistics and significance tests) so we can see which differences are robust.

1 comment

Thank you for the careful reading, particularly the question about seed variability in the small benchmark differences.

Results across training seeds.

We completed the requested three-seed evaluation at both scales across all four benchmarks. The table reports Full’s mean ± sample standard deviation over seeds 42, 43, and 44, alongside the Table 1 PPO reference. Scores are percentages.

ScaleMethodSWE-ProTBBFCLLCB
4BPPO25.8925.845.2527.48
4BFull29.91 ± 0.4827.72 ± 0.656.00 ± 0.1334.35 ± 0.76
9BPPO31.3728.0919.0038.93
9BFull36.16 ± 0.4137.83 ± 0.6519.08 ± 0.6941.22 ± 0.76

On SWE-Pro and Terminal-Bench, all three Full seeds at both scales exceed the PPO reference. The mean SWE-Pro gains are 4.02 points at 4B and 4.79 points at 9B.

For the two cases you highlighted, 9B BFCL remains close to PPO, and the LCB mean matches Base at 41.22%, above PPO’s 38.93%. These benchmarks evaluate transfer from repository-repair training to tool use and standalone code generation. Efficiency also improves: paired BFCL evaluations show 10.93% fewer turns at 4B and 16.12% fewer at 9B.

How credit errors affect an update.

The seed experiments measure variability across complete training runs. To examine how credit errors enter learning, we separately tested the effect of reversing signs at different training-coefficient magnitudes. This complements the reported 83.43% weighted sign agreement on high-impact MC labels.

For each of four 64-trajectory batches, Full, Flip-small, and Flip-large independently take one update from the same 4B step-64 actor and optimizer state. Flip-small reverses the smallest 20% of nonzero coefficients; Flip-large reverses the largest 20%. Each intervention flips 1,820 turns in total. Keeping the model, batch, and coefficient magnitudes fixed makes the sign changes the intervention being tested.

We evaluate the updated actors in the environment using the correctness–efficiency utility and a shared Pre-derived length reference. Here ΔU=Uˉ−UˉPre\Delta U=\bar U-\bar U_{\mathrm{Pre}}.

ConfigurationMean utility Uˉ\bar UΔU\Delta U vs. Pre
Pre−0.6200660
Full−0.555921+0.064145
Flip-small−0.554276+0.065790
Flip-large−0.613487+0.006579

The mean improvement stays close to Full when small coefficients are flipped, but falls by 89.7% when large coefficients are flipped. Thus, an unweighted error rate treats together sign errors with markedly different effects on the update.

Shuffled-credit controls.

The shuffle controls test whether credit is assigned to the right positions. Section 6.2 includes whole-trajectory and within-half shuffling. In the separate one-update experiment in Table 4, Full achieves ΔU=+0.1035\Delta U=+0.1035, compared with +0.0166+0.0166 for Shuffled.

Table 4 uses a shared 4B actor and four independent update batches. We will clarify this in the caption. Together, the shuffle and Flip experiments examine where credit is placed and how strongly errors at those positions affect learning.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. Our additional experiments examine credit reliability, sensitivity to sign errors, and the robustness of policy improvements.

Effect of credit errors.

For each of four fixed batches of 64 trajectories, Full, Flip-small, and Flip-large independently take one update from the same 4B step-64 actor and optimizer state. We reverse the smallest or largest 20% of nonzero training coefficients: 1,820 turns per intervention, preserving coefficient magnitudes, token masks, and the loss denominator.

Environment utility is +1 for efficient successful completion, 0 for other successes, and −1 for checker failure, using a shared Pre-derived length reference.

ConfigurationMean utilityChange vs. Pre
Pre−0.6200660
Full−0.555921+0.064145
Flip-small−0.554276+0.065790
Flip-large−0.613487+0.006579

Flip-small retains a mean utility gain close to Full. Reversing the same number of large coefficients reduces the gain by 89.7%, showing that sign errors have different effects depending on their training weight.

Frozen-selector reliability as the actor evolves.

Following the remaining question about policy dependence, we evaluated the frozen selector at 4B seed42 updates 0, 64, and 128 on the same 32 tasks, with 96 sampled turns per checkpoint. MC targets were recomputed using each checkpoint’s actor for alternative actions and continuations. High-impact weighted sign agreement is 74.92%, 74.08%, and 70.45%; negative precision changes from 47.17% to 53.49%, and recall from 44.64% to 46.94%. This audit measures agreement with current-policy targets and identifies some late-stage drift.

Additional comparisons and robustness checks.

With three evaluator calls per trajectory and 128 actor updates for both configurations, Full achieves 30.41% on 4B SWE-Pro versus 26.30% for Initial-repeat3. The improvement therefore persists after matching the evaluator-call count.

We completed three-seed evaluations of Full at both scales across all four benchmarks. All three seeds at both scales exceed the PPO reference on SWE-Pro and Terminal-Bench. The additional outcome-only, critic-free GRPO baseline is evaluated at both scales. Full achieves higher success rates in seven of eight settings and uses 16.41–40.95% fewer turns across SWE-Pro, Terminal-Bench, and BFCL.

Independent MC continuation replication preserves 58/60 signs, including all 24 strong-credit positions. Re-scoring the same continuations with success-only utility increases the zero-credit share from 14.15% to 44.67%, showing how the efficiency term preserves action distinctions when eventual success is saturated.

Our individual responses provide the full protocols and results, evaluator-adaptation and prompt-disagreement analyses, and the breakdown of annotation cost and prompt-development/online-scoring budgets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.