acceptodds
Under review as a conference paper at ICLR 2027

CoVER: Reinforcement Learning with Code-Verified Process Rewards in Stateful Code Execution Environments

Abstract

Large language models increasingly rely on external code execution for mathematical and code-intensive reasoning. However, most RL post-training assigns credit only through final-answer correctness, while prior process-supervision methods are largely designed for static traces and often require costly step labels or learned evaluators. We introduce **CoVER** (**Co**de-**Ve**rified Process **R**ewards), a reinforcement learning framework that turns stateful code execution into online process supervision. CoVER trains LLMs through iterative execution and self-correction in a persistent sandbox, using lightweight process rewards derived from execution-grounded sandbox signals. To optimize mixed agent–environment trajectories, we propose PR-GRPO, a process-reward variant of GRPO that applies masked log-probabilities and masked KL regularization only to policy-generated tokens and assigns credit with depth-wise step-level advantages. We construct CoVERBench by standardizing public reasoning sources and solver-verified tasks into a unified benchmark spanning mathematical reasoning, logical deduction, and constraint satisfaction. On CoVERBench, CoVER improves Qwen3-8B to 93.7% held-out validation accuracy, versus 82.1% for outcome-only GRPO under a matched protocol, with non-overlapping bootstrap confidence intervals. CoVER also transfers to unseen math and code benchmarks.

Primary area:
reinforcement learning
Keywords:
Decision making, reinforcement learning, and control, Generalization and multi-task learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5gfyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CoVER, an RL framework that uses code-verified process rewards from stateful execution environments to improve LLM reasoning. It proposes PR-GRPO, an extension of GRPO that incorporates step-level credit assignment to optimize multi-turn tool interactions. Experiments show that CoVER improves Qwen3-8B’s accuracy on CoVERBench from 82.1% to 93.7% compared to outcome-only GRPO. The results also show improved generalization to unseen mathematical reasoning and code generation benchmarks.

Critical strengths
  1. The paper introduces CoVERBench, a comprehensive benchmark covering mathematical reasoning, logical deduction, and constraint satisfaction tasks. It provides a systematic setting for both training and evaluating multi-turn tool-integrated reasoning.

  2. CoVER shows strong generalization across multiple unseen mathematical reasoning and code generation benchmarks. The consistent improvements suggest that code-verified process rewards help develop transferable reasoning capabilities beyond the training distribution.

Critical weaknesses
  1. CoVER seems similar to THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning (ICLR 2026), which also combines execution-based step-level rewards, hierarchical RL, and error correction for tool-integrated mathematical reasoning. The authors should include THOR as a baseline in the experiments and also discuss the conceptual differences between the two methods to make CoVER’s contribution clearer.

  2. PR-GRPO groups trajectories based on their interaction depth, assuming that steps at the same depth are comparable. However, different trajectories may perform different reasoning actions at the same step, making depth-based grouping potentially misaligned with actual reasoning progress. This raises concerns about whether the advantages accurately reflect the contributions of individual actions.

  3. For each step, CoVER mainly measures format compliance, execution success, and claim grounding, which do not necessarily reflect the correctness of intermediate reasoning steps. Successfully executed code may still implement an incorrect mathematical procedure or make no meaningful progress toward solving the problem.

Questions for authors

See Weakness.

4 comments
AuthorsOpenReview ↗

Reponse to Reviewer

Thank you for highlighting THOR, depth-group comparability and process-reward scope. The revised PDF adds matched controls and audits, and clarifies these distinctions.

W1: THOR and matched controls

We add THOR to Table 11 and correct two contrasts: THOR uses episode-level GRPO, not learned-critic PPO, and already separates policy tokens from executor observations. Neither masking nor step-level execution feedback is specific to CoVER.

The distinction is signal source and cost. THOR resamples single-step candidates from failed-execution prefixes, using binary execution success and summed episode/step losses. CoVER scores every executed turn in the existing batch for execution, fabricated feedback/claim grounding, format and efficiency. We therefore charge THOR’s prefix rollouts to the generation budget.

SettingAcc.%Err.%Rec.%Gen. tokens
THOR port, prefix rollouts charged90.36.161.81.00×
THOR port, native budget91.25.464.31.47×
CoVER93.73.274.71.00×

Ports share CoVER’s warm-up checkpoint, RL prompts, persistent sandbox and verifier, retaining THOR’s episode filtering and positive-sample objective. Inference-time backtracking is unused under shared decoding.

These are single-seed point estimates; Table 13’s reduced-scale variability does not establish significance for these comparisons. Released THOR Qwen3-8B scores 85.9%, contextual only because its training corpora and tool stack differ.

W2: What depth grouping establishes

Equal depth does not imply equal state, reasoning stage or remaining horizon. Prop. 3.3 establishes centering, not unbiased/causal action credit; return-to-go includes later decisions. We replace the stronger horizon-comparability claim with stratification by interaction depth.

Audit: 300 turns, two annotators, four stages (setup/compute/verify/conclude), agreement 0.79. 41.6% of depth-matched pairs differ in stage. Within-group remaining-horizon SD is 1.12 at depth 1 and 0.94 at depth 3. Singletons comprise 3.9% of nonempty groups/0.9% of policy tokens and provide no relative signal.

Controls fix rewards, initialization, environment, κ=80% and budget. Pooled-turn/anchor-state rows reuse the GTPO/GiGPO runs in our other responses.

GroupingAcc.%Rec.%
Sequence-level89.258.3
Pooled-turn90.663.1
Anchor-state91.466.9
Active-depth93.774.7
Active-depth × action type93.975.2

Separating code/final-answer turns adds only 0.2 points, insufficient to establish improvement on single runs. Active-depth exceeds sequence-level credit by 4.5 points and anchor-state by 2.3, but statistical superiority remains unestablished. Only 18.7% of actions have same-anchor partners, limiting GiGPO’s intra-group term. These results support depth as useful, inexpensive stratification—not semantic equivalence.

W3: Process rewards and reasoning correctness

Matched probes receive format +0.1/execution +0.2, except fabricated feedback, which involves no execution:

Probe (turn)GroundingEfficiencyTotal
Correct/wrong-executable/irrelevant/claim-free; no contradicted claim (2)00+0.3
Wrong + contradicted numeric claim (2)−0.50−0.2
Repeated computation (4)0−0.2+0.1
Fabricated feedback (2)−0.50−0.4

Executable wrong/no-progress procedures can therefore receive the same reward as correct turns. 90/146 residual errors (61.6%) involve executable but incorrect reasoning, rising from 42.9% on easy problems to 64.6% on hard problems, versus 1.4% code-level failures (Tables 8–9).

Efficiency leaves padded executable turns +0.2/+0.1/0 at turns 3/4/5, negative from 6: discouragement, not prohibition. R_out=2.0 does not guarantee protection either: terminal contrast vanishes in all-incorrect groups and cancels in all-correct groups only when discounted terminal contributions coincide. Process differences can still yield order-one normalized advantages.

Across 2,000 rollouts per checkpoint, warm-up→convergence: irrelevant execution 2.6→1.3%, repetition 4.9→2.1%, claim-free turns 30.1→28.6%, fabricated feedback 0.4→0.05%; turns/solved problem 3.4→2.6 (M.3). The irrelevance detector requires completed trajectories and is audit-only. These observations counter increasing gaming, not its possibility.

The revised conclusion defines the rewards as proxies for execution reliability and claim grounding, not correct intermediate reasoning. Executable flawed strategies can pass; CoVER assumes safe, resettable environments with mechanically verifiable feedback.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer,

Thank you again for the care you have put into evaluating our work. Your comments have guided our revisions, and we have now provided detailed responses, additional experimental results, and an updated PDF with the changes highlighted in blue.

As we enter the second half of the rebuttal period, may we ask whether these updates have resolved your main concerns? We would value the opportunity to clarify anything that remains unclear before the discussion closes. If you are satisfied with the new evidence and revisions, we would sincerely appreciate your consideration of an upward revision to your score.

Thank you for continuing the discussion with us.

Thank you for the response. I still find CoVER’s methodological contribution to be limited compared to THOR. Also, my second concern was only partially resolved. The authors clarify the limits of depth grouping, but do not directly show how grouping different reasoning stages affects credit assignment. Separating code from final-answer turns does not address this issue, since code turns can serve different reasoning stages. I will keep my score.

AuthorsOpenReview ↗

Reply to Reviewer (Round 2)

Thank you for the follow-up. We respectfully believe that CoVER’s contribution should be assessed through its concrete methodological differences and controlled experimental evidence, rather than its shared use of execution feedback with THOR.

CoVER changes both how process supervision is collected and how it enters optimization. THOR branches additional single-step rollouts from failed-execution prefixes and combines step-level execution-success losses with episode-level optimization. CoVER obtains execution, grounding, format, and efficiency signals from every executed turn of the existing rollout batch, and uses depth-normalized returns without additional prefix sampling. This distinction has measurable practical consequences: under the matched protocol in Appendix L.2, CoVER achieves 93.7% accuracy versus THOR’s 90.3% at equal generation budget. Even with 1.47× generation tokens, the THOR port reaches 91.2%. These comparisons go beyond a conceptual distinction: they demonstrate a favorable observed performance–budget trade-off.

On depth grouping, we agree that code-versus-answer separation is not a test of reasoning-stage alignment; our justification rests primarily on the matched estimator comparisons. Holding process rewards, initialization, sandbox, retention, and budget fixed, active-depth normalization achieves 93.7% accuracy and 74.7% recovery, compared with 89.2%/58.3% for sequence-level credit and 90.6%/63.1% for pooled-turn normalization (Table 18). These are single-seed comparisons, but they directly isolate the estimator’s practical contribution from reward design and tool access. Semantic stage alignment and downstream learning effectiveness are distinct questions: the former requires a stage-conditioned credit analysis, while the latter is directly evaluated by these controls. Depth grouping is an inexpensive statistical stratification, not an assertion that all actions at the same depth are semantically identical.

Taken together, the paper contributes a distinct rollout-efficient supervision and optimization design, controlled evidence of its utility, and a multi-domain benchmark. The complete framework also preserves its advantage on the independent, selection-free test split: 92.9% versus 81.4% for matched outcome-only GRPO. We respectfully ask you to give these methodological distinctions and controlled results careful consideration when assessing the contribution, and to reconsider whether they warrant an upward revision of your score. We would welcome further discussion of any specific aspect of this evidence.

Reviewer_TGnaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoVER trains an LLM to reason through repeated interaction with a persistent Python sandbox. After supervised warm-up, it combines final-answer rewards with feedback on execution, formatting, claim grounding, and efficiency. PR-GRPO updates only model-generated tokens and compares future returns among trajectories at the same turn depth. On CoVERBench, Qwen3-8B reaches 93.7% accuracy versus 82.1% for matched outcome-only GRPO. Further experiments examine transfer, recovery, and inference cost.

Critical strengths
  1. The ablations show where the improvement comes from. Under a matched protocol, process rewards raise GRPO accuracy from 82.1% to 89.2%; depth-wise credit raises it further to 93.7%. Keeping all trajectories still reaches 93.1%. These results support gains from process supervision and credit assignment beyond simply providing sandbox access.

  2. The evaluation covers useful aspects of tool interaction. The paper measures execution errors, recovery, and end-to-end latency including sandbox overhead. It also reports reduced-scale training-seed results. The process rewards need neither a learned evaluator nor dense human step labels.

Critical weaknesses
  1. The 20% retention result needs explanation. Appendix F.1 keeps one of five trajectories. Algorithm 1 then applies Eq. 4, which gives zero advantage for a singleton and removes the task-reward gradient. Interpreting the reported 88.5% accuracy therefore requires the starting SFT accuracy and the actual normalization procedure.

  2. The turn-level baseline comparisons need clearer controls. Table 4 includes GiGPO and GTPO, but does not establish whether they share CoVER's initialization, process rewards, and training budget. Without these details, the gains cannot be attributed specifically to the credit-assignment rule rather than differences in supervision.

Questions for authors
  1. Is normalization performed before or after retention? How are groups with one trajectory handled? Please report the initial SFT accuracy for the 20% experiment and how often singleton groups occur with default retention.

  2. Do GiGPO and GTPO use the same SFT checkpoint, prompts, process rewards, sandbox, and rollout budget as CoVER? Please state any differences and, where possible, provide a comparison with these settings matched.

3 comments
AuthorsOpenReview ↗

Reponse to Reviewer

We thank the reviewer for recognizing the mechanism decomposition and interaction-level evaluation. We clarify the implementation and provide matched GiGPO/GTPO controls in the revised PDF.

W1 & Q1: Normalization, singletons and SFT

The submitted specification was inconsistent: Algorithm 1 and two §4 sentences place retention before normalization, while Eq. (5) omits retention. Under retained-set normalization, κG=1 gives μₜ=Rᵢ,ₜ and Âᵢ,ₜ=0; δ>0 cannot restore task gradients.

(i) Order. Code normalizes all active rollouts, Gₜ={i≤G:Tᵢ≥t}, before selecting retained updates. At κ=20%, logs show 97.3% nonzero optimized-token advantages, mean |Â|=0.78. We align Algorithm 1, both §4 sentences, Eq. (5)’s retained index set/normalizer, and Eq. (4)/Prop. 3.3’s index universe. Results remain unchanged.

(ii) Singletons. At Â=0, the clipped task term vanishes, leaving KL-only regularization. There is no fallback or separate terminal-reward gradient.

(iii) SFT baseline. Under Table 14’s identical development problems, sandbox, verifier, nine-turn horizon, 8,192-token cap and decoding, warm-up scores 71.3%, distinct from direct-answering Qwen3-8B (62.4%) and the sandbox-equipped backbone (66.8%). Thus κ=20% gains 17.2 points (71.3→88.5), and default retention gains 22.4 (71.3→93.7).

(iv) Default-retention frequency. Fig. 6 and Appendix K report:

Depth123456–9
Mean active group size5.004.934.213.082.141.62
Singleton groups (%)0.00.01.16.418.337.5
Policy tokens (%)41.229.616.87.43.11.9

Singletons comprise 3.9% of nonempty prompt–depth groups/0.9% of policy tokens. Zero-variance nonsingletons separately add 1.6%/1.1%.

The retention sweep—88.5/91.6/92.8/93.7/93.1% at κ=20/40/60/80/100%—is nonmonotone. Ranking by cumulative process reward changes selection bias and update volume without changing normalization groups; it is not a pure budget ablation. Keeping all rollouts reaches 93.1%, +11.0 points over matched outcome-only GRPO (I.1).

W2 & Q2: Matched GiGPO/GTPO

Table 4’s methods share CoVER’s Qwen3-8B SFT checkpoint, RL prompts, sandbox, verifier, rollout budget and decoding, but use their own reward/credit designs. Appendix L adds estimator-level controls with identical CoVER rewards:

ControlShared setting
Initialization/promptsSame SFT checkpoint; same CoVERBench IDs/order
RewardsCoVER R_proc, R_out and γ
Environment/maskPersistent sandbox, verifier, policy-token mask
Rollouts/retentionG=5, κ=80%, 9 turns, 8,192 tokens
BudgetSame generation-token budget/update count
Optimizer/KL/clippingSame shared settings
Selection/evaluationSame checkpoint rule, split and decoding
Estimator (identical rewards)Acc.%Err.%Rec.%
Sequence-level (Table 3)89.27.858.3
GTPO, pooled-turn90.66.363.1
GiGPO, episode + anchor-state91.45.566.9
PR-GRPO, active-depth93.73.274.7

PR-GRPO leads all three metrics in these single-run comparisons; statistical significance remains unestablished. Table 13’s reduced-scale seed variability does not establish significance for these full-scale comparisons.

Only 18.7% of actions have same-anchor-state partners, limiting activation of GiGPO’s intra-group term. The +2.3-point gap characterizes this stateful-code setting, not GiGPO’s performance in its intended embodied/web domains.

GTPO follows Ding et al., arXiv:2511.14846, §4.2/Eq. (7), pooling turns across trajectories rather than using the first preprint’s per-turn normalization. The estimator-only control excludes embedding-similarity shaping. With their own rewards, GTPO and GiGPO score 88.1% and 86.9%, respectively, on CoVERBench.

The revision adds these controls, clarifies Table 4 in §5.2, labels its rows + SFT + GiGPO/GTPO, and specifies the GTPO formulation.

AuthorsOpenReview ↗

Gentle Follow-up

Dear Reviewer,

With the rebuttal period now more than halfway through, we wanted to check whether you have had an opportunity to review our responses and updated manuscript. The revised PDF includes our additional experiments and clarifications, with changes marked in blue.

We would welcome your feedback on whether these additions resolve the issues you raised, and we remain available to address any outstanding questions. If the new results and revisions have sufficiently addressed your concerns, we kindly ask you to consider increasing your score. We sincerely appreciate your time and careful consideration.

The rebuttal and revisions satisfactorily resolve all my questions. I have raised my score.

Reviewer_G3qZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoVER, a reinforcement learning framework for training LLMs to solve reasoning tasks through multi-turn interaction with a persistent code-execution sandbox. CoVER combines supervised warm-up with online process rewards for format compliance, execution success, claim grounding, and interaction efficiency, alongside a terminal correctness reward. Its optimization algorithm, PR-GRPO, uses depth-wise group-normalized returns to assign turn-level credit and restricts policy optimization and KL regularization to policy-generated tokens. The authors also introduce CoVERBench, which combines mathematical reasoning, structured reasoning, and solver-verified logic tasks under a unified execution protocol. CoVER achieves 93.7% validation accuracy compared with 82.1% for a matched outcome-only GRPO baseline, and reports improvements in external benchmark performance, error recovery, and inference efficiency.

Critical strengths

*Relevant problem and practical supervision strategy. The paper addresses the limitations of terminal-only rewards in stateful tool-integrated reasoning. Deriving intermediate feedback directly from sandbox execution offers a practical alternative to human step annotations or learned process reward models.

*Well-controlled comparison against outcome-only GRPO. The main baseline shares the SFT checkpoint, sandbox, verifier, rollout budget, decoding configuration, and policy-token masking. This makes the substantial CoVERBench improvement more informative than a comparison confounded by unequal tool access or initialization.

*Broad ablation and robustness analysis. The paper examines reward components, step-level advantages, policy-token masking, trajectory retention, decoding variability, and reduced-scale training seeds. Retaining all rollouts still achieves 93.1%, suggesting that trajectory selection alone does not explain the improvement.

*Evaluation beyond aggregate accuracy. The analysis includes external math and code benchmarks, traceback-based recovery, residual error categories, token consumption, and end-to-end latency. These complementary measurements provide a broader picture of the framework’s practical behavior.

Critical weaknesses

The single-rollout retention experiment appears inconsistent with the stated advantage estimator. Algorithm 1 retains trajectories before computing depth-wise group statistics. With five sampled rollouts and 20% retention, only one trajectory remains per prompt, making its centered advantage zero at every active depth under the stated estimator. Nevertheless, Table 14 reports 88.5% accuracy. This discrepancy raises an important reproducibility concern about the actual normalization group, computation order, or an undocumented fallback. Singleton active groups may also arise at later depths under the default setting.

The claim-grounding reward is insufficiently specified and its semantic validity is not established. Appendix C.1 defines abstract claim-extraction and checking functions without sufficiently detailing their operational rules, coverage, or error rates. It is also unclear how execution-consistency checks are separated from task-specific correctness checks using verifier metadata. More generally, successful execution does not establish meaningful reasoning progress: irrelevant but executable code can receive positive rewards, and avoiding extractable claims may circumvent grounding checks. The existing reward–correctness correlation does not rule out these behaviors.

The algorithmic contribution is not sufficiently isolated from established multi-turn RL techniques. Policy-token masking is necessary for correctly handling mixed agent–environment trajectories, while the theoretical results primarily establish likelihood factorization and the zero-centering property of normalized returns. Although GiGPO and GTPO are evaluated on external benchmarks, the main CoVERBench experiments do not directly compare them under identical process rewards, sandbox access, and training budgets. Consequently, the incremental value of the proposed depth-wise estimator over closely related alternatives remains uncertain.

Evidence for generalization is weaker than the main in-domain result. External improvements over the strongest reported multi-turn baseline are relatively small, and their statistical reliability is not established. On AIME, the improvement over the original Qwen3-8B backbone amounts to four additional solved problems out of 120. Moreover, the CoVERBench validation set is used to select the retention ratio and report the headline result, rather than reserving an independent test set. These limitations warrant more cautious claims about robust generalization.

Questions for authors

Could you clarify whether advantages are computed before or after trajectory retention, and provide the corresponding implementation? When only one rollout is retained—or only one remains active at a later depth—how do you avoid zero advantages? Please reconcile this with the 88.5% result at 20% retention.

Could you provide the concrete claim-extraction and verification rules, including their coverage and false-positive/false-negative rates, and specify which checks access task-specific correctness information? How does the reward behave on irrelevant but executable code, repeated computations, and trajectories that avoid making mechanically extractable claims?

Could you compare PR-GRPO with GiGPO and GTPO on CoVERBench while holding process rewards, initialization, sandbox access, trajectory retention, and training budget fixed? Such a comparison would help establish whether the proposed depth-wise credit assignment contributes beyond the reward design and standard masking.

Could you provide paired, problem-level uncertainty estimates for the external benchmark improvements, include a matched outcome-only GRPO transfer baseline, and evaluate the final selected configuration on an independent CoVERBench test split not used for hyperparameter selection?

4 comments
AuthorsOpenReview ↗

Reponse to Reviewer

We thank the reviewer for a precise review. We then give the checker audit, the matched GiGPO/GTPO comparison, paired intervals and an independent test split. A revised PDF accompanies this reply.

W1 & Q1: Retention/singletons

The submitted order was inconsistent: Algorithm 1/two §4 sentences retain before normalization; Eq. (5) omits retention. Literally, κ=20% gives μₜ=Rᵢ,ₜ and Âᵢ,ₜ=0; δ cannot help.

Code normalizes all active samples, Gₜ={i≤G:Tᵢ≥t}, then retains updates. At κ=20%, 97.3% of optimized tokens have nonzero Â, mean |Â|=0.78. We correct Algorithm 1, §4, Eqs. (4)–(5) and Prop. 3.3; results unchanged. K documents the computation; code release remains pending.

Matched SFT scores 71.3%, versus the different direct/sandbox backbones’ 62.4/66.8%; κ=20/80% gains 17.2/22.4 points. The sweep, 88.5/91.6/92.8/93.7/93.1% at κ=20/40/60/80/100%, changes selection bias and update volume—not just budget.

At κ=80%, singletons comprise 3.9% of nonempty groups/0.9% of policy tokens; zero-variance nonsingletons add 1.6%/1.1% (Fig. 6; K). No fallback: Â=0 leaves KL-only, no task credit. Singleton frequency increases with depth; 76% of recoveries occur at depths 1–4. ≥5-turn episode recovery improves to 68.2% versus sequence-level 54.1%, but does not isolate late-depth recovery.

W2 & Q2: Checkers/hacking

Hᶠᵇ detects policy-emitted <execution_result>: origin, not correctness. Hᶜˡᵃⁱᵐ checks five claim classes, returns 1/0/⊥, penalizing only 0 (M.1). Numeric/symbolic and output-to-answer checks use observations/runtime state; assignments, constraints and test outcomes additionally use solver/harness metadata—task-aware supervision, although y* enters only R_out.

Audit: 600 stratified turns, two annotators, κ=0.83. Claims occur in 71.4% of turns; 82.6% of extracted claims are decidable. Conditional FPR/FNR: 1.8/12.9%, excluding extraction misses/abstentions.

Probes receive format +0.1/execution +0.2:

Probe (turn)GroundingEfficiencyTotal
Correct/wrong-executable/irrelevant/claim-free; no contradicted claim (2)00+0.3
Wrong + contradicted claim (2)−0.50−0.2
Repeated computation (4)0−0.2+0.1

Execution therefore does not certify progress. Efficiency leaves padding +0.2/+0.1/0 at turns 3/4/5, negative from 6: discouragement, not prohibition. R_out=2.0 does not guarantee protection: terminal contrast vanishes for all-incorrect groups and cancels for all-correct groups only with equal discounted contributions. Process differences can still yield order-one advantages.

Across 2,000 rollouts/checkpoint, warm-up→convergence: irrelevant execution 2.6→1.3%, repetition 4.9→2.1%, claim-free 30.1→28.6%, fabricated feedback 0.4→0.05%; turns/solved problem 3.4→2.6 (M.3). These audits counter increasing gaming, not its possibility. 90/146 residual errors involve executable but incorrect reasoning (Table 8): the rewards proxy execution reliability/grounding, not intermediate correctness.

W3 & Q3: Matched GiGPO/GTPO

Table 4 shares SFT initialization, prompts, sandbox, verifier, rollout budget and decoding, but uses each method’s rewards. Appendix L additionally fixes CoVER rewards, mask, G=5, κ=80%, horizon, token cap, budget and checkpoint rule:

EstimatorAcc.%Err.%Rec.%
Sequence-level89.27.858.3
GTPO, pooled-turn90.66.363.1
GiGPO, episode + anchor-state91.45.566.9
PR-GRPO, active-depth93.73.274.7

Single runs: Table 13’s reduced-scale SD does not establish significance here. Only 18.7% of actions have same-anchor partners, limiting GiGPO’s intra-group term and cross-domain interpretation.

GTPO uses Ding et al., arXiv:2511.14846, pooled-turn Eq. (7), excluding similarity shaping in this control. Own-reward GTPO/GiGPO: 88.1/86.9%. Masking is necessary bookkeeping; Prop. 3.3 proves centering, not causal credit.

W4 & Q4: Generalization/test

Matched outcome-only GRPO; identical decoding; 10,000 paired problem-bootstrap resamples (N):

Benchmark (n)GRPOCoVERΔ [95% CI]
AIME (120)48.350.8+2.5 [−1.7,+6.7]
GSM8K (1,319)91.293.9+2.7 [+1.6,+3.8]
HumanEval (164)89.090.9+1.9 [−1.2,+5.0]
MBPP (378)88.490.7+2.3 [+0.5,+4.1]
BigCodeBench-Hard (148)24.325.8+1.5 [−2.7,+5.7]

Only GSM8K/MBPP exclude zero against GRPO; reliability over GTPO/GiGPO remains unestablished. AIME adds four solves over the original backbone. We narrow transfer claims.

The original split is now development, since it selects retention/checkpoint. With configuration and selection protocol frozen, 1,000 additional problems from families outside the original partition, used for no tuning, yield 92.9% versus GRPO’s 81.4%. The 11.5-point margin persists independently; the 0.8-point development–test drop does not isolate selection optimism because the problems/families differ.

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer,

As we pass the midpoint of the rebuttal period, we would like to kindly follow up on our response. We have addressed your comments and uploaded a revised PDF, with changes highlighted in blue for ease of review.

We would greatly appreciate hearing whether our clarifications and additional experiments have addressed your concerns. If any questions remain, we would be happy to discuss them further. If you find that the revisions and new evidence satisfactorily resolve your main concerns, we would be grateful if you would consider raising your score.

Thank you for your time and thoughtful feedback.

Thanks for your reply. I think my question on W2 are still not be solved. As is shown in your table, the effect of rewards seems limited and I am worried that the reward cannot distiguish the reasoning progress and wrong execuction. I will keep my score.

AuthorsOpenReview ↗

Reply to Reviewer (Round 2)

Thank you for following up. We understand your remaining concern, and we agree with its central premise: successful execution does not establish meaningful reasoning progress, and our reward cannot reliably distinguish every incorrect-but-executable computation from a useful one. Table 22 deliberately exposes this limitation; we do not claim that the terminal reward or efficiency penalty eliminates it. Our revised claim is narrower: execution and grounding checks provide useful, inexpensive auxiliary supervision alongside outcome rewards, rather than a complete verifier of intermediate reasoning. We respectfully hope to distinguish this limitation in semantic coverage from the separate question of whether these signals meaningfully improve learning.

For that second question, we would particularly appreciate your consideration of the controlled ablations, rather than the reward–correctness correlation alone. With sequence-level credit retained, adding process rewards increases accuracy from 82.1% to 89.2% (Table 3), supporting their usefulness without relying on the proposed depth-wise estimator. Within the full framework, removing the execution reward reduces accuracy from 93.7% to 85.3%, while removing the combined fabricated-feedback/claim-grounding penalty reduces it to 88.4% (Figure 7b). The latter does not isolate claim checking from fabricated-feedback detection, but it does support the contribution of the combined component. Furthermore, the complete system reaches 92.9% versus 81.4% for matched outcome-only GRPO on the independent, selection-free test split (Appendix N). This is system-level evidence, not proof of semantic validity, but it shows that the benefits extend beyond making code executable or increasing the proxy reward.

We also take the circumvention risk seriously. In the audited rollouts, irrelevant execution decreases from 2.6% to 1.3%, repetition from 4.9% to 2.1%, and claim-free turns from 30.1% to 28.6% (Table 23). These observations do not rule out gaming or undetected reasoning errors; they provide limited empirical evidence against increasing exploitation of the identified loopholes. Our contribution is therefore not to solve intermediate semantic verification, but to demonstrate that partial, execution-grounded supervision can improve final task success despite its documented blind spots. Your feedback has helped us make that distinction explicit. We sincerely hope you will weigh these controlled experiments and the independent test result alongside the remaining limitation, and consider whether they warrant a more favorable assessment of the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.