- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces CoVER, an RL framework that uses code-verified process rewards from stateful execution environments to improve LLM reasoning. It proposes PR-GRPO, an extension of GRPO that incorporates step-level credit assignment to optimize multi-turn tool interactions. Experiments show that CoVER improves Qwen3-8B’s accuracy on CoVERBench from 82.1% to 93.7% compared to outcome-only GRPO. The results also show improved generalization to unseen mathematical reasoning and code generation benchmarks.
Critical strengths
-
The paper introduces CoVERBench, a comprehensive benchmark covering mathematical reasoning, logical deduction, and constraint satisfaction tasks. It provides a systematic setting for both training and evaluating multi-turn tool-integrated reasoning.
-
CoVER shows strong generalization across multiple unseen mathematical reasoning and code generation benchmarks. The consistent improvements suggest that code-verified process rewards help develop transferable reasoning capabilities beyond the training distribution.
Critical weaknesses
-
CoVER seems similar to THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning (ICLR 2026), which also combines execution-based step-level rewards, hierarchical RL, and error correction for tool-integrated mathematical reasoning. The authors should include THOR as a baseline in the experiments and also discuss the conceptual differences between the two methods to make CoVER’s contribution clearer.
-
PR-GRPO groups trajectories based on their interaction depth, assuming that steps at the same depth are comparable. However, different trajectories may perform different reasoning actions at the same step, making depth-based grouping potentially misaligned with actual reasoning progress. This raises concerns about whether the advantages accurately reflect the contributions of individual actions.
-
For each step, CoVER mainly measures format compliance, execution success, and claim grounding, which do not necessarily reflect the correctness of intermediate reasoning steps. Successfully executed code may still implement an incorrect mathematical procedure or make no meaningful progress toward solving the problem.
Questions for authors
See Weakness.
4 comments
Kind Reminder
Dear Reviewer,
Thank you again for the care you have put into evaluating our work. Your comments have guided our revisions, and we have now provided detailed responses, additional experimental results, and an updated PDF with the changes highlighted in blue.
As we enter the second half of the rebuttal period, may we ask whether these updates have resolved your main concerns? We would value the opportunity to clarify anything that remains unclear before the discussion closes. If you are satisfied with the new evidence and revisions, we would sincerely appreciate your consideration of an upward revision to your score.
Thank you for continuing the discussion with us.
Thank you for the response. I still find CoVER’s methodological contribution to be limited compared to THOR. Also, my second concern was only partially resolved. The authors clarify the limits of depth grouping, but do not directly show how grouping different reasoning stages affects credit assignment. Separating code from final-answer turns does not address this issue, since code turns can serve different reasoning stages. I will keep my score.
Reply to Reviewer (Round 2)
Thank you for the follow-up. We respectfully believe that CoVER’s contribution should be assessed through its concrete methodological differences and controlled experimental evidence, rather than its shared use of execution feedback with THOR.
CoVER changes both how process supervision is collected and how it enters optimization. THOR branches additional single-step rollouts from failed-execution prefixes and combines step-level execution-success losses with episode-level optimization. CoVER obtains execution, grounding, format, and efficiency signals from every executed turn of the existing rollout batch, and uses depth-normalized returns without additional prefix sampling. This distinction has measurable practical consequences: under the matched protocol in Appendix L.2, CoVER achieves 93.7% accuracy versus THOR’s 90.3% at equal generation budget. Even with 1.47× generation tokens, the THOR port reaches 91.2%. These comparisons go beyond a conceptual distinction: they demonstrate a favorable observed performance–budget trade-off.
On depth grouping, we agree that code-versus-answer separation is not a test of reasoning-stage alignment; our justification rests primarily on the matched estimator comparisons. Holding process rewards, initialization, sandbox, retention, and budget fixed, active-depth normalization achieves 93.7% accuracy and 74.7% recovery, compared with 89.2%/58.3% for sequence-level credit and 90.6%/63.1% for pooled-turn normalization (Table 18). These are single-seed comparisons, but they directly isolate the estimator’s practical contribution from reward design and tool access. Semantic stage alignment and downstream learning effectiveness are distinct questions: the former requires a stage-conditioned credit analysis, while the latter is directly evaluated by these controls. Depth grouping is an inexpensive statistical stratification, not an assertion that all actions at the same depth are semantically identical.
Taken together, the paper contributes a distinct rollout-efficient supervision and optimization design, controlled evidence of its utility, and a multi-domain benchmark. The complete framework also preserves its advantage on the independent, selection-free test split: 92.9% versus 81.4% for matched outcome-only GRPO. We respectfully ask you to give these methodological distinctions and controlled results careful consideration when assessing the contribution, and to reconsider whether they warrant an upward revision of your score. We would welcome further discussion of any specific aspect of this evidence.
Reponse to Reviewer
Thank you for highlighting THOR, depth-group comparability and process-reward scope. The revised PDF adds matched controls and audits, and clarifies these distinctions.
We add THOR to Table 11 and correct two contrasts: THOR uses episode-level GRPO, not learned-critic PPO, and already separates policy tokens from executor observations. Neither masking nor step-level execution feedback is specific to CoVER.
The distinction is signal source and cost. THOR resamples single-step candidates from failed-execution prefixes, using binary execution success and summed episode/step losses. CoVER scores every executed turn in the existing batch for execution, fabricated feedback/claim grounding, format and efficiency. We therefore charge THOR’s prefix rollouts to the generation budget.
Ports share CoVER’s warm-up checkpoint, RL prompts, persistent sandbox and verifier, retaining THOR’s episode filtering and positive-sample objective. Inference-time backtracking is unused under shared decoding.
These are single-seed point estimates; Table 13’s reduced-scale variability does not establish significance for these comparisons. Released THOR Qwen3-8B scores 85.9%, contextual only because its training corpora and tool stack differ.
Equal depth does not imply equal state, reasoning stage or remaining horizon. Prop. 3.3 establishes centering, not unbiased/causal action credit; return-to-go includes later decisions. We replace the stronger horizon-comparability claim with stratification by interaction depth.
Audit: 300 turns, two annotators, four stages (setup/compute/verify/conclude), agreement 0.79. 41.6% of depth-matched pairs differ in stage. Within-group remaining-horizon SD is 1.12 at depth 1 and 0.94 at depth 3. Singletons comprise 3.9% of nonempty groups/0.9% of policy tokens and provide no relative signal.
Controls fix rewards, initialization, environment, κ=80% and budget. Pooled-turn/anchor-state rows reuse the GTPO/GiGPO runs in our other responses.
Separating code/final-answer turns adds only 0.2 points, insufficient to establish improvement on single runs. Active-depth exceeds sequence-level credit by 4.5 points and anchor-state by 2.3, but statistical superiority remains unestablished. Only 18.7% of actions have same-anchor partners, limiting GiGPO’s intra-group term. These results support depth as useful, inexpensive stratification—not semantic equivalence.
Matched probes receive format +0.1/execution +0.2, except fabricated feedback, which involves no execution:
Executable wrong/no-progress procedures can therefore receive the same reward as correct turns. 90/146 residual errors (61.6%) involve executable but incorrect reasoning, rising from 42.9% on easy problems to 64.6% on hard problems, versus 1.4% code-level failures (Tables 8–9).
Efficiency leaves padded executable turns +0.2/+0.1/0 at turns 3/4/5, negative from 6: discouragement, not prohibition. R_out=2.0 does not guarantee protection either: terminal contrast vanishes in all-incorrect groups and cancels in all-correct groups only when discounted terminal contributions coincide. Process differences can still yield order-one normalized advantages.
Across 2,000 rollouts per checkpoint, warm-up→convergence: irrelevant execution 2.6→1.3%, repetition 4.9→2.1%, claim-free turns 30.1→28.6%, fabricated feedback 0.4→0.05%; turns/solved problem 3.4→2.6 (M.3). The irrelevance detector requires completed trajectories and is audit-only. These observations counter increasing gaming, not its possibility.
The revised conclusion defines the rewards as proxies for execution reliability and claim grounding, not correct intermediate reasoning. Executable flawed strategies can pass; CoVER assumes safe, resettable environments with mechanically verifiable feedback.