acceptodds
Under review as a conference paper at ICLR 2027

Beyond Plan Selection: Attributing and Improving Execution in RLVR

Abstract

Does reinforcement learning with verifiable rewards (RLVR) improve reasoning by selecting better plans or by executing them more reliably? We study this question through a matched plan-conditioned interface that allows Base and RLVR-trained selectors and executors to be interchanged while keeping the execution context fixed for each task–plan pair. Using a 1.5B model in two controlled two-plan environments, we separate realized performance contributions from available routing opportunity and the effects of targeted training. In arithmetic, the prespecified component-replacement analysis assigns 2.27 percentage points more improvement to execution than selection (95% CI: 1.14–3.41), although a common-repeat sensitivity analysis does not establish this ordering. Motivated by this diagnosis, we evaluate Execution-Focused RLVR (EF-RLVR), which removes the direct selector-policy loss while retaining shared parameters and KL regularization.EF-RLVR improves accuracy over Standard Joint RLVR by 1.28 points across eight fresh paired training seeds (95% CI: 0.77–1.79). A factorial follow-up and a supplementary direct comparison support a larger benefit from selector-term removal than executor up-weighting. In Search, input-dependent routing remains beneficial after training under both structured and final-answer scoring, but neither the original selector objective nor a uniform-action-weighting variant learns the conditional routing rule. These results show how component attribution can guide a useful training intervention, while demonstrating that available selection opportunity need not translate into learned routing.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with Verifiable Rewards (RLVR), Large Language Model Reasoning, Plan Selection, Plan-Conditioned Execution, Intervention-Based Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dAVWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether RLVR improves reasoning primarily through better plan selection or more reliable execution. The authors propose a matched plan-conditioned interface to separate these contributions through component replacement. Experiments on arithmetic and search tasks suggest that execution contributes more to RLVR gains in the arithmetic setting. Based on this finding, the authors introduce EF-RLVR, which improves accuracy over standard joint RLVR by 1.28 percentage points across eight paired training seeds.

Critical strengths

1.The matched interface and component-replacement analysis enable controlled comparisons, supported by confidence intervals and independent training-seed confirmation.

2.Separating plan selection from execution provides a useful framework for understanding where RLVR gains originate.

Critical weaknesses

1.All experiments use a single 1.5B model and two controlled two-plan environments. It remains unclear whether the findings extend to larger models, more diverse reasoning tasks, or realistic multi-step agent settings.

2.Although the selection- and execution-directed procedures match rollout counts and optimizer updates, they differ substantially in gradient-bearing token counts and optimization geometry. Therefore, the observed advantage does not necessarily establish that execution-focused optimization is intrinsically superior.

3.In the search setting, neither the original selector objective nor uniform action weighting learns the conditional routing rule. The experiments demonstrate the limitation of the tested objectives but do not identify an effective solution.

Questions for authors

1.The conclusion that execution contributes more than plan selection appears sensitive to the repetition protocol. Under the common-repeat setting, the confidence interval includes zero. How should we interpret the claim that execution is the dominant contributor to RLVR improvements?

2.In the search setting, the selector fails to learn conditional routing despite the demonstrated benefits of routing. What do the authors believe is the main cause of this failure?

3.The selection-directed and execution-directed approaches differ substantially in the number of gradient-bearing tokens. How can we distinguish the effect of optimizing execution from the effect of having a much larger effective optimization budget?

Reviewer_uqS3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether RLVR improves plan selection or plan execution through a matched interface that permits controlled replacement of Base and trained selectors and executors. Its contributions include component attribution, execution-focused RLVR that removes the direct selector-policy loss, and experiments demonstrating that available routing opportunities can remain unexploited after training. LLM baselines are included, Base, Standard Joint RLVR, and selector/executor-directed variants, but all experiments use DeepSeek-R1-Distill-Qwen-1.5B on two controlled environments.

Critical strengths
  1. The canonical execution interface enables interpretable component comparisons by preventing selector-generated rationale from changing the executor’s input.

  2. EF-RLVR improves acc by 1.28% points over Standard Joint RLVR across eight fresh paired seeds, with all eight differences positive and a 95% CI of [0.77, 1.79].

  3. The factorial ablation usefully distinguishes selector-loss removal from executor up-weighting, supporting the former as the main contributor within the tested configuration.

Critical weaknesses
  1. One small model, two synthetic environments, and two predefined plans provide limited evidence that the findings extend to natural reasoning, where plan construction and execution interact.

  2. The execution-over-selection contrast becomes statistically inconclusive under common-repeat evaluation (1.61 pp; 95% CI [−0.32, 3.56]), while the interaction estimate substantially attenuates; attribution uncertainty also remains conditional on the evaluated checkpoints.

  3. Although the factorial study supports selector-loss removal, comparisons against the specified joint objective do not establish whether the benefit survives stronger optimization tuning or explain whether gradient conflict, noise, or regularization causes it.

  4. Search selectors remain strongly biased toward label A, and without a successful learned-routing control, their failure offers limited insight into whether the obstacle is representation, initialization, or the training objective.

Questions for authors
  1. Do the attribution pattern and EF-RLVR advantage persist with another model or a less constrained reasoning task, especially where selection has substantial measurable headroom?

  2. Can you repeat the component-swap analysis across independent training seeds with equally many evaluation repetitions per cell, reporting both the attribution contrast and interaction?

  3. Does EF-RLVR retain its advantage after tuning selector learning rates, loss weights, and KL regularization, and can gradient-alignment diagnostics explain why selector updates hurt?

  4. Can supervised routing or counterbalanced-label training provide a successful routing control, helping identify why Original-S and Uniform-S remain near Always-A?

Reviewer_3hDzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper attributes RLVR gains to plan selection and execution through matched selector–executor swaps. It proposes EF-RLVR, which removes the direct selector-policy loss and improves Arithmetic accuracy by 1.28 pp over Standard Joint RLVR across eight fresh paired runs.

Critical strengths
  1. Motivation is meaningful.

Distinguishing the contributions of plan selection and execution is an interesting question.

  1. Empirical validation with multiple seeds is well done.

The authors conduct independent confirmation experiments with eight seeds.

Critical weaknesses
  1. The scope of the Arithmetic diagnosis is too limited.

My biggest concern is that the primary diagnosis is limited to restricted tasks and a model. All 1,024 Arithmetic evaluation instances use the same expression, ((X+Y)Z−K)((X+Y)Z-K), with only the integer inputs changing. Therefore, even with 1,024 evaluation cases, the roles of selection and execution were not verified across different problem structures or various planning spaces. Consequently, it remains unclear whether the observed pattern reflects a general property of RLVR or is specific to this task. To substantiate the core diagnosis and the usefulness of EF-RLVR, the analysis should be repeated across different expression structures and plan sets.

  1. Raises questions about generalizability to other models and broader RLVR behavior.

While the eight-seed confirmation supports reproducibility within the tested configuration, it does not address generalization across models, scales or problem structures. Given the paper's broader claims about RLVR, validation on additional model families and more diverse reasoning tasks would be more informative than extensive replication within a single DeepSeek-R1-Distill-Qwen-1.5B model and a fixed arithmetic expression.

  1. The optimization mechanism of EF-RLVR remains unresolved.

As the authors acknowledged in Section 5, gradient conflict, noise, and other shared-parameter optimization effects remain unresolved. It is also unclear whether the advantage persists against joint training with better tuned selector-loss weights. Combined with the narrow scope of the diagnosis, these limitations make it difficult to determine when and why execution-focused training should be preferred.

  1. Some implementation details are missing.

Appendix B provides detailed training settings for the formal Arithmetic S/E study, but it is unclear which numerical settings carry over to the separate Standard–EF and Search experiments.

Questions for authors
  1. Do the execution-dominant attribution results and EF-RLVR improvements generalize to different problem structures and more diverse plan sets?

  2. All experiments use DeepSeek-R1-Distill-Qwen-1.5B. Could the authors validate their findings on other model families and scales, such as Gemma-3-4B or Gemma-3-12B? At least, could they repeat the four-cell attribution analysis on an additional model, using diverse arithmetic expressions and plan structures rather than the same single expression?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.