acceptodds
Under review as a conference paper at ICLR 2027

Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards

Abstract

Reinforcement learning with verifiable rewards has become a common way to improve explicit reasoning in large language models, but final-answer correctness alone does not reveal whether the reasoning trace is faithful, reliable, or useful to the model that consumes it. This outcome-only signal can reinforce traces that are right for the wrong reasons, overstate reasoning gains by rewarding shortcuts, and propagate flawed intermediate states in multi-step systems. To this end, we propose TraceLift, a planner-executor training framework that treats reasoning as a consumable intermediate artifact. During planner training, the planner emits tagged reasoning. A frozen executor turns this reasoning into the final artifact for verifier feedback, while an executor-grounded reward shapes the intermediate trace. This reward multiplies a rubric-based Reasoning Reward Model (RM) score by measured uplift on the same frozen executor, crediting traces that are both high-quality and useful. To make reasoning quality directly learnable, we introduce TRACELIFT-GROUPS, a rubric-annotated reason-only dataset built from math and code seed problems. Each example is a same-problem group containing a high-quality reference trace and multiple plausible flawed traces with localized perturbations that reduce reasoning quality or solution support while preserving task relevance. Extensive experiments on code and math benchmarks show that this executor-grounded reasoning reward improves the two-stage planner-executor system over execution-only training, suggesting that reasoning supervision should evaluate not only whether a trace looks good, but also whether it helps the model that consumes it.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large language model, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hr1wOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TraceLift, a framework for training a reasoning planner to guide a frozen executor. The planner is trained with GRPO using execution correctness and a reasoning-quality score weighted by the executor’s improvement over a no-plan baseline. The authors construct TRACELIFT-GROUPS, a dataset of reference and perturbed reasoning traces, to train the reasoning reward model. Experiments on coding and mathematics report improvements over execution-only training across three model families.

Critical strengths
  1. The method measures whether a reasoning plan actually helps a fixed executor. This provides a concrete basis for rewarding useful plans beyond their apparent quality.

  2. The experiments cover coding and mathematics across three model families. Reward ablations, including a constant-RM control, provide evidence that the proposed components contribute to performance.

  3. Analyses of reasoning length and human preferences provide additional evidence about plan quality. The length results help address whether the gains simply come from producing longer traces.

Critical weaknesses
  1. The training-cost comparison is incomplete. TraceLift requires reward-model training and repeated executor sampling to estimate uplift, yet no comparison under matched total training compute is reported. It remains unclear whether the gains persist when execution-only training receives a comparable budget.

  2. The reward-component ablations focus on Qwen2.5-7B coding tasks. Similar ablations on mathematics and other model families would help establish whether each component consistently contributes to the reported improvements.

  3. The reward model is trained on synthetically perturbed reasoning traces. The main experiments do not directly establish how reliably it evaluates naturally occurring planner errors, which may differ from the errors introduced during dataset construction.

  4. The main results are averaged over three runs without reporting variability estimates. The large SVAMP improvement also warrants explanation: the gain over execution-only training is 21.00 points with LoRA, compared with 5.66 points under full-parameter training (Table 4). This indicates that the size of the benefit depends substantially on the adaptation setting.

Questions for authors
  1. Could you report the total training cost and compare TraceLift with execution-only training under a matched compute budget?

  2. Does the advantage over the constant-RM control also hold on mathematics and other model families?

  3. How accurately does the RM evaluate naturally occurring errors in planner-generated traces, beyond the synthetic perturbations used to construct its training data?

Reviewer_6A7HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TraceLift, a reinforcement learning framework that treats reasoning trajectories as intermediate artifacts in a two-stage planner–executor pipeline. Under this framework, a trainable planner generates an intermediate reasoning trace, while a frozen executor consumes this trace to generate the final code or math artifact.

To prevent the planner from exploiting outcome shortcuts (e.g., hallucinated or flawed reasoning that accidentally yields a correct answer), the policy is trained using Group Relative Policy Optimization (GRPO) with a composite reward: verifier feedback on the executor's output plus an executor-grounded reasoning score.

The latter is computed as the product of an offline rubric-based Reasoning Reward Model (RM) score and the empirical "uplift" measured on the frozen executor over a no-reasoning baseline. To train the offline RM, the authors construct TRACELIFT-GROUPS, containing 6,000 problem-reasoning groups derived from GSM8K and OpenCodeReasoning with rule-based/agentic localized perturbations. Experiments are conducted on code (HumanEval, MBPP, LiveCodeBench) and math (GSM8K, SVAMP, MATH500) benchmarks using 4B–8B models.

Critical strengths
  • The research question this paper is trying to attack, i.e., verifiable outcome-only rewards can reinforce unsound, superficial or misleading reasoning, is well motivated and central to post training of reasoning models.
  • The authors propose to combine an intrinsic rubric-based trajectory quality score with an empirical executor-utility uplift metric; this is conceptually sound to ensure the intermediate reasoning is both valid and actionable.
  • The authors construct TRACELIFT-GROUPS. This provides systematic coverage of distinct perturbation categories rather than relying on random, ad hoc corruptions.
Critical weaknesses

Problem framing and experiment setup

  • The paper has a mismatch between conceptual framing and evaluated tasks. The paper frames the method using agentic and planner-executor concepts, but the actual empirical evaluation is entirely on standard single-turn code and math benchmarks.
  • Following previous point, the paper lacks multi-step or agentic evaluations. In its current form, the planner in the paper simply outputs a CoT rationale and the executor generates the code/answer in one forward step; therefore it does not test the actual planner-executor architecture's central questions which are handling environment feedback, executing intermediate actions or long-horizon trajectory planning.
  • The benchmarks the paper opts to evaluate on are heavily saturated and too simple for the models adopted. New benchmarks or more challenging tasks need to be evaluated on.

Technical details

  • Table 3 and table 15 have inconsistency in K-sample rollout ablation. In Proposition 1 and Proposition 6, the authors prove that the empirical uplift estimator u^k\hat{u}_k is an unbiased estimator and its variances decreases monotonically as K increases, but empirical results show that moving from K=3 to K=5 leads to noticeable performance regression on all coding benchmarks. Also IMO setting K<10 is less pragmatic compared to how people use group-based algorithms in practice.
  • TRACELIFT-GROUPS uses 3,000 seed problems from GSM8K training set to train the Reason RM. The downstream evaluation is performed on the GSM8K test set and GSM-Hard. There is potential train-test contamination here because GSM benchmarks share the same question template.
  • Compute matched comparison. For each problem during GRPO training, TraceLift samples K executor rollouts with the candidate trace and K baseline executor rollouts without reasoning, in addition to the standard executor rollout for RexecR_{\textit{exec}} with K=3. In total it requres up to 2K+1 forward calls to the executor per candidate trajectory. While the Exec-only baseline seems to be trained with K=1 executor call.
  • Missing baselines. In its currently covered benchmarks, the authors should include PRM baselines or LLM-as-a-Judge methods for process reward, among many other potential baseline choices, for example Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation. shows you can try to distill LLM judge labels into lite scale RMs for process correctness.
Questions for authors
  • Additional baselines.
  • Compute matched experiments.
  • Larger group size.
  • Generalization to agentic tasks.
Reviewer_KXc8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a “Planner + Frozen Executor” framework to reward both final-answer correctness and intermediate reasoning quality. However, the added complexity appears insufficiently justified, particularly given that the evaluation focuses on standard single-turn benchmarks that small open-weight models with approximately 4B parameters can already largely solve in a single pass. Although the paper reports performance improvements, these results do not convincingly establish the need for the proposed architecture. Overall, I find the motivation underdeveloped, the technical contribution limited, and the soundness of the approach insufficiently substantiated.

Critical strengths

Finding a way to not only reward final outcome but also the reasoning process (CoT faithfulness) is an important problem to work on, and at least this paper tries to tackle that.

Critical weaknesses
  • Unjustified Planner–Executor split. For static, single-turn benchmarks such as GSM8K and HumanEval, a single model can generate reasoning and an answer in one pass. The need for a two-stage pipeline—which also constrains the Exec-only baseline—is insufficiently justified.

  • Unequal compute budgets. The paper lacks compute-matched baselines across the benchmark suite. TraceLift uses 2K+1=72K+1 = 7 executor rollouts per trace plus a 7B reward-model forward pass, versus one rollout for Exec-only. Training both for 600 steps does not control for this disparity.

  • Missing baselines and ablations. The evaluation lacks PRM baselines and sufficient ablations to justify the pipeline’s design choices and hyperparameters.

  • Unclear presentation. The paper is difficult to follow, starting with the abstract. More precise language and clearer explanations of the method and its motivation are needed.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.