acceptodds
Under review as a conference paper at ICLR 2027

Learning Reachable Plans: PlanGap-Guided Distillation and Plan-Contrastive Policy Optimization for Reasoning

Abstract

Reinforcement learning with verifiable rewards improves long chain-of-thought (Long-CoT) reasoning in large language models (LLMs). However, outcome-supervised Group Relative Policy Optimization (GRPO) derives a single response-level advantage from each rollout's terminal reward and applies it across all tokens. To support fine-grinded advantages, we introduce a Plan-and-Action structure that places a concise high-level plan before detailed execution. We first validate this structure through a PlanGap-guided distillation, which extracts plans from verified teacher solutions and selects those with smaller teacher–student gaps in mean plan negative log-likelihood to prioritize strategies accessible to the student. We then propose Plan-Contrastive Group Relative Policy Optimization (PC-GRPO), which operates independently of the distillation stage and requires no SFT cold start. Under a fixed verifier budget, PC-GRPO samples multiple plans and multiple executions per plan, using their outcomes to estimate plan reliability and construct separate learning signals for plan selection and plan-conditional execution. It further contrasts plans with higher and lower estimated values using only plan tokens, providing plan-specific supervision without privileged solutions, an external teacher, or additional full-trajectory scoring for the contrastive signal. Experiments across diverse reasoning domains, model families, and scales show improved reasoning performance and more efficient plan supervision and policy optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t8tsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the insufficient credit assignment between the planning and execution stages in large language model reasoning. It proposes a PlanGap-guided selective distillation strategy and a PC-GRPO optimization method to improve reasoning performance through plan reliability estimation and plan contrastive learning. The motivation is clear, the method design is reasonably well justified, and the experimental evaluation is relatively comprehensive. However, the independent contribution of the plan contrastive mechanism has not been consistently validated, and several theoretical assumptions, as well as their connections to empirical performance improvements, require more rigorous justification. Overall, this work demonstrates certain practical value, but the current evidence is insufficient to fully substantiate its central claims regarding methodological effectiveness and novelty.

Critical strengths
  1. It decomposes the reasoning process into high-level planning and concrete execution, providing a reasonable structural foundation for assigning separate supervision signals to different stages.
  2. Experiments on Qwen3-series models and Olmo-3-7B-Think demonstrate the applicability of the proposed method across different model scales and architectures.
Critical weaknesses
  1. The overall framework is largely a combination of existing techniques rather than a fundamentally new optimization paradigm. The paper combines the Plan-and-Action structure, teacher–student likelihood gap, hierarchical reward decomposition, and contrastive advantage modulation. However, each component is closely related to existing techniques. Overall, the proposed method appears to be more of a recombination of existing mechanisms rather than a fundamentally new paradigm for reasoning optimization.
  2. A poor plan may lead to a correct answer through self-correction during execution. However, the current mechanism may assign positive feedback to such a plan, which deviates from the goal of accurately assessing plan quality.
Questions for authors
  1. In Table 2, PlanGap-Q2 and Q3 perform noticeably worse than Raw-SFT, while Q4 shows a performance recovery. How do the authors explain this phenomenon?
  2. When a model starts with an incorrect plan but subsequently self-corrects during execution and arrives at the correct answer, could PC-GRPO mistakenly reinforce the original plan?
  3. According to Equations (2) and (3), when all executions under the same plan either succeed or fail, their execution residuals become zero, preventing the corresponding execution tokens from receiving effective learning signals. This situation may occur frequently, particularly for relatively easy or difficult problems. How do the authors address this issue? Could it prevent the model from further optimizing successful but inefficient reasoning trajectories?
2 comments

We sincerely thank you for carefully reading our manuscript and for your thoughtful, constructive questions. We added a Student-NLL control and completed the Flat-versus-PC checkpoint sweep. These results help clarify the contribution and the meaning of feedback under self-correction and uniform outcomes.

W1: Contribution beyond combining components

The contribution is supervision allocation at an explicit plan interface. Offline, PlanGap selects where to add plan supervision while retaining the same questions and verified executions. Online, repeated executions expose between-plan value differences and within-plan residuals under a fixed verifier budget. PC-GRPO allocates these signals to plan and execution tokens, then applies bounded contrast to plan tokens using outcome-ranked siblings, without an external teacher or privileged solutions.

The new Student-NLL control tests the teacher-relative selection rule: PlanGap-Q1 reaches 64.0 AIME Avg. versus 56.3 (+7.7) on Qwen3-4B-Instruct-2507/Math10K (Avg@10, single checkpoints). Table 6(b) also shows compatibility with subsequent PC-GRPO: Stage2-Plan gains 2.78 points versus 1.67 for Stage2-Raw, widening their gap from 0.83 to 1.94. This initialization is distinct from PlanGap-selected sparse supervision.

Table 3 compares the full framework with matched standard GRPO. The completed Flat comparison covers all twenty checkpoints at steps 20–400 for 1.7B, 4B, and 8B, showing scale- and stage-dependent differences rather than uniform PC superiority. The credit controls clarify the tradeoff discussed below.

W2 / Q2: Can self-correction reinforce a bad plan?

Yes. PC-GRPO estimates conditional success under the current rollout policy, including its ability to recover, rather than intrinsic plan correctness or adherence. An incorrect plan can receive positive credit if recovered executions raise its estimated value above the group mean.

Repeated executions nevertheless allow separate feedback for a plan and a successful execution. With K=4, M=2 and outcome pairs (1,0), (1,1), (1,1), (0,0), the first plan has Q=0.5, below the group mean 0.625. PC can assign it negative relative plan credit while giving its successful execution a positive residual of 0.5.

Flat also rewards the original plan in a successful self-corrected trajectory. Averaged across repeated executions before clipping, with a common normalization denominator, Flat's plan advantages are proportional to Q−Qbar too. PC's distinction is explicit allocation and scaling of plan/execution credit. If both executions recover successfully, PC may still reinforce the original plan; repeated outcomes cannot identify whether success came from following or correcting it. We will clarify this conditional-utility interpretation.

Q1: Non-monotonic PlanGap quartiles

Additional PlanGap Q1/Q2/Q3/Q4 averages are 64.0/50.5/53.2/59.4. Every treatment retains the same 10,000 questions and verified executions; only the 2,500 plan-prefixed rows change. Thus, harder questions are not removed, but selected subsets may differ in content and difficulty. Q1 supports low-gap allocation in this setting; the cause of Q2/Q3 degradation remains unresolved. PlanGap is not a monotonic difficulty index.

Q3: Zero execution residuals

When two executions agree, their within-plan residuals are zero:

OutcomesPlan value QExecution residuals R−Q
(1,1)1(0,0)
(0,0)0(0,0)
(1,0)0.5(+0.5,−0.5)

Plan credit can remain nonzero if Q differs from sibling plans' values. PC uses between-plan differences for plan selection and within-plan differences for execution feedback. When outcomes agree, the execution-residual term gives those execution tokens no direct policy-gradient signal. Flat retains cross-plan execution feedback if the overall prompt group contains different outcomes. This is a concrete cost of separating the signals.

For two independent executions, disagreement occurs with probability 2q(1−q), so residual feedback is sparse near both very low and very high success rates. Terminal correctness also rewards short and long successful trajectories equally. Our wall-clock comparisons measure training efficiency, not an objective directly optimizing each successful execution's length. We will distinguish these efficiency notions explicitly.

Thank you for your detailed responses and additional experiments. While these clarifications are helpful, I still have concerns regarding the novelty of the proposed framework. Therefore, I maintain my original rating (2).

Reviewer_RtnbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies explicit planning in language model reasoning through two independently applicable methods. PlanGap-guided distillation selectively introduces plan supervision based on differences between teacher and student likelihoods. PC-GRPO uses repeated executions of the same plan to construct separate learning signals for planning and execution, and further refines the plan-level signal through token-level contrast. The paper evaluates these methods across multiple model sizes, model families, and reasoning benchmarks.

Critical strengths

1.The research problem is meaningful, and the overall motivation is intuitive. Distinguishing strategy selection from execution provides a useful perspective on how reasoning models learn. Whether a teacher’s strategy is appropriate for a student, and how success or failure should be attributed to planning versus execution, are both worthwhile questions. Explicit plans offer a natural interface for investigating these issues.

  1. The methods are clearly presented and carefully formalized. The paper provides explicit mathematical definitions of the plan representation, supervision selection, reward decomposition, and contrastive adjustment. These formulations make the source and intended role of each learning signal reasonably clear. The design also considers practical concerns such as training stability and computational cost.

  2. The empirical evaluation is broad and informative. The experiments cover different model sizes and families, cross-domain performance, component ablations, and training efficiency. The paper also reports mixed and negative results, which help readers assess the methods’ applicability and limitations.

Critical weaknesses

Beyond the questions listed below, my main question is: why should the SFT and RL approaches be presented together in a single paper? The paper currently reads more like two related contributions presented side by side than a unified investigation developing a central claim. I do not think the two methods must form an inseparable training pipeline. However, the paper would benefit from explaining what additional insight emerges from studying them together that would not emerge from studying either method independently.

Questions for authors

Questions on PlanGap-Guided Distillation

  1. What additional value does the teacher’s score provide beyond the student’s score alone? If the goal is to identify plans that a student can more readily learn, selecting plans directly by student likelihood seems like a natural baseline. How does PlanGap compare with this simpler approach, and what does the teacher–student difference capture that student likelihood does not?

  2. How should we interpret the non-monotonic results across PlanGap groups? In Table 2, the lowest-gap group, Q1, performs best, but Q2 and Q3 perform substantially worse—even worse than the highest-gap group, Q4. What explains this pattern? Could differences in problem difficulty or plan content across groups contribute to the results, rather than the gap primarily reflecting how learnable a plan is for the student?

  3. How well does the benefit of low-gap selection generalize across students and tasks? The direct evaluation of PlanGap-based selection is concentrated on Qwen3-4B and Math10K. Does low-gap selection consistently outperform random selection for students with different capabilities or on other tasks?

Questions on PC-GRPO

  1. How much additional benefit comes from the proposed credit assignment and contrastive adjustment? In Table 6a, full PC-GRPO matches the flat-credit baseline, while Table 5 shows that the contrastive component does not improve every model. How much of the overall gain can be attributed to the sampling structure and separate normalization of the plan and execution segments, and how much comes from the proposed credit assignment and plan contrast?

  2. Are two executions per plan sufficient to reliably estimate relative plan quality? Under the default configuration, a single accidental success or failure can change a plan’s ranking. This ranking affects both the plan-level advantage and the selection of positive and negative reference plans. Could the authors validate these estimates using more executions per plan and quantify how reliable the rankings obtained from only two executions are?

  3. Does the contrastive score capture useful strategic content or primarily similarity in wording? A token becoming more likely when conditioned on a higher-success reference plan may indicate useful strategic information, but it could also reflect shared wording or method names. Can the authors provide evidence that this score helps identify strategic content worth reinforcing, beyond surface-level similarity between plans?

1 comment

We sincerely thank you for carefully reading our manuscript and for your detailed, constructive questions. We added a Student-NLL control and completed the Flat-versus-PC checkpoint sweep. Together with the composition study, these results clarify the connection between offline supervision and online credit assignment.

W1: Why study SFT and RL together?

Both methods use an explicit plan interface: offline selection allocates plan supervision, while online optimization allocates feedback between plan selection and execution. Table 6(b) tests their compatibility:

InitializationBefore PC-GRPOAfterGain
Instruct-250754.8658.06+3.20
Stage2-Raw75.1476.81+1.67
Stage2-Plan75.9778.75+2.78

The Plan–Raw gap widens from 0.83 to 1.94 AIME-average points, supporting the continued usefulness of plan supervision during PC-GRPO. This study uses Stage2-Plan, not PlanGap-selected sparse supervision. The methods remain independently usable: PC-GRPO requires no SFT cold start, and PlanGap requires no RL.

Q1: What does the teacher score add?

Student-NLL ranks by student mean token NLL, E_S; PlanGap uses E_S − E_T, so the teacher score can change rankings even for similar student NLL. On Qwen3-4B-Instruct-2507/Math10K, the additional Avg@10 evaluation gives 64.0 AIME Avg. for PlanGap-Q1 versus 56.3 for Student-NLL-Q1 (+7.7). Both retain the same 10,000 examples and add plans to 2,500 rows. These single-checkpoint results support an incremental teacher-relative signal in this setting.

Q2: Why do Q2/Q3 underperform Q4?

Additional PlanGap Q1/Q2/Q3/Q4 averages are 64.0/50.5/53.2/59.4. All questions and verified executions remain in training; only plan-prefix placement changes. The selected subsets can still differ in content and difficulty. Q1 supports low-gap allocation, but the pattern is not monotonic, and the cause of the middle-quartile degradation remains unresolved.

Q3: Generalization

Direct PlanGap allocation evidence, including Random25 and Student-NLL controls, is limited to Qwen3-4B-Instruct-2507/Math10K. We will state this scope explicitly.

Q4: Beyond nested sampling and segment normalization

Flat retains the K-plan/M-execution structure and separate plan/execution loss aggregation, applying response-level outcome advantages to both spans. These structural factors are therefore controlled. Table 6(a) gives both Flat and PC-GRPO 70.28 AIME Avg.

We evaluated every saved checkpoint at steps 20, 40, ..., 400 with seed 42, temperature 1.0, and Avg@12. Equal-weight means over all twenty checkpoints are:

ModelFlatPC-GRPOPC / Flat / tied checkpoints
Qwen3-1.7B43.9943.427 / 12 / 1
Qwen3-4B70.2270.1011 / 8 / 1
Qwen3-8B72.3572.6511 / 8 / 1

At 1.7B, Flat leads later in training. At 8B, PC leads by 1.50 points over steps 20–100, while means over 320–400 are equal; 4B is close overall. These correlated checkpoints are descriptive trajectories, not independent training repetitions. Structured credit is not uniformly better than Flat.

Table 6(a)'s component controls show +1.39 points from adding contrast to hierarchical credit and +2.36 from preserving rather than shuffling token-weight placement. Flat versus PC changes credit and contrast together; the no-contrast control isolates contrast. These results separate full-framework gains over standard GRPO from component benefits. A possible tradeoff is that Flat retains execution updates when outcomes within a plan agree, whereas PC reserves execution credit for within-plan variation; the trajectories do not establish this as their cause.

Q5: Are two executions enough?

For M independent executions, the empirical success rate Q is unbiased with variance q(1−q)/M. At M=2, Q is 0, 0.5, or 1, so rankings are noisy. Without contrast, the fixed-budget K=2/M=4 and K=4/M=2 settings yield 45.97 and 46.53 AIME Avg. This supports the practical diversity/precision tradeoff, not ranking reliability. That would require comparing fixed plans' M=2 rankings against additional held-out executions. Bounded contrast does not remove sampling uncertainty.

Q6: Strategy versus surface wording

The contrast is an outcome-anchored token-likelihood signal on plan tokens. The +2.36-point placement control supports its utility, but does not separate strategic content from lexical overlap. We will describe it as execution-informed token supervision, not a validated semantic importance label.

Reviewer_VNRaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the credit assignment problem and capacity gap in training large language models (LLMs) for complex, multi-step mathematical and scientific reasoning. The authors propose an explicit "Plan-and-Action" structure that decouples high-level strategic reasoning (<plan>...</plan>) from low-level execution steps.

To effectively train models under this paradigm, the paper introduces two primary methods:

  1. PlanGap-Guided Selective Distillation (Plan-SFT): A data selection framework for Supervised Fine-Tuning . High-level plans extracted from verified teacher trajectories are prioritized based on "PlanGap"—the log-likelihood difference between teacher and student models. By distilling plans that fall into the student's "reachable" regime (low PlanGap), data efficiency is improved while preserving performance gains.

  2. Plan-Contrastive Group Relative Policy Optimization (PC-GRPO): An online reinforcement learning algorithm operating without SFT cold-start. Using a nested sampling scheme (KK plans ×\times MM executions per plan), PC-GRPO explicitly factorizes rewards into between-plan reliability (QkQ_k) and within-plan execution residuals (Dk,mD_{k,m}) . It applies fine-grained contrastive credit assignment specifically on plan tokens using bounded modulation, anchoring updates to terminal verifier feedback (R∈{0,1}R \in \{0, 1\}).

The authors evaluate their methods across Qwen3 (1.7B to 14B) and Olmo-3-7B-Think models across benchmarks such as AIME24/25, AMC23, GPQA-Diamond, demonstrating improvements in accuracy and wall-clock training efficiency compared to standard GRPO and related baselines.

Critical strengths
  1. Clear Structural Decoupling for Credit Assignment. Decoupling reasoning into high-level plan tokens (Tc\mathcal{T}_c) and execution tokens (Ty\mathcal{T}_y) provides a principled solution to the credit assignment dilemma in CoT reasoning, preventing local execution errors from improperly penalizing good high-level strategies .

  2. Principled Data Selection via PlanGap. Using PlanGap to filter teacher trajectories addresses the "capacity gap" in knowledge distillation. Prioritizing plans within the student's "zone of proximal development" enables the model to retain 86.6% of full-distillation performance gains using only 25% of plan-augmented data (PlanGap-Q1).

Critical weaknesses
  1. Inconsistent Cross-Domain Performance (Code Regressions). While mathematical and scientific reasoning benchmarks see steady gains, cross-domain transfer to coding exhibits noticeable regressions compared to standard Raw-SFT baselines. Specifically, in Appendix Table 11, performance on CodeForces drops from 25.09% (Raw-SFT) to 21.78% (Plan-SFT). The forced explicit plan structure may not naturally align with code generation paradigms.

  2. High Run-to-Run Variance and Sensitivity. Multi-seed evaluation in Appendix Table 9 reveals notable variance across independent runs, recording a standard deviation of 4.32 on AIME25 for PC-GRPO. This suggests that the plan contrastive objective and advantage modulation are sensitive to random seeds and sampling dynamics.

  3. Scale Instability of Plan-Token Contrastive Rewards. Ablation studies in Table 5 demonstrate that the addition of plan-token contrastive adjustment does not yield monotonic gains across all model scales. In certain configurations, enabling contrastive modulation leads to slight performance drops on specific benchmarks, raising questions about hyperparameter robustness.

Questions for authors
  1. Mitigating Coding Domain Degradation. What causes the performance drop on CodeForces (25.09% →\to 21.78%, Appendix Table 11) when applying Plan-SFT? Does the explicit plan structure induce over-abstraction in tasks where implementation details heavily influence the high-level architecture?

  2. NLL vs. Strategy Complexity. Since PlanGap relies on student NLL on teacher plans, how do you prevent the algorithm from systematically favoring overly simplistic or trivial plans over harder but necessary strategic steps?

1 comment

We sincerely thank you for carefully reading our manuscript and for your thoughtful, constructive questions. We added a Student-NLL control, completed the Flat-versus-PC checkpoint sweep, and clarify below the evidence for plan supervision, its domain dependence, and variability.

W1 / Q1: CodeForces regression and domain fit

Table 11 shows a CodeForces decrease from 25.09 (Raw-SFT) to 21.78 (Plan-SFT). Our explicit plan style originated in mathematical reasoning. The shared compression prompt (Appendix E.5) reduces traces to two or three strategic steps, emphasizing variables, equations, and transformations while omitting detailed calculations.

SFT trajectories and output requirements differ across domains: mathematics often ends with a short \boxed{} answer, whereas coding requires an executable program. Reusing this compression style without adapting it to coding may underrepresent data structures, boundary cases, or input/output handling. This is a plausible contributor to the regression, not an isolated cause established by a prompt ablation.

In the domain-matched OpenThoughts3 experiment, Plan-SFT improves LiveCodeBench from 44.85 to 58.32: medium problems rise from 34.63 to 60.60 and hard problems from 13.14 to 29.67, while easy problems decline from 77.84 to 75.09. Thus, the current construction can benefit coding, but transfer is uneven. We will clarify the plan style's mathematical origin, training sources, and compression settings.

W2: AIME25 variability

Table 9's three PC-GRPO runs score 34.17, 42.78, and 37.78, giving 38.24 ± 4.32 versus GRPO's 37.13 ± 0.58. The higher mean is directional evidence; this spread does not support a robustness claim from three runs. We will retain individual results and distinguish training seeds from checkpoints and decoding repetitions. Repeated evaluation of one checkpoint is not an independent training run.

W3: Contrastive modulation across scales

At matched K=4, M=2, contrast changes AIME Avg. by +1.11, −0.14, +0.97, +0.97, and −0.14 for Qwen3-1.7B, 4B, 8B, 14B, and Olmo-3-7B-Think. It helps three of five models and is configuration-dependent. On Qwen3-4B, contrast scales 0.25/0.50/0.75 yield 72.09/72.78/72.22. Equation (6) bounds the adjustment and preserves the verifier-derived plan advantage's sign.

The full method improves over matched standard GRPO (Table 3). Table 5 isolates contrast; Flat versus PC changes the credit rule and contrast together. The completed sweep of twenty checkpoints (steps 20–400) at three sizes does not show uniform superiority over Flat. We will keep these claims separate.

Q2: Does PlanGap favor trivial plans?

Student-NLL ranks by student mean token NLL, E_S; PlanGap ranks by E_S − E_T. The teacher reference changes the ranking from absolute student likelihood to a proxy for relative reachability. Mean-token scoring avoids a direct total-length penalty. Neither score directly measures strategic complexity or correctness. Plans come from verified executions, which do not independently certify every compressed plan.

In the new control, all treatments retain the same 10,000 examples; only the 2,500 plan-prefixed rows change. Quartiles ascend by each score. Additional Avg@10 results are:

QuartileStudent-NLL: AIME24 / AIME25 / Avg.PlanGap: AIME24 / AIME25 / Avg.
Q160.3 / 52.3 / 56.366.0 / 62.0 / 64.0
Q259.7 / 56.3 / 58.051.0 / 50.0 / 50.5
Q348.7 / 46.7 / 47.754.7 / 51.7 / 53.2
Q455.0 / 50.7 / 52.961.7 / 57.0 / 59.4

PlanGap-Q1 exceeds Student-NLL-Q1 by 5.7/9.7/7.7 points on AIME24/AIME25/Avg. These single-checkpoint results support the teacher-relative selection signal for Qwen3-4B-Instruct-2507 on Math10K, without ruling out content or complexity biases. Q4 outperforming Q2/Q3 also shows that PlanGap is not a monotonic difficulty index.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.