- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the insufficient credit assignment between the planning and execution stages in large language model reasoning. It proposes a PlanGap-guided selective distillation strategy and a PC-GRPO optimization method to improve reasoning performance through plan reliability estimation and plan contrastive learning. The motivation is clear, the method design is reasonably well justified, and the experimental evaluation is relatively comprehensive. However, the independent contribution of the plan contrastive mechanism has not been consistently validated, and several theoretical assumptions, as well as their connections to empirical performance improvements, require more rigorous justification. Overall, this work demonstrates certain practical value, but the current evidence is insufficient to fully substantiate its central claims regarding methodological effectiveness and novelty.
Critical strengths
- It decomposes the reasoning process into high-level planning and concrete execution, providing a reasonable structural foundation for assigning separate supervision signals to different stages.
- Experiments on Qwen3-series models and Olmo-3-7B-Think demonstrate the applicability of the proposed method across different model scales and architectures.
Critical weaknesses
- The overall framework is largely a combination of existing techniques rather than a fundamentally new optimization paradigm. The paper combines the Plan-and-Action structure, teacher–student likelihood gap, hierarchical reward decomposition, and contrastive advantage modulation. However, each component is closely related to existing techniques. Overall, the proposed method appears to be more of a recombination of existing mechanisms rather than a fundamentally new paradigm for reasoning optimization.
- A poor plan may lead to a correct answer through self-correction during execution. However, the current mechanism may assign positive feedback to such a plan, which deviates from the goal of accurately assessing plan quality.
Questions for authors
- In Table 2, PlanGap-Q2 and Q3 perform noticeably worse than Raw-SFT, while Q4 shows a performance recovery. How do the authors explain this phenomenon?
- When a model starts with an incorrect plan but subsequently self-corrects during execution and arrives at the correct answer, could PC-GRPO mistakenly reinforce the original plan?
- According to Equations (2) and (3), when all executions under the same plan either succeed or fail, their execution residuals become zero, preventing the corresponding execution tokens from receiving effective learning signals. This situation may occur frequently, particularly for relatively easy or difficult problems. How do the authors address this issue? Could it prevent the model from further optimizing successful but inefficient reasoning trajectories?
2 comments
Thank you for your detailed responses and additional experiments. While these clarifications are helpful, I still have concerns regarding the novelty of the proposed framework. Therefore, I maintain my original rating (2).
We sincerely thank you for carefully reading our manuscript and for your thoughtful, constructive questions. We added a Student-NLL control and completed the Flat-versus-PC checkpoint sweep. These results help clarify the contribution and the meaning of feedback under self-correction and uniform outcomes.
W1: Contribution beyond combining components
The contribution is supervision allocation at an explicit plan interface. Offline, PlanGap selects where to add plan supervision while retaining the same questions and verified executions. Online, repeated executions expose between-plan value differences and within-plan residuals under a fixed verifier budget. PC-GRPO allocates these signals to plan and execution tokens, then applies bounded contrast to plan tokens using outcome-ranked siblings, without an external teacher or privileged solutions.
The new Student-NLL control tests the teacher-relative selection rule: PlanGap-Q1 reaches 64.0 AIME Avg. versus 56.3 (+7.7) on Qwen3-4B-Instruct-2507/Math10K (Avg@10, single checkpoints). Table 6(b) also shows compatibility with subsequent PC-GRPO: Stage2-Plan gains 2.78 points versus 1.67 for Stage2-Raw, widening their gap from 0.83 to 1.94. This initialization is distinct from PlanGap-selected sparse supervision.
Table 3 compares the full framework with matched standard GRPO. The completed Flat comparison covers all twenty checkpoints at steps 20–400 for 1.7B, 4B, and 8B, showing scale- and stage-dependent differences rather than uniform PC superiority. The credit controls clarify the tradeoff discussed below.
W2 / Q2: Can self-correction reinforce a bad plan?
Yes. PC-GRPO estimates conditional success under the current rollout policy, including its ability to recover, rather than intrinsic plan correctness or adherence. An incorrect plan can receive positive credit if recovered executions raise its estimated value above the group mean.
Repeated executions nevertheless allow separate feedback for a plan and a successful execution. With K=4, M=2 and outcome pairs (1,0), (1,1), (1,1), (0,0), the first plan has Q=0.5, below the group mean 0.625. PC can assign it negative relative plan credit while giving its successful execution a positive residual of 0.5.
Flat also rewards the original plan in a successful self-corrected trajectory. Averaged across repeated executions before clipping, with a common normalization denominator, Flat's plan advantages are proportional to Q−Qbar too. PC's distinction is explicit allocation and scaling of plan/execution credit. If both executions recover successfully, PC may still reinforce the original plan; repeated outcomes cannot identify whether success came from following or correcting it. We will clarify this conditional-utility interpretation.
Q1: Non-monotonic PlanGap quartiles
Additional PlanGap Q1/Q2/Q3/Q4 averages are 64.0/50.5/53.2/59.4. Every treatment retains the same 10,000 questions and verified executions; only the 2,500 plan-prefixed rows change. Thus, harder questions are not removed, but selected subsets may differ in content and difficulty. Q1 supports low-gap allocation in this setting; the cause of Q2/Q3 degradation remains unresolved. PlanGap is not a monotonic difficulty index.
Q3: Zero execution residuals
When two executions agree, their within-plan residuals are zero:
Plan credit can remain nonzero if Q differs from sibling plans' values. PC uses between-plan differences for plan selection and within-plan differences for execution feedback. When outcomes agree, the execution-residual term gives those execution tokens no direct policy-gradient signal. Flat retains cross-plan execution feedback if the overall prompt group contains different outcomes. This is a concrete cost of separating the signals.
For two independent executions, disagreement occurs with probability 2q(1−q), so residual feedback is sparse near both very low and very high success rates. Terminal correctness also rewards short and long successful trajectories equally. Our wall-clock comparisons measure training efficiency, not an objective directly optimizing each successful execution's length. We will distinguish these efficiency notions explicitly.