acceptodds
Under review as a conference paper at ICLR 2027

SocraticPO: Policy Optimization via Interactive Guidance

Abstract

Outcome-based reinforcement learning (RL) improves large language model reasoning by reinforcing correct responses, but leaves failed attempts without a path to correction. Inspired by Socratic teaching, in which a human teacher uses questions and hints to prompt learners to examine and repair their own mistakes, we introduce SocraticPO (Socratic Policy Optimization) to model failed reasoning as a teacher–student correction process. After an independent attempt fails, the teacher provides guidance conditioned on that attempt, and the student generates its own revision, shifting learning from outcome selection to guided correction. Rewarding assisted and independent solutions equally can encourage reliance on the teacher. SocraticPO therefore uses assistance-aware, batch-adaptive reward decay to favor solutions requiring less guidance while preserving non-negative normalized advantages for successful corrections. Across four SciKnowEval scientific reasoning domains and GSM8K, SocraticPO outperforms conventional RL and self-distillation baselines in unassisted accuracy under a controllable additional inference budget. Further analyses show that a teacher's answer accuracy does not necessarily predict its guidance quality and that the gains do not require a consistent reduction in teacher–student KL divergence, highlighting the advantage of text-level guidance over answer or distribution transfer.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_Y4aXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to provide text-level guidance to students' failed reasoning attempts through the teacher model, and combine the batch adaptive reward decay mechanism to encourage students to independently correct their mistakes. Although the motivation of this study has some intuitive inspiration, it has serious flaws in terms of core innovation, the rigor of experimental design, and actual computational benefits. Overall, this method fails to make substantive contributions beyond the existing iterative fine-tuning or process supervision paradigms, and the experimental results are not sufficient to support its core claims. Therefore, it is recommended to reject it.

Critical strengths
  1. The research motivation is quite intuitive and enlightening. Introducing the Socratic questioning mechanism from the field of education into the post-training stage of large language models provides a new perspective for how to utilize failure trajectories.

  2. The author noticed that the fixed exponential decay might lead to the problem where the corrected correct answers obtain negative advantage values. To address this, the author attempted to ensure that the samples that were successfully corrected would obtain non-negative advantage values by smoothing the batch means. This mathematical reasoning is logically consistent.

Critical weaknesses
  1. The core innovation is severely lacking. This method is essentially still a simple variant of the existing process supervision or iterative correction methods. The so-called interactive guidance does not break away from the traditional framework where the teacher model generates feedback and the student model optimizes strategies. It has an extremely blurred boundary with existing self-distillation or feedback-based reinforcement learning methods, lacking a true paradigm shift.

  2. Although the paper provides theoretical proofs regarding reward decay, these proofs are based on highly simplified assumptions. In actual large language model training, the sample complexity within batches varies greatly. Simple smoothing averages are difficult to stably control the variance of the advantage function. The paper lacks in-depth empirical analysis of the training instability.

  3. Introducing multi-round interactive guidance inevitably leads to a significant increase in training and inference costs. However, the paper only shows a minor performance improvement on limited scientific reasoning and mathematical datasets. This marginal gain cannot prove the rationality of its high computational cost and lacks feasibility in practical applications.

Questions for authors

see weakness

1 comment

We sincerely thank the reviewer for recognizing the intuitive motivation of applying Socratic teaching to policy optimization and for providing detailed comments on the theoretical and empirical limitations.

We address the concerns below.

W1. Novelty Beyond Existing Process Supervision and Iterative Correction

SocraticPO focuses on the interaction between two specific components:

  1. Attempt-conditioned diagnostic guidance: The teacher examines the student's actual failed reasoning and provides targeted feedback rather than merely evaluating the final answer or generating an independent demonstration.
  2. Assistance-aware credit assignment: The student-generated correction is evaluated by the task verifier, while its reward depends on the amount of assistance required.

The key distinction is that the method aims to turn failed attempts into positive learning opportunities without assigning equal credit to independent and assisted success.

We will expand the related-work discussion to include SCoRe, RISE, Reflexion, Self-Refine, and related self-correction methods.

We will also narrow our novelty claims and explain the differences in feedback structure, optimization objectives, and reward assignment.

W2. Batch Heterogeneity and Training Stability

Our reward construction uses the active sample set at each round and a smoothing pseudocount beta to update the preceding mean reward.

Theorem 1 establishes deterministic properties of the resulting reward assignment under this construction. However, we agree that these properties do not fully characterize the variance of policy-gradient estimates or guarantee optimization stability under heterogeneous batches.

We will therefore distinguish the mathematical properties established by Theorem 1 from the broader empirical question of training stability.

W3. Computational Cost and Practical Benefits

As reported in Figure 6, self-guided SocraticPO at K=2 increases training time per step from 0.0349 h to 0.0507 h. External teacher guidance further increases this cost.

We will therefore add matched student-generation and wall-clock comparisons to determine whether targeted correction provides more useful learning signals than spending the same computational budget on independent rollouts.

Reviewer_Dk5LOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SocraticPO uses teacher hints after failed attempts to generate student correction trajectories. Its reward rule sets the reward for a later success using the running mean of previous attempt rewards. Combined with global normalization, this preserves non-negative advantages for successful corrections while favoring earlier successes. The paper evaluates unassisted performance with one Qwen3-4B student on four SciKnowEval domains and GSM8K, and studies blind retries, reward discounts, different teachers, and training time.

Critical strengths
  1. A novel LLM RL algorithm of SocraticPO is proposed by using teacher hints after failed attempts to generate student correction trajectories, and the credit assignment strategy is correspondingly designed.

  2. The reward design addresses a concrete optimization issue: a correct assisted attempt can otherwise receive a negative centered advantage. The recursive construction gives a clear non-negativity property while preserving a preference for earlier success; this is more specific than simply adding teacher feedback.

  3. The blind-retry and discount controls examine plausible alternative explanations. Blind retries score 75.4975.49, while the self-guided adaptive version reaches 78.9578.95; removing discounting also hurts. These results support the value of guidance together with differentiated rewards.

  4. The method only requires textual teacher feedback, and evaluation is unassisted. Testing self-guidance as well as two external teachers makes the training setup practically interesting, while the reported time measurements expose the additional cost.

Critical weaknesses
  1. The reward design remains somewhat heuristic. The non-negativity guarantee is useful, but it does not establish that the proposed rule leads to better independent reasoning.

  2. The evaluation is not sufficiently convincing for the broader reasoning claims. Four benchmarks are scientific multiple-choice tasks, while the mathematics evaluation is limited to GSM8K. More challenging mathematical reasoning and coding tasks (AIME, LiveCodeBench, etc.) would provide stronger evidence that the method improves reasoning beyond these relatively narrow settings.

  3. The experiments use only a single Qwen3-4B-Instruct student. Results on additional model families, larger models, and stronger reasoning models are needed to assess whether the gains generalize beyond this particular configuration.

  4. The practical motivation for teacher-guided RL needs further clarification. In settings where RL aims to advance a model’s capabilities, a stronger teacher may not be readily available; when one is available, direct distillation is a natural alternative. The self-guidance results partly address this concern, but the paper should better establish when SocraticPO is preferable to simpler distillation or self-correction approaches under comparable supervision and compute budgets.

  5. LLM RL training is already resource-intensive, and SocraticPO increases per-step training time by approximately 40% even with self-guidance (Fig. 6), raising concerns about whether the performance gains justify the additional computational cost.

Questions for authors

Please see Critical Weaknesses.

1 comment

We sincerely thank the reviewer for the positive assessment of our reward construction and the detailed suggestions regarding generalization, teacher availability, and computational efficiency.

We address the concerns below.

W1. The Heuristic Nature of the Reward Design

Theorem 1 establishes three specific properties of the batch-adaptive reward assignment: non-negativity of successful corrections, monotonicity with respect to the number of required interventions, and saturation decay. These properties characterize the credit-assignment behavior induced by our reward construction.

The empirical contribution of the reward design is supported by our ablation study. Under self-guidance, removing reward decay reduces the average accuracy from 0.7895 to 0.7641, while using a fixed decay factor of 0.8 achieves 0.7824.

W2. Limited Evaluation on Reasoning Benchmarks

Our evaluation focuses on verifiable reasoning, consistent with the setting targeted by closely related RL-for-reasoning work. In particular, our use of SciKnowEval follows the ICML work SDPO (Reinforcement Learning via Self-Distillation), which evaluates on the same scientific reasoning benchmark.

[1] Reinforcement Learning via Self-Distillation, ICML 2026.

W3. Generalization Across Model Scales and Families

We acknowledge that all main experiments currently use Qwen3-4B-Instruct as the student, with thinking mode disabled. This is mainly due to the limited computational cost, and we will supplement more experiments under model parameters in the future.

W4. Why Teacher-Guided RL Instead of Direct Distillation?

Teacher-guided RL offers two practical differences from direct distillation. First, it supports black-box teachers because only textual feedback is required; no teacher logits, token probabilities, or tokenizer alignment are needed. Second, the teacher diagnoses the student's actual failed trajectory, while the student must generate the corrected response and is encouraged by reward decay to solve with less assistance later. This makes the signal closer to guided reflection than answer imitation. Consistently, the teacher--student KL study in Figure 6 shows no systematic KL decrease as task performance improves, although this does not rule out all behavioral transfer.

Our self-guidance results provide an additional perspective. Even when the student itself provides guidance, SocraticPO reaches 0.7895 at K=2 and 0.8066 at K=3, compared with 0.7807 for Reinforce++.

These results suggest that the interaction structure does not fundamentally require a stronger external teacher, although additional correction attempts increase computational cost.

W5. Computational Overhead

We agree that SocraticPO introduces additional computational costs. However, we do not claim that SocraticPO is computationally cheaper than conventional RL. Its motivation is to extract more useful learning signals from unsuccessful attempts by introducing targeted correction.

To assess whether this additional computation is justified, we will provide matched student-generation and wall-clock comparisons, reporting unassisted accuracy as a function of total computational cost.

Reviewer_TrZoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SocraticPO, a policy optimization method that turns failed reasoning rollouts into multi-turn teacher-guided corrections. When the student's initial attempt fails, a teacher provides a diagnostic hint conditioned on that attempt, and the student generates a revised solution. To prevent over-reliance on guidance, the authors introduce a batch-adaptive reward decay scheme that preserves non-negative normalized advantages for successful corrections while favoring earlier success. Experiments on four SciKnowEval domains and GSM8K with Qwen3-4B-Instruct show gains in unassisted accuracy over Reinforce++, GRPO, GSPO, and SDPO.

Critical strengths
  1. Using text-level diagnostic hints to turn failed rollouts into positive training signals is intuitive, and the observation in Figure 2b that naive exponential reward decay can induce negative normalized advantages for late corrections is a valid point.
Critical weaknesses
  1. There is a fundamental mismatch between the hint-conditioned training objective and the unassisted test-time distribution, and Theorem 1 is largely trivial.

    • Context distribution gap: For any corrected attempt (k≥2k \ge 2), the policy gradient in Eq. (8) maximizes log⁡πθ(y(k)∣x,y(1),f(1))\log \pi_{\theta}(y^{(k)} \mid x, y^{(1)}, f^{(1)}), which is conditioned on both the failed attempt and the teacher's hint. At test time, however, the model is evaluated on πθ(y∣x)\pi_{\theta}(y \mid x) without any hint. Unlike context distillation, there is no objective term or KL constraint that transfers the hint-conditioned behavior back to the unassisted policy πθ(y∣x)\pi_{\theta}(y \mid x). Prior work on RL for self-correction [1] has already shown that directly reinforcing second-turn revisions does not reliably improve first-turn accuracy and can induce behavioral collapse.

    • Triviality and flaw of Theorem 1: Theorem 1 merely states that if the reward at round kk is set to the running mean μk−1\mu_{k-1}, then the running mean is non-increasing (μk−1≥μt\mu_{k-1} \ge \mu_t), which is an elementary arithmetic identity rather than a theoretical guarantee on policy improvement. Worse, Property 3 ("saturation decay") reveals a design flaw: when all remaining failed problems in round tt are successfully corrected (pt=1p_t = 1), their normalized advantages become identically zero (a^i(t)=0\hat{a}_i^{(t)} = 0), meaning the model receives zero gradient signal from those successful repairs.

  2. The experimental comparison is unfair in compute/rollout budget, and the paper overlooks the established literature on multi-turn self-correction RL.

    • Unfair rollout and runtime budget: As shown in Figure 6 (right), SocraticPO requires 1.45×1.45\times to 2.28×2.28\times more training time per step than Reinforce++ (0.0507-0.0796 h/step vs. 0.0349 h/step) and generates additional student and teacher rollouts on failed prompts. However, the baselines in Table 1 are not given a matched number of student rollouts or matched training time. Notably, self-guided SocraticPO at K=3K=3 (0.8066) even outperforms using a 27B teacher at K=2K=2 (0.7987), suggesting that much of the gain simply comes from sampling more attempts per problem.

    • Missing comparisons with self-correction RL: Training LLMs via multi-turn "attempt →\to critique/feedback →\to revision" loops is already a well-established paradigm [1-5]. None of these closely related works are discussed or compared. Moreover, in Table 2, under self-guidance, a simple fixed 0.8 decay already achieves 0.7824 (vs. 0.7895 for the proposed adaptive decay), showing that the core technical contribution brings only a +0.7% difference.

[1] Training Language Models to Self-Correct via Reinforcement Learning

[2] Recursive Introspection: Teaching Language Model Agents How to Self-Improve

[3] Reflexion: Language Agents with Verbal Reinforcement Learning

[4] Self-Refine: Iterative Refinement with Self-Feedback

[5] Generating Sequences by Learning to Self-Correct

Questions for authors
  1. Since Eq. (8) for k≥2k \ge 2 only updates πθ(y(k)∣x,y(1),f(1))\pi_{\theta}(y^{(k)} \mid x, y^{(1)}, f^{(1)}) conditioned on the teacher's hint, what theoretical mechanism ensures that this gradient improves the unassisted policy πθ(y∣x)\pi_{\theta}(y \mid x) at test time?

  2. By Property 3 of Theorem 1, whenever the teacher's hints help the student fix all remaining failed samples in round tt (pt=1p_t = 1), every corrected sample in that round receives zero advantage (a^i(t)=0\hat{a}_i^{(t)} = 0) and contributes nothing to the gradient update. Why is throwing away the training signal in the most successful correction round desirable?

  3. Can the authors compare SocraticPO with Reinforce++ and GRPO under a matched student-rollout budget and matched wall-clock training time (Figure 6)?

  4. Can the authors clarify the conceptual novelty of SocraticPO over existing multi-turn self-correction RL methods [1-5], and explain why the proposed batch-adaptive decay only improves over a fixed 0.8 decay by 0.7 points under self-guidance (Table 2)?

1 comment

We sincerely thank the reviewer for the insightful and technically detailed feedback. We particularly appreciate the concerns regarding the context distribution gap, the saturation property of our reward design, and the computational fairness of our comparisons. We address each point below.

Q1. Context Distribution Gap Between Training and Inference

Our motivation is that guided correction provides additional supervision about how to repair errors encountered by the current policy. Although the corrected response is generated under an augmented context, its gradients still update the shared student parameters. Together with the negative advantage assigned to failed independent attempts, this creates an optimization process that penalizes unsuccessful behaviors while exposing the student to successful revisions.

Our existing Figure 3 provides preliminary evidence of this transfer. It separately tracks first-attempt accuracy (k=1), which is evaluated without teacher assistance, and guided accuracy after subsequent correction attempts. The first-attempt curves show improvements during training on the Biology and Material benchmarks.

Q2. Saturation Property and Zero Advantage

We agree that these final-round corrected responses do not contribute a positive policy-gradient signal under the current normalization rule.

However, this does not imply that the entire trajectory contributes no learning signal. Earlier failed attempts still receive negative advantages, and successful responses in earlier rounds retain positive advantages when subsequent failures reduce the running mean. Moreover, Appendix A shows that the final-round centered advantage is proportional to (1−pt)(1-p_t); it is strictly positive whenever some active examples remain incorrect (pt<1p_t<1), and becomes zero only in the fully saturated case pt=1p_t=1. This conservative behavior avoids over-rewarding responses that become uniformly correct only after the maximum amount of assistance.

Q3. Matched Student-Rollout and Wall-Clock Budgets

We agree that equal training steps do not imply equal compute. We will add two matched-budget experiments: (1) Reinforce++ with the same total student-rollout budget, measured by realized student-generated tokens/attempts, and (2) Reinforce++ trained for the same wall-clock budget. We will report unassisted accuracy under both settings and revise our claims according to the results.

Q4. Conceptual Novelty over Existing Self-Correction RL

We appreciate the reviewer pointing out the closely related literature. We agree that multi-turn self-correction and iterative refinement are established research directions, and we do not intend to claim that SocraticPO is the first method to train models through multiple correction attempts. Our contribution is more specific.

First, SocraticPO introduces attempt-conditioned diagnostic guidance. Rather than simply asking the student to reconsider its answer, the teacher examines the actual failed reasoning trajectory and provides a targeted question, critique, or next-step hint.

Second, we explicitly distinguish the reward assigned to independent success from that assigned to teacher-assisted correction. The batch-adaptive reward construction is designed to preserve non-negative normalized advantages for successful corrections while favoring earlier success.

Third, the teacher only needs to provide textual feedback. We do not require access to teacher logits or a teacher-generated target distribution.

We will substantially expand the discussion of SCoRe, RISE, Reflexion, Self-Refine, and Generating Sequences by Learning to Self-Correct, and clarify that our contribution concerns teacher-guided correction and assistance-aware credit assignment rather than multi-turn self-correction itself.

Reviewer_q7VhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SocraticPO, a policy optimization method that turns failed student rollouts into multi-turn teacher–student correction trajectories. When the student model fails on its first independent attempt, a teacher model (either the student itself or an external model like Qwen3.5-27B or Gemma3-27B) generates a text-level diagnostic hint conditioned on the failed attempt, and the student generates a revised response. To discourage reliance on teacher hints while keeping normalized advantages non-negative for successful revisions, the authors propose a batch-adaptive reward decay rule based on the smoothed mean of preceding attempts (Theorem 1). Using Qwen3-4B-Instruct as the student, the authors evaluate the method on four multiple-choice science subsets from SciKnowEval and GSM8K.

Critical strengths

Intuitive motivation and batch-adaptive reward formulation: Turning failed rollouts into guided revisions rather than simply assigning zero reward is a sensible motivation for extracting learning signals from mistakes. In addition, the observation that fixed exponential reward decay can push successful multi-turn revisions below the batch mean (yielding negative normalized advantages) is a useful insight, and the proposed smoothed-mean reward rule (Theorem 1) provides a clean way to guarantee non-negative credit while favoring earlier successes.

Critical weaknesses
  1. Gains appear driven by option elimination on multiple-choice tasks rather than genuine reasoning improvements. Four of the five benchmarks (Chemistry, Physics, Biology, Material) are 4-option multiple-choice tasks (A/B/C/D) evaluated solely by the final option letter, with very small test sets (50 to 210 questions, totaling 434 questions across all four tasks; Appendix B.3). In a 4-choice setting with first-attempt accuracy p_1, simply eliminating the rejected option and guessing uniformly among the remaining choices brings a minimum accuracy gain of \frac{1}{3}(1 - p_1) at k=2 (recovering 33.3% of failed attempts) and \frac{2}{3}(1 - p_1) cumulatively at k=3 (recovering 66.7% of failed attempts), which the verifier rewards as long as the final letter matches. Conversely, on GSM8K (Math)—the only open-ended benchmark where option elimination is impossible and which makes up 75.2% (1,319 questions) of all test examples—SocraticPO shows virtually no improvement over standard RL baselines: Reinforce++ achieves 0.9484 and GRPO achieves 0.9500, whereas SocraticPO (K=2) gets 0.9484 with self-guidance and drops to 0.9438 with Qwen3.5-27B. This strongly suggests that the reported gains come from option elimination on the small multiple-choice tasks rather than learning useful reasoning knowledge.
  2. In Eq. 8, the policy gradient for a successful revision at step k \ge 2 maximizes \log \pi_\theta(y_i^{(k)} \mid c_i^{(k)}), where the context c_i^{(k)} = x \parallel y_i^{(1)} \parallel f_i^{(1)} includes both the initial failed attempt and the teacher's diagnostic hint (e.g., in Appendix B.1, the reinforced student response begins with "I apologize for the earlier error..."). At test time, however, the model is evaluated strictly in an unassisted setting conditioned only on x. For difficult problems where the first attempt fails, the update only applies a negative advantage to \pi_\theta(y^{(1)} \mid x) and reinforces revising an answer given a critique, without directly optimizing a correct reasoning path conditioned solely on x.
  3. In Section 5.4 and Figure 5, the authors compare training with (gt) and without (w/o gt) teacher access to ground-truth answers. The w/o gt scores in Figure 5 match the main results in Table 1, implying the main experiments do not give the teacher ground-truth access. However, the teacher prompt template in Appendix B.2 explicitly includes {reference solution} and {ground truth block} as inputs. Moreover, in Table 1, the standalone accuracy of the Gemma3-27B teacher on Chemistry (0.4625), Physics (0.5866), and Material (0.6987) is lower than that of the untuned Qwen3-4B student (0.5071, 0.6664, and 0.7413). It is difficult to explain how a teacher that performs worse than the student on these specialized science tasks can accurately diagnose the student's errors without access to the ground-truth solution to yield the best student performance (0.8113).
  4. All experiments use a single student model size (Qwen3-4B-Instruct with thinking mode disabled), leaving generalization across model scales or thinking-enabled models untested. Furthermore, while Section 4.3 states that SocraticPO uses Reinforce++ as its optimization backend, Table 5 in Appendix B.4 indicates that SocraticPO enables asymmetric clipping (\epsilon-high = 0.28) and rollout importance-sampling clipping (\rho = 2), whereas both are disabled for the Reinforce++ baseline.
Questions for authors
  1. Option elimination vs. genuine reasoning on open-ended tasks: In a 4-way multiple-choice task, simply eliminating rejected options provides a minimum accuracy boost of \frac{1}{3}(1 - p_1) at k=2 and \frac{2}{3}(1 - p_1) at k=3 by random guessing alone, and SocraticPO shows almost zero gain over GRPO and Reinforce++ on the only open-ended task (GSM8K). How can you rule out that the gains primarily come from option elimination rather than learning useful reasoning? How does SocraticPO perform on harder open-ended benchmarks (e.g., MATH or AIME)?
  2. Teacher access to ground truth in Table 1 vs. Appendix B.2: Did the teachers in the main experiments (Table 1) have access to the reference solution and ground truth as shown in the Appendix B.2 prompt template, or were they run without ground truth as implied by Figure 5? If they did not have access to ground truth, how is Gemma3-27B able to provide accurate diagnostic feedback on Chemistry and Physics when its own standalone accuracy is lower than the untuned 4B student's?
  3. Aligning backend hyperparameters with Reinforce++: In Table 5, why are \epsilon-high asymmetric clipping (0.28) and rollout IS clipping (2) enabled for SocraticPO but disabled for the Reinforce++ baseline? How much of the performance gap between SocraticPO and Reinforce++ remains when both use the exact same clipping configuration?
1 comment

We sincerely thank the reviewer for the detailed and constructive feedback. We appreciate the recognition of our motivation and assistance-aware reward formulation. We address the concerns below.

Q1. Option Elimination vs. Genuine Reasoning

Thank you for raising this important concern. We would first like to clarify a key aspect of our evaluation protocol.

Our evaluation follows the unassisted mean@16 protocol. For each test question, we independently sample 16 responses conditioned only on the original question. No previous answers, teacher feedback, or answer-elimination information is provided across these generations. Importantly, mean@16 computes the average correctness of these independent responses, rather than selecting the best response among 16 attempts (i.e., pass@16).

Therefore, the proposed improvement bound of (1/3)(1-p_1) for K=2 and (2/3)(1-p_1) for K=3 describes a sequential answer-elimination procedure, which is not the evaluation protocol used in our experiments. Such a mechanism cannot directly explain the reported improvement in independent unassisted accuracy.

Our w/o targeted guidance (w/o tg) ablation is the closest control for this concern: it preserves an additional attempt after failure but replaces attempt-conditioned diagnosis with a fixed retry prompt.(Of course, the experiment is test on no previous answers, teacher feedback, or answer-elimination information)

SettingUnassisted Avg.
Fixed retry (w/o tg)0.7549
Self-guidance0.7895
Qwen3.5-27B guidance0.7987

Thus, merely indicating failure and allowing another attempt is insufficient to explain the gain; the diagnostic content of the guidance matters.

Q2. Teacher Access to Ground Truth

Thank you for identifying the ambiguity in Appendix B.2.

The main experimental results should be distinguished from the ground-truth-assisted setting in Figure 5. For the configurations shown in Figure 5, the w/o-GT results reproduce the corresponding main results in Table 1. Specifically, the self-guidance and Qwen3.5-27B configurations are evaluated without teacher access to ground-truth answers.

Appendix B.2 intentionally presents a unified template. Access to privileged information is controlled by a hyperparameter: when disabled, the reference-solution and ground-truth blocks are left empty; when enabled for the GT ablation, these fields are populated. We will clarify this control mechanism in the appendix rather than introduce separate templates.

In the final main experiments, the teacher does not use the ground-truth answer. Ground truth is used only by the verifier to determine correctness and by the dedicated GT ablation in Figure 5.

Regarding the teacher's lower standalone accuracy, we believe that solving a problem independently and diagnosing an error in an existing reasoning trajectory are fundamentally different capabilities. A teacher does not necessarily need to know the final answer to identify a local contradiction, an incorrect numerical reference, or an invalid reasoning step. For example, in Appendix B.1, the teacher identifies an inconsistency between the student's statement about material density and the values explicitly provided in the question. This diagnosis relies on checking the student's reasoning against the given information rather than independently solving the entire problem.

Q3. Aligning Backend Hyperparameters with Reinforce++

Thank you for your remind, We acknowledge that this was our negligence and we wil modify in the new version this setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.