- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to provide text-level guidance to students' failed reasoning attempts through the teacher model, and combine the batch adaptive reward decay mechanism to encourage students to independently correct their mistakes. Although the motivation of this study has some intuitive inspiration, it has serious flaws in terms of core innovation, the rigor of experimental design, and actual computational benefits. Overall, this method fails to make substantive contributions beyond the existing iterative fine-tuning or process supervision paradigms, and the experimental results are not sufficient to support its core claims. Therefore, it is recommended to reject it.
Critical strengths
-
The research motivation is quite intuitive and enlightening. Introducing the Socratic questioning mechanism from the field of education into the post-training stage of large language models provides a new perspective for how to utilize failure trajectories.
-
The author noticed that the fixed exponential decay might lead to the problem where the corrected correct answers obtain negative advantage values. To address this, the author attempted to ensure that the samples that were successfully corrected would obtain non-negative advantage values by smoothing the batch means. This mathematical reasoning is logically consistent.
Critical weaknesses
-
The core innovation is severely lacking. This method is essentially still a simple variant of the existing process supervision or iterative correction methods. The so-called interactive guidance does not break away from the traditional framework where the teacher model generates feedback and the student model optimizes strategies. It has an extremely blurred boundary with existing self-distillation or feedback-based reinforcement learning methods, lacking a true paradigm shift.
-
Although the paper provides theoretical proofs regarding reward decay, these proofs are based on highly simplified assumptions. In actual large language model training, the sample complexity within batches varies greatly. Simple smoothing averages are difficult to stably control the variance of the advantage function. The paper lacks in-depth empirical analysis of the training instability.
-
Introducing multi-round interactive guidance inevitably leads to a significant increase in training and inference costs. However, the paper only shows a minor performance improvement on limited scientific reasoning and mathematical datasets. This marginal gain cannot prove the rationality of its high computational cost and lacks feasibility in practical applications.
Questions for authors
see weakness
We sincerely thank the reviewer for recognizing the intuitive motivation of applying Socratic teaching to policy optimization and for providing detailed comments on the theoretical and empirical limitations.
We address the concerns below.
W1. Novelty Beyond Existing Process Supervision and Iterative Correction
SocraticPO focuses on the interaction between two specific components:
The key distinction is that the method aims to turn failed attempts into positive learning opportunities without assigning equal credit to independent and assisted success.
We will expand the related-work discussion to include SCoRe, RISE, Reflexion, Self-Refine, and related self-correction methods.
We will also narrow our novelty claims and explain the differences in feedback structure, optimization objectives, and reward assignment.
W2. Batch Heterogeneity and Training Stability
Our reward construction uses the active sample set at each round and a smoothing pseudocount beta to update the preceding mean reward.
Theorem 1 establishes deterministic properties of the resulting reward assignment under this construction. However, we agree that these properties do not fully characterize the variance of policy-gradient estimates or guarantee optimization stability under heterogeneous batches.
We will therefore distinguish the mathematical properties established by Theorem 1 from the broader empirical question of training stability.
W3. Computational Cost and Practical Benefits
As reported in Figure 6, self-guided SocraticPO at K=2 increases training time per step from 0.0349 h to 0.0507 h. External teacher guidance further increases this cost.
We will therefore add matched student-generation and wall-clock comparisons to determine whether targeted correction provides more useful learning signals than spending the same computational budget on independent rollouts.