- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 3
Read review
Paper summary
The paper proposes IP-OPD, an OPD method that evaluates teacher-proposed reasoning blocks through student continuations. By comparing the outcomes of teacher interventions and student blocks, the method constructs bidirectional preferences for block-level optimization. Experiments on mathematical reasoning benchmarks show improvements over several baselines.
Critical strengths
-
The proposed method is clean, intuitive, and clearly motivated.
-
The paper provides meaningful analysis and well-designed diagnostics.
-
The experiments include training-round-matched, outcome-information-matched, and compute-matched comparisons, which help assess the effectiveness of the proposed method and objective.
Critical weaknesses
- Limited evaluation across model scales and families.
The main experiments are conducted on Qwen3-1.7B-Base, with an additional 0.6B experiment in the appendix. Experiments with at least a 4B student and another model family beyond Qwen3 would strengthen the empirical evidence.
- Limited evaluation across task domains.
All experiments focus on mathematical reasoning. Additional experiments on other domains would help assess the cross-domain generalization of IP-OPD.
- Missing training dynamics analysis.
The paper trains for three epochs on 7,500 problems and reports endpoint results without evaluation learning curves. At the final checkpoint, GRPO already slightly outperforms Vanilla OPD. However, OPD and GRPO may exhibit different learning dynamics, with OPD potentially benefiting from faster early-stage improvement. It is unclear whether IP-OPD retains this advantage or primarily achieves its improvements after extended reward-based training. Evaluation curves would help clarify this distinction. If IP-OPD mainly benefits from longer training, comparisons with stronger teacher-guided reward-based methods, such as LUFFY and Prefix-RFT, would also be important.
Questions for authors
-
Could the authors extend the experiments to larger students, such as Qwen3-4B, and ideally another model family?
-
Could the authors evaluate IP-OPD on other verifiable reasoning domains?
-
Could the authors provide evaluation learning curves for IP-OPD, Vanilla OPD, EOPD, and GRPO? In particular, does IP-OPD retain the early-stage learning efficiency of OPD? If its improvements mainly emerge after extended training, could the authors include teacher-guided reward-based baselines such as LUFFY and Prefix-RFT?
I appreciate the authors' additional experiments, which address many of my concerns. However, several issues remain.
Cross-domain evaluation. While the K&K experiments demonstrate the applicability of IP-OPD beyond mathematical reasoning, the benchmark appears largely saturated. Logic-RL achieves performance similar to IP-OPD, and comparisons with other OPD baselines are missing. Therefore, it remains unclear whether IP-OPD provides additional benefits over existing OPD methods in this domain.
GRPO training dynamics and stronger baselines. I appreciate the authors providing the training curves. The OPD baselines appear to plateau in later training, which is understandable. However, GRPO also plateaus after approximately 1–2 epochs, while IP-OPD continues to improve. I think further explanation of GRPO's training behavior is needed.
I also remain interested in comparisons with LUFFY and Prefix-RFT, as they share similar motivations and approaches with IP-OPD and would serve as relevant baselines.
Presentation. Some newly added sections are unnecessarily verbose and awkwardly written. For instance, I do not see why the sentence "LUFFY and Prefix-RFT are not evaluated, so no superiority over them is claimed" appears in the revised manuscript's discussion of training dynamics. It seems that the authors did not even carefully review the revised manuscript before submission, which raises concerns about the overall quality of the revision.
Overall, the rebuttal has strengthened the paper, but I believe these concerns remain unresolved.
Thank you for providing a detailed response and conducting additional experiments.
The fixed-buffer and three-seed results for native-winning preferences, as well as the common-budget comparison with DOPD-adapt and CR-style, address my two primary concerns. The clarification regarding novelty is reasonable. I am increasing my score to 3.