acceptodds
Withdrawn from ICLR 2027

Closing the Self-Teaching Loop for Open-Ended Generation: Sibling-Conditioned Distillation Meets Self-Verification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-teaching improves language models by learning from their own generations and assessments, but extending this loop to open-ended generation remains challenging because response quality is multidimensional and task-native verification is generally unavailable. Rubrics provide a natural specification in this setting, but do not by themselves close the learning loop: how should a fixed rubric be grounded in the model's current responses, and how should stronger completed responses be distinguished from weaker ones? We introduce , which combines local and global routes to address these challenges. The local route uses same-prompt siblings to ground rubric guidance in current-policy realizations, while the global route reuses rubric-conditioned forward pass to derive continuous response scores for group-relative optimization. Across Qwen3-4B and Qwen3-8B on medical and science tasks, S²PO improves over the base model by 3.09–5.26 points and outperforms both RGSD and 32B-judge GRPO in all four in-domain settings. These gains persist under out-of-distribution evaluation, with improvements of 2.46–5.14 points over the base model across both model scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-teaching, on-policy distillation, open-ended generation, self-verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GGcfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Sibling and Self-Verification Policy Optimization, a framework for improving open-ended generation through self-teaching. Experiments on Qwen3-4B and Qwen3-8B across medical and scientific tasks show improvements over RGSD and GRPO, with additional ablation studies supporting the contributions of both components.

Critical strengths
  1. The proposed sibling-conditioned distillation provides an interesting approach to incorporating information from multiple on-policy responses into dense token-level supervision.

  2. The experiments demonstrate consistent improvements across multiple benchmarks and model scales.

Critical weaknesses
  1. The paper only evaluates Qwen3-4B/8B in Non-Thinking Mode and lacks comparisons with stronger recent OPD/OPSD methods, particularly those combining RL with self-distillation. Evaluation in Thinking Mode would further strengthen the results.

  2. The proposed method largely combines existing on-policy self-distillation and GRPO techniques with sibling-conditioned guidance and self-verification. While the combination is potentially useful, the individual components build heavily on prior work, and the paper does not sufficiently establish a substantial methodological advance beyond existing approaches.

Questions for authors

How does S²PO differ substantially from recent OPD/OPSD methods that combine RL with self-distillation? Why are these methods not included as baselines, especially given that adding distillation to self-verification yields only 0.71 and 0.53 percentage-point improvements in Table 2?

Reviewer_SPW3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes S²PO, a self-teaching framework for open-ended generation that combines sibling-conditioned distillation with trajectory-level self-verification. The local route leverages same-prompt responses to construct rubric-guided token-level distillation targets through matched logit corrections, while the global route uses a frozen same-model verifier to produce continuous rewards for GRPO. Experiments on medical and science tasks with Qwen3-4B and Qwen3-8B demonstrate improvements over rubric-guided self-distillation and external-judge GRPO, including gains on out-of-domain benchmarks.

Critical strengths
  1. I think the paper is well-motivated. It identifies two meaningful challenges in open-ended self-teaching: translating rubric specifications into effective token-level supervision and obtaining reliable trajectory-level feedback without external verification.
  2. The method is generally sound. The proposed matched-correction mechanism offers an interesting way to incorporate sibling information into distillation, while the pass-reusing self-verifier efficiently provides trajectory-level rewards.
  3. I think the empirical analyses and results are somewhat comprehensive, although some aspects requires deeper investigation (see weaknesses below).
Critical weaknesses
  1. Insufficient evidence for the incremental contribution of matched correction. While matched correction is a central methodological contribution, its incremental benefit under full joint training remains unclear. Table 3(b) shows improvements over rubric-only and direct sibling-conditioned distillation, but only on Qwen3-4B Medical at step 100 without self-verification. Table 2(b) compares η\eta = 0 and η\eta = 0.8 under joint training, also only at step 100. Meanwhile, Table 2(a) shows modest gains of 0.71 and 0.53 points from adding dense distillation to self-verifier GRPO, without isolating the contribution of matched correction itself. In my opinion, comparisons against simpler distillation targets under full joint training at convergence, across models and domains, are needed to establish the benefit of the proposed mechanism.
  2. Limited comparisons with stronger rubric-based optimization methods. The main experiments compare S²PO only against RGSD and external-judge GRPO, leaving its competitiveness against stronger rubric-based methods insufficiently established. In particular, CriPO [1] also combines rubric-based RL with on-policy self-distillation for open-ended generation, making it a closely related baseline. RuscaRL [2] further explores rubric-guided reinforcement learning for open-ended tasks, offering a stronger alternative to standard GRPO. Comparisons with these methods under matched training and evaluation settings would provide more convincing evidence of S²PO's effectiveness relative to existing approaches.
  3. Limited fine-grained evaluation of multidimensional response quality. While multidimensional quality is a central motivation, the paper reports only aggregated rubric scores without examining improvements across individual criteria. It remains unclear whether the gains reflect broad improvements or mask regressions in important dimensions such as factual accuracy and safety. Criterion-level analysis, particularly for positive and negative HealthBench criteria, would better substantiate the paper's claims.

[1] Xia, M., Yang, Y., Ye, C., Zhu, S., Yang, S., Zhu, G., Zhang, Y., Peng, C., Wang, H., & Wang, S. (2026). Enhancing Rubric-based RL via Self-Distillation. arXiv preprint arXiv:2607.18082. [2] Zhou, Y., Li, S., Liu, S., et al. (2026). Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for Open-Ended LLM Reasoning. Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306, 165222–165251.

Questions for authors

Please see weakness above.

Reviewer_mKniOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a self-teaching framework, which leverages siblings to guide the self-distillation process.

Critical strengths
  • Basically, the reviewer agrees that if we can explore more within the group, we can gain more. I believe this is what this paper did.
Critical weaknesses
  • This paper is pretty hard to follow. Honestly, I do not fully understand the paper at this time. The Figures in the paper do not help me better understand the proposed method. For example, Figure 1 and Figure 2, it seems that the motivation in Figure 1 is not well correlated with the procedures in Figure 2. The reviewer can understand the motivation in Figure 1, but cannot associate the methodology in Figure 2 with Figure 1.
  • It would be nice if the authors could provide legends in Figure 2. It is hard to understand the symbols in Figure 2 and what the flow describes.
  • I think the authors can also discuss some self-play works. Self-play and self-teaching basically share the same idea.
  • It would also be great if the authors could add more comparisons of related works in Table 1. And results on popular benchmarks like AlpacaEval, AGI-Eval, etc.
Questions for authors
  • Please check the Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.