acceptodds
Under review as a conference paper at ICLR 2027

OpenSIR: Open-Ended Self-Improving Reasoner

Abstract

Self-play promises to advance LLM reasoning without human-annotated data, yet existing methods lack open-ended learning and yield only marginal or even negative gains on post-trained models. We introduce Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework where a single LLM alternates teacher and student roles to generate and solve novel problems without external verifiers or annotated data. From a single seed, OpenSIR sustains open-ended exploration via diversity rewards pushing toward unfamiliar concepts and difficulty calibration keeping problems learnable. Across seven math benchmarks, OpenSIR consistently improves all models, averaging +3.6 on instruction models and +3.1 on reasoning models, outperforming recent self-play baselines. Without any annotated data, it surpasses GRPO baselines trained on over 7K examples. Despite training only on self-generated math, OpenSIR is the only self-play method that transfers to general reasoning, improving reasoning models by at least +4.4 points across three benchmarks. Our analysis shows that, with diversity rewards, generated problems progress from basic arithmetic and algebra to advanced topics such as calculus and optimisation, reaching regions of the problem space beyond human-curated datasets. Without diversity rewards, problems concentrate in a narrow region of this space and accuracy falls on both mathematical and general reasoning. Difficulty calibration depends on teacher-student co-evolution: freezing the teacher destabilises problem difficulty and lowers accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, reinforcement learning in agents, llm agents, math reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gEyjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

OpenSIR studies verifier-free self-improvement for mathematical reasoning. One policy generates and solves questions, using majority-vote pseudo-answers and rewards based on solvability, solution length, and embedding distance from earlier questions. The paper reports gains on mathematical and general-reasoning benchmarks across four post-trained models. The results support the recipe in the tested setting.

Critical strengths
  1. The paper addresses the important challenge of improving mathematical reasoning without a large labeled question pool or an executable verifier. Its shared-policy training loop is concrete, and the reported gains span four starting models.

  2. Diversity and teacher-training ablations, seed sensitivity, and pass@k analyses give useful evidence that the full recipe works in the tested mathematical setting.

Critical weaknesses
  1. The incremental contribution is not yet convincing to me. Seed-based augmentation and embedding diversity have precedents in DaaR; RL-based mathematical question synthesis with complexity and answer-consistency signals appears in MathSmith; and Absolute Zero and R-Zero already explore self-play problem generation. OpenSIR's verifier-free shared-policy combination may be useful, but the paper does not isolate a sufficiently distinctive insight from it.

  2. The central scores are defined mathematically but insufficiently validated as proxies for what the paper claims to reward. Majority answer agreement does not establish answer correctness or a well-posed question; longer solutions need not mean harder reasoning; and question-embedding distance need not imply a new solution skill (pages 3–4). Table 3 calls external-model agreement “validity,” while the topic and embedding analyses do not rule out familiar templates (pages 7–9). With a shared generator/solver and a pool containing every format-valid question, independent checks of question quality, pseudo-answers, and skills are needed to support the proposed mechanism.

  3. The strongest efficiency and open-endedness claims outpace the experiments. The “exactly 2×” cost claim counts selected update rollouts, while Algorithm 1 samples solutions for candidate questions before selection; total generated tokens and equal-compute results are missing (pages 22–23). Performance plateaus after roughly 300 steps in the longer run, so sustained open-ended discovery is not shown. A per-example evaluation-overlap audit is also absent; notably, the separate AIME 2024 seed ablation evaluates on AIME 2024 (page 16). These gaps limit the interpretation of the gains as efficient, sustained transfer.

Questions for authors
  1. What is the specific insight beyond the closest data-diversity and mathematical-synthesis methods? Could you separate the diversity reward from the prompt's instruction to generate a conceptually different question, and test whether it yields distinct solution skills at a matched question count?

  2. On a representative sample across training stages, could you independently assess well-posedness, pseudo-answer correctness, and solution-skill novelty? How often do mathematically invalid questions remain in the historical pool, and how do length and embedding-distance scores relate to these checks?

  3. Could you report total candidate questions, solution rollouts, generated tokens, and GPU hours, clarify what the 2× cost figure includes, and provide an equal-compute comparison? Please also report per-example evaluation-overlap checks and whether the AIME 2024 seed ablation's result changes after excluding that seed from its AIME 2024 evaluation (Appendix B.5, B.9).

Reviewer_CR5JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenSIR trains one LLM to generate and solve math problems, starting from a single seed. It combines majority-vote labels with difficulty and diversity rewards, improving four post-trained models on math benchmarks with gains that transfer to general reasoning.

Critical strengths
  1. The gains hold across four instruction-tuned and reasoning models, and extend to three general-reasoning benchmarks. This makes the recipe interesting for improving models that have already received substantial post-training.

  2. Removing diversity rewards lowers both math and general-reasoning accuracy, supporting their role beyond producing different-looking questions. The appendix also examines seed sensitivity, longer training, solution length, and teacher updates.

Critical weaknesses
  1. The student is rewarded for matching its own majority answer. If that answer is wrong, RL can reinforce the mistake and penalize correct minority answers? Since the same model generates later training data, these errors can feed back into subsequent rounds.

Section 3.3 acknowledges label noise, but agreement with GPT-5 does not establish actual answer correctness.

  1. SQLM [1] already trains one shared model in both proposer and solver roles using RL. It needs only a topic prompt, rewards moderately difficult questions, and uses majority-vote answers as labels. These are also central parts of OpenSIR. The embedding-based diversity reward is a difference, but the paper does not discuss or compare with SQLM. SeRL [2] also combines majority-vote rewards with difficulty and diversity filters, though it does not directly train question generation with RL.

[1] Lili Chen et al. Self-Questioning Language Models. arXiv:2508.03682, 2025.

[2] Wenkai Fang et al. SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data. arXiv:2505.20347, 2025.

Questions for authors
  1. How many selected training problems fall outside the target solve-rate range? A human check of their majority answers would help distinguish incorrect labels from ill-posed questions, beyond the existing problem-validity assessment.

  2. Does the 2x cost estimate include solutions sampled for problems later discarded? A GRPO comparison at matched generated tokens or GPU-hours would help assess whether the gains justify the additional compute.

  3. How does OpenSIR differ from SQLM beyond its reward design? Comparing with SQLM under the same training setup would help establish what the new recipe adds.

Reviewer_FSLEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces OpenSIR, an asymmetric self-play framework that starts from a single seed problem. A single model alternatively acts as both a teacher (generating problems) and a student (solving problems). The teacher generates math questions with a solvability/solution-length/diversity question reward, and the student is trained to solve them with majority-vote answers. The paper shows performance improvement over self-play baselines across both instruction-tuned and reasoning models, and math/general-reasoning benchmarks. It also ablates the different components of the teacher reward.

Critical strengths
  • The demonstration that existing self-play methods fail on reasoning/post-trained models is interesting, and serves as good motivation for the proposed method.
  • Comprehensive set of experiments: multiple models (covering different families, and both instruction-tuned and reasoning models), math benchmarks, general reasoning benchmarks, and multiple seeds per experiment.
  • OpenSIR surpasses GRPO baselines trained on real data
  • Fairly comprehensive ablations on the different components of the teacher reward.
Critical weaknesses
  • The paper mentions that results are over 3 seeds, but I couldn't find quantification of error in the main results.
  • The main setup/method is quite similar to existing works from late 2025/early 2026. Overall the specifics of the setup and experiments on post-trained models differentiate OpenSIR sufficiently, but I would recommend much better positioning compared to existing work, especially in the related works section, which fails to mention quite a few relevant works (examples include [1-3] - for instance, [1] also discusses task diversity). I think it would be a good idea to discuss other teacher reward formulations from the literature more extensively, also beyond the papers I mentioned below.

[1] Li, G., He, J., Wang, S., Zhang, D., Liu, R., Zhang, R., Yao, Z., Fang, J., Guo, H., & Wang, J. (2026). R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training. ICML, 2026.

[2] Bailey, L., Wen, K., Dong, K., Hashimoto, T. and Ma, T., 2026. Scaling self-play with self-guidance. arXiv preprint arXiv:2604.20209.

[3] Sundaram S, Quan J, Kwiatkowski A, Ahuja K, Ollivier Y, Kempe J. Teaching models to teach themselves: Reasoning at the edge of learnability. ICML, 2026.

Questions for authors

Main questions:

  • Can the authors provide mean/stdev and/or the min/max over seeds for their main results?
  • Can the authors expand their discussion of related work to better position their teacher reward formulation against other asymmetric self-play works in the literature, and how it differs?

Clarification:

  • Majority voting (described in section 2.2) - The paper claims that majority voting can quantify the reliability of the problem ("low solve rates suggest inconsistent solutions that may indicate flawed problem formulations"). Can this be separated from problem difficulty? It seems that a problem that is well-formulated but difficult for the student would also have a low solve-rate, despite being potentially pedagogically useful. Conversely, a problem that the student is incorrectly confident on could have a good solve rate while actually reinforcing an incorrect belief. Section 3.3 partially addresses this, but it may be the case that good-but-difficult problems are still getting thrown away; is there a way of disentangling this?
  • Figure 2 showing the increase in task diversity compared to R-Zero was quite nice. Was this at the end of an R-Zero run? If so, did that look different at the beginning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.