acceptodds
Under review as a conference paper at ICLR 2027

Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

Abstract

Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforms R-Zero by 17.32 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Self-evolving, Reinforcement Learning, Math Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_tUh8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why R-Zero-style self-evolution suffers from performance collapse after multiple rounds. The authors identify two problems, invalid questions being increasingly selected by answer-consistency filtering, and repeated mathematical tasks that cannot be detected by lexical similarity. R-Quest introduces validity-aware solver training and semantic novelty assessment to improve generated question quality. Experiments show better results on math, code and general reasoning, and more stable performance over ten rounds.

Critical strengths
  1. The analysis of performance collapse is interesting. Especially the finding that answer-consistency filtering actually increases the proportion of invalid questions. The repaired-question experiment in Fig. 3 provides some evidence that these invalid questions really hurt training.
  2. The long-term results are quite good. I also like the observation in Fig. 7 that stronger novelty control does not always help. Too much diversity may also prevent learning harder variants of existing problems. This part makes sense.
Critical weaknesses
  1. One concern is what really prevents the performance collapse. R-Quest uses GPT-5.6-Sol to construct validity training data, and keeps replaying these annotated samples during every round of solver training. So there is still external supervision continuously involved. It is unclear how much of the improvement comes from this replay and how much from the proposed online validity feedback. The existing ablations do not clearly separate these effects. This matters particularly when the main failure of R-Zero is accumulating incorrect pseudo-labels.
  2. Another issue is the validity assessment itself. From Fig. 6(a,b), the final balanced accuracy and invalid recall imply that recall on valid questions is only around 60%. So quite many valid questions are also rejected. I wonder whether difficult but valid questions are more likely to be rejected. Also, Fig. 6(d) compares pseudo-label accuracy on questions generated by different methods. These question distributions could have very different difficulty. Higher accuracy may partly come from generating easier or more predictable questions, rather than improved solver reliability. The current evidence cannot fully distinguish these explanations.
Questions for authors
  1. Can you compare R-Quest with a baseline using the same validity-initialized solver and 10% replay data, but without the online validity gate? Does it still collapse after ten rounds?

  2. In Fig. 6, the valid-question recall seems quite low. What is the false rejection rate under the actual nine-response majority gate? Are the rejected valid questions systematically more difficult?

  3. For Fig. 6(d), can you evaluate the pseudo-label accuracy of both solvers on the same set of valid generated questions? This would help understand whether the improvement mainly comes from better question selection or better solver supervision.

Reviewer_J7thOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines performance collapse in self-evolving reasoning models and attributes it to invalid self-generated questions and mathematically equivalent repeats missed by lexical diversity filters. The authors propose R-Quest, using solver validity judgments and frozen-model novelty feedback to filter data and reward questioners. Across 12 benchmarks and two model families, R-Quest achieves the best average performance and sustains gains over ten rounds, surpassing R-Zero by 17.32 points.

Critical strengths
  • The paper offers a well-motivated diagnosis of self-evolution failure modes, identifying and quantifying two concrete bottlenecks: invalid questions and task-level repetition. In particular, the finding that answer-consistency filtering reduces the valid-question rate to 27.64% by round five provides a strong and useful empirical insight. This is consequential because it explains why R-Zero-style self-evolution degrades and motivates the proposed feedback mechanisms.

  • The proposed method, R-Quest, is simple, general, and empirically effective, combining validity-aware initialization, solver-derived validity feedback, and a sampled pairwise novelty gate. It improves over R-Zero, OCNR, and R-Diverse across two model families and 12 benchmarks, and sustains gains over ten rounds. The ablations in Table 3 and Figure 5 further support the importance of both validity and novelty feedback.

Critical weaknesses
  • The method relies heavily on GPT-5.6-Sol for validity annotation, question repair, answer verification, and novelty grouping, so the definition of a "valid" question is effectively outsourced to a strong proprietary judge. This may bias both the solver's learned validity preferences and the original-vs-repaired comparison, and the paper reports little on human agreement or judge robustness.

  • The novelty check is somewhat shallow. It only compares each new question against a small random sample from the same batch, and it does not remember repeated question types across rounds. As a result, similar tasks can still creep back in over time, and the comparison budget feels chosen by trial and error rather than by principle.The paper's result shows how easily this trade-off can tip too far toward diversity at the expense of performance.

Questions for authors
  • The training settings state that all evaluated methods use validity-aware initialization and 10% replay. Does this include R-Zero, OCNR, and R-Diverse? If so, how were these methods adapted to use the same initialization? If not, how do you ensure that R-Quest’s gains are not simply due to a stronger initialization?

  • GPT-5.6-Sol is used for annotation, repair, answer verification, and clustering. Which parts of the training loop require GPT-5.6-Sol, and which are only for analysis? Can the validity-aware initialization dataset be reproduced without GPT-5.6-Sol or a comparably strong model?

  • The novelty gate uses a frozen base model, while the analysis uses GPT-5.6-Sol for clustering. What is the agreement rate, false-positive rate, and false-negative rate of the frozen base model’s SAME_TYPE judgments relative to GPT-5.6-Sol or human labels? How sensitive is performance to this judge’s errors?

  • The validity gate relies on the solver outputting INVALID. Could this penalize difficult but valid questions, or encourage excessive rejection?

Reviewer_qxGsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

R-Quest investigates why repeated questioner–solver self-training deteriorates, identifying invalid questions and repeated mathematical exercise patterns as failure modes. It initializes the solver to recognize invalid questions, then uses its judgments in questioner rewards and solver-data filtering, with labeled replay to retain this capability. A frozen base model supplies novelty feedback through sampled question-pair comparisons. The paper evaluates twelve benchmarks across two backbones and reports a ten-round Qwen experiment in which R-Quest preserves its gains while R-Zero deteriorates.

Critical strengths

The repair experiment supports the proposed validity diagnosis. Figure 3(b) compares two datasets of 1,273 questions containing the same 766 valid questions; replacing the other 507 invalid questions with repaired versions increases the average across seven mathematical benchmarks from 46.54 to 49.91. Keeping the dataset size and valid portion fixed makes this an informative intervention linking question quality to subsequent solver performance.

Figure 5 directly examines late collapse over ten rounds. R-Quest finishes at 51.92 while R-Zero ends at 34.60; the variant without validity ends at 37.84, whereas the variant without novelty retains 49.22. These trajectories and the supervision-quality diagnostics in Figure 6 support the central stability claim. Tables 1 and 2 additionally show the highest domain averages on both backbones, including transfer from mathematical training to code generation and general reasoning.

Critical weaknesses

The reliability of the stability and component conclusions is not quantified across independent training runs. Figure 5 shows a large late-round separation, but seed counts and uncertainty across training runs are not reported. Smaller differences particularly need calibration: Table 1’s mathematical margins over R-Diverse are 1.82 and 2.08 points, while Table 3’s mathematical gain from novelty is 0.62. Figure 6(c)’s Wilson intervals concern sampled question validity. Please report seed counts and uncertainty across runs for the trajectories and method differences, with independent replications if these are single runs.

The validity-removal controls need explicit definitions before their results can establish which intervention prevents collapse. Validity enters through initialization, labeled replay, questioner rewards, and solver-data filtering, but Tables 3 and 9 do not specify which are retained in the validity-removal variant. Section 4.3.1 also leaves the frozen validity judge’s model/checkpoint unspecified. Please identify the components held fixed in each variant and the frozen judge, clarifying whether these comparisons isolate validity feedback while holding the additional labeled training constant.

Questions for authors

Does “w/o validity” retain the same initialization, 10% labeled replay, and novelty mechanism while disabling only validity-based questioner rewards and solver-data filtering?

Could you confirm which Table 8 settings apply across the trained Table 1 methods, identify exceptions, and provide the common benchmark prompt templates, token limits, AMC/AIME sampling temperature/top-p/top-k, and Qwen3-32B judging prompt and settings?

For each method and round, what fraction of generated questions survives filtering, and are those questions reused or resampled to complete the stated solver updates?

Evaluating the same saved base, reproduced R-Diverse, and R-Quest checkpoints under both evaluation protocols would be a useful optional check of evaluation sensitivity. The original R-Diverse paper uses GPT-4o rather than Qwen3-32B and reports a different base score. This requires no retraining in the original framework; documenting the known protocol differences would also clarify the comparison’s scope.

An optional mechanism comparison would use an ordinary solver without validity-aware initialization or validity replay, with a fixed stronger external model supplying only validity judgments for questioner rewards and solver-data filtering. Keeping novelty and the solver-update schedule fixed, retaining solver-generated answer pseudo-labels, and reporting verifier cost would help assess the tradeoff involved in internalizing validity assessment. VHG, already cited in Section 5, is a related precedent.

1 comment

I thank the authors for the thorough rebuttal and the extra compute put into addressing my concerns.

⚬ Multi-seed stability: The 3-seed runs (Figure 9, Table 17) confirm the stability gains and margins over baselines are solid and well beyond run-to-run noise. Though it would be nice to see per-benchmark score+spread.

⚬ Validity mechanism: The new ablation (R-Zero + Init. + Replay dropping to 27.90% in Table 13) is convincing. It shows that replay alone doesn't fix things; the online validity gate is what actually stops the questioner from exploiting bad questions.

⚬ Internalization vs. external verifier: Table 18 neatly proves that internalizing validity matches a strong external judge while replacing a recurring ~$1,400 API cost with a one-time ~$200 setup.

The paper is stronger now empirically. Score raised.

Reviewer_qDoUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why reasoning models can lose performance over successive rounds of generating and solving their own training problems. The authors identify two failure modes in R-Zero. Filtering questions by answer consistency can increase the proportion of invalid questions, and lexical similarity can miss mathematically equivalent problems with different wording. Their method, R-Quest, trains the solver to recognize invalid questions and uses its judgments to reward the question generator and filter the solver’s training data. A frozen base model also compares sampled question pairs to discourage repeated problem types. On Qwen3-4B-Base, R-Quest reaches an average score of 51.92 across seven math benchmarks in round ten, 5.75 points above the initial model and 17.32 points above R-Zero at the same round.

Critical strengths

The authors hold 766 valid questions fixed and replace the remaining 507 invalid questions with repaired versions. Training on the repaired dataset improves the average math score by 3.37 points. This controlled comparison strengthens the claim that invalid questions contribute to performance degradation.

By round five, the five most frequent problem types account for 69% of sampled R-Zero questions, compared with 24% for R-Quest. Together with the ablation that removes novelty feedback, this analysis links the proposed mechanism to both the distribution of generated problems and downstream performance.

Critical weaknesses

R-Quest first trains its solver using validity labels and reference answers from GPT-5.6-Sol, whereas R-Zero does not receive an equivalent warm start. A more informative comparison would initialize both methods with the same pretrained validity-aware solver. This would show how much additional benefit comes from using its judgments to reward question generation and filter subsequent training data.

The second backbone is evaluated for only five rounds. Moreover, of the 17.32-point gap at round ten, 11.57 points reflect R-Zero falling below its initial score, while 5.75 points reflect R-Quest improving above its own starting point. The results therefore support the claim that R-Quest mitigates late-stage collapse more strongly than the claim that additional rounds reliably produce further gains. Reporting the gain from round five to round ten across independent runs and on the second backbone would clarify that distinction.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.