acceptodds
Under review as a conference paper at ICLR 2027

Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning

Abstract

As Reinforcement Learning with Verifiable Rewards (RLVR) substantially improves the reasoning abilities of large language models (LLMs), a new bottleneck emerges: more training problems become saturated, that is, the LLM answers the questions correctly for nearly every rollout. On such problems, rewards provide little useful learning signal. While collecting harder problems is a natural response, it is costly and increasingly difficult. We propose failure-prefix conditioning, a simple method that unlocks the remaining signal in saturated problems by shifting exploration toward failure-prone reasoning states. By conditioning on prefixes of rare incorrect trajectories, the method improves the model’s ability to recover from misleading early reasoning. We observe that failure-prefix conditioning consistently improves performance where standard RLVR stalls, and achieves gains comparable to training on newly collected medium-difficulty problems. We further analyze the model's robustness, finding that our method reduces performance degradation under misleading failure prefixes, albeit with a mild trade-off in adherence to correct early reasoning. Finally, we demonstrate that an iterative approach, which refreshes failure prefixes during training, unlocks additional gains after performance plateaus. Overall, our results show that saturated problems still contain valuable learning signal, and that failure-prefix conditioning provides an effective way to unlock it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RLVR, LLM, language model, reinforcement learning, GRPO, saturation, reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vNqbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the diminishing learning signal in RLVR when training problems become saturated. The authors propose Failure-Prefix Conditioning, which constructs new training prompts by appending prefixes of incorrect reasoning trajectories to saturated problems. Experiments show that the method improves performance where standard RLVR stalls, achieves gains comparable to training on newly collected medium-difficulty problems, and enhances recovery from misleading reasoning prefixes.

Critical strengths
  1. The paper identifies an important challenge in RLVR: extracting useful learning signals from saturated problems. Failure-Prefix Conditioning offers a straightforward approach that does not require modifying the RL algorithm.

  2. The method substantially outperforms standard RLVR on saturated problems and achieves results comparable to training on newly collected medium-difficulty problems. The comparisons with multiple baselines and ablation studies support its effectiveness.

Critical weaknesses
  1. Limited analysis of improvements in reasoning capability versus reliability. Failure-Prefix trains on failures from saturated problems, which may primarily improve reasoning reliability rather than the ability to solve more challenging problems. Improving reliability is itself valuable, but the paper could provide a deeper analysis distinguishing these effects, such as through pass@k, pass^k, and per-problem success rate distributions.

  2. Experiments are restricted to a 1.5B model, leaving scalability to stronger models with rarer failures unclear.

Questions for authors
  1. How does Failure-Prefix compare with the Medium baseline in terms of pass@k, pass^k, and per-problem success rate distributions? Does it primarily improve reasoning reliability or expand the range of solvable problems?

  2. Can combining Failure-Prefix with the harder problems used in the Medium baseline improve both reasoning reliability and problem-solving capability?

Reviewer_L2nLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims at RLVR-based large model training on saturated datasets, i.e., on data where models generate almost correct rollouts. To this end, it proposes a simple method called failure-prefix conditioning, whose core idea is to select parts of incorrect rollout trajectories as prefixes, inserting them into prompts to construct a new dataset for training. It also proposes to update failure prefixes iteratively to obtain additional performance gains. The experimental results on five math reasoning datasets and three other datasets of different domains (e.g., coding) validate that the proposed method improves the performance of DeepSeek-R1-Distill-Qwen-1.5B.

Critical strengths
  1. The target of this paper is practical, as current models are becoming powerful and consuming more data, fully exploiting existing data is as important as finding more hard data.

  2. The proposed method is easy to follow and implement. It simply sweeps through prefix lengths on training data, then generates a new training set, without changes in model architectures or optimization objectives.

  3. The results in Sec 5.1 show that the proposed method particularly improves the robustness of the proposed method against misleading reasoning, instead of general performance gains, which fulfills the goal of this paper.

Critical weaknesses
  1. My major concern about this paper is whether the proposed method can be applied to other models of different architectures and scales. In this paper, only a rather small model is tested (DeepSeek-R1-Distill-Qwen-1.5B), it is unclear whether larger models (e.g., 7B) still yield incorrect prefixes and benefit from the proposed method.

  2. The proposed method is mainly compared with the vanilla RLVR and variants that are trained with data of different difficulty levels. It is unclear whether the proposed method is competitive against state-of-the-art methods, especially those also solving the RLVR saturation problem, e.g., rejection sampling and difficulty‑aware sampling.

  3. The performance gains of the proposed method against the variant trained on medium-difficulty samples are marginal (Table 1, 44.5 vs. 44.0 in mask reasoning and 52.6 vs. 52.3 in other domains).

Questions for authors
  1. Have you compared the cost of peer question generation methods with that of the proposed method? Considering the performance of the proposed method, if medium-level data can be generated efficiently, the practical value of the proposed method might be constrained.

  2. Could you provide the results against other GRPO variants (e.g., DAPO, ByteDance Seed 2025) to verify the unique advantage of the proposed method?

Reviewer_KDZNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether nearly saturated problems can still provide useful learning signals for reinforcement learning with verifiable rewards. It proposes failure-prefix conditioning: collecting rare incorrect solutions, conditioning generation on prefixes of those solutions, and training on the resulting continuations. Prefix lengths are selected to target an intermediate success rate, while a simpler fixed-truncation variant achieves similar results.

Experiments with DeepSeek-R1-Distill-Qwen-1.5B on nearly saturated MATH problems show improvements across mathematical and other reasoning benchmarks over standard RLVR on the same problems or the full MATH training set. The gains are comparable to training on an equally sized set of medium-difficulty problems. Further analysis finds improved recovery from prefixes of failed solutions, accompanied by reduced benefit from prefixes of successful solutions. A second round of collecting failures and reconstructing the conditioned dataset yields modest additional gains.

Critical strengths

The main strength is the empirical demonstration that nearly saturated training problems can still support useful learning when exploration is conditioned on prefixes of failed solutions. On the same subset of MATH problems, failure-prefix conditioning achieves a mean math benchmark accuracy of 44.5%, compared with 40.7% for standard RLVR; success-prefix conditioning also provides little improvement. These comparisons support the value of targeting failure-prone continuations, with gains comparable to training on an equally sized set of medium-difficulty problems. This is consequential because it demonstrates a way to extract additional value from existing high-accuracy training data.

Critical weaknesses
  1. The practical advantage over alternative exploration strategies is not established under comparable total compute. Dataset construction requires substantial sampling: Appendix E reports 3.32 billion tokens for saturation identification and another 1.94 billion for prefix selection. The comparisons do not establish whether allocating a comparable total budget—including data preparation—to broader exploration or additional baseline training would yield similar or greater gains. The response-length controls in Appendix F do not resolve this issue, and the cheaper staged-sampling procedure changes the data-selection rule without reporting downstream performance. Given the conceptual proximity to existing prefix-conditioning approaches and the small differences relative to medium-difficulty training, comparisons with methods targeting stalled learning, such as BroRL, would help establish the distinctive value of the proposed intervention. This is central to assessing whether the method provides a meaningful practical advance.

  2. The experiments do not establish applicability to the stronger-model, highly saturated regime motivating the work. All experiments use a single 1.5B model, and training uses questions with observed failures among 128 rollouts, excluding those answered correctly in all 128 attempts. As failure probabilities decrease, collecting the required trajectories becomes increasingly expensive. Moreover, stronger models may differ in their existing ability to recover from misleading prefixes. Consequently, the results demonstrate usefulness in the tested near-saturation setting but leave unresolved whether comparable benefits persist for stronger models and more extreme saturation. This limits support for the paper’s broader motivation.

Questions for authors
  1. Compute fairness and competing approaches. What are the total data-preparation and training costs for each method? Under a comparable total budget, how does failure-prefix conditioning compare both with allocating additional rollouts or training steps to the standard baseline and with a relevant competing exploration method, such as BroRL? Also, does the cheaper staged-sampling procedure in Appendix E preserve downstream performance? Since it stops at the first failure after an initial 16/16 screen, it does not reproduce the original 121–127/128 selection criterion.

  2. Model scale, recovery ability, and model family. Could you provide evidence addressing the following distinct aspects of the stronger-model motivation?

    • Failure collection: For a substantially larger model—for example, in the 20B–100B range—how frequently can incorrect trajectories be sampled on the proposed training problems, and what sampling cost does this imply? This question concerns inference only, without requiring training at that scale.
    • Incremental training benefit: For a moderately larger model that remains feasible to train, such as a 4B or 8B model, does failure-prefix conditioning improve ordinary benchmark performance and recovery beyond the model’s existing ability to correct misleading trajectories? A limited before/after comparison would help establish whether the benefit persists as baseline recovery improves.
    • Model-family dependence: Is there evidence that these benefits extend beyond the DeepSeek-R1-Distill-Qwen model used here? Evidence from another family would help distinguish a general effect from behavior specific to this model or its distillation history.
  3. Variability and checkpoint selection. How many independent training seeds support the reported results, and how were checkpoints selected? Could you report variability across repeated runs for the key comparisons, particularly against medium-difficulty training and for the additional 0.3–0.5-point gains from iterative refreshing? The 32 evaluation samples per question do not quantify variation across training runs.

Comments on the paper

1 comment
Area_Chair_H9cVOpenReview ↗

Please engage with the authors' rebuttal

Dear reviewers,

Given that we have only a few days until the closing of the rebuttal period, please take into account the effort that the author spent on their rebuttal, and feel free to engage with them (or at least give an acknowledgement that you have read it). Feel free to ignore this message if you have done so.

Best regards,

Your AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.