acceptodds
Under review as a conference paper at ICLR 2027

Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining

Abstract

Reinforcement learning with verifiable rewards (RLVR) is widely used to improve reasoning in language models, yet growing evidence suggests that it mostly sharpens what the base model can already do. This raises a practical question: when RL fails, what is the base model missing? We look inside the RL policy's own rollouts and find that failures are localized: whether a rollout fails is strongly predicted by a small set of characteristic -grams, ranging from reflection phrases such as "let's recheck" to specific operations such as "the inverse of A". These failures persist throughout RL training, although they are not beyond the model's reach: other sampled answers to the same question are often correct. Building on this observation, we use the failure -grams to retrieve text from the pretraining corpus and continue pretraining the base model on it before RL. The resulting corpus has only 35M tokens, 3–4 orders of magnitude smaller than typical mid-training. We evaluate this approach on OLMo2 and Qwen3 models trained with RL on (1) GSM8K; (2) a mixture of math, commonsense, and knowledge questions; and (3) general math and science reasoning. Despite its size, the corpus yields consistent gains: our approach outperforms continued pretraining on the same amount of text sampled from the pretraining corpus across 10 out of 11 settings, including by points on GSM8K when inserted before mid-training. The retrieved text also transfers across model families: text retrieved for OLMo2-1B's failures improves Qwen3-4B on the mixture by points over the same baseline. Together, our results suggest that where RL fails is a cheap and effective signal for selecting pretraining data.

Primary area:
interpretability and explainable AI
Keywords:
language models, reinforcement learning, continue pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XHwoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether failures observed during reinforcement learning with verifiable rewards (RLVR) can guide data selection for continued pretraining (CPT). The authors identify n-grams associated with low success rates in RL rollouts, use them as queries to retrieve relevant text from large-scale pretraining corpora, and perform targeted CPT before resuming RL with the same training setup. The experiments cover OLMo2 and Qwen3 models on mathematical and general reasoning tasks. Targeted CPT generally outperforms matched CPT controls based on content-neutral retrieval queries, using a relatively small data budget of 35M tokens. The paper also examines failure n-gram statistics, CPT insertion points, token budgets, and cross-model transfer.

Critical strengths
  1. A simple approach to using RL failures for data selection. The paper proposes a practical way to use failure signals from RL rollouts to guide CPT data selection. A key strength is that the method uses information already available during RL training, without requiring additional data attribution models or manually annotated reasoning behaviors. The resulting pipeline is relatively simple and could be incorporated into existing RLVR training workflows.

  2. Broad empirical evaluation with informative ablations. The paper evaluates multiple model families, reasoning tasks, and training stages. The comparisons with matched CPT controls and the analyses of failure n-grams, CPT insertion points, token budgets, and cross-model transfer help characterize the method's behavior. In particular, the finding that a relatively small amount of targeted CPT data can match or outperform larger CPT budgets suggests that data selection can be an important factor.

Critical weaknesses
  1. Limited direct evidence for the proposed failure-repair mechanism. The paper motivates targeted CPT as a way to address reasoning failures that RL alone may struggle to overcome. However, the current experiments provide limited direct evidence that the observed downstream improvements arise from repairing these particular failures. For example, Table 14 shows that in the OLMo2-1B GSM8K setting, the success rate among rollouts containing the selected failure n-grams increases from 25.7% without CPT to 28.7% with matched CPT and 29.2% with targeted CPT. The gap between targeted CPT and the matched CPT control is only 0.5 percentage points on this metric. Moreover, in some settings, including Qwen3-0.6B on GSM8K and Qwen3-4B on the multi-task mixture, this conditional success rate decreases after targeted CPT, even though final evaluation accuracy improves. These observations do not invalidate the method. In particular, the same n-gram can appear in different reasoning contexts, and the distribution of trajectories containing a given n-gram can change across policies. However, this also means that the reported conditional statistics do not directly measure whether CPT improves the model's ability to recover from the original failure states. A more direct evaluation would examine continuation success from fixed reasoning prefixes associated with the identified failures. It would also be useful to investigate whether targeted CPT increases the availability of informative rollout groups during subsequent RL. Without such evidence, it remains difficult to distinguish the proposed failure-repair mechanism from more general changes in the model's generation distribution or improvements due to domain-relevant pretraining.

  2. Modest gains with limited statistical validation and unclear advantages over alternative data-selection strategies. Although targeted CPT outperforms the matched retrieval controls in most reported settings, its improvements over the no-CPT baseline are often relatively small and are not consistently positive. For example, in Table 4, the OLMo2-1B GSM8K result with CPT inserted before mid-training improves from 37.4% to 38.2%, whereas the released-base multi-task result decreases from 52.6% to 51.5%. As discussed in Appendices A and D, most results rely on a single RL seed, and some reported improvements are comparable to the observed run-to-run variation. This makes it difficult to assess how consistently the gains would hold across independent runs. The concern is particularly relevant for larger models and the General Reasoner setting, where some comparisons are made at matched training steps rather than at convergence. Another question is how much of the improvement comes specifically from using RL failure signals, rather than simply retrieving task-relevant or domain-relevant data. Table 13 shows that 35M tokens of curated mathematical data (MegaMath-Web-Pro) achieve 60.7% accuracy, compared with 60.6% for targeted CPT in the OLMo2-1B GSM8K setting. While this comparison does not negate the potential benefit of an automated, model-specific selection method, it suggests that high-quality domain-specific CPT can achieve similar performance in at least this setting. The matched content-neutral retrieval controls provide a useful starting point, but additional comparisons with queries derived from task prompts or successful RL rollouts would help isolate the contribution of failure-specific information. Such comparisons would be particularly informative if the retrieval budgets and relevant query statistics were matched.

Questions for authors

Q1. Can the authors provide more direct evidence that targeted CPT improves recovery from the identified failures? Would it be possible to evaluate the original and CPT-modified policies on identical, held-out reasoning prefixes associated with the selected failure n-grams? Comparing continuation success before and after CPT, ideally also against the matched CPT control and after subsequent RL, would help separate improvements in local recoverability from changes in the distribution of states visited by the policy. Relatedly, does targeted CPT increase the fraction of RL rollout groups containing both successful and unsuccessful responses, especially for prompts associated with the identified failures? This would help test whether targeted CPT increases the availability of non-degenerate learning signals during subsequent RL.

Q2. How much of the benefit comes specifically from failure-guided selection, and how robust is this benefit? The comparison with MegaMath-Web-Pro in Table 13 suggests that high-quality domain-specific data may yield similar improvements. Could the authors clarify when failure-guided retrieval offers an advantage over domain-matched CPT, particularly when accounting for variability across RL seeds? Additionally, have the authors considered using task-prompt n-grams or n-grams from successful rollouts as retrieval queries, under comparable data budgets? These controls would help determine whether the gains are specific to failure signals or primarily arise from retrieving relevant training data.

2 comments

Thank you for the constructive review. We ran the analyses you proposed and report them as they came out.

Q1a. Recovery from the identified failures. We agree that phrase-conditioned rates do not isolate recoverability, and the single-seed Table 14 overstated the difference: pooled over three stage-1 RL seeds, a(G) on the 66 five-grams is 24.3 (no CPT), 28.3 (control), 28.5 (ours), so both CPT corpora raise it equally; ours differs in a higher entry rate (7.6% vs 5.2%). The markers are high-precision, low-recall (76% of marked rollouts fail vs a 58% base rate; 9% of failures covered). We have removed the claim of localized repair; what remains is that failure n-grams select a corpus that beats a frequency-matched control at the paper's RL budget, by a mechanism that is open. The evaluation you propose, continuation success from identical held-out prefixes before and after RL, is running for seven settings and will follow in a separate comment. The a(G) decreases you note (Qwen3-0.6B GSM8K, Qwen3-4B multi-task) concern states seen in only 0.5-1.5% of rollouts, so those rates are noisy; we no longer rest any claim on a(G).

Q1b. Informative rollout groups. Over steps 150-210 (3 seeds, 94k groups per condition), the share of GRPO groups with mixed rewards is 38.7% (no CPT), 37.3% (control), 38.3% (ours); among groups with a wrong rollout 50.1 / 48.7 / 50.5%; among groups with a marked rollout 38.1 / 38.6 / 39.2%. Ours is slightly above the control on all three (+1.0, +1.8, +0.6) and on par with no CPT, but the differences are too small to explain the accuracy gain on their own.

Q2. Failure-guided vs domain data. The matched control is already label-free retrieval from the same index ("1", "first", "step 1"; the paper's "content-neutral queries" is fixed). Against it, ours wins in all 7 settings with per-question outputs, 3 individually significant (stage-1 GSM8K p = 0.006, Qwen3-0.6B GSM8K p = 0.025, Qwen3-4B multi-task p < 1e-4), and pooled across settings Stouffer's z = 5.2 (general response, point 4). At stage 1 we added both controls you suggest. MegaMath-Web-Pro (35M, 3 seeds) reaches 36.7 +- 1.7, below ours (38.2 +- 1.5; p = 0.047) and no better than no CPT. The task-prompt baseline (eight frequent GSM8K-prompt phrases, 3 seeds) reaches 38.7 +- 1.5, tied with ours (p = 0.51). Its query list is disjoint from ours, and the documents it retrieves do not overlap with ours (reflection phrases vs question phrasing; no shared document; see general response, point 3). So two disjoint corpora, one selected by the model's failures and one by the task prompts, give the same gain on stage-1 GSM8K. Failure n-grams need no task prompts and, in the General Reasoner setting, name specific missing operations that no prompt list produces; whether that matters is untested. Task-prompt controls for released-base GSM8K and the mixture, and their combination with ours, are queued; we will add what finishes and report the rest in the final version, whichever way they come out. At the released base, MegaMath matches us (60.7 vs 60.7) and single-phrase corpora land at 59.1-60.7, so that setting does not separate data choices.

Statistical validation and retention. Paired tests for every Table 2 cell are in the general response; averaging the last three checkpoints moves no number by more than 0.6. A longer-RL comparison for every condition will follow in a separate comment. Log-likelihood benchmarks are unchanged across 35M-1B CPT tokens (reply to ADJf); the large-budget drop is format drift, not forgetting, and is the same for random data.

Thank you for running the additional experiments. The held-out prefix results provide little evidence of localized repair, and task-prompt retrieval performs comparably in both stage-1 settings. The gains over no CPT also disappear with longer RL in the two settings tested. Therefore, I will maintain my score.

Reviewer_ADJfOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes using failures observed during reinforcement learning with verifiable rewards (RLVR) to select data for continued pretraining (CPT). It identifies n-grams associated with low final-answer accuracy in RL rollouts, retrieves text containing these phrases through exact matching, and trains the original base model on approximately 35M retrieved tokens before rerunning RL. The authors interpret these phrases as markers of reasoning states that RL struggles to improve.

Experiments cover OLMo2 and Qwen3 models on GSM8K, a multi-task mixture, and broader math and science benchmarks. The paper reports mostly modest improvements over matched-budget control CPT, some cross-model transfer, and analyses of phrase-conditioned accuracy and pass@k. Most settings use a single RL seed. These results suggest that failure-associated phrases may provide a useful data-selection signal, while the proposed interpretation as localized capability repair remains less firmly established.

Critical strengths

The paper proposes a simple, actionable way to use failure-associated patterns in RL rollouts to select continued-pretraining data. Exact-match retrieval makes the approach straightforward to implement without requiring a teacher model or synthetic demonstrations. This practical simplicity is its strongest positive aspect, although the marginal gains and limited statistical validation leave its advantage over simpler data-selection strategies uncertain.

Critical weaknesses

The reported improvements are extremely marginal. Gains over no CPT are frequently below one percentage point, most settings use a single RL seed, and the authors acknowledge that the three-seed GSM8K improvement before mid-training (+0.8 points) is within noise. A curated math-data CPT baseline also matches the method in one setting. Without stronger statistical validation and a compute-matched comparison against additional RL training, the experiments do not convincingly establish a reliable benefit from the proposed pipeline.

The method’s scalability and computational benefit remain unestablished. Increasing the CPT budget from 35M to 1B tokens generally yields flat or worse performance in Table 5. This may reflect forgetting or optimization effects from continued pretraining on a narrow distribution, rather than demonstrating that a small corpus sufficiently repairs the diagnosed deficits. The pipeline also requires an initial RL run to obtain rollouts, followed by mining, retrieval, CPT, and another RL run. Without a comparison against continued RL under the same total compute budget—or clear accounting of which diagnostic costs can be reused—the paper does not establish that this additional pipeline is preferable to spending the resources on further RL training.

Questions for authors

Reliability of the gains: Do you have additional seed results or uncertainty estimates showing that the improvements over no CPT are reproducible, particularly beyond the OLMo2-1B GSM8K setting? The reported gains are often small relative to the variability discussed in Appendix D.

Compute-matched comparison: Is there evidence that the proposed pipeline outperforms continuing RL under an equivalent total compute budget? Please clarify the cost of obtaining diagnostic rollouts, mining and retrieval, CPT, and rerunning RL, distinguishing reusable costs from those incurred for each new setting. Were the RL baselines trained to convergence?

Scaling and capability retention: How do you distinguish diminishing benefits from the selected data from forgetting or optimization effects as the CPT budget increases? Do you have capability-retention measurements or controlled results explaining the degradation in Table 5? This distinction matters for assessing whether the method offers a scalable training strategy.

3 comments
AuthorsOpenReview ↗

Rebuttal 1/2

Thank you for the review. We address reliability, compute and scaling in turn, and we start by conceding the point you quote: +0.8 over no CPT on stage-1 GSM8K is within seed noise, as Appendix D says.

Reliability of the gains. All conditions are evaluated on the same test questions, so we ran paired exact McNemar tests on per-question outcomes (last RL step; the mean of the last three checkpoints changes nothing by more than 0.6).

Settingno CPTmatched controloursp vs controlp vs none
GSM8K / OLMo2-1B after stage 1 (3 seeds)37.4 +- 1.136.2 +- 0.838.2 +- 1.50.0060.32
multi-task / OLMo2-1B after stage 143.245.345.80.29< 1e-4
GSM8K / OLMo2-1B released base (3 seeds)59.3 +- 0.760.0 +- 1.060.6 +- 0.90.430.08
GSM8K / Qwen3-0.6B74.573.075.40.0250.37
GSM8K / Qwen3-4B91.991.792.30.330.58
multi-task / Qwen3-0.6B52.452.252.80.220.59
multi-task / Qwen3-4B72.972.073.9< 1e-40.003

Ours is above the matched control in 7 of 7 settings (two-sided sign test p = 0.016; the settings share models and corpora, so this is indicative, not independent evidence) and above no CPT in 7 of 7, with 3 and 2 comparisons individually significant. The two largest-sample settings (multi-task, 5,804 test items) are significant at p <= 0.003 against no CPT. Pooling the paired evidence answers your reliability question more directly than any single cell: on the questions where ours and the matched control disagree, ours is right 54% of the time and the control 46%, in the same direction in all seven settings (Stouffer's z = 5.2, p ≈ 2e-7; still p = 4e-4 without the strongest setting), i.e. about one extra correct answer per 100 test questions. Against no CPT the split is 54/46 as well (z = 5.2). This is an average effect across settings that share models and corpora, so we read it as evidence that the gain is real, not that it holds in every setting. We do not have more RL seeds for the Qwen3 settings within the discussion period; the single-seed cells are labeled as such. The General Reasoner rows are matched-step comparisons; one control entry was mis-typed and one row (Qwen3-4B) is removed from the headline table because its advantage reverses at later steps (details in the reply to sqKa).

AuthorsOpenReview ↗

Rebuttal 2/2

Compute-matched comparison. Measured on H100s:

StageCostReused?
Diagnostic RL runGSM8K: 2.6 h x 4 GPUs = 10 GPU-h; multi-task: 47 GPU-hit is the standard RL run itself
Miningminutes of CPU on saved rolloutsper setting
Retrieval8 min - 4.4 h of 8 CPUs per query set (local index)index reused everywhere
CPT, 35M tokens0.4 GPU-h (< 5% of one GSM8K RL run); 1B tokens: 9.5 GPU-hcorpus reused across model families (OLMo2-1B -> Qwen3)
Second RL runsame as the diagnostic run-

So the pipeline costs one extra RL run plus a negligible CPT. The comparison you ask for, no-CPT RL continued to twice the steps against ours at the paper's budget, and all three conditions trained to the same longer length (3 seeds each), will be reported in a follow-up comment before the discussion closes, together with the same test on the multi-task setting. On convergence: training-rollout accuracy on GSM8K rises from 33% at step 100 to 36% at step 210, so the 210-step recipe was still improving slowly; we will state the convergence status explicitly with those results. Two points on cost independent of that comparison: the diagnostic run is the standard RL run itself, and one mined corpus transfers across model families (OLMo2-1B -> Qwen3-0.6B/4B), so in a multi-model setting the extra cost is one 0.4 GPU-hour CPT per model. The General Reasoner runs cover 1 of the recipe's 30 epochs and are labeled as matched-step comparisons.

Scaling and capability retention. We measured zero-shot HellaSwag, ARC-easy/challenge, PIQA, WinoGrande and MMLU for every CPT checkpoint before RL. For OLMo2-1B the six-task mean is 60.4 (base), 60.4 / 60.5 / 60.0 (ours 35M / 100M / 1B), 60.5 / 60.4 (random 35M / 1B); for Qwen3-4B 70.2 (base), 71.1 / 71.2 / 71.4 (ours), 71.8 (random 1B); MMLU moves by at most 1 point anywhere. Nothing is forgotten in the log-likelihood sense. What changes with budget is generation behavior: the share of GSM8K generations that end without a final answer is 19% for the Qwen3-4B base, 4% after 35M targeted tokens, 8% after 100M, and 66% after 1B (51% after 1B random tokens); MATH-500 shows the same pattern (5% -> 20%). This explains the Table 5 degradation as format drift of a base model under long CPT with a fresh optimizer, equally for random data (Table 10: random 1B lowers the eight-task mean from 37.5 to 28.5). It also means part of the pre-RL gain at 35M comes from better answer formatting, which we now state. We present the method as a small intervention of about 35M tokens, not as a scalable replacement for mid-training, and we have revised the conclusion accordingly.

Thank you for the additional experiments and clarifications. The marginal improvements and limited evidence of scalability remain my main concerns, as increasing the CPT budget does not yield sustained gains. I therefore maintain my score.

Reviewer_sqKaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Targeted continued pretraining uses low-success-rate phrases in reinforcement-learning trajectories as exact-match queries to retrieve approximately 35 million tokens from large text corpora. The retrieved text is used to continue pretraining the original base checkpoint before rerunning RL. Experiments with OLMo and Qwen models examine different insertion points, tasks, data budgets, and transfer of the retrieved corpus between models. The results suggest that this is a useful data-selection heuristic, but the evidence does not yet establish that failure-specific information is responsible for the gains or that the identified states are being directly repaired.

Critical strengths
  1. The feedback-to-data pipeline is concrete and economical in its data requirements. Existing RL trajectories provide the query signal, and exact-match retrieval constructs a small CPT corpus without requiring a separate large-scale data-generation procedure. The published query lists make the selection process inspectable.
  2. The evaluation explores meaningful design choices. The paper includes no-CPT and CPT controls, different insertion points and token budgets, replay, high-accuracy queries, curated mathematics data, and corpus transfer across model families. These experiments provide useful context for where targeted CPT helps or fails.
  3. The negative results are valuable. The paper reports detrimental effects of larger CPT budgets, cases where replay does not resolve them, and a setting where MegaMath matches the proposed selection method. These observations discourage an overly simple conclusion that more retrieved data is always beneficial.
  4. The state-level analysis exposes the mechanism to scrutiny. The paper reports phrase-level and phrase-set-level behavior rather than only final task scores.
Critical weaknesses

W1. The correspondence between query sets and reported results needs clarification.** The stage-1 experiment is associated with both four manually selected queries and a sixteen-query set, the General Reasoner set-level accuracy appears to be a phrase average, and the scope of the 10/11 headline comparison is unclear. Please provide a single mapping from each result to its query set, corpus, checkpoint, budget, and evaluation split, and correct the affected labels.

W2. The repair analysis compares changing prefix populations. Phrase-conditioned success is measured over prefixes that can change after CPT, confounding recovery with context changes; the reported rates are 29.2% versus 28.7% across 66 phrases and 8.6% versus 13.1% for the ambiguously labeled sixteen-query group. After clarifying the query labels, please compare the policies on identical held-out marked prefixes and report how much of the failure set those markers cover.

W3. The contribution of failure-based selection could be separated more clearly from domain and reflective content. The headline queries include generic reflective expressions, while MegaMath-Web-Pro [1] already matches the proposed method in the released-base GSM8K setting (60.7 versus 60.6). Please test a mathematics-corpus baseline and retrieval without failure labels in the headline stage-1 setting under matched token counts and CPT/RL settings, using the same source corpus for the retrieval controls.

W4. Headline gains and insertion-point comparisons would benefit from consistent reference points. The stage-1 GSM8K gain is 2.0 points over the CPT control but only 0.8 over no CPT, and the released-base multi-task result falls from 52.6 without CPT to 51.5 with the method. Please report both references consistently and account for the query, corpus, optimizer, and schedule changes accompanying the insertion-point comparison.

W5. The descriptions of supervision and automation could be more precise. Failure mining uses verifier rewards, and the headline query set includes manual selection.

Questions for authors

The response would be easier to assess with a single experiment-to-query mapping as its starting point. I suggest using that mapping to address W1–W4, connecting the reported gains to the failure-selection and repair analyses. I will increase my score if the reconciled results and matched controls support those interpretations.

  1. What does validation-mined mean in each experiment, and how are these examples and retrieved documents separated from the final evaluation sets?
2 comments
AuthorsOpenReview ↗

Rebuttal 1/2

Thank you for the precise reading; you were right that several labels were inconsistent. As you suggested we start from one mapping and address W1-W5 from it.

W1. Table R1: every result, its query set and its run. All corpora 35M tokens unless noted. "Frequency control" = retrieval on the three most frequent n-grams of the model's own rollouts ("1", "first", "step 1") from the same index with the same pipeline (the paper called this "content-neutral queries"; we now describe it precisely).

ResultModel / taskInsertionQueries#SelectionIndexRL stepSeeds
Tab. 2 r1OLMo2-1B / GSM8Kafter stage 1reflection set4hand-picked from mined candidatesOlmo-mix (API)2103
Tab. 2 r2OLMo2-1B / multi-taskafter stage 1same CPT model as r14hand-pickedOlmo-mix (API)5951
Tab. 2 r3OLMo2-1B / GSM8Kreleased baseself-mined n=516automaticDolmino+OLMoE (local)2103
Tab. 2 r4-5Qwen3-0.6B, 4B / GSM8Kreleased baser1 set (transfer)4hand-pickedOlmo-mix (API)2101
Tab. 2 r6Qwen3-0.6B / multi-taskreleased baseself-mined n=420automaticDolmino+OLMoE5951
Tab. 2 r7Qwen3-4B / multi-taskreleased baser1 set (transfer)4hand-pickedOlmo-mix (API)5951
Tab. 2 r8Qwen3-1.7B / Gen. Reasonerreleased baseQwen3-4B-mined n=3 (transfer)9automaticDolmino+OLMoE226 / 1001
Tab. 2 r9Qwen3-4B / Gen. Reasonerreleased baser1 set (transfer)4hand-pickedOlmo-mix (API)351
Tab. 5 GR row, Tab. 10Qwen3-4B before RLreleased baseself-mined n=3, 35M/100M/1B9automaticDolmino+OLMoE01
Tab. 3/14/15, Fig. 3OLMo2-1B / GSM8Kafter stage 166 mined five-grams (never retrieved)66automatic-150-2101 and 3

Corrections this produced: (a) Table 15's caption said "16 queries" for the stage-1 corpus; it has 4, and the 16 are the released-base set. (b) Tables 3, 14 and 15 and Fig. 3 use the 66 five-grams, not the retrieval queries. (c) Table 3's General Reasoner row reported mean a(g) = 16.9 -> 27.7; set-level a(G) is 23.6 -> 27.3 at step 100. (d) Table 7's last row attributed the budget runs of Table 5 to the validation-mined n=5 set; they use the self-mined n=3 set. (e) Two General Reasoner controls in Table 2 were mis-entered: the Qwen3-1.7B control run ends at step 100 and scores 29.8 (none 34.1 and ours 35.0 at step 100; 31.7 and 34.3 at step 226), and the Qwen3-4B entry 43.9 was our own run at step 85, not a control; the random-document control for that model scores 45.0 at step 70, and the no-CPT run overtakes ours at later steps (47.8 vs 43.9 at step 85). We move the Qwen3-4B General Reasoner row out of Table 2 and report all evaluated steps in the appendix; the "10 of 11" statement becomes "9 of 11" and is defined as: settings at 35M tokens, ours vs the matched control, Tables 2 and 8.

W2. Changing prefix populations. We agree that phrase-conditioned success rates mix recovery with changes in which prefixes each policy visits, and the single-seed numbers in Table 14 overstated the difference between conditions: pooled over the three stage-1 RL seeds, a(G) on the 66 five-grams is 24.3 (no CPT), 28.3 (matched control), and 28.5 (ours), i.e. both CPT corpora raise it equally, while ours raises the entry rate (6.8% -> 7.6%) and the control lowers it (-> 5.2%). On coverage: the 66 five-grams mark 6.8% of rollouts, cover 9.0% of failing rollouts, and 76% of marked rollouts fail (base failure rate 58%); the four retrieval queries mark 1.4%, cover 2.0%, precision 83%. We now call them high-precision, low-recall markers of low-value states, we no longer attribute the accuracy gain to recovery from the diagnosed states, and we state that which states the +2.0 gain over the control comes from is open. The comparison you propose, continuation success of every policy from identical held-out prefixes that end at a failure n-gram, is running; we will report it in a follow-up comment as soon as it finishes.

AuthorsOpenReview ↗

Rebuttal 2/2

W3. Failure selection vs domain and reflective content. The matched control is label-free retrieval from the same index with the same budget (Table R1), and ours beats it by 2.0 +- 0.9 at stage 1 (paired McNemar over 3 seeds, p = 0.006). In the same stage-1 setting, MegaMath-Web-Pro at 35M tokens reaches 36.7 +- 1.7 (3 RL seeds: 36.9 / 38.3 / 35.0), below ours (38.2 +- 1.5; paired McNemar pooled p = 0.047) and no different from no CPT (37.4 +- 1.1, p = 0.35). We also ran the task-prompt baseline suggested by Reviewer XHwo: retrieval on eight frequent GSM8K-prompt phrases from a local Dolmino-mix index (35M tokens, 3 RL seeds). Its query list is disjoint from ours, and the documents it retrieves do not overlap with ours. Ours are reflection phrases mined from failing rollouts ("let's correct", "still incorrect", "solve it correctly", "wait"); the baseline's are question phrasing ("How much money does", "twice as many", ...). Across the two retrieved corpora, no document appears in both, 0.2% of our documents contain any task-prompt phrase, and 0.9% of the baseline's documents contain a reflection query. It reaches 38.7 +- 1.5 (37.5 / 40.4 / 38.2), statistically tied with ours (paired p = 0.51) and above the frequency control (p = 0.0005) and MegaMath (p = 0.007). So two disjoint corpora, one selected by the model's failures and one by the task prompts, give the same gain on stage-1 GSM8K, while curated math data does not; combining the two is an obvious next step, which we have queued. Where we would expect a difference is where the failure n-grams name missing operations rather than task phrasing (the General Reasoner setting) and on the task mixture; those controls are queued (released-base GSM8K on two models, multi-task on two models) but may not finish before the discussion closes, and the General Reasoner one (two days per RL run) cannot. We will add whatever finishes and report the rest in the final version, whichever way they come out. (Caveat: this control was retrieved from the local Dolmino-mix index, ours from the public Olmo-mix index.) At the released base we agree MegaMath matches us (60.7 vs 60.7); single-phrase corpora ("wait", "step 1", "first", "not possible") score 59.1-60.7 there as well, so that setting does not separate data choices, and our claim there is only parity with curated data without labels.

W4. Reference points. Every table now carries both references. On stage-1 GSM8K, +0.8 over no CPT is within seed noise (p = 0.32); +2.0 over the control is not (p = 0.006) at the paper's 210-step budget. The released-base multi-task drop (52.6 -> 51.5) is in the main text. For the insertion comparison we state that the two insertion points necessarily differ in optimizer state and schedule (stage 1 continues the pretraining optimizer; the released base needs a fresh one) and call it the same procedure, not an identical pipeline.

W5. Agreed. The method uses no supervision beyond the verifier reward RL already uses; the four stage-1 queries were chosen by hand from the mined candidates, all other corpora were selected automatically after removing formatting artifacts; the General Reasoner reward is a model-based verifier.

Q: validation-mined. n-grams mined from rollouts of the RL-trained Qwen3-4B on the training splits of GSM8K, MATH and MMLU (at most 10K prompts per set, 32 rollouts each), disjoint from every test split. We also ran 13-gram decontamination of the retrieved corpora against the GSM8K test questions and solutions: 0 of 41,733 documents of the stage-1 corpus, 0 of 918,479 of its control, 0 of 50,000 MegaMath documents. The scan of the released-base retrieval corpora (tens of GB) is still running and will be reported in a follow-up comment.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response 1/2

We thank the reviewers for careful and constructive reviews. Three concerns were shared: (i) inconsistent labels between query sets and results, (ii) whether the gains come from repairing the diagnosed failure states, and (iii) whether failure-based selection beats simpler data choices and further RL. We ran the analyses and controls the reviewers asked for. Several of them changed what we can claim, and we report them as they came out. A revised PDF (changes in blue) is uploaded.

1. Reconciled bookkeeping (sqKa W1). We built a single table (Table R1 in the reply to sqKa) mapping every reported number to its query set, selection procedure, index, RL step, and seed count. Preparing it surfaced errors that we have corrected: (a) the state-level analysis set of Tables 3, 14, 15 and Fig. 3 is the 66 automatically mined five-grams, not the retrieval queries; (b) the stage-1 corpus uses 4 queries, not 16; (c) Table 3's General Reasoner column reported mean a(g) rather than a(G); (d) the matched control is not "content-neutral random queries" but retrieval on the three most frequent n-grams of the model's own rollouts ("1", "first", "step 1") from the same index, i.e., label-free retrieval with the same pipeline; (e) two General Reasoner control entries in Table 2 were wrong (details in the reply to sqKa and XHwo). No GSM8K or multi-task accuracy changes.

2. What the failure n-grams are. On the stage-1 GSM8K runs (3 RL seeds, 468k training rollouts per condition, steps 150-210), a rollout that contains a failure n-gram fails 72-91% of the time against a 58% base failure rate, but the n-grams cover only 9% of failing rollouts (2% for the four retrieval queries). They are high-precision, low-recall markers of low-value states, and we now describe them that way. Pooled over the three seeds, the success rate from the marked states rises by the same amount with our corpus and with the matched control (24.3 -> 28.5 vs 24.3 -> 28.3), so the single-seed Table 1 comparison overstated the difference; the revision reports the pooled numbers and no longer attributes the accuracy gain to the diagnosed states.

3. Is failure-based selection the active ingredient? (sqKa W3, XHwo Q2). The matched control already is label-free retrieval from the same index (point 1d). In the headline stage-1 setting (35M tokens, 3 RL seeds, same CPT and RL recipe) we added two further controls. MegaMath-Web-Pro at 35M tokens reaches 36.7 +- 1.7 (3 RL seeds: 36.9 / 38.3 / 35.0), below ours (38.2 +- 1.5; paired McNemar pooled p = 0.047) and no different from no CPT (37.4 +- 1.1, p = 0.35): at stage 1, curated in-domain data does not reproduce the effect of failure-guided retrieval. The second control is the task-prompt baseline suggested by Reviewer XHwo: retrieval on eight frequent GSM8K-prompt phrases (35M tokens, 3 RL seeds). Its query list is disjoint from ours, and the documents it retrieves do not overlap with ours. Ours are reflection phrases mined from failing rollouts ("let's correct", "still incorrect", "solve it correctly", "wait"); the baseline's are question phrasing ("How much money does", "twice as many", ...). Across the two retrieved corpora, no document appears in both, 0.2% of our documents contain any task-prompt phrase, and 0.9% of the baseline's documents contain a reflection query. It reaches 38.7 +- 1.5 (37.5 / 40.4 / 38.2), statistically tied with ours (paired p = 0.51) and above the frequency control (p = 0.0005) and MegaMath (p = 0.007). So two disjoint corpora, one selected by the model's failures and one by the task prompts, give the same gain on stage-1 GSM8K; combining the two is an obvious next step, which we have queued. Whether this holds where the failure n-grams name specific missing operations rather than task phrasing, as in the General Reasoner setting ("the inverse of A", "reduced row echelon form"), or on the four-task mixture, is the open question; task-prompt controls for the released-base GSM8K runs (OLMo2-1B, 3 seeds; Qwen3-0.6B) and the multi-task runs (Qwen3-0.6B; OLMo2-1B at stage 1) are queued but may not finish before the discussion closes, and the General Reasoner control (two days per RL run) cannot. We will add whatever finishes to the paper and report the rest in the final version, whichever way they come out. At the released base we agree that MegaMath matches us (60.7 vs 60.7, single seed); our claim there is only that label-free, model-specific selection matches curated in-domain data.

AuthorsOpenReview ↗

General response 2/2

4. Reliability (ADJf, XHwo). All conditions are evaluated on the same test questions, so we ran paired exact McNemar tests on per-question outcomes. Ours beats the matched control in all 7 settings that have per-question outputs (two-sided sign test p = 0.016); 3 of the 7 are individually significant (stage-1 GSM8K pooled over 3 seeds p = 0.006; Qwen3-0.6B GSM8K p = 0.025; Qwen3-4B multi-task p < 1e-4). Against no CPT, ours is higher in 7 of 7 but only 2 are significant (stage-1 multi-task p < 1e-4, Qwen3-4B multi-task p = 0.003); we state explicitly that +0.8 over no CPT on stage-1 GSM8K is within seed noise. Individually the per-setting gains are about one point, and each test set is too small to resolve that reliably on its own. Pooled, the evidence is clear: on the test questions where ours and the matched control disagree, ours is right 54% of the time and the control 46% (2,273 vs 1,952 such answers), with the same direction in all seven settings (Stouffer's z = 5.2, p ≈ 2e-7; p = 4e-4 with the strongest setting removed). That is about one extra correct answer per 100 test questions on average. The combined test shows an average effect across settings, not one in every setting, and the settings share models and corpora. The General Reasoner rows are matched-step comparisons and are relabeled as such.

5. Compute and scaling (ADJf). CPT on 35M tokens costs 0.4 GPU-hours, under 5% of one GSM8K RL run (10 GPU-hours on 4 H100s); the pipeline's real cost is the second RL run. We will report a compute-matched comparison (no-CPT RL for twice the steps, and ours and the matched control trained to the same length, 3 seeds each) in a follow-up comment before the discussion closes, together with the same test on the multi-task setting. On scaling, the 1B-token checkpoints lose nothing on log-likelihood benchmarks (HellaSwag, ARC, PIQA, WinoGrande, MMLU within +-0.5 of the base for OLMo2-1B and slightly above it for Qwen3-4B, random data included), while 66% of their GSM8K generations end without a final answer (vs 19% for the base model). The budget effect is format drift of a base model under long CPT with a fresh optimizer, not forgetting, and it is the same for random data.

6. Wording. "Without labels or a teacher" becomes "no supervision beyond the verifier reward RL already uses"; we disclose that the four stage-1 queries were chosen by hand from the mined candidates and that the General Reasoner reward is a model-based verifier.

AuthorsOpenReview ↗

Follow-up (1/2)

We thank the reviewers again. The experiments we described as running have finished, and we report them here; all are included in the revised PDF uploaded today (changes in blue; held-out prefixes in Appendix J, longer RL in Appendix K).

1. Task-prompt baseline in four more settings (XHwo Q2; sqKa W3). In our response we reported the stage-1 GSM8K tie between the task-prompt baseline and ours, from disjoint query lists and non-overlapping documents. The baseline has now been run in four more settings:

Settingno CPTmatched controltask-prompt baselineours
GSM8K / OLMo2-1B after stage 1 (3 seeds; posted)37.4 +- 1.136.2 +- 0.838.7 +- 1.538.2 +- 1.5
GSM8K / OLMo2-1B released base (3 seeds)59.3 +- 0.760.0 +- 1.058.7 +- 0.960.6 +- 0.9
GSM8K / Qwen3-0.6B74.573.074.575.4
multi-task / Qwen3-0.6B (20 mixture-prompt phrases)52.452.252.652.8
multi-task / OLMo2-1B after stage 1 (same 20 phrases)43.245.345.745.8

After stage 1 the baseline matches ours on the mixture as on GSM8K (45.7 vs 45.8, p = 0.77). On Qwen3-0.6B it lies between no CPT and ours on both GSM8K and the mixture (no difference significant). On the released base, where both corpora come from the same local index, it falls below ours (paired McNemar p = 0.009) and is no better than no CPT. So prompt-guided retrieval can match failure-guided retrieval when CPT is inserted before mid-training, but on the released base, with the source corpus held fixed, failure-guided retrieval is clearly better.

2. Decontamination of the released-base corpora (sqKa Q). 47 of 5,540,126 retrieved documents (0.001%) of our self-mined corpus and 31 of 955,198 (0.003%) of the high-accuracy control share a 13-gram with a GSM8K test item, mostly common word-problem templates ("the cost of an adult ticket is 12 and a child ticket is"). The two are exposed at a similar rate; with the stage-1 corpora (0 hits) every GSM8K corpus is now scanned.We thank the reviewers again. The experiments we described as running have finished, and we report them here; all are included in the revised PDF uploaded today (changes in blue; held-out prefixes in Appendix J, longer RL in Appendix K).

AuthorsOpenReview ↗

Follow-up (2/2)

3. held-out-prefix evaluation (sqKa W2; XHwo Q1a)

Following the suggestion of Reviewers sqKa and XHwo, we evaluated every policy on the same held-out prefixes: training rollouts of the no-CPT policy that contain a failure five-gram, cut right after it, plus unmarked rollouts cut at the same position; 16 continuations per prefix at the RL temperature, scored with each task's reward function. Continuation success after RL (no CPT / matched control / ours):

Settingmarked prefixesposition-matched unmarked
GSM8K / OLMo2-1B after stage 1 (1,000 prefixes, 3 seeds)18.2 / 19.2 / 18.432.6 / 31.8 / 33.4
GSM8K / OLMo2-1B released base (1,000, 3 seeds)10.6 / 9.5 / 10.146.1 / 45.0 / 44.8
GSM8K / Qwen3-0.6B (1,000)15.5 / 14.2 / 15.237.9 / 37.1 / 37.0
GSM8K / Qwen3-4B (150)35.3 / 40.5 / 39.885.4 / 92.5 / 92.1
multi-task / Qwen3-0.6B (312)13.4 / 14.4 / 12.360.7 / 60.2 / 60.5
multi-task / OLMo2-1B after stage 1 (749)19.8 / 17.4 / 17.627.9 / 25.6 / 25.8
multi-task / Qwen3-4B (1,000, mostly TriviaQA)10.0 / 5.4 / 11.961.6 / 53.4 / 63.1
General Reasoner / Qwen3-1.7B, step 100 (1,000)4.1 / 4.0 / 4.58.1 / 7.8 / 7.8

Paired bootstrap intervals on the marked column are about +-1 point (+-2 for the smaller sets). In seven of eight settings, ours does not recover from the marked states better than both baselines. In Qwen3-4B multi-task it does (+1.9 over no CPT, 95% CI +1.2 to +2.7; +6.5 over the control), but by the same margin as on unmarked prefixes, so it continues better from the same prefixes in general rather than specifically from the marked ones. (For strong models few five-grams meet the paper's support threshold; in five settings we used the 30 lowest-accuracy five-grams with at least 20 occurrences instead, and the Qwen3-4B multi-task set uses the 0.1% support of Table 1.) This confirms the reading in our response: the accuracy gain is not explained by local repair of the states the n-grams mark, and the n-grams are best understood as high-precision markers that select useful pretraining data; where the gain is largest (Qwen3-4B multi-task), the repaired model continues better from the same prefixes than both baselines.

4. longer RL (ADJf; XHwo)

We continued every condition of the two multi-task settings for 300 more GRPO steps from its step-595 endpoint (fresh optimizer, single seed).

no CPTmatched controlours
Qwen3-4B, step 595 -> +30072.9 -> 73.772.0 -> 71.973.9 -> 74.1
OLMo2-1B after stage 1, step 595 -> +30043.2 -> 46.645.3 -> 46.245.8 -> 46.4

On OLMo2-1B after stage 1 the three conditions converge to within half a point. On Qwen3-4B, the setting with the largest gain, the lead over no CPT shrinks to +0.4 (p = 0.30), but the lead over the matched control survives longer RL in full (74.1 vs 71.9, paired p < 1e-4).

In sum: on OLMo2-1B after stage 1 longer RL closes the gap, and we will say so in the final version rather than describe the method as a higher RL endpoint there. The advantage over the frequency-matched control holds at the paper's budgets across all seven settings (pooled z = 5.2, as reported), and where it was largest it survives 300 additional RL steps intact.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.