- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether failures observed during reinforcement learning with verifiable rewards (RLVR) can guide data selection for continued pretraining (CPT). The authors identify n-grams associated with low success rates in RL rollouts, use them as queries to retrieve relevant text from large-scale pretraining corpora, and perform targeted CPT before resuming RL with the same training setup. The experiments cover OLMo2 and Qwen3 models on mathematical and general reasoning tasks. Targeted CPT generally outperforms matched CPT controls based on content-neutral retrieval queries, using a relatively small data budget of 35M tokens. The paper also examines failure n-gram statistics, CPT insertion points, token budgets, and cross-model transfer.
Critical strengths
-
A simple approach to using RL failures for data selection. The paper proposes a practical way to use failure signals from RL rollouts to guide CPT data selection. A key strength is that the method uses information already available during RL training, without requiring additional data attribution models or manually annotated reasoning behaviors. The resulting pipeline is relatively simple and could be incorporated into existing RLVR training workflows.
-
Broad empirical evaluation with informative ablations. The paper evaluates multiple model families, reasoning tasks, and training stages. The comparisons with matched CPT controls and the analyses of failure n-grams, CPT insertion points, token budgets, and cross-model transfer help characterize the method's behavior. In particular, the finding that a relatively small amount of targeted CPT data can match or outperform larger CPT budgets suggests that data selection can be an important factor.
Critical weaknesses
-
Limited direct evidence for the proposed failure-repair mechanism. The paper motivates targeted CPT as a way to address reasoning failures that RL alone may struggle to overcome. However, the current experiments provide limited direct evidence that the observed downstream improvements arise from repairing these particular failures. For example, Table 14 shows that in the OLMo2-1B GSM8K setting, the success rate among rollouts containing the selected failure n-grams increases from 25.7% without CPT to 28.7% with matched CPT and 29.2% with targeted CPT. The gap between targeted CPT and the matched CPT control is only 0.5 percentage points on this metric. Moreover, in some settings, including Qwen3-0.6B on GSM8K and Qwen3-4B on the multi-task mixture, this conditional success rate decreases after targeted CPT, even though final evaluation accuracy improves. These observations do not invalidate the method. In particular, the same n-gram can appear in different reasoning contexts, and the distribution of trajectories containing a given n-gram can change across policies. However, this also means that the reported conditional statistics do not directly measure whether CPT improves the model's ability to recover from the original failure states. A more direct evaluation would examine continuation success from fixed reasoning prefixes associated with the identified failures. It would also be useful to investigate whether targeted CPT increases the availability of informative rollout groups during subsequent RL. Without such evidence, it remains difficult to distinguish the proposed failure-repair mechanism from more general changes in the model's generation distribution or improvements due to domain-relevant pretraining.
-
Modest gains with limited statistical validation and unclear advantages over alternative data-selection strategies. Although targeted CPT outperforms the matched retrieval controls in most reported settings, its improvements over the no-CPT baseline are often relatively small and are not consistently positive. For example, in Table 4, the OLMo2-1B GSM8K result with CPT inserted before mid-training improves from 37.4% to 38.2%, whereas the released-base multi-task result decreases from 52.6% to 51.5%. As discussed in Appendices A and D, most results rely on a single RL seed, and some reported improvements are comparable to the observed run-to-run variation. This makes it difficult to assess how consistently the gains would hold across independent runs. The concern is particularly relevant for larger models and the General Reasoner setting, where some comparisons are made at matched training steps rather than at convergence. Another question is how much of the improvement comes specifically from using RL failure signals, rather than simply retrieving task-relevant or domain-relevant data. Table 13 shows that 35M tokens of curated mathematical data (MegaMath-Web-Pro) achieve 60.7% accuracy, compared with 60.6% for targeted CPT in the OLMo2-1B GSM8K setting. While this comparison does not negate the potential benefit of an automated, model-specific selection method, it suggests that high-quality domain-specific CPT can achieve similar performance in at least this setting. The matched content-neutral retrieval controls provide a useful starting point, but additional comparisons with queries derived from task prompts or successful RL rollouts would help isolate the contribution of failure-specific information. Such comparisons would be particularly informative if the retrieval budgets and relevant query statistics were matched.
Questions for authors
Q1. Can the authors provide more direct evidence that targeted CPT improves recovery from the identified failures? Would it be possible to evaluate the original and CPT-modified policies on identical, held-out reasoning prefixes associated with the selected failure n-grams? Comparing continuation success before and after CPT, ideally also against the matched CPT control and after subsequent RL, would help separate improvements in local recoverability from changes in the distribution of states visited by the policy. Relatedly, does targeted CPT increase the fraction of RL rollout groups containing both successful and unsuccessful responses, especially for prompts associated with the identified failures? This would help test whether targeted CPT increases the availability of non-degenerate learning signals during subsequent RL.
Q2. How much of the benefit comes specifically from failure-guided selection, and how robust is this benefit? The comparison with MegaMath-Web-Pro in Table 13 suggests that high-quality domain-specific data may yield similar improvements. Could the authors clarify when failure-guided retrieval offers an advantage over domain-matched CPT, particularly when accounting for variability across RL seeds? Additionally, have the authors considered using task-prompt n-grams or n-grams from successful rollouts as retrieval queries, under comparable data budgets? These controls would help determine whether the gains are specific to failure signals or primarily arise from retrieving relevant training data.
2 comments
Thank you for running the additional experiments. The held-out prefix results provide little evidence of localized repair, and task-prompt retrieval performs comparably in both stage-1 settings. The gains over no CPT also disappear with longer RL in the two settings tested. Therefore, I will maintain my score.
Rebuttal
Thank you for the constructive review. We ran the analyses you proposed and report them as they came out.
Q1a. Recovery from the identified failures. We agree that phrase-conditioned rates do not isolate recoverability, and the single-seed Table 14 overstated the difference: pooled over three stage-1 RL seeds, a(G) on the 66 five-grams is 24.3 (no CPT), 28.3 (control), 28.5 (ours), so both CPT corpora raise it equally; ours differs in a higher entry rate (7.6% vs 5.2%). The markers are high-precision, low-recall (76% of marked rollouts fail vs a 58% base rate; 9% of failures covered). We have removed the claim of localized repair; what remains is that failure n-grams select a corpus that beats a frequency-matched control at the paper's RL budget, by a mechanism that is open. The evaluation you propose, continuation success from identical held-out prefixes before and after RL, is running for seven settings and will follow in a separate comment. The a(G) decreases you note (Qwen3-0.6B GSM8K, Qwen3-4B multi-task) concern states seen in only 0.5-1.5% of rollouts, so those rates are noisy; we no longer rest any claim on a(G).
Q1b. Informative rollout groups. Over steps 150-210 (3 seeds, 94k groups per condition), the share of GRPO groups with mixed rewards is 38.7% (no CPT), 37.3% (control), 38.3% (ours); among groups with a wrong rollout 50.1 / 48.7 / 50.5%; among groups with a marked rollout 38.1 / 38.6 / 39.2%. Ours is slightly above the control on all three (+1.0, +1.8, +0.6) and on par with no CPT, but the differences are too small to explain the accuracy gain on their own.
Q2. Failure-guided vs domain data. The matched control is already label-free retrieval from the same index ("1", "first", "step 1"; the paper's "content-neutral queries" is fixed). Against it, ours wins in all 7 settings with per-question outputs, 3 individually significant (stage-1 GSM8K p = 0.006, Qwen3-0.6B GSM8K p = 0.025, Qwen3-4B multi-task p < 1e-4), and pooled across settings Stouffer's z = 5.2 (general response, point 4). At stage 1 we added both controls you suggest. MegaMath-Web-Pro (35M, 3 seeds) reaches 36.7 +- 1.7, below ours (38.2 +- 1.5; p = 0.047) and no better than no CPT. The task-prompt baseline (eight frequent GSM8K-prompt phrases, 3 seeds) reaches 38.7 +- 1.5, tied with ours (p = 0.51). Its query list is disjoint from ours, and the documents it retrieves do not overlap with ours (reflection phrases vs question phrasing; no shared document; see general response, point 3). So two disjoint corpora, one selected by the model's failures and one by the task prompts, give the same gain on stage-1 GSM8K. Failure n-grams need no task prompts and, in the General Reasoner setting, name specific missing operations that no prompt list produces; whether that matters is untested. Task-prompt controls for released-base GSM8K and the mixture, and their combination with ours, are queued; we will add what finishes and report the rest in the final version, whichever way they come out. At the released base, MegaMath matches us (60.7 vs 60.7) and single-phrase corpora land at 59.1-60.7, so that setting does not separate data choices.
Statistical validation and retention. Paired tests for every Table 2 cell are in the general response; averaging the last three checkpoints moves no number by more than 0.6. A longer-RL comparison for every condition will follow in a separate comment. Log-likelihood benchmarks are unchanged across 35M-1B CPT tokens (reply to ADJf); the large-budget drop is format drift, not forgetting, and is the same for random data.