acceptodds
Under review as a conference paper at ICLR 2027

Task Reformulation Enables Reinforcement Learning From Hard Reasoning Problems

Abstract

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning abilities of large language models (LLMs), but it cannot learn from problems that are too difficult to solve under the current policy, since these yield no reward signal and therefore no gradient. We propose a simple solution based on task reformulation. We transform challenging open-ended problems into cognitively simpler variants (such as multiple-choice and cloze formats) that preserve the original answer while reducing the effective search space and providing denser learning signals. These variants span a spectrum from discriminative to generative tasks, so a model can first learn from the structured, easier formats and transfer that knowledge back to the original problems. Building on this insight, we introduce Cog-DRIFT, a framework that automatically constructs reformulated variants and organizes them into an adaptive curriculum, progressing from easier to harder formats as the model improves. This lets the model learn from problems that previously yielded zero signal under standard RL post-training, without distilling reasoning from any stronger external model. Cog-DRIFT improves on the originally unsolvable hard problems (+10.11% for Qwen and +8.64% for Llama in absolute gains) and also generalizes to held-out datasets. Across two models and six challenging reasoning benchmarks, it outperforms standard GRPO and strong guided-exploration baselines that supply stepwise cues or partial solutions on average, improving the average accuracy by +4.72% (Qwen) and +3.23% (Llama) over the strongest baseline in our main comparison while needing only the gold answer. We further show that Cog-DRIFT improves pass@k at test time and that the curriculum improves sample efficiency. Cog-DRIFT outperforms GRPO when GRPO is given 8 times more rollouts or 4 times longer generations. Overall, our results highlight task reformulation and curriculum learning as an effective paradigm for overcoming the exploration barrier in LLM post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, LLM post-training, Reinforcement Learning with Verifiable Reward, RLVR

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b95UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Cog-DRIFT, a framework that enables reinforcement learning from hard reasoning problems by reformulating open-ended questions into multiple-choice and cloze variants, thereby reducing task difficulty and providing more accessible learning signals. It further employs an instance-level curriculum that progressively transitions each problem from easier to harder formats based on the model's performance. The method is evaluated on two language models across six reasoning benchmarks and compared with several strong baselines. The results show consistent improvements in average accuracy and suggest that learning from reformulated problems can improve performance on both the original hard problems and held-out benchmarks.

Critical strengths
  1. The paper proposes Cog-DRIFT, which reformulates hard open-ended problems into multiple-choice and cloze variants to reduce task difficulty, thereby enabling models to learn more effectively from challenging problems. The method is simple and easy to implement, and constructing these variants requires no additional information beyond the gold answers.
  2. The paper conducts extensive experiments and ablation studies on the proposed method, evaluating two models across six benchmarks covering mathematical reasoning, scientific knowledge, and date understanding. It also compares Cog-DRIFT against strong baselines and investigates the effects of different reformulation strategies, curriculum designs, rollout budgets, and generation lengths. These experiments provide a comprehensive evaluation of the method's effectiveness and the contributions of its key components.
Critical weaknesses
  1. The core mechanism of Cog-DRIFT can be viewed as answer-related scaffolding: multiple-choice and cloze formats provide additional answer information to facilitate exploration. This is conceptually related to prior work such as STaR [1], which uses gold answers to elicit reasoning traces when ordinary generation fails. The authors should clarify the fundamental difference between their approach and such answer-guided methods.
  2. The manuscript contains many issues in its presentation and clarity. Figure 2(b) uses the same horizontal-axis categories as Figure 2(a), although they represent training formats rather than evaluation formats, which may cause confusion. Figure 3 also lacks an explicit horizontal-axis label. In Table 2, the term “number of rollouts” is ambiguous: does it refer to responses sampled per group or total rollouts per optimization step? Moreover, the paper does not clearly specify the stopping criteria or optimizer update counts for different rollout settings, making the training-budget comparison difficult to assess.
Questions for authors
  1. How does Cog-DRIFT fundamentally differ from prior answer-guided approaches such as STaR? Since multiple-choice and cloze reformulations provide candidate answers or partial answer information, can the authors isolate the benefit of this reformulation from the more general effect of providing answer-related guidance?
  2. The training set is filtered to retain problems that Qwen cannot solve within 64 rollouts but for which GPT-5.4's majority answer across three runs agrees with the ground truth. While this filtering helps remove flawed or unanswerable questions, it also selects problems that a stronger model can solve. Consequently, the reported results may primarily demonstrate effectiveness on problems that are difficult for smaller models but remain within the capability of stronger models. How well would the method generalize to problems that are also difficult for stronger models?
  3. Figure 3 shows relatively modest Pass@k improvements over the base model on AIME24 and AIME25. Since the paper aims to enable learning from problems that the original policy cannot solve, these limited gains raise questions about how much transferable reasoning capability is actually acquired. Could the authors provide further analysis? Why are the gains more limited on mathematical benchmarks?

[1] Zelikman, Eric, et al. "Star: Bootstrapping reasoning with reasoning." Advances in Neural Information Processing Systems 35 (2022): 15476-15488.

4 comments
AuthorsOpenReview ↗

Response to Reviewer b95U

We thank the reviewer for recognizing that "the method is simple and easy to implement" and that our experiments "provide a comprehensive evaluation." We address the comments below.


 

W1 & Q1: Cog-DRIFT resembles answer-guided methods such as STaR; the benefit of reformulation should be separated from providing answer information.

The submission already compares against STaR-style answer guidance (Appendix E.4). Appending the gold answer to the question reaches near-ceiling in-domain accuracy but collapses on held-out MATH500 (58.60), far below the 82.20 of our reformulation mixture under the same RFT protocol. In Table 9, adding STaR-style rationalized OEQ trajectories to a single reformulation lowers OmniMATH-Hard below the untrained model (e.g., OEQ + 4-MCQ: 12.39 vs. 15.67).

The mechanisms also differ in three ways. (1) STaR reveals the answer in the prompt and fine-tunes on rationales conditioned on it. Cog-DRIFT never reveals which candidate is correct, so the model must still derive the answer and discriminate among distractors in MCQ, and must produce the full value from partial digits in cloze. (2) If the model were verifying candidates backward, the most discriminative format should transfer best, yet transfer improves as the format becomes more generative (4-choice 11.1%, 10-choice 15.5%, cloze 18.9%; Section 3). (3) Cog-DRIFT is on-policy RL with a curriculum that removes the scaffold per instance, so the final policy trains on the unaided open-ended problem.

Cog-DRIFT also outperforms guided-exploration methods that inject reasoning guidance (HINT and QuestA, Appendix E.2, Table 7) by +4.67% on average while requiring only the gold answer. To isolate this effect under RL, we add an answer-hint GRPO baseline (Gold-Appended):

BMHOmni-HardAIME24AIME25GPQADate
Zero-shot0.015.6746.9241.7123.5787.71
Gold-Appended0.911.2730.0033.3320.1985.89
Cog-DRIFT10.1118.5851.7440.2836.0188.75

Gold-Appended underperforms even the zero-shot model, and Cog-DRIFT outperforms it. Its training reward plateaus quickly, but the gains do not transfer once the gold answer is removed at test time.


 

W2: Presentation issues in Figures 2 and 3, ambiguous "number of rollouts" in Table 2, and unclear stopping criteria/update counts.

We thank the reviewer for the detailed suggestions and revised the PDF accordingly. We made four changes in the revised paper. (1) Figure 2(b) now has the x-axis label "Training format", which distinguishes it from Figure 2(a), where the x-axis shows evaluation formats. (2) Figure 3 now has the x-axis label "k (number of samples)". (3) The Table 2 caption now states that n is the number of responses sampled per prompt, i.e., the GRPO group size (rollout_n in Table 3). (4) Appendix B now states that, for all methods, we select the checkpoint at training step 400. The text changes are highlighted in blue.


 

Q2: The GPT-5.4 agreement filter selects problems that a stronger model can solve, so effectiveness on problems hard for strong models is unclear.

The GPT-5.4 filter removes flawed question-answer pairs (incorrect gold answers, incomplete questions, or missing figures; Appendix E.7). It does not select by difficulty, and Cog-DRIFT itself needs only the gold answer. We test this directly in Appendix C.2 (Table 5). The unfiltered Dhard (8,922 problems) includes problems that GPT-5.4 fails to solve or answers inconsistently with the gold label. Training on it, Cog-DRIFT on Qwen still reaches 82.2 on MATH500 and 33.1 on GPQA, above the untrained model (78.8 and 23.57) and on par with the filtered set (81.6 and 36.0). The Binary Filter (5,922 problems), which never requires GPT-5.4 to solve the problem, gives comparable results (81.8 / 34.7 for Qwen).


 

Q3: Pass@k improvements on AIME24/25 are modest; why are math gains more limited?

AIME24 and AIME25 have 30 problems each, so a +3.33% gain at k=128 is one additional solved problem. The base model is also close to saturation on AIME24 at k=128 (around 90% in Figure 3). At pass@1, where headroom is larger, Cog-DRIFT improves AIME24 by +4.82% over zero-shot and +19.87% over GRPO (Table 1). At matched 32k evaluation, Cog-DRIFT reaches 56.66 / 46.67 on AIME24 / AIME25 versus 52.22 / 43.34 for GRPO (Table 6).

As further evidence, we re-sample the base model with 1,024 samples on BMH and keep the 48 problems that remain unsolved (0/1,024). On this subset:

kBaseGRPOCog-DRIFT
10.00.00.7
80.05.28.0
320.08.58.3
640.011.112.4
2560.018.422.2
10240.029.235.4

Cog-DRIFT reaches 35.4% at k=1,024 versus 29.2% for GRPO, which widens the gap at a larger k.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer b95U,

There are only 3 days left in the discussion period. In our response, we have addressed your concerns regarding the distinction from existing answer-guided methods, the benefits of reformulation, and the effectiveness of our approach on challenging problems and under larger sampling budgets. We have also updated the PDF accordingly to incorporate your comments. If there is anything still unresolved, please let us know, and we are happy to answer. We would be grateful if you would consider revisiting your score in light of our responses.

Thank you for your response which resolves my questions. I keep my original score.

Dear Reviewer b95U,

Thank you very much for your follow-up and for acknowledging that our response has resolved your questions. We appreciate your time and feedback throughout the review process.

Given that your original concerns have been addressed, we were wondering if you might be willing to reconsider your original score in light of our clarifications, additional experiments, and revisions to the manuscript.

If there are any remaining concerns that prevent you from revisiting your score, we would be more than happy to address them or provide further clarification.

Thank you again for your time and consideration!

Reviewer_Q7poOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reformulates hard reasoning problems into multiple-choice and cloze variants, then gradually returns them to the original open-ended format during RL. Progression is determined by the student’s success rate on each problem. The trained models are evaluated on the original problems and held-out benchmarks.

Critical strengths
  • The approach offers a way to improve a larger model without updating its weights. Notably, the trained 7B concept generator outperforms an untuned 32B generator, suggesting that learning useful guidance may be more effective than simply increasing the generator’s size.

  • Careful experiments examine how decoding parameters affect repeated sampling. Further analyses of mismatched concepts and concept counts help clarify the contributions of problem-specific guidance and multiple approaches to the observed gains.

Critical weaknesses
  • Although the hard problems are selected because all 64 sampled attempts fail, standard GRPO still improves Qwen’s performance on them. Does Cog-DRIFT develop new reasoning capabilities, or does it primarily increase the likelihood of sampling solutions the model can already produce? Would this conclusion hold if a larger, independent sampling budget were used to identify unsolved problems?

  • Evaluation is limited to two small models, one of which is Llama3.2-3B, and a compact training set filtered by GPT-5.4 answer agreement. Does the method remain effective with larger reasoning models and training data that combine easy and hard problems? Given the uneven gains across benchmarks, could the authors report results across independent training seeds to establish the reliability of the improvements?

  • Are the method and all baselines compared under matched total cost or wall-clock time budgets, including data filtering, reformulation generation, and training? Since different question formats may produce different response lengths, does the reported sample-efficiency advantage remain under the same generated-token or GPU-hour budget?

Questions for authors

Same as the “Critical Weaknesses” section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Q7po (Part 1)

We thank the reviewer for their time and effort. We would like to gently note that the Strengths section refers to a trained 7B concept generator, a 32B generator, and analyses of mismatched concepts and concept counts, none of which appear in our paper, so this part may have been intended for a different submission. The weaknesses do concern our work, and we address them below.


 

W1 & Q1: Standard GRPO also improves Qwen on the hard set; unclear whether Cog-DRIFT learns new reasoning, and whether this holds with a larger sampling budget for filtering.

We discuss this behavior in Section 6. The GRPO gain on BMH is specific to Qwen, which is known to improve even under weak or spurious rewards (Shao et al., 2026). For Llama, GRPO stays at 0.0 on BMH while Cog-DRIFT reaches 8.64. GRPO's in-domain gain also does not transfer, as it lowers the held-out average for both models (Table 1).

On a larger sampling budget, Table 2 already scales GRPO's rollouts from 8 to 64. BMH rises only from 6.37 to 8.43, below Cog-DRIFT with n=8 (10.11), and the average also stays below (38.26 vs. 40.92).

Following the reviewer's suggestion, we also re-filter the hard set with a larger independent budget. We re-sample the base model with 1,024 samples on BMH and keep the 48 problems that remain unsolved (0/1,024). On this subset:

kBaseGRPOCog-DRIFT
10.00.00.7
80.05.28.0
320.08.58.3
640.011.112.4
2560.018.422.2
10240.029.235.4

Cog-DRIFT reaches 35.4% coverage at k=1,024 versus 29.2% for GRPO, so the advantage persists under a much larger filtering budget.

[Paper Update] We revised the claims in Section 1 and Section 6 from "acquires new reasoning capabilities / new knowledge" to "improves performance on problems for which the base model produces no correct solutions within the tested sampling budget." We also removed the phrase "reflecting the acquisition of new knowledge" from the pass@k paragraph in Section 6. The revised text is highlighted in blue.


 

W2 & Q2: Only two small models and a compact GPT-5.4-filtered training set; unclear if the method scales to larger reasoning models and mixed-difficulty data, and only evaluation seeds are reported.

Data filtering. Cog-DRIFT does not depend on the GPT-5.4 filter. Appendix C.2 (Table 5) reports training on the unfiltered Dhard (8,922 problems) and with a lighter Binary Filter (5,922 problems), and Cog-DRIFT remains effective in both. GPT-5.4 is used only to remove noisy labels and contributes no reasoning traces to training.

Larger model. We train gpt-oss-20b on the same Dhard as in the submission. A model-specific hard set (pass@64 = 0 under gpt-oss-20b) would be ideal, but evaluating just 300 problems takes more than three days on our hardware, so we use the Qwen-based Dhard as an approximation.

gpt-oss-20bBMHAIME24AIME25OmniMATHGPQADateAvg
Zero-shot11.759.455.810.732.882.142.1
GRPO11.965.450.410.934.782.242.6
Cog-DRIFT12.665.456.510.836.584.044.3

Cog-DRIFT achieves the highest average of 44.3, outperforming GRPO by +1.7% and the zero-shot model by +2.2%. This is preliminary evidence that the benefits extend to larger reasoning models, even when the training data is not selected for the target model.

[Paper Update] We added this limitation to the Dataset paragraph of Section 5. The revised text states that “We note as a limitation that building Dhard from each target model's own pass@64 = 0 problems would be a more direct test.” The added sentence is highlighted in blue.

AuthorsOpenReview ↗

Response to Reviewer Q7po (Part 2)

Mixed-difficulty data. Using Llama-3.2-3B-Instruct, we sample 360 hard problems (pass@64 = 0) and 360 easier problems (pass@64 > 0), train GRPO on these 720 problems, and compare it with Cog-DRIFT trained only on hard problems.

MethodBMHOmni-HardAIME24AIME25GPQADateAvg
Zero-shot0.05.13.80.415.354.913.3
GRPO (easy+hard)6.36.95.00.817.251.814.7
Cog-DRIFT8.67.04.20.423.362.817.7

Cog-DRIFT averages 17.7, outperforming GRPO trained on mixed-difficulty data by +3.0% (17.7 vs. 14.7). Adding easier, solvable problems to GRPO does not match the benefit of reformulating the hard ones.

Training stability. Following the reviewer's suggestion, we train Cog-DRIFT with 3 independent seeds and report mean ± std across training runs. Given the limited rebuttal period, Zero-shot and GRPO use 3 evaluation seeds as a reference.

MethodBMHOmni-HardAIME24AIME25GPQADate
Zero-shot0.0 ± 0.015.67 ± 1.146.92 ± 3.341.71 ± 4.523.57 ± 1.287.71 ± 0.9
GRPO6.37 ± 1.916.37 ± 0.731.87 ± 2.727.29 ± 5.431.81 ± 2.189.53 ± 1.0
Cog-DRIFT9.21 ± 1.319.18 ± 2.450.00 ± 3.346.67 ± 6.736.41 ± 1.588.18 ± 0.7

Across three training seeds, Cog-DRIFT has a higher mean than GRPO on five of the six benchmarks.


 

W3 & Q3: Are methods compared under matched total cost (filtering, reformulation, training), and does the sample-efficiency advantage hold under matched tokens or GPU-hours?

(1) Data filtering is identical for all methods, since every baseline trains on the same 958 problems. (2) Reformulation is a one-time cost of three variants per problem with the same Qwen3-4B-Instruct-2507 model (Section 3), about 2.9k short generations. (3) All GRPO-based methods use the same rollout number (n=8), response limit (8k), batch size, and learning rate (Appendix B, Table 3). The submission also tests two compute-scaled variants of GRPO, 8 times more rollouts (Table 2) and 4 times longer generations at evaluation (Figure 4, Table 6), and Cog-DRIFT outperforms both.

Following the reviewer's suggestion, we report generated tokens and GPU hours over 100 training steps, excluding the one-time filtering and reformulation costs:

MethodRollouts per questionGenerated tokens / stepGPU-seconds / stepGPU-hours (100 steps)Optimizer updates (100 steps)
GRPO (n=8)8236k1,70347.3800
GRPO (n=64)642,062k6,493180.4800
Cog-DRIFT8260k1,75348.7800

At the same n=8, Cog-DRIFT uses only +2.9% more GPU-seconds per step than GRPO (1,753 vs. 1,703). GRPO with n=64 uses about 3.7 times as many GPU-seconds per step as Cog-DRIFT (6,493 vs. 1,753) and still underperforms it.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer Q7po,

There are only 3 days left in the discussion period. In our response, we have addressed your questions regarding the improvements over GRPO, whether Cog-DRIFT enables the learning of new reasoning capabilities, the consistency of gains under larger sampling budgets, scalability to larger reasoning models and mixed-difficulty data, robustness across training seeds, and the computational cost breakdown. We have also updated the PDF accordingly to incorporate your comments. If there is anything still unresolved, please let us know, and we are happy to answer. We would be grateful if you would consider revisiting your score in light of our responses.

Thank you for the response and clarifications. After considering the authors’ replies, I have decided to maintain my original score.

Reviewer_eA1VOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies learning from hard reasoning problems on which the initial policy produces no correct answer in 64 sampled attempts. Its main contribution is to use answer-preserving task reformulations—4-option MCQ, 10-option MCQ, and digit-masked cloze—to make successful exploration more accessible without teacher-provided reasoning traces. Cog-DRIFT organizes these formats and the original open-ended questions into an adaptive, per-instance GRPO curriculum, advancing each problem when rollout accuracy reaches a threshold.

Experiments with Qwen3-4B-Instruct-2507 and Llama3.2-3B-Instruct report the best average performance among the compared methods across six benchmarks, including improvements on the held-out BigMath-Hard split and transfer to other datasets. Additional analyses examine format transfer, curriculum design, and performance under larger sampling and generation budgets.

Critical strengths
  1. Using output format to adjust problem difficulty is a simple and useful contribution. The reformulations require only the question and gold answer, and Section 3 demonstrates increased solvability and transfer back to the original open-ended problems under RFT. This suggests a practical way to obtain learning signals from hard examples without requiring teacher reasoning traces.

  2. The results provide evidence of benefits beyond memorizing training answers: Cog-DRIFT improves on the held-out BigMath-Hard split and achieves the highest six-benchmark average for both models. Comparisons with guided-exploration methods, larger GRPO rollout budgets, and longer test-time generations strengthen the empirical case, although they do not fully isolate the source of the gains.

Critical weaknesses
  1. The robustness and breadth of generalization remain uncertain. Excluding BMH and GPQA, Qwen's average improvement is 1.84 percentage points over zero-shot and 0.45 over GRPO with 64 rollouts. For Llama, gains on these remaining benchmarks are concentrated in Date Understanding. The paper averages repeated evaluations but does not report uncertainty intervals or robustness across training seeds. GPQA contributes substantially to the overall gain, and familiarity with MCQ formats may contribute to this improvement; this is useful transfer, but does not by itself establish broadly improved open-ended reasoning. Similarly, the small pass@128 gains do not distinguish newly acquired reasoning from increased sampling probability of existing successful behaviors. These limitations reduce my confidence in the strength of the generalization claims.

  2. The contribution of the full curriculum is not sufficiently isolated from prompt diversity. Even with equal gradient steps, training on several formulations exposes the model to more varied prompts. The RFT transfer and mixture experiments do not establish which stages are necessary under RL, while reversing the curriculum changes both its starting difficulty and subsequent exposure. The gold-answer control also uses RFT, so it does not establish the mechanism of the RL gains. Adaptive curricula with fewer formats would help determine whether the four-stage progression is necessary or whether simpler reformulation schemes retain the benefits. This matters because the paper attributes its gains to both reformulation and the proposed progression.

Questions for authors
  1. How robust are the main gains across independent training seeds? Please report uncertainty for the key comparisons, distinguishing variation across training runs from repeated evaluation of a single checkpoint.

  2. Can you compare the four-stage adaptive curriculum with shorter adaptive progressions, such as 4-MCQ → cloze → open-ended and 4-MCQ → open-ended or cloze → open-ended? Keeping the update budget fixed and controlling prompt diversity where feasible would help assess what the intermediate stages contribute. Table 9 motivates this question under RFT but does not answer it under RL.

  3. What evidence distinguishes the reported pass@k improvements from making already possible solutions more likely? Without such evidence, would the authors narrow the claim of acquiring new knowledge to improved performance within the tested sampling budgets?

  4. Which model, checkpoint, and evaluation settings are used in Figure 5? Please reconcile its OmniMATH-Hard accuracy range with the corresponding Table 1 results, since this figure supports the curriculum's sample-efficiency claim.

5 comments
AuthorsOpenReview ↗

Response to Reviewer eA1V (Part 1)

We thank the reviewer for recognizing that "using output format to adjust problem difficulty is a simple and useful contribution" and that it offers "a practical way to obtain learning signals from hard examples without requiring teacher reasoning traces." We address the comments below.


 

W1 & Q1: Generalization gains are small and uneven, GPQA may benefit from MCQ familiarity, and robustness across independent training seeds is unclear.

Size of the gains. Under the matched rollout budget of Table 1 and excluding BMH and GPQA, Cog-DRIFT improves over GRPO by +8.57% on Qwen and over the strongest baseline, NuRL, by +3.65%. It also stays ahead of GRPO with n=64 (+0.45%), which uses 8 times the rollout budget.

GPQA and MCQ familiarity. We evaluate GPQA-Diamond with answer matching (Appendix C.1, following Chandak et al., 2025), so selecting an option letter won’t be counted as successful. Under this protocol, Cog-DRIFT gains +12.44% and +4.20% over the base model and GRPO on Qwen, and +8.02% and +11.31% on Llama. We additionally evaluate GPQA with the options removed entirely, using Qwen3-4B-Instruct with answers judged by Qwen3.8-27B:

GPQA Acc.
Zero-shot32.0
GRPO33.5
Cog-DRIFT35.8

The improvement persists with all answer options removed.

Training seeds. Following the reviewer's suggestion, we train Cog-DRIFT with 3 independent seeds and report mean ± std across training runs. Given the limited rebuttal period, Zero-shot and GRPO use 3 evaluation seeds as a reference.

BMHOmni-HardAIME24AIME25GPQADate
Zero-shot0.0 ± 0.015.67 ± 1.146.92 ± 3.341.71 ± 4.523.57 ± 1.287.71 ± 0.9
GRPO6.37 ± 1.916.37 ± 0.731.87 ± 2.727.29 ± 5.431.81 ± 2.189.53 ± 1.0
Cog-DRIFT9.21 ± 1.319.18 ± 2.450.00 ± 3.346.67 ± 6.736.41 ± 1.588.18 ± 0.7

The gains are not limited to a single training run.

[Paper Update] We added a second average to Table 1 that excludes BMH and GPQA (Avg.†), next to the original six-benchmark average. We also colored the Table 1 column headers to separate the in-domain benchmark (BMH) from the out-of-domain benchmarks, and added two sentences in Section 6 that explain why we report both averages. All changes are highlighted in blue in the revised paper.


 

W2 & Q2: The contribution of the full four-stage curriculum is not isolated from prompt diversity under RL; shorter adaptive progressions should be compared.

The submission already has two RL controls that fix the set of prompt formats and change only the curriculum. (1) Random permutation (Table 10(b)) uses the same four formats in a random order, and the average drops from 40.92 to 35.61. (2) The uniform mixture in Figure 5 (right) shows all four formats at 25% each throughout training, and its OmniMATH-Hard accuracy plateaus while the curriculum keeps improving. Prompt diversity alone therefore does not explain the gains. We agree that the reversed order changes the starting difficulty, so we regard the random permutation as the cleaner control.

Following the reviewer's suggestion, we also compare shorter adaptive progressions under GRPO with the same number of optimizer updates:

CurriculumBMHOmniAIME24AIME25GPQADateAvg.
4-MCQ → OEQ5.119.152.140.836.989.340.6
Cloze → OEQ3.013.350.636.521.785.635.1
4-MCQ → Cloze → OEQ5.916.254.640.231.389.539.6
4-MCQ → 10-MCQ → Cloze → OEQ (ours)10.118.951.740.336.088.841.0

The full four-stage progression has the highest average (41.0), driven by BMH (10.1 vs. 5.1 for 4-MCQ → OEQ). The two-stage 4-MCQ → OEQ curriculum remains competitive on the held-out benchmarks. Adding the 10-MCQ stage to the three-stage curriculum improves BMH by +4.2% and GPQA by +4.7%, so the intermediate stage helps most on the hardest problems.

We agree that the gold-answer control (Appendix E.4) is run under RFT. We therefore add an RL version, in which we append the gold answer to the prompt during GRPO and remove it with the same adaptive schedule (Gold-Appended):

BMHOmni-HardAIME24AIME25GPQADate
Zero-shot0.015.6746.9241.7123.5787.71
Gold-Appended0.911.2730.0033.3320.1985.89
Cog-DRIFT10.1118.5851.7440.2836.0188.75

Gold-Appended underperforms the zero-shot model on five of the six benchmarks, and Cog-DRIFT outperforms it on all six. Its training reward plateaus quickly, but the gains do not transfer once the gold answer is removed at test time.

AuthorsOpenReview ↗

Response to Reviewer eA1V (Part 2)

Q3: Pass@k gains may reflect higher sampling probability of existing solutions rather than newly acquired reasoning.

We agree that pass@k at a finite k cannot fully distinguish the two (Appendix A). Our setting is defined by a bounded sampling budget, so the practical question is whether a method improves performance within that budget.

[Paper Update] As the reviewer suggested, we revised the claims in Section 1 and Section 6 from "acquires new reasoning capabilities / new knowledge" to "improves performance on problems for which the base model produces no correct solutions within the tested sampling budget." We also removed the phrase "reflecting the acquisition of new knowledge" from the pass@k paragraph in Section 6. The revised text is highlighted in blue.

As further evidence, we re-sample the base model with 1,024 samples on BMH and keep the 48 problems that remain unsolved (0/1,024). On this subset:

kBaseGRPOCog-DRIFT
10.00.00.7
80.05.28.0
320.08.58.3
640.011.112.4
2560.018.422.2
10240.029.235.4

Cog-DRIFT reaches 35.4% coverage at k=1,024 versus 29.2% for GRPO (+6.2%), so it makes previously unsolved problems accessible within the tested sampling budget.


 

Q4: Figure 5 OmniMATH-Hard accuracy range does not match Table 1; clarify the model, checkpoint, and evaluation settings.

We thank the reviewer for catching this. We updated the Figure 5 caption to state that the results are with Llama3.2-3B-Instruct, that all checkpoints reported elsewhere (e.g., Table 1) are selected at step 400, and that we train longer in Figure 5 only to show the longer-run trend. The added text is highlighted in blue.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer eA1V,

There are only 3 days left in the discussion period. In our response, we have addressed your questions on generalization gains, MCQ familiarity, robustness across independent training seeds, the contribution of the full four-stage curriculum, and clarified the pass@k gains. We have also updated the PDF accordingly to incorporate your comments. If there is anything still unresolved, please let us know, and we are happy to answer. We would be grateful if you would consider revisiting your score in light of our responses.

Comments on the paper

2 comments
Area_Chair_7SUSOpenReview ↗

Please engage with the authors' rebuttal

Dear reviewers,

Given that we have only a few days until the closing of the rebuttal period, please take into account the effort that the author spent on their rebuttal, and feel free to engage with them (or at least give an acknowledgement that you have read it). Feel free to ignore this message if you have done so.

Best regards,

Your AC.

AuthorsOpenReview ↗

Summary of Additional Rebuttal Experiments

  1. Robustness across independent training seeds (eA1V W1/Q1; Q7po W2/Q2)
    We trained Cog-DRIFT with three independent random seeds and reported mean ± standard deviation across six benchmarks. Cog-DRIFT outperformed GRPO in mean accuracy on five of six benchmarks. GRPO and zero-shot results were also evaluated with three evaluation seeds, although these do not capture training-run variance.

  2. Open-ended GPQA evaluation without answer options (eA1V W1)
    To rule out the possibility that improvements on GPQA primarily arise from familiarity with multiple-choice formats, we removed all answer options and evaluated models using free-form answer generation. Cog-DRIFT achieved 35.8%, compared with 33.5% for GRPO and 32.0% for zero-shot, demonstrating improvements even without multiple-choice options.

  3. Shorter adaptive curriculum ablations (eA1V W2/Q2)
    To isolate the contribution of individual curriculum stages, we compared the full four-stage curriculum with three shorter progressions under matched optimizer-update budgets. The full curriculum achieved the highest overall average (41.0%), compared with 40.6% for the two-stage 4-MCQ → OEQ curriculum. Its largest advantage was on the hardest in-domain problems (BMH: 10.1% vs. 5.1%), while shorter progressions remained competitive on held-out benchmarks.

  4. Gold-Appended GRPO baseline (eA1V W2; b95U W1/Q1)
    To distinguish task reformulation from simply providing gold-answer guidance, we introduced an RL baseline that appends the gold answer during GRPO training and gradually removes it using the same adaptive schedule. Gold-Appended underperformed the zero-shot model on five of six benchmarks, while Cog-DRIFT outperformed it on all six. This supports the importance of reformulation beyond answer disclosure and addresses comparisons with STaR-style guidance.

  5. Larger sampling-budget evaluation on previously unsolved problems (eA1V Q3; Q7po W1/Q1; b95U Q3)
    We independently re-sampled the base model with 1,024 attempts per problem and identified 48 BMH problems that remained unsolved. On this subset, Cog-DRIFT achieved 35.4% pass@1024 coverage, compared with 29.2% for GRPO and 0% for the base model. This demonstrates that the advantage persists even on problems unsolved under a substantially larger initial sampling budget, without claiming to prove acquisition of entirely new reasoning capabilities.

  6. Scaling to a larger reasoning model (Q7po W2/Q2)
    We extended our experiments to gpt-oss-20b, using the same Qwen-derived hard training set. Cog-DRIFT achieved an average accuracy of 44.3%, outperforming GRPO (42.6%) and zero-shot (42.1%). These preliminary results suggest that the benefits extend beyond the two smaller models in the original submission, while acknowledging that the hard set was not constructed specifically for gpt-oss-20b.

  7. Mixed-difficulty training comparison (Q7po W2/Q2)
    To test whether adding easier problems to standard GRPO could replace reformulation, we trained GRPO on 360 hard and 360 easier problems using Llama-3.2-3B-Instruct. Cog-DRIFT trained only on hard problems achieved an average accuracy of 17.7%, compared with 14.7% for mixed-difficulty GRPO. This indicates that simply incorporating solvable examples does not reproduce the benefits of reformulating challenging problems.

  8. Computational cost and training-efficiency analysis (Q7po W3/Q3; also clarifies b95U W2)
    We measured generated tokens, GPU time, and optimizer updates over 100 training steps. Cog-DRIFT required 48.7 GPU-hours, compared with 47.3 for GRPO (n=8) and 180.4 for GRPO (n=64). At the same rollout budget, Cog-DRIFT incurred only 2.9% additional GPU time per step, while GRPO (n=64) consumed approximately 3.7× more GPU time per step than Cog-DRIFT and still underperformed it. The reported training-cost comparison excludes one-time filtering and reformulation costs.


 

Overall

These eight additions strengthen the empirical evidence for Cog-DRIFT along several dimensions: robustness, generalization beyond multiple-choice evaluation, curriculum design, differentiation from answer-guided learning, performance under larger sampling budgets, scalability, comparisons with mixed-difficulty training, and computational efficiency.

We also revised the manuscript to clarify the scope of our capability claims, distinguish reformulation from curriculum-specific benefits, and address the presentation and experimental-protocol issues raised by the reviewers. All updates are highlighted in blue.

We thank reviewers and ACs for their time and effort on help strengthen our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.