- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Cog-DRIFT, a framework that enables reinforcement learning from hard reasoning problems by reformulating open-ended questions into multiple-choice and cloze variants, thereby reducing task difficulty and providing more accessible learning signals. It further employs an instance-level curriculum that progressively transitions each problem from easier to harder formats based on the model's performance. The method is evaluated on two language models across six reasoning benchmarks and compared with several strong baselines. The results show consistent improvements in average accuracy and suggest that learning from reformulated problems can improve performance on both the original hard problems and held-out benchmarks.
Critical strengths
- The paper proposes Cog-DRIFT, which reformulates hard open-ended problems into multiple-choice and cloze variants to reduce task difficulty, thereby enabling models to learn more effectively from challenging problems. The method is simple and easy to implement, and constructing these variants requires no additional information beyond the gold answers.
- The paper conducts extensive experiments and ablation studies on the proposed method, evaluating two models across six benchmarks covering mathematical reasoning, scientific knowledge, and date understanding. It also compares Cog-DRIFT against strong baselines and investigates the effects of different reformulation strategies, curriculum designs, rollout budgets, and generation lengths. These experiments provide a comprehensive evaluation of the method's effectiveness and the contributions of its key components.
Critical weaknesses
- The core mechanism of Cog-DRIFT can be viewed as answer-related scaffolding: multiple-choice and cloze formats provide additional answer information to facilitate exploration. This is conceptually related to prior work such as STaR [1], which uses gold answers to elicit reasoning traces when ordinary generation fails. The authors should clarify the fundamental difference between their approach and such answer-guided methods.
- The manuscript contains many issues in its presentation and clarity. Figure 2(b) uses the same horizontal-axis categories as Figure 2(a), although they represent training formats rather than evaluation formats, which may cause confusion. Figure 3 also lacks an explicit horizontal-axis label. In Table 2, the term “number of rollouts” is ambiguous: does it refer to responses sampled per group or total rollouts per optimization step? Moreover, the paper does not clearly specify the stopping criteria or optimizer update counts for different rollout settings, making the training-budget comparison difficult to assess.
Questions for authors
- How does Cog-DRIFT fundamentally differ from prior answer-guided approaches such as STaR? Since multiple-choice and cloze reformulations provide candidate answers or partial answer information, can the authors isolate the benefit of this reformulation from the more general effect of providing answer-related guidance?
- The training set is filtered to retain problems that Qwen cannot solve within 64 rollouts but for which GPT-5.4's majority answer across three runs agrees with the ground truth. While this filtering helps remove flawed or unanswerable questions, it also selects problems that a stronger model can solve. Consequently, the reported results may primarily demonstrate effectiveness on problems that are difficult for smaller models but remain within the capability of stronger models. How well would the method generalize to problems that are also difficult for stronger models?
- Figure 3 shows relatively modest Pass@k improvements over the base model on AIME24 and AIME25. Since the paper aims to enable learning from problems that the original policy cannot solve, these limited gains raise questions about how much transferable reasoning capability is actually acquired. Could the authors provide further analysis? Why are the gains more limited on mathematical benchmarks?
[1] Zelikman, Eric, et al. "Star: Bootstrapping reasoning with reasoning." Advances in Neural Information Processing Systems 35 (2022): 15476-15488.
4 comments
A gentle reminder
Dear Reviewer b95U,
There are only 3 days left in the discussion period. In our response, we have addressed your concerns regarding the distinction from existing answer-guided methods, the benefits of reformulation, and the effectiveness of our approach on challenging problems and under larger sampling budgets. We have also updated the PDF accordingly to incorporate your comments. If there is anything still unresolved, please let us know, and we are happy to answer. We would be grateful if you would consider revisiting your score in light of our responses.
Thank you for your response which resolves my questions. I keep my original score.
Dear Reviewer b95U,
Thank you very much for your follow-up and for acknowledging that our response has resolved your questions. We appreciate your time and feedback throughout the review process.
Given that your original concerns have been addressed, we were wondering if you might be willing to reconsider your original score in light of our clarifications, additional experiments, and revisions to the manuscript.
If there are any remaining concerns that prevent you from revisiting your score, we would be more than happy to address them or provide further clarification.
Thank you again for your time and consideration!
Response to Reviewer b95U
We thank the reviewer for recognizing that "the method is simple and easy to implement" and that our experiments "provide a comprehensive evaluation." We address the comments below.
W1 & Q1: Cog-DRIFT resembles answer-guided methods such as STaR; the benefit of reformulation should be separated from providing answer information.
The submission already compares against STaR-style answer guidance (Appendix E.4). Appending the gold answer to the question reaches near-ceiling in-domain accuracy but collapses on held-out MATH500 (58.60), far below the 82.20 of our reformulation mixture under the same RFT protocol. In Table 9, adding STaR-style rationalized OEQ trajectories to a single reformulation lowers OmniMATH-Hard below the untrained model (e.g., OEQ + 4-MCQ: 12.39 vs. 15.67).
The mechanisms also differ in three ways. (1) STaR reveals the answer in the prompt and fine-tunes on rationales conditioned on it. Cog-DRIFT never reveals which candidate is correct, so the model must still derive the answer and discriminate among distractors in MCQ, and must produce the full value from partial digits in cloze. (2) If the model were verifying candidates backward, the most discriminative format should transfer best, yet transfer improves as the format becomes more generative (4-choice 11.1%, 10-choice 15.5%, cloze 18.9%; Section 3). (3) Cog-DRIFT is on-policy RL with a curriculum that removes the scaffold per instance, so the final policy trains on the unaided open-ended problem.
Cog-DRIFT also outperforms guided-exploration methods that inject reasoning guidance (HINT and QuestA, Appendix E.2, Table 7) by +4.67% on average while requiring only the gold answer. To isolate this effect under RL, we add an answer-hint GRPO baseline (Gold-Appended):
Gold-Appended underperforms even the zero-shot model, and Cog-DRIFT outperforms it. Its training reward plateaus quickly, but the gains do not transfer once the gold answer is removed at test time.
W2: Presentation issues in Figures 2 and 3, ambiguous "number of rollouts" in Table 2, and unclear stopping criteria/update counts.
We thank the reviewer for the detailed suggestions and revised the PDF accordingly. We made four changes in the revised paper. (1) Figure 2(b) now has the x-axis label "Training format", which distinguishes it from Figure 2(a), where the x-axis shows evaluation formats. (2) Figure 3 now has the x-axis label "k (number of samples)". (3) The Table 2 caption now states that n is the number of responses sampled per prompt, i.e., the GRPO group size (rollout_n in Table 3). (4) Appendix B now states that, for all methods, we select the checkpoint at training step 400. The text changes are highlighted in blue.
Q2: The GPT-5.4 agreement filter selects problems that a stronger model can solve, so effectiveness on problems hard for strong models is unclear.
The GPT-5.4 filter removes flawed question-answer pairs (incorrect gold answers, incomplete questions, or missing figures; Appendix E.7). It does not select by difficulty, and Cog-DRIFT itself needs only the gold answer. We test this directly in Appendix C.2 (Table 5). The unfiltered Dhard (8,922 problems) includes problems that GPT-5.4 fails to solve or answers inconsistently with the gold label. Training on it, Cog-DRIFT on Qwen still reaches 82.2 on MATH500 and 33.1 on GPQA, above the untrained model (78.8 and 23.57) and on par with the filtered set (81.6 and 36.0). The Binary Filter (5,922 problems), which never requires GPT-5.4 to solve the problem, gives comparable results (81.8 / 34.7 for Qwen).
Q3: Pass@k improvements on AIME24/25 are modest; why are math gains more limited?
AIME24 and AIME25 have 30 problems each, so a +3.33% gain at k=128 is one additional solved problem. The base model is also close to saturation on AIME24 at k=128 (around 90% in Figure 3). At pass@1, where headroom is larger, Cog-DRIFT improves AIME24 by +4.82% over zero-shot and +19.87% over GRPO (Table 1). At matched 32k evaluation, Cog-DRIFT reaches 56.66 / 46.67 on AIME24 / AIME25 versus 52.22 / 43.34 for GRPO (Table 6).
As further evidence, we re-sample the base model with 1,024 samples on BMH and keep the 48 problems that remain unsolved (0/1,024). On this subset:
Cog-DRIFT reaches 35.4% at k=1,024 versus 29.2% for GRPO, which widens the gap at a larger k.