acceptodds
Withdrawn from ICLR 2027

Starting Near the Goal: Curriculum Reinforcement Learning for Training Initially Weak LLM Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards is difficult for initially weak language-model policies. An initially weak policy may rarely discover successful trajectories, and terminal rewards provide little direct guidance for intermediate reasoning decisions. To address these challenges, we propose Curriculum Reinforcement Learning (CRL), a training framework that begins exploration near the goal state and uses outcome feedback to adjust curriculum. Starting with shorter continuations reduces the difficulty of obtaining successful outcomes early in training, helping to alleviate the RL cold-start problem. Adaptive curriculum strength and infix reinforcement then support policy improvement by adjusting external guidance and reinforcing reasoning patterns from the policy’s own successful experience. Together, these mechanisms make curriculum responsive to the needs of RL and facilitate the transition to autonomous reasoning as reference guidance is gradually withdrawn. Extensive evaluations across multiple mathematical reasoning benchmarks show that CRL improves both reasoning performance and training efficiency. The code is available at: https://anonymous.4open.science/r/LLM-CRL

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, LLM Reasoning, Initially Weak LLM Policies

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_W1HBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CRL for initially weak LLMs, combining progressively shortened reference prefixes with adaptive prefix-loss weighting and reinforcement of successful generated infixes. Experiments on Qwen3.5-0.8B and 2B show improved mathematical reasoning performance and training efficiency.

Critical strengths
  • The paper addresses the practical cold-start problem in RL training and presents a relatively simple approach for incorporating reference solutions into policy optimization.
Critical weaknesses
  • Limited novelty. CRL closely builds on existing prefix-based approaches such as UFT and R³. The adaptive weighting and infix reinforcement appear to be incremental modifications, with limited additional methodological insight.
  • Modest performance gains. The aggregate improvements over SFT-RFT are only 0.71 and 0.61 percentage points, and CRL does not consistently outperform strong baselines across individual benchmarks. These results provide limited evidence of a substantial practical advantage.
  • Limited evaluation scale. Experiments cover only 0.8B and 2B models from the same family. Although these models fit the motivation of weak initial policies, the evaluation does not establish whether the benefits extend to larger models facing similarly difficult tasks.
Questions for authors
  • Do the authors have evidence that CRL remains effective on larger models in settings where their initial success rates are low?
  • Does the preference for a 10% infix persist when the effective auxiliary-loss strength is controlled across different infix lengths?
Reviewer_srgXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CRL for training initially weak LLM policies. It combines a reference-prefix curriculum with adaptive supervision weights and additional imitation of successful continuation infixes.

Critical strengths

The paper addresses an important problem: sparse rewards make it difficult for initially weak policies to benefit from reinforcement learning.

Critical weaknesses
  1. The main framework closely follows prior work, particularly R³ and UFT. The incremental contributions are an outcome-dependent weighting rule for prefix supervision and additional imitation of the initial segment of successful continuations.

  2. Experiments use only two small models from the Qwen3.5 family. The benchmarks are also dated and potentially contaminated.

Questions for authors
  1. I do not consider the curriculum-based combination of RL and SFT novel. Beyond the two incremental weighting modifications, are there other methodological contributions that distinguish this work?
  2. Do you have results on other model families or larger models, and on newer benchmarks such as HMMT 2026 and AIME 2026?
Reviewer_wkFdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Curriculum Reinforcement Learning (CRL) to train initially weak LLM policies with sparse outcome rewards. CRL initializes rollouts from reference-solution prefixes and gradually shortens them to encourage increasingly autonomous generation. It adapts curriculum strength based on rollout success and prefix length, combining a weighted prefix-likelihood loss with additional likelihood updates on the initial segments of successful continuations. Experiments on mathematical reasoning tasks using Qwen3.5-0.8B and Qwen3.5-2B demonstrate its effectiveness.

Critical strengths

The paper addresses a practical challenge: weak policies often generate too few successful trajectories for effective outcome-based RL. It proposes CRL and demonstrates its effectiveness on mathematical reasoning tasks using two Qwen models, comparing against SFT–RFT, UFT, Prefix-RFT, and R³.

Critical weaknesses
  1. CRL builds on ideas explored in R³, UFT, and Prefix-RFT. The paper should clarify what methodological advance it offers beyond these approaches.
  2. In Table 1, CRL improves over SFT–RFT from 42.76 to 43.47 at 0.8B (+0.71 percentage points) and from 60.63 to 61.24 at 2B (+0.61). These small gains provide limited evidence of its potential benefits at larger model scales.
  3. Small models are often trained through distillation from stronger models. The paper should clarify when CRL would offer a practical advantage over this alternative.
  4. CRL’s overall aggregate advantage does not hold when the same aggregation is restricted to OOD tasks. It also trails SFT–RFT on GSM8K at both sizes. At 2B, its OOD macro-average is approximately 39.48, compared with 39.99 for R³.
Questions for authors

I would appreciate it if the authors could kindly address the concerns raised in the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.