acceptodds
Under review as a conference paper at ICLR 2027

Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoning

Abstract

Reinforcement learning has emerged as a powerful paradigm for unlocking reasoning capabilities in language models. However, relying on sparse rewards makes this process highly sample-inefficient, as models must navigate vast search spaces with minimal feedback. While classic curriculum learning aims to mitigate this by ordering data based on complexity, prior works have primarily targeted small datasets and do not directly transfer to the large-scale settings typical of modern language model training. Furthermore, the right ordering for a specific model is often unclear. To address this, we propose Goldilocks, an adaptive data-selection strategy that uses a Selector network to predict the standard deviation of rewards across the model's rollouts for each candidate question. The Selector prioritizes questions with high predicted reward variability, corresponding to questions that are neither too easy nor too hard for the model’s current capabilities (Goldilocks principle), while training the model with GRPO. By leveraging the model's performance on seen samples, the Selector continuously adapts to the model's evolving abilities. Across the OpenMathReasoning and Polaris datasets, Goldilocks consistently improves over standard GRPO, requiring up to 78% fewer optimization steps to reach the corresponding GRPO performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language models, Reinforcement Learning, Curriculum learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_4156OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces Goldilocks, a method for estimating the standard deviation of rewards for candidate questions during model rollouts. It is used to filter for questions of medium difficulty, which are then employed to train GRPO. The authors conducted experiments on two benchmarks, demonstrating consistent improvements over standard GRPO while also reducing training time.

Critical strengths
  1. The author uses a selector to predict reward variance, prior to rollout generation, to choose which problems to address. This approach generalizes to unseen problems and dynamically adapts as model capabilities evolve.

  2. The paper provides a theoretical motivation for GRPO based on its binary reward gradient and zero-variance batch properties, demonstrating that problems of moderate difficulty can yield stronger learning signals.

  3. They claim to consistently outperform GRPO across multiple models on OpenMathReasoning and Polaris, while reducing the number of optimization steps by up to 78% in certain settings.

Critical weaknesses
  1. It is a well-established consensus that problems of moderate difficulty can enhance model performance; this conclusion is trivial. What, then, are the advantages of the proposed method compared to direct offline rollouts?

  2. The experiments cover only mathematical reasoning and binary verifiable rewards; scenarios such as code generation, theorem proving, or noisy/graded rewards have not yet been evaluated.

  3. This method requires additional GPU resources to run and update the Selector, and it relies on an online replay buffer and continuous feedback; consequently, its end-to-end cost and long-term scalability remain limited.

Questions for authors
  1. Does the reward variance predicted by the selector truly correspond to an effective learning signal for GRPO? How can this be demonstrated?

  2. Do the candidate pool size KK and ϵ\epsilon-greedy exploration introduce selection bias?

  3. Can the method be extended to non-binary rewards or non-mathematical tasks?

3 comments

We thank the reviewer for the feedback. We address each point below and add the new experiments to the revised manuscript (Appendix M).

W1, Q1: Why online selection rather than offline rollouts, and does it improve the GRPO learning signal?

We agree that the benefit of intermediate-difficulty questions is well established. Our contribution is to identify such questions adaptively and before rollout generation, particularly in large-scale settings where question-specific histories are limited and difficulty changes as the Model improves.

Offline difficulty estimation requires generating rollouts for the entire dataset before training. For our 812K-question pool, using eight rollouts per question would require approximately 6.5 million additional generations, equivalent to 23% of the rollouts in a full GRPO run. Moreover, these estimates remain fixed as the Model's capabilities evolve.

To evaluate this, we add an offline curriculum (Appendix M.3, Figure 26) that selects questions solved 2--6 times out of eight initial rollouts. At 9,200 training steps, Goldilocks achieves 40.5% validation accuracy, compared with 37.2% for GRPO and 31.1% for offline selection, even before accounting for the additional offline generation cost. This demonstrates the importance of continuously adapting the curriculum.

We also directly examine whether Goldilocks provides stronger GRPO learning signals by comparing the Model's gradient norms with and without the Selector, alongside the fraction of zero-variance rollout groups.

MetricMethodOLMo2-1BQwen2.5-1.5BQwen3-4BPhi-4-miniQwen2.5-7B
Zero-variance (%) ↓GRPO74.557.139.935.541.4
PCL90.863.340.132.544.4
Goldilocks61.833.916.916.421.5
Gradient norm ↑GRPO0.1950.1740.2460.1230.342
PCL0.2480.1880.2780.1520.427
Goldilocks0.4430.2310.3430.2140.455

Goldilocks consistently produces the lowest fraction of zero-variance groups and the largest average gradient norms across all five models. Notably, it also produces larger gradients than PCL, which uses the same Model batch size. Together with the improvements in validation accuracy, these findings support our hypothesis that adaptive reward-variability-based selection yields more informative GRPO updates. We have added these analyses to the revised manuscript (Appendix M.2, Table 8, Figure 25).

W2/Q3: Generalization to other domains and rewards.

We extend our evaluation to AppWorld (Appendix M.4, Table 9 and Figure 27), a multi-turn agentic benchmark where the Model interacts with applications through Python APIs and receives verifiable feedback from state-based unit tests. Goldilocks is applied without modifying its selection mechanism.

ModelMethodTest-Normal (%)Test-Challenge (%)
Qwen3.5-4BGRPO70.144.3
Qwen3.5-4BGoldilocks73.044.4
Qwen3.5-9BGRPO75.151.1
Qwen3.5-9BGoldilocks78.957.8

Goldilocks improves task completion on both models, with particularly strong gains on Qwen3.5-9B, including a 6.7 percentage-point improvement on Test-Challenge. These results demonstrate that adaptive reward-variability-based selection also benefits multi-turn agentic tasks beyond mathematical reasoning.

For more granular rewards, the method naturally extends by replacing p^(1−p^)\sqrt{\hat p(1-\hat p)} with the empirical standard deviation of rollout rewards. The binary target is a special case of this general formulation, requiring no change to the selection framework. Note that even if a partial reward (e.g., 0.5) is assigned to all the generation group of a GRPO question, there would still be no gradient.

W3: Computational cost and scalability.

Goldilocks operates under the same eight-GPU hardware budget as GRPO, allocating seven GPUs to Model training and one to the Selector. Importantly, the Selector is trained in parallel with the Model, so its updates do not introduce additional sequential overhead to Model optimization. Despite using only 7/87/8 of GRPO's training batch per optimization step, Goldilocks consistently achieves higher validation accuracy across all five models.

The Selector is lightweight, using a 0.6B-parameter network and a replay buffer of only 128 samples. Its fixed one-GPU allocation represents a decreasing fraction of the total hardware budget as training scales. Our experiments on a large pool further demonstrate that the improvements persist throughout extended training, supporting the scalability of Goldilocks.

Q2: Do the candidate pool and ε\varepsilon-greedy exploration introduce selection bias?

Goldilocks intentionally biases selection toward informative questions, while maintaining broad coverage of the training distribution. Rather than selecting globally from the highest-scoring questions, the Selector chooses among K=8K=8 randomly sampled candidates. Furthermore, with probability ε=0.2\varepsilon=0.2, a candidate is selected uniformly at random, ensuring that all questions retain a nonzero probability of selection and preventing the Selector from relying exclusively on its current predictions.

Our ablations across different candidate-pool sizes and exploration probabilities (Table 3) show that Goldilocks consistently improves over GRPO, indicating that its effectiveness does not depend on a narrowly tuned selection policy. Moreover, improvements suggest that the adaptive curriculum supports generalization rather than merely favoring a restricted subset of training questions.

We thank the reviewer again would be happy to answer any further questions or concerns.

Thanks for your experiments and your effort! The authors have addressed most of my concerns, so I have decided to raise my rating.

Reviewer_VJyDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Goldilocks uses an online Selector to predict reward variability and prioritize questions likely to produce both correct and incorrect answers before generating rollouts. Experiments across five models show improvements over GRPO, with one setting also demonstrating wall-clock and token savings after accounting for Selector overhead. The approach is practical, but the experiments do not yet clearly isolate the benefit of predicting variability or establish its advantage over existing difficulty-aware methods. The data split and the connection between the theoretical analysis and finite-group GRPO also need clarification.

Critical strengths
  1. Useful selection mechanism. Making the decision before rollout can avoid spending generation compute on uninformative questions. The Selector reuses feedback already collected for policy training, keeping the procedure straightforward.

  2. Consistent empirical gains. The results span several model families, and the four-seed experiment reports 44.46 ± 0.30 versus 40.33 ± 0.58 for GRPO. This supports the improvement beyond a single run.

  3. Attention to practical cost. The Selector shares the eight-GPU budget with the policy. For Qwen2.5-7B on OpenMathReasoning, the paper reports 46% less training time and 51% fewer generated tokens to reach the GRPO reference accuracy.

Critical weaknesses
  1. Relevant comparisons are missing. Table 3 reports “DAPO Loss,” but it is unclear whether this includes dynamic sampling with filtering and batch replenishment. A loss-only ablation would not test that mechanism. The evaluation also lacks GHPO, SPEED-RL, and DOTS comparisons. GHPO uses partial solution guidance for all-incorrect groups [Liu et al., 2025], while SPEED-RL and DOTS reduce wasted generation through screening or difficulty prediction [Zhang et al., 2025; Sun et al., 2025]. These alternatives matter for assessing what Goldilocks adds to existing approaches.

  2. The target's contribution remains unclear. PCL already predicts question difficulty before rollout. Appendix K gives a plausible reason to predict variability instead, but the comparison also changes Selector training settings, including the learning rate. Table 3 does not include an ablation that changes only the prediction target and corresponding selection rule.

  3. The evaluation leaves questions about generalization. Appendix D describes disjoint 5,000-question validation sets, but does not specify deduplication, grouping of repeated solution records, or the number of unique training questions. These details matter for the large-pool claim in Sec. 5. Validation accuracy is also used for tuning, and the reported results do not include a separate final test set or external math benchmarks.

  4. The mathematical argument needs a clearer scope. Sec. 3 substitutes population reward moments for the empirical group statistics used by GRPO. Eq. 2 retains a question-dependent conditional-gradient term, so reward variability alone does not rank actual gradient magnitudes. Eq. 3 is an upper bound under bounded token gradients, not a guarantee of faster learning. Appendix K correctly identifies the learned target as expected empirical variability, but its informative-group bounds do not establish a ranking of learning progress. The authors acknowledge some of these limits; the distinction should remain explicit throughout.

Questions for authors
  1. Add a focused baseline study. Compare with GHPO, full DAPO dynamic sampling, and at least one of SPEED-RL or DOTS on a representative model-dataset pair. Match initialization and hardware/time budgets, and count rejected rollouts and auxiliary-model costs. For GHPO, account for solution-trace supervision with an equally guided control. Report accuracy without hints, including on a fixed held-out subset that the initial model rarely solves.

  2. Isolate the prediction target. Keep the Goldilocks backbone, replay buffer, update schedule, exploration, and candidate budget fixed; replace the variability target with success-rate prediction and selection near 0.5. Compare accuracy, zero-variance fraction, and elapsed time.

  3. Make the evaluation reproducible. Document the dataset revision, unique-question counts, and grouping rule used before splitting. Add an untouched test set and a small external benchmark suite. Specify the verifier, decoding settings, and Selector update cadence needed to reproduce these results.

  4. Connect the analysis to the implemented update. State explicitly where population normalization approximates finite-group GRPO. Clarify that Eq. 3 motivates avoiding extreme success probabilities without proving optimal prompt ordering. An empirical check relating Selector scores to actual gradient norms would help assess this approximation.

2 comments

We thank the reviewer for the suggestions. We conducted additional baseline comparisons, controlled target ablations, and external evaluations to clarify the efficiency, generalization, and theoretical scope of Goldilocks.

W1, Q1: Comparison with additional RL methods.

We extend our evaluation to full DAPO dynamic sampling, SPEED-RL, and GHPO, as well as combinations of Goldilocks with SPEED-RL and GHPO. Importantly, these methods are not curriculum-learning approaches but address complementary aspects of RL efficiency. SPEED-RL reduces generation effort on uninformative questions through rollout screening, but still incurs the cost of sampling them. This overhead can be particularly significant in settings involving long reasoning trajectories or tool execution, where even preliminary rollouts can be expensive. GHPO instead provides hints to help the model solve difficult questions.

Since full solution traces are unavailable in our large-scale setting, we use the ground-truth final answer as the hint, while keeping evaluation hint-free. Unlike these methods, Goldilocks learns to prioritize informative questions before rollout generation. Therefore, SPEED-RL and GHPO can be applied on top of curriculum methods such as Goldilocks, making their mechanisms complementary rather than direct alternatives. We evaluate these combinations under matched initialization and an eight-GPU hardware budget.

OLMo2-1B

MethodAcc. (%)Steps
GRPO11.815,200
DAPO dynamic sampling13.13,000
SPEED-RL9.32,200
GHPO10.49,000
Goldilocks15.312,200
SPEED-RL + Goldilocks17.64,400
GHPO + Goldilocks18.19,400

Held-out accuracy averaged over the final five evaluations and training steps reached at matched training time (86 h).

These results highlight the distinction between optimization-step efficiency and end-to-end training efficiency. While DAPO and SPEED-RL filter uninformative rollout groups, Goldilocks selects promising questions before generation, allowing it to achieve higher accuracy than either method under the same wall-clock budget.

The combination experiments further demonstrate complementarity. Adding Goldilocks to SPEED-RL improves accuracy from 9.3% to 17.6%, while combining it with GHPO improves accuracy from 10.4% to 18.1%, the highest among all compared methods. These results show that adaptive curriculum selection can be effectively combined with existing RL efficiency techniques.

W2, Q2: Isolating the prediction target.

We performed the controlled comparison described in our response to Reviewer vJBf. With all other Selector settings fixed, success-probability and reward-variability regression perform comparably well, while the informative-group proxy is also competitive. These results show that the adaptive selection procedure is an important contributor to Goldilocks' gains over PCL.

W3, Q3: Reproducibility.

We thank the reviewer for this suggestion. We have expanded the reproducibility details in the revised manuscript (Appendix M.5). Our dataset contains 812,009 questions, from which we randomly sample a fixed test set of 5,000 questions. We explicitly verify that there is no overlap between the training and test sets, and use the same test set across all methods to ensure consistent comparisons. We also extended our evaluation to the AppWorld dataset, which provides official test splits (Appendix M.4 and our response to Reviewer 4156, W2/Q3).

We use math-verify to verify final answers. Evaluation uses greedy decoding. The Selector is updated after every four new labels using a replay buffer of 128 examples, with learning rate 10−510^{-5}, candidate pool size K=8K=8, and exploration probability ϵ=0.2\epsilon=0.2. We tuned only the learning rate and batch size of vanilla GRPO to establish a strong baseline. These selected settings were then reused across all methods, without further method-specific tuning.

W4, Q4: Scope of the theoretical analysis.

We agree that the theoretical motivation should be distinguished from the implemented finite-group GRPO updates. Section 3 uses population reward moments as an idealization of group normalization. The resulting gradient bound motivates avoiding extreme success probabilities but does not establish an optimal ordering of prompts.

To empirically validate the proposed mechanism, we added a comparison of the Model's gradient norms during training with and without the Goldilocks Selector. Goldilocks consistently produces larger gradient norms throughout training than vanilla GRPO, indicating stronger optimization signals from the selected questions. We also measure the fraction of zero-variance rollout groups to assess how frequently each method produces informative GRPO updates.

MetricMethodOLMo2-1BQwen2.5-1.5BQwen3-4BPhi-4-miniQwen2.5-7B
Zero-variance (%) ↓GRPO74.557.139.935.541.4
PCL90.863.340.132.544.4
Goldilocks61.833.916.916.421.5
Gradient norm ↑GRPO0.1950.1740.2460.1230.342
PCL0.2480.1880.2780.1520.427
Goldilocks0.4430.2310.3430.2140.455

Goldilocks achieves the lowest zero-variance fraction and the largest average gradient norm across all five models. Importantly, Goldilocks also produces larger gradients than PCL, which uses the same Model batch size. Together with the improved validation accuracy, these results provide empirical evidence that Goldilocks selects questions yielding more informative GRPO updates, although gradient magnitude alone does not guarantee improved learning. We include this analysis and clarify the theoretical assumptions in the revised manuscript (Appendix M.2, Figure 25 and Table 8).

We thank the reviewer again would be happy to answer any further questions or concerns.

Comments on the paper

2 comments

We thank the reviewer for the constructive suggestions. We conducted additional experiments to isolate the contribution of the Selector's prediction target and investigate alternative predictors for the usefulness of questions.

W1, W2, Q1, Q2: Comparison of prediction targets.

We compare three targets within the same Goldilocks framework: success probability pp, reward variability p(1−p)\sqrt{p(1-p)}, and the zero-variance probability proxy pG+(1−p)Gp^G+(1-p)^G. We keep the Selector architecture, replay buffer, learning rate, update schedule, candidate pool, and exploration probability fixed, changing only the prediction target and its corresponding selection rule.

MethodOLMo2-1BQwen2.5-1.5BPhi-4-mini
GRPO11.434.239.3
PCL12.932.836.8
Goldilocks (Success probability)14.837.443.3
Goldilocks (Informative-group proxy)17.036.044.1
Goldilocks (variability)15.036.843.0

Held-out accuracy (%), averaged over the final five evaluations at matched training steps (11,600 / 6,400 / 11,600).

The results provide two insights. First, success-probability and reward-variability regression achieve comparable accuracy under identical training and selection settings. Both improve over PCL, highlighting the importance of Goldilocks' adaptive selection procedure beyond the particular prediction target.

Second, the informative-group proxy achieves the highest accuracy on two of the three models and remains competitive on the third. This supports our central motivation: prioritizing questions that are likely to produce informative GRPO rollout groups. We included this comparison in Appendix M.1 (Table 7 and Figure 24) and clarify that reward variability is one effective choice within this broader framework. We thank the reviewer again for their useful suggestion.

We thank the reviewer and would be happy to answer any further questions or concerns.

As the discussion period draws to a close, we would like to sincerely thank all reviewers for their constructive feedback. We have conducted several additional experiments to address the concerns raised and incorporated the results into the revised manuscript (Appendix M). We summarize the main findings below.

1. Prediction-Target Ablations

We compare reward variability, success probability, and informative-group prediction under identical Selector settings.

MethodOLMo2-1BQwen2.5-1.5BPhi-4-mini
GRPO11.434.239.3
PCL12.932.836.8
Success probability14.837.443.3
Informative-group proxy17.036.044.1
Goldilocks (variability)15.036.843.0

The results show that different prediction targets perform competitively under the same adaptive selection framework, highlighting the importance of the selection procedure itself.

2. Additional RL Baselines

We extend our comparisons to full DAPO dynamic sampling, SPEED-RL, and GHPO, including combinations with Goldilocks.

MethodOLMo2-1B Accuracy (%)Training Steps
GRPO11.815,200
DAPO dynamic sampling13.13,000
SPEED-RL9.32,200
GHPO10.49,000
Goldilocks15.312,200
SPEED-RL + Goldilocks17.64,400
GHPO + Goldilocks18.19,400

Goldilocks achieves higher accuracy than the individual baselines, while combining it with SPEED-RL or GHPO leads to further improvements. These results demonstrate that adaptive curriculum selection is complementary to existing RL efficiency techniques.

3. GRPO Learning Signals

We additionally measure zero-variance rollout groups and policy-gradient norms to investigate whether Goldilocks selects questions that provide more informative GRPO updates.

MetricMethodOLMo2-1BQwen2.5-1.5BQwen3-4BPhi-4-miniQwen2.5-7B
Zero-variance (%) ↓GRPO74.557.139.935.541.4
PCL90.863.340.132.544.4
Goldilocks61.833.916.916.421.5
Gradient norm ↑GRPO0.1950.1740.2460.1230.342
PCL0.2480.1880.2780.1520.427
Goldilocks0.4430.2310.3430.2140.455

Across all five models, Goldilocks consistently produces fewer zero-variance groups and larger average gradient norms, supporting our hypothesis that the selected questions provide stronger GRPO optimization signals.

4. Offline Curriculum Comparison

We compare Goldilocks against an offline curriculum that selects questions of intermediate difficulty using initial Model rollouts.

MethodQwen2.5-1.5B Accuracy (%)
Offline curriculum + GRPO31.1
GRPO37.2
Goldilocks40.5

Goldilocks substantially outperforms offline selection, demonstrating the importance of continuously adapting question difficulty as the Model improves.

5. Generalization Beyond Mathematics

We extend Goldilocks to AppWorld, a multi-turn agentic benchmark involving Python tool execution and verifiable task completion.

ModelMethodTest-Normal (%)Test-Challenge (%)
Qwen3.5-4BGRPO70.144.3
Goldilocks73.044.4
Qwen3.5-9BGRPO75.151.1
Goldilocks78.957.8

Goldilocks improves task completion on both models, with particularly strong gains on Qwen3.5-9B, including a 6.7 percentage-point improvement on Test-Challenge. These results demonstrate the applicability of adaptive question selection beyond mathematical reasoning.

We have also expanded the reproducibility details (Appendix M.5) and clarified the scope and assumptions of our theoretical analysis.

We sincerely appreciate the reviewers' time and thoughtful suggestions. We hope these additional results and clarifications have addressed their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.