- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work introduces Goldilocks, a method for estimating the standard deviation of rewards for candidate questions during model rollouts. It is used to filter for questions of medium difficulty, which are then employed to train GRPO. The authors conducted experiments on two benchmarks, demonstrating consistent improvements over standard GRPO while also reducing training time.
Critical strengths
-
The author uses a selector to predict reward variance, prior to rollout generation, to choose which problems to address. This approach generalizes to unseen problems and dynamically adapts as model capabilities evolve.
-
The paper provides a theoretical motivation for GRPO based on its binary reward gradient and zero-variance batch properties, demonstrating that problems of moderate difficulty can yield stronger learning signals.
-
They claim to consistently outperform GRPO across multiple models on OpenMathReasoning and Polaris, while reducing the number of optimization steps by up to 78% in certain settings.
Critical weaknesses
-
It is a well-established consensus that problems of moderate difficulty can enhance model performance; this conclusion is trivial. What, then, are the advantages of the proposed method compared to direct offline rollouts?
-
The experiments cover only mathematical reasoning and binary verifiable rewards; scenarios such as code generation, theorem proving, or noisy/graded rewards have not yet been evaluated.
-
This method requires additional GPU resources to run and update the Selector, and it relies on an online replay buffer and continuous feedback; consequently, its end-to-end cost and long-term scalability remain limited.
Questions for authors
-
Does the reward variance predicted by the selector truly correspond to an effective learning signal for GRPO? How can this be demonstrated?
-
Do the candidate pool size and -greedy exploration introduce selection bias?
-
Can the method be extended to non-binary rewards or non-mathematical tasks?
3 comments
Q2: Do the candidate pool and -greedy exploration introduce selection bias?
Goldilocks intentionally biases selection toward informative questions, while maintaining broad coverage of the training distribution. Rather than selecting globally from the highest-scoring questions, the Selector chooses among randomly sampled candidates. Furthermore, with probability , a candidate is selected uniformly at random, ensuring that all questions retain a nonzero probability of selection and preventing the Selector from relying exclusively on its current predictions.
Our ablations across different candidate-pool sizes and exploration probabilities (Table 3) show that Goldilocks consistently improves over GRPO, indicating that its effectiveness does not depend on a narrowly tuned selection policy. Moreover, improvements suggest that the adaptive curriculum supports generalization rather than merely favoring a restricted subset of training questions.
We thank the reviewer again would be happy to answer any further questions or concerns.
Thanks for your experiments and your effort! The authors have addressed most of my concerns, so I have decided to raise my rating.
We thank the reviewer for the feedback. We address each point below and add the new experiments to the revised manuscript (Appendix M).
W1, Q1: Why online selection rather than offline rollouts, and does it improve the GRPO learning signal?
We agree that the benefit of intermediate-difficulty questions is well established. Our contribution is to identify such questions adaptively and before rollout generation, particularly in large-scale settings where question-specific histories are limited and difficulty changes as the Model improves.
Offline difficulty estimation requires generating rollouts for the entire dataset before training. For our 812K-question pool, using eight rollouts per question would require approximately 6.5 million additional generations, equivalent to 23% of the rollouts in a full GRPO run. Moreover, these estimates remain fixed as the Model's capabilities evolve.
To evaluate this, we add an offline curriculum (Appendix M.3, Figure 26) that selects questions solved 2--6 times out of eight initial rollouts. At 9,200 training steps, Goldilocks achieves 40.5% validation accuracy, compared with 37.2% for GRPO and 31.1% for offline selection, even before accounting for the additional offline generation cost. This demonstrates the importance of continuously adapting the curriculum.
We also directly examine whether Goldilocks provides stronger GRPO learning signals by comparing the Model's gradient norms with and without the Selector, alongside the fraction of zero-variance rollout groups.
Goldilocks consistently produces the lowest fraction of zero-variance groups and the largest average gradient norms across all five models. Notably, it also produces larger gradients than PCL, which uses the same Model batch size. Together with the improvements in validation accuracy, these findings support our hypothesis that adaptive reward-variability-based selection yields more informative GRPO updates. We have added these analyses to the revised manuscript (Appendix M.2, Table 8, Figure 25).
W2/Q3: Generalization to other domains and rewards.
We extend our evaluation to AppWorld (Appendix M.4, Table 9 and Figure 27), a multi-turn agentic benchmark where the Model interacts with applications through Python APIs and receives verifiable feedback from state-based unit tests. Goldilocks is applied without modifying its selection mechanism.
Goldilocks improves task completion on both models, with particularly strong gains on Qwen3.5-9B, including a 6.7 percentage-point improvement on Test-Challenge. These results demonstrate that adaptive reward-variability-based selection also benefits multi-turn agentic tasks beyond mathematical reasoning.
For more granular rewards, the method naturally extends by replacing p^(1−p^) with the empirical standard deviation of rollout rewards. The binary target is a special case of this general formulation, requiring no change to the selection framework. Note that even if a partial reward (e.g., 0.5) is assigned to all the generation group of a GRPO question, there would still be no gradient.
W3: Computational cost and scalability.
Goldilocks operates under the same eight-GPU hardware budget as GRPO, allocating seven GPUs to Model training and one to the Selector. Importantly, the Selector is trained in parallel with the Model, so its updates do not introduce additional sequential overhead to Model optimization. Despite using only 7/8 of GRPO's training batch per optimization step, Goldilocks consistently achieves higher validation accuracy across all five models.
The Selector is lightweight, using a 0.6B-parameter network and a replay buffer of only 128 samples. Its fixed one-GPU allocation represents a decreasing fraction of the total hardware budget as training scales. Our experiments on a large pool further demonstrate that the improvements persist throughout extended training, supporting the scalability of Goldilocks.