- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GOLF, an RL framework that leverages group-level natural-language feedback to operationalize refinement as a learnable meta-capability. Specifically, it first aggregates group-level NL feedback including external critiques and intra-group alternative attempts to generate refined responses, which are then adaptively injected as off-policy samples into RL training. The policy's generation and refinement capabilities are jointly optimized so that the two roles co-evolve. Experiments on both verifiable and non-verifiable benchmarks show GOLF outperforming baselines such as GRPO and Critique-GRPO, and a series of ablations further support the conclusion that group-level NL feedback aggregation, combined with joint generator–refiner optimization, delivers consistent performance gains.
Critical strengths
-
The experimental evaluation of this paper is extensive: it spans verifiable and non-verifiable benchmarks, and includes ablations on reward-model scale, feedback granularity, and step efficiency, providing comprehensive validation of GOLF's effectiveness.
-
GOLF jointly optimize the generator and refiner within a single RL step, which is a relatively novel design choice.
Critical weaknesses
-
In terms of novelty, GOLF appears to be a combination of Critique-GRPO and LUFFY. Although the authors highlight two differences: (a) GOLF's NL feedback aggregates both external critiques and intra-group alternative attempts, but the "intra-group alternative attempts" is just concatenating critiques of all rollouts, which is not a substantial distinction. (b) GOLF jointly optimizes the generator and refiner, yet the community already has extensive literature on co-evolutionary training, so this is not particularly novel either.
-
GOLF concatenates multiple rollout trajectories and their critiques from the same group to form the refinement prompt, which may limit its applicability to agentic tasks. Agentic trajectories are much longer, and packing an entire group of them into a single context would easily exceed the context window.
-
The reported step efficiency is somewhat misleading. GOLF incorporates an additional refiner rollout, so it rollouts roughly twice as many samples as GRPO, this additional cost is not accounted for in the step-efficiency evaluation.
Questions for authors
-
The adaptive injection is triggered only when all N rollouts in a group fail. The motivation for this exact threshold is not clearly justified. Why is this design chosen? Is it justified theoretically or experimentally, and how would the results change under a different threshold η?
-
I noticed that Critique-GRPO reports 60% on AIME24 in its original paper, whereas in this paper Critique-GRPO achieves only 55.49% on AIME24. How do the authors explain this discrepancy? In addition, the numbers for many compared methods cannot be found in their original papers, did the authors retrain these compared methods on the same datasets, and if so, how was fairness in retraining ensured?