acceptodds
Under review as a conference paper at ICLR 2027

Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning

Abstract

Reinforcement learning (RL) for large language model (LLM) post-training relies on scalar outcome rewards that convey only *whether* the model succeeded, not *why* it failed, forcing blind exploration that stalls in failure-dominated regimes where advantages collapse. Yet rich natural language (NL) feedback, from execution traces to textual critiques, often arises as a byproduct of the reward system itself but remains largely discarded. We propose GOLF, an RL framework that leverages **G**r**O**up-level **L**anguage **F**eedback to operationalize refinement as a learnable *meta-capability*. Rather than refining each failure in isolation, GOLF aggregates external critiques and intra-group attempts into a unified context, surfacing systemic error patterns that broaden solution diversity. The resulting refinements are adaptively injected as off-policy scaffolds in failure-dominated regimes, while generation and refinement are jointly optimized so that improved self-refinement yields stronger scaffolds that accelerate exploration. Across verifiable and non-verifiable tasks, GOLF achieves the best average score in every setting: it surpasses the strongest NL-feedback baseline, Critique-GRPO, by +11.86 points on Llama-3.1-8B-Instruct, and its gains hold consistently as Qwen3 policies scale from 4B to 14B. It also matches the baseline's final performance in up to fewer steps, while sustaining broader exploration and cultivating test-time self-refinement that generation-only RL fails to develop.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Natural Language Feedback

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JJjeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GOLF, an RL framework that leverages group-level natural-language feedback to operationalize refinement as a learnable meta-capability. Specifically, it first aggregates group-level NL feedback including external critiques and intra-group alternative attempts to generate refined responses, which are then adaptively injected as off-policy samples into RL training. The policy's generation and refinement capabilities are jointly optimized so that the two roles co-evolve. Experiments on both verifiable and non-verifiable benchmarks show GOLF outperforming baselines such as GRPO and Critique-GRPO, and a series of ablations further support the conclusion that group-level NL feedback aggregation, combined with joint generator–refiner optimization, delivers consistent performance gains.

Critical strengths
  1. The experimental evaluation of this paper is extensive: it spans verifiable and non-verifiable benchmarks, and includes ablations on reward-model scale, feedback granularity, and step efficiency, providing comprehensive validation of GOLF's effectiveness.

  2. GOLF jointly optimize the generator and refiner within a single RL step, which is a relatively novel design choice.

Critical weaknesses
  1. In terms of novelty, GOLF appears to be a combination of Critique-GRPO and LUFFY. Although the authors highlight two differences: (a) GOLF's NL feedback aggregates both external critiques and intra-group alternative attempts, but the "intra-group alternative attempts" is just concatenating critiques of all rollouts, which is not a substantial distinction. (b) GOLF jointly optimizes the generator and refiner, yet the community already has extensive literature on co-evolutionary training, so this is not particularly novel either.

  2. GOLF concatenates multiple rollout trajectories and their critiques from the same group to form the refinement prompt, which may limit its applicability to agentic tasks. Agentic trajectories are much longer, and packing an entire group of them into a single context would easily exceed the context window.

  3. The reported step efficiency is somewhat misleading. GOLF incorporates an additional refiner rollout, so it rollouts roughly twice as many samples as GRPO, this additional cost is not accounted for in the step-efficiency evaluation.

Questions for authors
  1. The adaptive injection is triggered only when all N rollouts in a group fail. The motivation for this exact threshold is not clearly justified. Why is this design chosen? Is it justified theoretically or experimentally, and how would the results change under a different threshold η?

  2. I noticed that Critique-GRPO reports 60% on AIME24 in its original paper, whereas in this paper Critique-GRPO achieves only 55.49% on AIME24. How do the authors explain this discrepancy? In addition, the numbers for many compared methods cannot be found in their original papers, did the authors retrain these compared methods on the same datasets, and if so, how was fairness in retraining ensured?

Reviewer_haw7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GOLF, an RL framework that co-evolves generation and refinement as a unified meta-capability rather than treating correction as an ad-hoc mechanism. GOLF aggregates group-level feedback from external critiques and intra-group rollouts, employing adaptive off-policy guidance in failure-dominated regimes. Across verifiable and non-verifiable tasks, GOLF achieves superior performance with up to 2.3× faster convergence, sustaining broader exploration and equipping models with robust test-time self-correction.

Critical strengths

this paper introduces GOLF, an RL framework that operationalizes refinement as a learnable meta-capability through group-level NL feedback aggregation, conduct extensive experiments on both verifiable and non-verifiable tasks, comprehensive analysis on exploration dynamics and test-time transfer.

Critical weaknesses

While the empirical gains are convincing, the conceptual novelty is incremental. The framework borrows substantially from Critique-GRPO and LUFFY, and its main distinct feature—concatenating failed rollouts with natural language critiques into the aggregated prompt—is an intuitive engineering design rather than a new algorithmic formulation.

Questions for authors

I would appreciate it if the authors could kindly address the concerns raised in the Critical Weaknesses section.

Reviewer_Q8LMOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds upon Critique-GRPO and LUFFY to address the exploration challenges in RLVR, particularly when models struggle to generate correct responses to difficult problems. The main methodological contribution is to jointly optimize the generator and refiner, while enabling the refiner to leverage aggregated feedback from multiple failed attempts rather than a single response. The authors evaluate the proposed framework across a diverse range of tasks using models from two different families, demonstrating consistent performance improvements over existing baselines.

Critical strengths

The paper evaluates multiple model families and scales across diverse tasks, which is valuable in RLVR domain. Also, the results show promising and consistent improvements over existing baselines, which I think could be valuable on this useful setting of hard exploration.

Critical weaknesses
  1. One key limitation is the lack of an ablation study that removes the training on refiner. I consider explicitly training the refiner to be one of the most novel components of the proposed framework, but its independent contribution to the overall performance is not clearly demonstrated.

  2. The proposed framework combines several components from existing methods, particularly Critique-GRPO and LUFFY, making the overall approach somewhat complicated and less elegant. While jointly training the refiner and generator appears to be the main methodological contribution, I am not fully convinced that this provides sufficient novelty.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.