acceptodds
Under review as a conference paper at ICLR 2027

Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs

Abstract

Reinforcement learning for large language models (LLMs) often relies on scalar rewards, a practice that discards valuable textual rationale buried in the rollouts and hampers training efficiency. While keeping and re-using the textual information may seem intuitive, we show that naive attempts are often counterproductive, risking either memorization from leaked solutions or behavior collapse due to irrelevant context. To address this, we propose Language-And-Numerical Policy Optimization (LANPO), a framework that explicitly separates the roles of feedback: language guides exploration, while numerical rewards drive optimization. Central to LANPO is a dynamically evolving experience pool accumulated from on-policy rollouts, which facilitates two principles to ensure the effectiveness of feedback: Reward-Agnostic Reflection for safe intra-sample self-correction and Relevant Abstraction to distill generalizable lessons from inter-sample experiences. Across the evaluated mathematical reasoning and coding benchmarks, LANPO consistently outperforms strong GRPO-trained baselines for 7B and 14B models in zero-shot accuracy, and further improves performance with feedback-augmented inference at test time.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Reinforcement Learning, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LoH3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  1. This paper proposes LANPO, which combines intra-sample reward-agnostic reflection and inter-sample relevant abstraction to guide exploration, while numerical rewards drive policy optimization.
  2. Experiments demonstrate improved zero-shot performance across mathematical reasoning and coding benchmarks, with further gains from feedback-augmented inference.
Critical strengths
  1. The paper presents a practical framework that uses language feedback to guide exploration and numerical rewards to optimize the policy. Experiments across multiple model sizes and both math and coding tasks show consistent improvements in average zero-shot performance.
  2. The framework mixes feedback-conditioned and from-scratch rollouts during training, allowing the resulting policy to solve problems independently while also benefiting from feedback at inference time.
Critical weaknesses
  1. The benefits of inter-sample feedback may depend on the coverage and density of related problems in the training data, a dependence that is not sufficiently characterized. Meanwhile, intra-sample feedback, which relies less on cross-problem data composition, yields only modest zero-shot gains on math and coding tasks. Given the additional computational cost and pipeline complexity, the current evidence does not clearly establish a compelling cost–benefit trade-off.
  2. The ablations do not sufficiently isolate the source of the gains. Inter-sample feedback combines experience summarization, relevance filtering, structured prompting, and a format reward, but the existing ablations primarily examine filtering. It therefore remains unclear whether the improvements arise from transferable knowledge in the summaries or from the other components.
Questions for authors

Q1: Table 10 reports a comparable-training-time comparison between LANPO-Both and GRPO. Do you also have results comparing LANPO-Intra with GRPO under the same training-time budget?

Q2: How does the effectiveness of inter-sample feedback depend on the availability of closely related training problems? Do the gains persist when highly relevant experiences are scarce?

Reviewer_f5biOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LANPO augments GRPO training with language feedback from an evolving experience pool. It supports reflection on previous attempts at the same problem and retrieval of summarized guidance from related problems. Feedback-conditioned and from-scratch rollouts are mixed during training. An initial SFT stage teaches feedback skills. Experiments cover mathematical reasoning with 4B, 7B, and 14B Qwen models and programming with a 7B model. Results show modest zero-shot gains and additional benefits from feedback-assisted inference in some settings.

Critical strengths
  1. Language guidance and numerical optimization have complementary roles. Sections 4.1 and 4.2 use language to suggest how to approach a problem, while numerical rewards determine which responses to reinforce. This allows previous rollouts to contribute strategies and error information beyond a correctness score. The verifier remains the optimization signal, so the method can explore self-generated advice without treating that advice as ground truth.

  2. Reflection uses information available at deployment. The model revisits its own previous attempt without receiving the gold answer or an oracle verdict. This gives exploration a concrete reasoning path to inspect and revise. Because the same feedback can be constructed at inference, the design avoids relying on assistance that is available only during training.

  3. The training design accommodates both experience reuse and independent reasoning. Summarization aims to preserve strategies that can transfer across problems, while mixing feedback-conditioned and from-scratch rollouts maintains practice in solving without assistance. This gives the framework useful deployment flexibility. A trained policy can operate independently or draw on its feedback skills when relevant experience is available.

Critical weaknesses
  1. The feedback-improvement loop is assumed rather than established. Section 4.2 optimizes answers conditioned on feedback, while Appendix C selects experiences using source correctness, relevance, and recency. Better source answers do not necessarily yield more faithful or transferable summaries. The paper does not show that refreshing the memory improves the usefulness of its contents, leaving the central bootstrapping claim insufficiently supported.

  2. The collapse analysis does not isolate useful strategy transfer. Appendix A.3 uses output distance to characterize context neglect. Yet a model can appropriately ignore guidance it already knows. Conversely, experience-evaluation prompts and header rewards can change the output without changing the underlying strategy. The metric therefore cannot distinguish useful adoption, adaptation, or rejection of advice, limiting its support for the claimed prevention of collapse.

  3. The role of abstraction is not distinguished from relevance filtering. Section 4.1 presents abstraction as central to transfer, but the main inter-sample ablation changes relevance filtering. Finding a related problem and extracting a general strategy are different contributions. Without comparing abstracted and raw guidance under the same retrieval conditions, the gains could come from selecting better examples, leaving the value of abstraction unclear.

  4. The retrieval rule is unclear for an evolving pool. Section 5 precomputes problem relatedness, but Appendix C.1 scores the relevance of the current feedback text. These approaches can select different experiences because similar problems may yield different lessons. The paper does not explain how summary updates affect retrieval scores. This ambiguity concerns the central connection between the changing memory and exploration.

Questions for authors
  1. How is the summarizer trained during RL, and what establishes that its lessons become more transferable as the policy improves?

  2. How does the collapse analysis distinguish failure to use helpful guidance from appropriate rejection or use of an already learned strategy? What evidence separates strategy transfer from changes in response format?

  3. Under the same retrieval and prompting conditions, what does abstraction add over raw solutions from related problems?

  4. Section 5 precomputes problem relatedness, whereas Appendix C.1 scores the current feedback text. Which quantity determines retrieval during training as the summaries evolve?

Reviewer_LwedOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LANPO reuses past solving experience during RL. The model can reflect on its own earlier attempts or use summaries from related problems. New experience is added to a pool during training. This text guides how the model explores, while numerical rewards still drive GRPO-style updates. Training also keeps answers without feedback, so the model can solve alone at test time.

Critical strengths

The method makes useful use of past attempts during RL. The gains remain when no feedback is provided at test time. Results cover math with 7B and 14B models and coding with a 7B model. The AIME24/25 comparison under comparable training time also supports the method.

Critical weaknesses

The feedback and filtering experiments support the full method. But I still cannot tell how much is gained by summarizing old solutions rather than using them directly. It is also unclear how much updating the experience pool helps compared with keeping a fixed pool.

Questions for authors

Could the authors quantify the separate contributions of the summaries and the online experience pool to the final performance while keeping all other parameters fixed?

Comments on the paper

2 comments

Thank you for the authors' responses. Most of my concerns are solved, and I have increased my score. One concerns still remain:

Although the retrieval helps in the settings the authors test, I’m still unsure whether it prevents behavior collapse. A model could comment on the retrieved example and then solve the problem the same way it would have without it. That alone could increase edit distance.

AuthorsOpenReview ↗

Thank you for your continued effort!

Dear Reviewer f5bi,

Thank you again for taking the time to revisit our responses and for raising your score. We are glad that most of your concerns have been addressed, and we appreciate your continued help in clarifying the remaining point. Below, we examine your point directly by auditing the raw responses.


We uniformly sampled 1,000 guidance-exposed responses from each of two Qwen2.5-7B policies: LANPO-Inter and its variant trained without relevance filtering. A GPT-6 Luna judge received the complete problem, retrieved guidance, generated solution, and the corresponding no-guidance solution. Policy identity and existing correctness labels were hidden.

Following the distinction you highlighted, we asked the judge to look for a substantive mathematical step or solution route that differed from its no-guidance counterpart and implemented a specific idea from the retrieved guidance. Merely mentioning, paraphrasing, or promising to use the experience did not satisfy this criterion.

Solution-level criterionLANPO-InterWithout training relevance filtering
Substantive reasoning change corresponding to retrieved guidance315/1,000 (31.5%)144/1,000 (14.4%)
Of these (the first row), changes judged to be appropriate use of the guidance211/1,000 (21.1%)90/1,000 (9.0%)

In particular, 21.1% of LANPO-Inter responses exhibit both appropriate use and a substantive reasoning change tied to the guidance, compared with 9.0% for the no-filter policy—an absolute difference of 12.1 percentage points in these sampled outputs. These cases provide more direct evidence on the point you raised: the observed changes involve the mathematical solution itself and correspond to the supplied guidance.


Our earlier tests show test-time retrieval increases average accuracy by 2.69 percentage points on Qwen2.5-7B mathematics and 2.32 points on coding. The new audit examines the mathematical behavior accompanying that performance effect: it identifies changes within the solution itself that correspond to the supplied guidance.

These results demonstrate that LANPO can act on test-time guidance within its mathematical reasoning, including appropriate changes to the solution procedure.

We hope this additional evidence helps address your remaining concern. Thank you again for the time and care you have put into reviewing our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.