acceptodds
Under review as a conference paper at ICLR 2027

Language Models Can Learn from Verbal Feedback Without Scalar Rewards

Abstract

LLMs are often trained with RL from human or AI feedback, yet such methods typically compress nuanced feedback into scalar rewards, discarding much of their richness and inducing scale imbalance. We propose treating verbal feedback as a conditioning signal. Inspired by language priors in text-to-image generation, which enable novel outputs from unseen prompts, we introduce the **feedback-conditional policy (FCP)**. FCP learns directly from response–feedback pairs, approximating the feedback-conditional posterior through maximum likelihood training on *offline* data. We further develop an *online* bootstrapping stage where the policy generates under positive conditions and receives fresh feedback to refine itself. This reframes feedback-driven learning as conditional generation rather than reward optimization, offering a more expressive way for LLMs to directly learn from verbal feedback.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Models, Verbal Feedback, Reinforcement Learning, Feedback-Conditional Policy

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_LGWDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the feedback-conditional policy (FCP), which uses verbal feedback as a conditioning signal instead of compressing it into a scalar reward.

  • Offline stage. FCP fits the feedback-conditional posterior Poff(o∣x,c)∝πref(o∣x) penv(c∣x,o)P_{\text{off}}(o\mid x,c)\propto\pi_{\text{ref}}(o\mid x)\,p_{\text{env}}(c\mid x,o) by maximum likelihood on (instruction, response, critique) triples, with the critique prepended to the prompt (Alg. 1, Eq. 1).
  • Online stage. The model samples under a positive condition c+c^+ drawn from a pool, has each response re-critiqued, and trains on the response paired with its actual critique (Alg. 2, Eq. 2).

Experiments use Qwen2.5-7B-Base with GPT-5-nano as critic. On five math benchmarks, FCP+Bootstrap reaches 38.7, against 36.4–38.4 for four GRPO variants (Table 1). On general reasoning it reaches 47.8, against 47.5 for RFT+GRPO (Table 2). Results on Qwen3-4B and Llama-3.1-8B are similar (Table 10). In creative writing, with a Qwen3.5-122B critic and an independent LitBench reward model as evaluator, FCP+Bootstrap is preferred over Base+GRPO and RFT+GRPO in 68.0% and 62.7% of head-to-head comparisons (Table 3). Further analyses cover:

  • controllability under different conditions (Table 4);
  • feedback quality (Table 5);
  • textualized-scalar and no-feedback controls (Table 6);
  • a weaker 1.5B critic (Table 7);
  • out-of-distribution test-time feedback (Table 12).
Critical strengths

1. A simple mechanism that uses all feedback, including mixed and negative, and matches strong RL baselines at similar cost. FCP needs only likelihood training on critique-annotated samples, without scalar reward computation, advantage estimation, or data filtering in the policy update. It still reaches GRPO-level accuracy on math (38.7 vs. 38.4, Table 1) and general reasoning (47.8 vs. 47.5, Table 2), and stays within 1.6 points of the best GRPO baseline on two other backbones (Table 10). Training time is comparable (7.1 h vs. 7.8 h for GRPO, Table 14). In creative writing, where response quality has multiple dimensions, an independent reward model prefers FCP+Bootstrap over both GRPO baselines (68.0% / 62.7% head-to-head, Table 3).

2. Controlled evidence that the verbal content matters, not just the conditioning mechanism. The three conditioning formats behave almost identically offline (28.1–29.8). Online, they diverge sharply: +10.2 with verbal critiques, +3.8 with textualized scores, and +0.8 with no feedback (Table 6). Richer critiques also beat binary labels (38.7 vs. 31.6, Table 5), and a critic weaker than the policy still helps (33.1 with a 1.5B critic, Table 7). The controllability study (Table 4: 68.5 under positive vs. 17.1 under negative conditions on MATH500) shows that the model actually uses the condition. These controls provide useful evidence for the contribution of verbal feedback beyond the conditioning mechanism alone.

Critical weaknesses

1. The relationship to prior methods and the scope of the claims would benefit from clearer positioning.

  • Positioning relative to closely related work. Feedback-conditioned likelihood training with hindsight relabeling has close precedents that are not cited. Chain of Hindsight (Liu et al., ICLR 2024) trains on responses paired with natural-language feedback. Quark (Lu et al., NeurIPS 2022) uses a related online loop with quantized reward tokens: sample under a high-reward token, re-score, relabel, and retrain. HIR (Zhang et al., ICML 2023) relabels instructions in hindsight. SteerLM (Dong et al., 2023) conditions on attribute labels. The Textualized-Scalar control in Table 6 resembles a Quark-style approach. Clarifying the differences would help establish what is new about using free-form critique as the condition; the margin over this control is particularly informative.
  • The "without relying on verifiers" claim (p. 2) needs qualification for the reasoning experiments. The GPT-5-nano prompt receives the reference final answer and judges correctness against it (Fig. 5). Although the policy update does not require scalar verifier rewards, its verbal feedback still carries reference-based correctness information. These experiments therefore do not establish learning without such supervision.
  • The population-level characterization leaves a generalization question. The optimum π∗=Poff(o∣x,c)\pi^*=P_{\text{off}}(o\mid x,c) is characterized on supp Poff(c∣x)\text{supp}\,P_{\text{off}}(c\mid x). This does not by itself establish finite-sample generalization when positive feedback conditions are transferred across prompts. It would be helpful to clarify the assumptions and empirical evidence supporting this use of the learned conditional policy.

2. More closely matched comparisons would help establish the benefits of verbal feedback over scalar rewards.

  • Reasoning. FCP+Bootstrap has similar reported point estimates to the strongest GRPO baselines (38.7 vs. 38.4; 47.8 vs. 47.5), but the number of independent training runs per method is not specified. Table 9 reports six FCP configurations, including the main configuration at 38.7 and a default at 35.3. The paper does not explain how the main configuration was selected or whether GRPO received a comparable tuning budget. The stated protocol describes repeated evaluation with greedy decoding, but does not make clear whether the reported ± include variation across independent training runs. With binary correctness feedback, FCP has a lower reported accuracy: Correctness-Only FCP scores 31.6, while GRPO with the rule-based verifier scores 36.4 (Tables 5, 1). This raises the question of whether the gap reflects the update rule, optimization settings, or other differences in the training protocol.
  • Creative writing. The two signals are not information-matched. Dimensions scored 6–10 are rendered as "Perfect." (Fig. 6), and c+c^+ requires "Perfect." on all four. FCP therefore targets a conjunctive threshold, while GRPO maximizes the mean of four 0–10 scores. A GRPO control with a more closely matched objective would help separate the contribution of verbal richness from that of the objective's shape; possible controls include the indicator that all dimensions are ≥ 6 or the minimum score across dimensions.
Questions for authors
  1. [Suggestion] Could you run GRPO in the creative-writing setting with a reward matched to FCP's target, e.g., 1[all four dimensions≥6]\mathbb{1}[\text{all four dimensions} \ge 6] or the minimum dimension score, from the same critic? If FCP still wins, that would substantiate the paper's central "verbal vs. scalar" claim. If the advantage disappears under this control, the current comparison would provide weaker evidence that verbal richness itself explains the gain. This is the comparison I would find most informative for assessing the central claim.
  2. [Clarification] How many independent training runs were performed per method in Tables 1–3, and what does "±" denote (std or SE, over which randomness)? Please clarify whether these values include variation across independently trained models. If additional training runs are feasible, a few seeds for FCP+Bootstrap and the strongest GRPO baseline would help assess the small reported margins.
  3. [Clarification and suggested control] Given that GPT-5-nano sees the reference answer (Fig. 5), could you clarify the intended scope of "without relying on verifiers"? As a suggested control, could you evaluate FCP+Bootstrap with reviewer-style feedback produced without the reference answer? This would clarify how the method behaves when reference-based correctness supervision is unavailable.
  4. [Clarification] How does FCP relate to Chain of Hindsight, Quark, HIR and SteerLM? In particular, what are the differences between Textualized-Scalar + online (Table 6) and Quark's loop? Making this relationship explicit would help interpret the baseline for the conditioning paradigm.
  5. [Clarification] Could you help explain the reported gap between FCP with Correctness-Only labels (31.6) and GRPO with the same binary verifier (36.4)? Does this reflect the conditional-MLE update itself (no explicit negative gradient under c+c^+), the online schedule, or the size of the condition space? An answer would clarify when FCP should be preferred over standard RL.
Reviewer_zKU5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Feedback-Conditional Policy (FCP), to learn directly from verbal feedback, which contains richer information than scalar reward. FCP first learns from offline response-feedback pairs by doing standard likelihood training with feedback prepended to the prompt. Then FCP conducts online training to bootstrap performance by conditioning explicitly on positive feedback and learn from newly collected critics. Experiments on math reasoning, general reasoning, and creative writing show that FCP achieves performance comparable to scalar-reward RL baselines on reasoning tasks and outperforms them on the evaluated creative-writing task.

Critical strengths
  1. This method proposes a novel learning paradigm on how to learn from verbal feedback rather than scalar reward, demonstrating the potential of verbal feedback as an alternative training signal for LLM post-training.
  2. This method uses standard maximum-likelihood training without requiring explicit reward optimization, or rule-based verifiers, which is simple and easy to reproduce.
Critical weaknesses
  1. The method relies on the quality of feedback model and feedback style, making it unclear whether the model truly learns reasoning skills or simply produce responses that satisfy the critic's preferences.
  2. During online training, it only learns from responses generated conditioned on positive feedback. It's unclear whether this procedure maintains sufficient exploration and output diversity, or how well this process avoids self-reinforcing errors or distribution collapse over longer training horizons.
  3. Although the paper's idea is presented as a flexible framework that can condition on verbal feedback, the range of verbal condition is limited (Table 15). The method can work with similar-style positive verbal condition, but might be hard to handle complicated free-style verbal condition. It's also unclear how this method can deal with multi-objective alignment.
Questions for authors
  1. The critic prompt in Figure 5 includes the reference answer for the correctness judgment. How does FCP perform when the critic has no reference answer? Without that comparison, how does FCP claim that it works without relying on verifiers?
  2. Have the authors observed collapse of output diversity over rounds in online training?
  3. Can the model handle arbitrary free-form instructions, rather than feedback resembling its training distribution?
  4. Can the model reliably trade off conflicting objectives, rather than merely following a condition that asks for everything to be good?
Reviewer_gGutOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the information loss that occurs when rich verbal feedback is converted into scalar rewards for LLM training. To address this issue, the authors propose Feedback-Conditional Policy (FCP), which models the relationship between responses and verbal feedback through conditional generation and maximum likelihood training. The method further introduces online bootstrapping, where responses are generated under positive feedback conditions and then relabeled with actual feedback. Experiments show that FCP achieves performance comparable to GRPO on mathematical and general reasoning tasks while outperforming scalar-reward baselines on creative writing.

Critical strengths

Clear Motivation and Simple Methodology

The paper identifies a meaningful limitation of scalar-reward-based learning and proposes a simple approach that directly incorporates verbal feedback into conditional generation. The ability to train using maximum likelihood without explicit reward optimization is an appealing aspect of the method.

Extensive Experimental Evaluation

The method is evaluated across mathematical reasoning, general reasoning, and creative writing tasks, with comparisons against several baselines, including GRPO. The ablation studies on feedback quality and conditioning strategies provide useful insights into the effectiveness of the proposed approach.

Critical weaknesses

Insufficient Analysis of the Online Bootstrapping Mechanism

Although Section 2.2 formulates online bootstrapping as conditional posterior learning, it remains unclear why training on actual feedback (c) systematically improves generation under the desired condition (c^+). The paper does not sufficiently analyze how the mismatch between these conditions affects policy improvement.

Unclear Contribution of Natural-Language Feedback Itself

Sections 4.1 and 4.2 compare different feedback types, including textualized scalar scores. However, the experiments do not fully distinguish the benefits of natural-language semantics from those of multi-dimensional evaluation signals. A structured, multi-dimensional scalar baseline would help clarify this distinction.

Limited Differentiation from Prior Work

Prior studies, such as Chain of Hindsight (ICLR 2024) and Hindsight Instruction Relabeling (ICML 2023), have explored feedback conditioning and hindsight relabeling through supervised learning. The paper does not sufficiently clarify its key methodological differences from these approaches, and direct experimental comparisons are missing.

Questions for authors

Online Bootstrapping Mechanism

Could the authors clarify under what conditions training on actual feedback (c) improves generation under (c^+), particularly when the two conditions differ?

Contribution of Verbal Feedback

Could the authors compare FCP against a baseline conditioned on structured, multi-dimensional scalar feedback, using the same evaluation attributes and feedback budget to isolate the contribution of natural-language semantics?

Differentiation from Prior Work

What are the key methodological differences between FCP and existing approaches such as Chain of Hindsight and Hindsight Instruction Relabeling? Could the authors provide direct comparisons using matched training data and feedback budgets?

Reviewer_qdamOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FCP learns from verbal critiques through conditional likelihood training and online bootstrapping. Using Qwen2.5-7B-Base, it approximately matches GRPO on reasoning with GPT-5-nano feedback and improves LitBench creative writing with Qwen3.5-122B-A10B feedback under an independent evaluator.

Critical strengths
  1. FCP reaches reported GRPO-level reasoning accuracy using a likelihood loss on critique-conditioned rollouts. It avoids reward scalarization, advantage estimation, and PPO-style clipping, which simplifies optimization.
  2. Verbal conditioning yields larger online gains than scalar conditioning (+10.2 versus +3.8 points, Table 6), which supports the usefulness of textual feedback.
Critical weaknesses
  1. Evaluation reliability: Table 1 matches Figure 4a's peak values (38.7, 38.4, 37.1), but checkpoint selection and training-run variability are unreported. If checkpoints were selected on test performance, the results may be optimistic, and held-out selection could change the ranking in either direction.
  2. Interpretation of writing gains: Figure 6 maps scores 6–10 to "Perfect." FCP targets this label separately across four dimensions, whereas GRPO averages the scores. The gains therefore do not separate the effect of rich verbal content from per-dimension threshold signals.
Questions for authors
  1. How were checkpoints selected, and what are the final-checkpoint results and the variation across independent training runs?
  2. Would conditioning on four per-dimension scores or pass/fail labels retain the creative-writing advantage?
  3. ALT already prepends textual feedback, samples under desired feedback, and retrains iteratively. What distinguishes FCP from ALT, and what allows richer critiques to work effectively?

Comments on the paper

1 comment

Thank you for the detailed response and the new experiments. My concerns have been addressed, and I will maintain my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.