- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes the feedback-conditional policy (FCP), which uses verbal feedback as a conditioning signal instead of compressing it into a scalar reward.
- Offline stage. FCP fits the feedback-conditional posterior by maximum likelihood on (instruction, response, critique) triples, with the critique prepended to the prompt (Alg. 1, Eq. 1).
- Online stage. The model samples under a positive condition drawn from a pool, has each response re-critiqued, and trains on the response paired with its actual critique (Alg. 2, Eq. 2).
Experiments use Qwen2.5-7B-Base with GPT-5-nano as critic. On five math benchmarks, FCP+Bootstrap reaches 38.7, against 36.4–38.4 for four GRPO variants (Table 1). On general reasoning it reaches 47.8, against 47.5 for RFT+GRPO (Table 2). Results on Qwen3-4B and Llama-3.1-8B are similar (Table 10). In creative writing, with a Qwen3.5-122B critic and an independent LitBench reward model as evaluator, FCP+Bootstrap is preferred over Base+GRPO and RFT+GRPO in 68.0% and 62.7% of head-to-head comparisons (Table 3). Further analyses cover:
- controllability under different conditions (Table 4);
- feedback quality (Table 5);
- textualized-scalar and no-feedback controls (Table 6);
- a weaker 1.5B critic (Table 7);
- out-of-distribution test-time feedback (Table 12).
Critical strengths
1. A simple mechanism that uses all feedback, including mixed and negative, and matches strong RL baselines at similar cost. FCP needs only likelihood training on critique-annotated samples, without scalar reward computation, advantage estimation, or data filtering in the policy update. It still reaches GRPO-level accuracy on math (38.7 vs. 38.4, Table 1) and general reasoning (47.8 vs. 47.5, Table 2), and stays within 1.6 points of the best GRPO baseline on two other backbones (Table 10). Training time is comparable (7.1 h vs. 7.8 h for GRPO, Table 14). In creative writing, where response quality has multiple dimensions, an independent reward model prefers FCP+Bootstrap over both GRPO baselines (68.0% / 62.7% head-to-head, Table 3).
2. Controlled evidence that the verbal content matters, not just the conditioning mechanism. The three conditioning formats behave almost identically offline (28.1–29.8). Online, they diverge sharply: +10.2 with verbal critiques, +3.8 with textualized scores, and +0.8 with no feedback (Table 6). Richer critiques also beat binary labels (38.7 vs. 31.6, Table 5), and a critic weaker than the policy still helps (33.1 with a 1.5B critic, Table 7). The controllability study (Table 4: 68.5 under positive vs. 17.1 under negative conditions on MATH500) shows that the model actually uses the condition. These controls provide useful evidence for the contribution of verbal feedback beyond the conditioning mechanism alone.
Critical weaknesses
1. The relationship to prior methods and the scope of the claims would benefit from clearer positioning.
- Positioning relative to closely related work. Feedback-conditioned likelihood training with hindsight relabeling has close precedents that are not cited. Chain of Hindsight (Liu et al., ICLR 2024) trains on responses paired with natural-language feedback. Quark (Lu et al., NeurIPS 2022) uses a related online loop with quantized reward tokens: sample under a high-reward token, re-score, relabel, and retrain. HIR (Zhang et al., ICML 2023) relabels instructions in hindsight. SteerLM (Dong et al., 2023) conditions on attribute labels. The Textualized-Scalar control in Table 6 resembles a Quark-style approach. Clarifying the differences would help establish what is new about using free-form critique as the condition; the margin over this control is particularly informative.
- The "without relying on verifiers" claim (p. 2) needs qualification for the reasoning experiments. The GPT-5-nano prompt receives the reference final answer and judges correctness against it (Fig. 5). Although the policy update does not require scalar verifier rewards, its verbal feedback still carries reference-based correctness information. These experiments therefore do not establish learning without such supervision.
- The population-level characterization leaves a generalization question. The optimum is characterized on . This does not by itself establish finite-sample generalization when positive feedback conditions are transferred across prompts. It would be helpful to clarify the assumptions and empirical evidence supporting this use of the learned conditional policy.
2. More closely matched comparisons would help establish the benefits of verbal feedback over scalar rewards.
- Reasoning. FCP+Bootstrap has similar reported point estimates to the strongest GRPO baselines (38.7 vs. 38.4; 47.8 vs. 47.5), but the number of independent training runs per method is not specified. Table 9 reports six FCP configurations, including the main configuration at 38.7 and a default at 35.3. The paper does not explain how the main configuration was selected or whether GRPO received a comparable tuning budget. The stated protocol describes repeated evaluation with greedy decoding, but does not make clear whether the reported ± include variation across independent training runs. With binary correctness feedback, FCP has a lower reported accuracy: Correctness-Only FCP scores 31.6, while GRPO with the rule-based verifier scores 36.4 (Tables 5, 1). This raises the question of whether the gap reflects the update rule, optimization settings, or other differences in the training protocol.
- Creative writing. The two signals are not information-matched. Dimensions scored 6–10 are rendered as "Perfect." (Fig. 6), and requires "Perfect." on all four. FCP therefore targets a conjunctive threshold, while GRPO maximizes the mean of four 0–10 scores. A GRPO control with a more closely matched objective would help separate the contribution of verbal richness from that of the objective's shape; possible controls include the indicator that all dimensions are ≥ 6 or the minimum score across dimensions.
Questions for authors
- [Suggestion] Could you run GRPO in the creative-writing setting with a reward matched to FCP's target, e.g., or the minimum dimension score, from the same critic? If FCP still wins, that would substantiate the paper's central "verbal vs. scalar" claim. If the advantage disappears under this control, the current comparison would provide weaker evidence that verbal richness itself explains the gain. This is the comparison I would find most informative for assessing the central claim.
- [Clarification] How many independent training runs were performed per method in Tables 1–3, and what does "±" denote (std or SE, over which randomness)? Please clarify whether these values include variation across independently trained models. If additional training runs are feasible, a few seeds for FCP+Bootstrap and the strongest GRPO baseline would help assess the small reported margins.
- [Clarification and suggested control] Given that GPT-5-nano sees the reference answer (Fig. 5), could you clarify the intended scope of "without relying on verifiers"? As a suggested control, could you evaluate FCP+Bootstrap with reviewer-style feedback produced without the reference answer? This would clarify how the method behaves when reference-based correctness supervision is unavailable.
- [Clarification] How does FCP relate to Chain of Hindsight, Quark, HIR and SteerLM? In particular, what are the differences between Textualized-Scalar + online (Table 6) and Quark's loop? Making this relationship explicit would help interpret the baseline for the conditioning paradigm.
- [Clarification] Could you help explain the reported gap between FCP with Correctness-Only labels (31.6) and GRPO with the same binary verifier (36.4)? Does this reflect the conditional-MLE update itself (no explicit negative gradient under ), the online schedule, or the size of the condition space? An answer would clarify when FCP should be preferred over standard RL.
Thank you for the detailed response and the new experiments. My concerns have been addressed, and I will maintain my positive score.