acceptodds
Withdrawn from ICLR 2027

FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) improves the reasoning capabilities of large-scale models. Group Relative Policy Optimization (GRPO) and its variants alternate between rollout sampling and policy update, updating the policy using sampled rollouts and their advantages. Unlike supervised learning with explicit ground-truth targets, these policy updates rely on high-quality rollouts as an implicit "teacher" to guide their direction. However, GRPO and similar algorithms sample only from the original prompt. For tasks beyond the policy's current capability, high-quality rollouts are rare, leaving updates without a meaningful gradient direction and causing training to stall. To address this issue, we propose FBOS-RL, a Feedback-Driven Bi-Objective Synergistic Reinforcement Learning framework. Specifically, we let the model perform Feedback-Guided Exploration Enhancement based on the feedback provided by the environment, and on top of this we design two mutually reinforcing training objectives: Exploitation-oriented Policy Alignment (EPA) and Exploration-oriented Capability Cultivation (ECC). Extensive experiments show improved rollout efficiency and higher peak validation performance. We match the total number of generated training rollouts with GRPO: GRPO samples 72 responses per original prompt, whereas FBOS-RL allocates the same budget to 8 initial and 64 feedback-guided responses. On TravelPlanner with Qwen3-8B, FBOS-RL achieves a relative gain of in peak final pass rate within the shared rollout budget. To reach GRPO's best final pass rate, GRPO requires approximately as many training rollouts as FBOS-RL. Additional experiments compare against On-Policy Self-Distillation (OPSD). FBOS-RL also exhibits higher policy entropy and lower gradient norms than GRPO throughout training. Code will be made publicly available.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning, reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AokkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

FBOS-RL modifies GRPO-style RL for LLM reasoning by adding a second, feedback-conditioned sampling round. For each prompt qq, the policy first draws nn responses, and a rule-based verifier returns both a reward and natural-language error feedback. Each response and its feedback are appended to qq to form a feedback-augmented prompt q~i\tilde q_i, from which kk further responses are sampled, matching GRPO's total of 72 rollouts per prompt. Two objectives are then optimized sequentially: EPA trains πθ(⋅∣q)\pi_\theta(\cdot\mid q) on the pooled group, using a cross-context ratio πθ(a∣q)/πθold(a∣q~i)\pi_\theta(a\mid q)/\pi_{\theta_{old}}(a\mid\tilde q_i) passed through f(ρ)=ρ/(ρ+0.1)f(\rho)=\rho/(\rho+0.1); ECC applies standard GRPO to πθ(⋅∣q~i)\pi_\theta(\cdot\mid\tilde q_i). Experiments on TravelPlanner (three backbones) and MiniF2F-Lean4 (one backbone) report higher peak pass rates and fewer rollouts to reach GRPO's best, along with higher entropy and GPQA-Diamond gains.

Critical strengths

Strict rollout-count matching against GRPO, together with an extra-update control, is more careful than is typical. However, it does not compensate for the evaluation and baseline issues below.

Critical weaknesses

First, the evaluation protocol leaks test information. TravelPlanner's 1,000-example test split and MiniF2F's test split serve as validation sets, and the reported numbers are peaks over training curves. Design choices are also compared on the same split (e.g., ff vs. gg in Appendix L, where gg substantially outperforms the method actually proposed). None of the headline gains is an unbiased held-out estimate.

Second, the comparison is not information-matched. FBOS-RL receives error-localized feedback, whereas GRPO sees only scalar rewards. The only feedback-using RL baseline, an adapted OPSD without any reward term, stays near zero pass rate, which points to a broken adaptation rather than a meaningful comparison.

Third, supporting evidence is thin. Training sets contain only 225 and 244 examples, and MiniF2F appears to use a single run with a non-standard score. The GPQA "generalization" starts from about 13%, below the 25% chance level, so it likely reflects format repair rather than reasoning, and no GRPO-trained model is compared.

Questions for authors
  1. Can you report final-checkpoint (or held-out-selected) results on a test split never used for checkpoint or design selection?

  2. Why does the adapted OPSD remain near zero while even untrained baselines score higher? Please compare with a reward-augmented OPSD or other feedback-driven RL methods such as LUFFY-style off-policy guidance.

  3. On GPQA-Diamond, what are the pre- and post-training format-error rates, and how does a GRPO-trained model perform?

  4. How many seeds were used on MiniF2F, and what is pass@kk under the standard protocol?

  5. Since gg outperforms ff, which transformation is the proposed method, and how sensitive are results to this choice on held-out data?

  6. Evaluating on tasks whose verifiers return only scalar rewards would clarify the method's scope.

Reviewer_AqYYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FBOS-RL, a GRPO-style RL framework for tasks whose rule-based verifier also returns natural-language feedback. At each step the policy samples n initial responses to the original prompt q; each response and its verifier feedback are concatenated into a Feedback-Augmented Prompt (FAP) q̃ = q ⊕ ans ⊕ F, from which k second-round responses are sampled. Two objectives are then optimized sequentially: (1) EPA updates the feedback-free policy π(·|q) on a mixed group of all n + nk responses, using a cross-context ratio π_θ(a|q)/π_θold(a|q̃) for second-round responses, transformed by the LUFFY-style f(ρ) = ρ/(ρ+0.1) without clipping; (2) ECC runs standard GRPO within each FAP group to improve the feedback-conditioned (self-correction) policy. The authors argue that the two objectives form a “positive bootstrapping flywheel”.

Critical strengths

A simple, practical mechanism with a large effect where it works. Resampling under verifier feedback and transferring those samples back to the feedback-free policy requires no external model. On Qwen3-8B / TravelPlanner, with rollouts matched, peak Final Pass Rate improves over GRPO by 21.9 points, and the three-seed bands are clearly separated. This is the main reason I think the direction is worth pursuing. Careful accounting and reasonably broad controls. Rollouts from both sampling stages count toward the budget; the GRPO w/ Extra Update baseline rules out “just one more gradient step”; both single-objective ablations are reported; and the limitations section states that matched rollouts are not matched compute. The w/o EPA ablation (π(·|q) is never trained directly, yet reaches ≈0.8 on Qwen3-14B) is itself an interesting finding about transfer from feedback-conditioned training.

Critical weaknesses

W1. EPA closely follows uncited prior work, while ECC, the genuinely new component, is validated on only one backbone, and baselines that use the same feedback are missing.

  • EPA has close, uncited precedents from mid-2025, more than a year before this submission. Guide-GRPO (Nath et al., arXiv June 2025; NeurIPS 2025) adds hints to the context for prompts on which all rollouts fail, normalizes advantages over the pooled guided and unguided rollouts, and uses the ratio π_θ(r|x)/π_θold(r|x̃) so that the no-hint policy is optimized; this is the same cross-context ratio as EPA. Critique-GRPO (Zhang et al., arXiv 2506.03106, 2025 versions) generates critique-conditioned refinements, mixes them into the GRPO group, and applies the LUFFY-style shaping π/(π+γ) to refinement tokens, which is essentially EPA's FAP branch with f. SDPO (Hübotter et al., 2026) uses the feedback-conditioned model as its own teacher, and the FAP “CRITICAL RULE” mirrors STaR's rationalization. In addition, the Introduction (p. 2) groups Self-Refine, Reflexion, SCoRe and RISE as methods that merely use feedback for data augmentation and overlook training the model's own ability to improve from feedback, whereas the Related Work (p. 3) acknowledges that SCoRe and RISE train correction. The Introduction also does not mention that EPA's transfer mechanism has precedents.
  • The paper does differ from these works in real ways, and these differences should be stated explicitly. First, feedback comes from a rule-based verifier, with no external LLM or reference solution; Guide's hints are written by GPT-4o from reference answers, and Critique-GRPO's main setting uses GPT-4o critiques. Second, every initial response is resampled densely (64 feedback-conditioned rollouts per prompt), whereas Guide triggers only when all rollouts fail and Critique-GRPO keeps a single refinement per prompt. Third, ECC explicitly trains the feedback-conditioned policy, which neither prior method does. The paper's own ablations on Qwen3-14B indicate that the gain over GRPO comes mainly from ECC: ≈0.80 final without ECC, ≈0.81 without EPA, 0.82 for GRPO's peak, and ≈0.85–0.87 for the full method. In other words, EPA on its own, the part that overlaps with prior work, does not beat GRPO, even though it uses the same feedback.
  • FBOS-RL receives per-constraint natural-language diagnostics, while GRPO receives only a scalar reward, and the paper never defines that reward for TravelPlanner. The only feedback-using baseline, the OPSD adaptation, uses a frozen base-model teacher (the same Llama-3.1-8B-Instruct that scores 0.43% without feedback; its performance with feedback is not reported), no task reward and fewer rollouts, on a single backbone; it stays near zero and is not informative. Needed: Guide-/Critique-GRPO adapted to use the same verifier feedback, SDPO with the current-policy teacher, and GRPO with a dense per-constraint reward that carries the same information without natural language. Why it matters: ECC is the most valuable part of the paper, but the evidence for it comes from a single backbone, the one with the smallest gain. On Qwen3-8B, where the gain is largest, there is no ablation, so it is unknown how much of the improvement comes from EPA, which overlaps with prior work. Without ablations on the other backbones and baselines that use the same feedback, the paper's contribution cannot be attributed. W2. The evaluation protocol is not standard, and several secondary claims are not supported.
  • The official TravelPlanner train and validation splits are used for training, and the 1,000-example test set serves as the “validation” set, evaluated every 5 steps (50–83 evaluations, depending on the backbone). The Δ numbers in Table 1 and the abstract are the maxima over these evaluations, and the × numbers use GRPO's maximum as the target; the same holds for the MiniF2F test split. Reporting the best checkpoint on the test set is common in RL-for-LLM papers, but the resulting absolute numbers are optimistic and the protocol is not standard; Appendix L (variant g reaching ≈0.8) also suggests that design iterations were made on the same set. There are only 3 seeds with min–max bands and no significance tests, and on Llama individual runs of both methods drop close to zero at points. Judging from the plots, FBOS-RL is still ahead at the end of training (≈0.66 vs. 0.56 on Llama, ≈0.87 vs. 0.81 on Qwen3-14B), so this affects how the numbers should be reported rather than the direction of the comparison. Please report the last checkpoint or the mean of the last few, with mean ± std over the three seeds.
  • It is unclear how the TravelPlanner test set was evaluated at all. The public test split omits the structured constraint fields that the official evaluator requires (budget, people_number, visiting_city_number, local_constraint), and the test annotations are held in a private dataset used only by the official leaderboard. The benchmark's README asks that test-set evaluation go through the leaderboard and prohibits converting test queries into structured formats for unauthorized evaluation. The paper nonetheless evaluates all 1,000 test examples every 5 steps in every run and does not explain how (see clarification question 2).
  • Matching rollouts does not match compute: 64 of the 72 rollouts are generated from longer FAP prompts, and ECC adds a second backward pass over 512 sequences, yet no GPU-hours are reported. GRPO uses a single, untuned allocation (72 samples × 8 prompts). On Qwen3-8B its curve is still rising when the budget ends, so the “peak within budget” and the 3.29× figure depend on where training is cut off.
  • Secondary claims overreach. (i) The lower gradient norm may follow from the design: in EPA's FAP branch, 64 of the 72 samples receive a per-token weight ρf′(ρ) = 0.1ρ/(ρ+0.1)² ≤ 0.25, and the paper does not say whether the plotted norm comes from the EPA or the ECC update, so a lower norm alone is not evidence of stability. Higher entropy may simply reflect training on samples that are low-probability under π(·|q), rather than stronger exploration. (ii) On GPQA-Diamond, Llama goes from 13.13% to 23.23%, both below the 25% chance level of a 4-choice task, which points to answer-extraction or format failures. Qwen3-14B scores 33.33% before and after, and there is no GRPO-trained comparison. (iii) On MiniF2F only 40 steps (9 evaluations) are shown: the curves overlap for the first 25 steps, and FBOS-RL leads only at the last three evaluations (≈0.44 vs. ≈0.37 at step 40). The metric is a custom +1/0/−1 score rather than pass@k, and no number is reported, yet the conclusion claims gains on MiniF2F. Why it matters: The unclear test-set evaluation affects the absolute numbers and whether they are comparable with the Table 1 baselines that appear to come from the original TravelPlanner paper (e.g., GPT-4-Turbo at 4.4). As long as FBOS-RL and GRPO were evaluated the same way, the comparison between them is unaffected. Until this is clarified, the absolute numbers in the table cannot be confirmed.
Questions for authors

Clarification requests:

  1. How is the TravelPlanner reward defined (binary final pass, fraction of satisfied constraints, or something else)? The official benchmark defines only evaluation metrics, not a training reward. Is the feedback text taken from the messages returned by the official constraint checks (e.g., “The restaurant in day 2 lunch … is repeated.”, “The room type should be …”)? The official budget check returns no message, so how is budget-related feedback produced? Please show several real feedback strings ({comment}).
  2. How was the 1,000-example test set evaluated locally? The public test split omits the structured fields that the official evaluator needs (budget, people_number, visiting_city_number, local_constraint); the official eval.py loads data only for the train and validation splits; and the test annotations are held in a private dataset used by the leaderboard. The README asks that test-set evaluation go through the official leaderboard and prohibits converting test queries into structured formats for unauthorized evaluation, yet the paper evaluates the test set every 5 steps in every run. Please explain where the test-set constraint annotations came from, and confirm the final numbers on the official leaderboard.
  3. Table 2 lists 23 / 21 / 21 epochs for Qwen3-8B / Llama / MiniF2F (≈647 / 591 / 640 steps at batch size 8), but the plotted curves end at ≈375 / 415 / 40 steps (only Qwen3-14B's 9 epochs, ≈253 steps, match its plot). Were the runs stopped early or the plots truncated, and how was the reported horizon chosen?
  4. MiniF2F: the initial prompt asks for a ```lean code block, while the FAP asks for reasoning followed by “[Final Lean Proof]”. How are second-round responses parsed and scored, and what happens when EPA pushes FAP-format responses into π(·|q)? Also, with a 2,048-token prompt limit, how are FAPs handled when they contain a previous proof (the response limit is 7,952 tokens) plus Lean errors?
Reviewer_4cQ4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses a known failure of GRPO in sparse reward setting: when every rollout for a prompt fails, the group has no reward variance, and the update is left without a gradient direction. FBOS-RL therefore re-allocates most of the rollout budget to second attempts.

Eight rollouts are drawn from the original prompt, a rule-based verifier returns a scalar reward together with a natural-language description of what failed, and prompt, failed rollout and feedback are concatenated into an augmented prompt from which 64 further rollouts are drawn. Two objectives then update the shared policy sequentially: one pools all 72 rollouts and trains the deployed policy, scoring the feedback-guided rollouts through a cross-context likelihood ratio left unclipped, and the other runs standard GRPO on the augmented-prompt groups (L205-299). Validation covers TravelPlanner with three backbones and MiniF2F-Lean4 with one, under matched rollout counts and three runs per configuration, reporting peak Final Pass Rate gains of 45.81, 7.91 and 5.94 percent over GRPO.

Critical strengths

Could not find any.

Critical weaknesses

Overall, this feels like an AI-generated paper (the title probably should have given a clue) with all its typical problems.

The core objective pre-exists in work that is not cited. Guide-GRPO (arXiv:2506.13923, June 2025) already uses the cross-context ratio, labelling it an off-policy correction, pools plain and guided rollouts into one normalized group, triggers guidance exactly when all unguided rollouts fail, and removes clipping on the guided branch while reporting divergence when the clip is retained.

This paper presents the same ratio at L242-246 as a surrogate which it introduces, and removes the same clipping at L859 with no argument given. Critique-GRPO (arXiv:2506.03106, ICML 2026) is the same two-round pipeline with a merged group and the same reshaping function, and its refinement branch already trains the feedback-free policy, so what is left is a different choice of denominator. GOLF (arXiv:2603.04597, March 2026) combines all of it, including the joint training which the authors of GOLF also call a virtuous cycle. Appendix K then reports that removing only the reshaping function leaves the method "much closer to GRPO than to FBOS-RL" on hard examples, which points at the borrowed component as load-bearing, and the comparison which would settle this is absent.

Beyond the attribution, there are clearly two hallucinated references:

  1. At L549-550, Tang et al., "Exploration–exploitation trade-off in reinforcement learning for large language models", is given as arXiv:2506.10202. That identifier resolves to "Q2E: Query-to-Event Decomposition for Zero-Shot Multilingual Text-to-Video Retrieval", by Dipta and Ferraro, a text-to-video retrieval paper with no author named Tang. The reference is used twice in text, at L054 for the two-stage structure of GRPO and at L122 for exploration limitations. No paper carrying the printed title is found in arXiv, Semantic Scholar or OpenAlex.

  2. At L585-586, Zheng et al., "Beyond GRPO: Tree-search enhanced reinforcement learning for reasoning", is given as arXiv:2502.10717. That identifier resolves to "Constraining Statistical Isotropy using 21cm Power Spectrum and Bispectrum", an observational cosmology paper filed under astro-ph.CO. It is the sole citation for tree search at L123, and again the printed title is not found in any of the three indices.

Questions for authors

Please position the method against Guide-GRPO, Critique-GRPO and GOLF component by component, and state which of the cross-context ratio, the pooled group, the reshaping function and the clip removal is claimed as new. RLTF (arXiv:2602.02482) derives the same estimator, proves it unbiased, and reports that removing the importance weighting improves stability and final performance, so the central design choice is one which prior work tested and rejected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.