- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FBOS-RL modifies GRPO-style RL for LLM reasoning by adding a second, feedback-conditioned sampling round. For each prompt , the policy first draws responses, and a rule-based verifier returns both a reward and natural-language error feedback. Each response and its feedback are appended to to form a feedback-augmented prompt , from which further responses are sampled, matching GRPO's total of 72 rollouts per prompt. Two objectives are then optimized sequentially: EPA trains on the pooled group, using a cross-context ratio passed through ; ECC applies standard GRPO to . Experiments on TravelPlanner (three backbones) and MiniF2F-Lean4 (one backbone) report higher peak pass rates and fewer rollouts to reach GRPO's best, along with higher entropy and GPQA-Diamond gains.
Critical strengths
Strict rollout-count matching against GRPO, together with an extra-update control, is more careful than is typical. However, it does not compensate for the evaluation and baseline issues below.
Critical weaknesses
First, the evaluation protocol leaks test information. TravelPlanner's 1,000-example test split and MiniF2F's test split serve as validation sets, and the reported numbers are peaks over training curves. Design choices are also compared on the same split (e.g., vs. in Appendix L, where substantially outperforms the method actually proposed). None of the headline gains is an unbiased held-out estimate.
Second, the comparison is not information-matched. FBOS-RL receives error-localized feedback, whereas GRPO sees only scalar rewards. The only feedback-using RL baseline, an adapted OPSD without any reward term, stays near zero pass rate, which points to a broken adaptation rather than a meaningful comparison.
Third, supporting evidence is thin. Training sets contain only 225 and 244 examples, and MiniF2F appears to use a single run with a non-standard score. The GPQA "generalization" starts from about 13%, below the 25% chance level, so it likely reflects format repair rather than reasoning, and no GRPO-trained model is compared.
Questions for authors
-
Can you report final-checkpoint (or held-out-selected) results on a test split never used for checkpoint or design selection?
-
Why does the adapted OPSD remain near zero while even untrained baselines score higher? Please compare with a reward-augmented OPSD or other feedback-driven RL methods such as LUFFY-style off-policy guidance.
-
On GPQA-Diamond, what are the pre- and post-training format-error rates, and how does a GRPO-trained model perform?
-
How many seeds were used on MiniF2F, and what is pass@ under the standard protocol?
-
Since outperforms , which transformation is the proposed method, and how sensitive are results to this choice on held-out data?
-
Evaluating on tasks whose verifiers return only scalar rewards would clarify the method's scope.