- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work proposes Heuristic-guided Branching-Replay Group Relative Policy Optimization (HBR-GRPO), which is particularly designed for multi-turn interactive reasoning tasks like Wordle and GuessWhat. The core idea is to break each task into individual turns and reward each action by the reduction it induces in the heuristic distance to the answer, which is measured by the size of the feasible candidate set. This work also combines it with a finite-horizon search value that favors answering once successful termination becomes preferable to further information gathering. It then groups the actions sampled from the same state into an independent GRPO group, and uses On-Policy Branching Replay to train on successor states that the current policy can actually reach. In terms of the training and evaluation on Qwen3-8B, HBR-GRPO reaches a Clemscore of 8.50 on Wordle and 48.33 on GuessWhat, outperforming SFT and SFT-GRPO baselines on six open-weight models and larger leaderboard models.
Critical strengths
This paper targets a real problem in multi-turn RL, where one terminal reward cannot tell good turns from bad ones, and addresses it with a clear and well-motivated design by assigning each action a verifiable turn-level reward, and comparing them only within the same state. The training and evaluation results also show great gain on the Wordle and GuessWhat tasks compared to all baselines.
Critical weaknesses
(1) To score each turn, HBR-GRPO needs to list all answers still consistent with the history, a fixed Answerer that labels every GuessWhat candidate, and a hand-built library of good reference actions. On HotpotQA, the reward is even computed directly from the gold supporting facts. However, the baselines including GRPO, IGPO, GiGPO only see whether the final answer is correct, so the comparison is actually unfair I believe.
(2) Before RL, HBR-GRPO is trained with the authors’ own SFT data built from solver-guided demonstrations, while the baselines use the original Playpen SFT. This SFT alone already raises the GuessWhat Clemscore from 13.33 to 35.0, before any RL (Tables 2 and 3). This is also unfair.
Questions for authors
In addition to those in weaknesses. Could the authors explain how HBR-GRPO could be applied to tasks where the set of possible answers cannot be enumerated and no reference action library is available?
4 comments
Response to the review ( part two )
Unfair S-SFT Initialization We acknowledge that this discrepancy existed in the original main table (Table 1). However, we would like to note that our subsequent horizontal comparisons against other RL algorithms in Tables 4 and 5 already used the exact same S-SFT initialization across all methods, ensuring a fair comparison on Wordle, GuessWhat, and HotpotQA where HBR-GRPO consistently achieves superior performance. Please also allow us to briefly explain our original rationale for Table 1: because the search-aligned S-SFT initialization is itself part of our proposed pipeline, we originally wanted Table 1 to cleanly separate the untouched Playpen baselines from the entirety of the contributions introduced in our paper, and therefore reported the original Playpen SFT initialization for the baselines.
That said, we fully agree that your concern regarding initialization fairness is valid. We have therefore re-trained and re-evaluated all six baseline model families in the main table using S-SFT initialization. The results are as follows:
Wordle:
| Model | Configuration | ACC (%) | Quality | Played (%) | Clemscore |
|---|---|---|---|---|---|
| Qwen3-8B | S-SFT-only | 7.74 | 6.22 | 50.60 | 3.15 |
| Qwen3-8B | S-SFT + Standard GRPO | 8.93 | 6.83 | 61.90 | 4.23 |
| Gemma-2-9B-IT | S-SFT-only | 8.93 | 6.42 | 56.55 | 3.63 |
| Gemma-2-9B-IT | S-SFT + Standard GRPO | 8.93 | 6.94 | 58.33 | 4.05 |
| Mistral-7B-Instruct-v0.3 | S-SFT-only | 0.00 | 0.00 | 73.21 | 0.00 |
| Mistral-7B-Instruct-v0.3 | S-SFT + Standard GRPO | 0.00 | 0.00 | 70.83 | 0.00 |
| Falcon-7B-Instruct | S-SFT-only | 0.00 | 0.00 | 79.17 | 0.00 |
| Falcon-7B-Instruct | S-SFT + Standard GRPO | 0.00 | 0.00 | 72.02 | 0.00 |
| GLM-4-9B-Chat | S-SFT-only | 7.14 | 6.62 | 86.31 | 5.71 |
| GLM-4-9B-Chat | S-SFT + Standard GRPO | 7.74 | 7.10 | 86.31 | 6.13 |
| DeepSeek-LLM-7B-Chat | S-SFT-only | 2.38 | 3.13 | 58.93 | 1.85 |
| DeepSeek-LLM-7B-Chat | S-SFT + Standard GRPO | 0.00 | 0.00 | 61.31 | 0.00 |
GuessWhat:
| Model | Configuration | ACC (%) | Quality | Played (%) | Clemscore |
|---|---|---|---|---|---|
| Qwen3-8B | S-SFT-only | 35.00 | 35.00 | 100.00 | 35.00 |
| Qwen3-8B | S-SFT + Standard GRPO | 35.00 | 35.00 | 100.00 | 35.00 |
| Gemma-2-9B-IT | S-SFT-only | 50.00 | 31.67 | 100.00 | 31.67 |
| Gemma-2-9B-IT | S-SFT + Standard GRPO | 48.33 | 36.67 | 100.00 | 36.67 |
| Mistral-7B-Instruct-v0.3 | S-SFT-only | 21.67 | 21.67 | 100.00 | 21.67 |
| Mistral-7B-Instruct-v0.3 | S-SFT + Standard GRPO | 26.67 | 25.00 | 100.00 | 25.00 |
| Falcon-7B-Instruct | S-SFT-only | 0.00 | 0.00 | 45.00 | 0.00 |
| Falcon-7B-Instruct | S-SFT + Standard GRPO | 0.00 | 0.00 | 48.33 | 0.00 |
| GLM-4-9B-Chat | S-SFT-only | 33.33 | 33.33 | 100.00 | 33.33 |
| GLM-4-9B-Chat | S-SFT + Standard GRPO | 31.67 | 31.67 | 100.00 | 31.67 |
| DeepSeek-LLM-7B-Chat | S-SFT-only | 21.67 | 21.67 | 100.00 | 21.67 |
| DeepSeek-LLM-7B-Chat | S-SFT + Standard GRPO | 33.33 | 31.67 | 100.00 | 31.67 |
Even under identical S-SFT initialization across all models, none of these baselines surpass HBR-GRPO on the primary metric (Clemscore: 8.50 on Wordle and 48.33 on GuessWhat).
Response to the review ( part three )
Applying HBR-GRPO to Scenarios Without Enumerable Answers or Reference Action Libraries First, we would like to note that our formulation of HBR-GRPO on HotpotQA already removes the requirement that the answer space must be enumerable: on HotpotQA, our HP reward is defined as the increment in Intersection-over-Union (IoU) between the model's selected evidence and the gold supporting facts, which does not require exhaustively enumerating all possible answers or evidence. Thus, the essential prerequisite for the HP reward is not an enumerable answer space, but rather the ability to construct a reliable heuristic progress evaluation function. In many practical domains, such progress signals can be naturally constructed. For instance, in LLM coding tasks, a viable heuristic progress function is the turn-over-turn increment in the number of passed unit test cases induced by each generated code patch (indeed, to our knowledge, several LLM coding benchmarks—particularly those targeting long-horizon software engineering capabilities such as SWE-CI and SWE-Marathon —already adopt similar progress-based scoring designs).
For scenarios where a pre-built reference action library is unavailable, one potential approach is to sample candidate actions directly from the policy itself and estimate their reference values via short-horizon rollouts (analogous to traditional Bellman action-value estimation) before performing within-group normalization, or alternatively, to query a stronger teacher model to generate reference actions at each visited state. However, we candidly note that these remain preliminary directions for future work; for the regimes where HBR-GRPO has demonstrated reliable empirical gains in the present paper, a reference action library is still required.
Thank you for the detailed responses and the additional experiments. I do not think it is enough to give me the reason to raise my rating. i think my original rating is very reasonable for overall quality. i do not recommend to accept this paper.
Response to the review ( part one )
Dear Reviewer,
We sincerely thank you for your time and effort in reviewing our manuscript, for recognizing the contributions of our work, and for raising such insightful and constructive questions. Below, we address each of your comments in detail.
Unfair Discrepancy in Reward Information We fully acknowledge this point. The primary reason for our reward design is that overcoming severe reward sparsity is one of the central challenges HBR-GRPO aims to address: for example, on Wordle, Qwen3-8B starts with nearly 0% accuracy and reaches only 7.74% even after S-SFT initialization, which motivated our effort to decouple step-level training signals from sparse terminal victories. We would also like to clarify that our reference action libraries are not manually handwritten on a per-instance basis: Wordle uses rule-based filtering and exact feedback transitions, while GuessWhat uses automated model generation, filtering, and verification. Likewise, on HotpotQA, the gold supporting facts are used strictly to compute training rewards and are never exposed as inputs to the policy.
Nevertheless, we completely understand your concern regarding reward-supervision parity. In addressing Reviewer UBJd's comment, we conducted an additional set of experiments on HotpotQA that strictly aligns the reward supervision (PVF) across methods to compare our grouping strategy against GiGPO's grouping strategy on an equal footing. We share these results below as well (where Groups D, E, and F are existing results from the paper included for reference):
Because Groups A, B, and C compare trajectory-level GRPO, GiGPO-style grouping, and HBR-GRPO's grouping under the exact same PVF reward supervision signal, we hope this comparison helps address your concern.