acceptodds
Under review as a conference paper at ICLR 2027

Heuristic-guided Branching-Replay Group Relative Policy Optimization

Abstract

Post-training has proven highly effective in improving the reasoning capabilities of large language models (LLMs). However, growing evidence suggests that reinforcement-learning-based post-training encounters a pronounced performance ceiling on long-horizon reasoning tasks. This limitation is particularly acute in multi-turn interactive reasoning, where learning is hindered by sparse terminal rewards, ambiguous trajectory-level credit assignment, and a compounding state-distribution mismatch between fixed replay trajectories and the evolving policy. In this paper, we propose HBR-GRPO, or Heuristic-guided Branching-Replay Group Relative Policy Optimization, a natural and effective extension of GRPO designed to be broadly applicable to multi-turn interactive reasoning. We adopt a heuristic-search perspective that decomposes trajectory-level training into individual turns and evaluates each action by the reduction it induces in the heuristic distance from the current state to the answer. We then use on-policy branching replay to reconnect these turn-level examples through transitions generated by the current policy, yielding genuine multi-turn interaction trajectories. HBR-GRPO substantially improves the performance of Qwen3-8B across the evaluated tasks. Notably, on one dataset where nearly all models with approximately 8B parameters listed on the clembench leaderboard achieve 0% accuracy, our method attains a nontrivial success rate.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language model post-training, reinforcement learning from verifiable rewards, multi-turn interactive reasoning, group relative policy optimization, turn-level credit assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8sX9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes Heuristic-guided Branching-Replay Group Relative Policy Optimization (HBR-GRPO), which is particularly designed for multi-turn interactive reasoning tasks like Wordle and GuessWhat. The core idea is to break each task into individual turns and reward each action by the reduction it induces in the heuristic distance to the answer, which is measured by the size of the feasible candidate set. This work also combines it with a finite-horizon search value that favors answering once successful termination becomes preferable to further information gathering. It then groups the actions sampled from the same state into an independent GRPO group, and uses On-Policy Branching Replay to train on successor states that the current policy can actually reach. In terms of the training and evaluation on Qwen3-8B, HBR-GRPO reaches a Clemscore of 8.50 on Wordle and 48.33 on GuessWhat, outperforming SFT and SFT-GRPO baselines on six open-weight models and larger leaderboard models.

Critical strengths

This paper targets a real problem in multi-turn RL, where one terminal reward cannot tell good turns from bad ones, and addresses it with a clear and well-motivated design by assigning each action a verifiable turn-level reward, and comparing them only within the same state. The training and evaluation results also show great gain on the Wordle and GuessWhat tasks compared to all baselines.

Critical weaknesses

(1) To score each turn, HBR-GRPO needs to list all answers still consistent with the history, a fixed Answerer that labels every GuessWhat candidate, and a hand-built library of good reference actions. On HotpotQA, the reward is even computed directly from the gold supporting facts. However, the baselines including GRPO, IGPO, GiGPO only see whether the final answer is correct, so the comparison is actually unfair I believe.

(2) Before RL, HBR-GRPO is trained with the authors’ own SFT data built from solver-guided demonstrations, while the baselines use the original Playpen SFT. This SFT alone already raises the GuessWhat Clemscore from 13.33 to 35.0, before any RL (Tables 2 and 3). This is also unfair.

Questions for authors

In addition to those in weaknesses. Could the authors explain how HBR-GRPO could be applied to tasks where the set of possible answers cannot be enumerated and no reference action library is available?

4 comments
AuthorsOpenReview ↗

Response to the review ( part one )

Dear Reviewer,

We sincerely thank you for your time and effort in reviewing our manuscript, for recognizing the contributions of our work, and for raising such insightful and constructive questions. Below, we address each of your comments in detail.

Unfair Discrepancy in Reward Information We fully acknowledge this point. The primary reason for our reward design is that overcoming severe reward sparsity is one of the central challenges HBR-GRPO aims to address: for example, on Wordle, Qwen3-8B starts with nearly 0% accuracy and reaches only 7.74% even after S-SFT initialization, which motivated our effort to decouple step-level training signals from sparse terminal victories. We would also like to clarify that our reference action libraries are not manually handwritten on a per-instance basis: Wordle uses rule-based filtering and exact feedback transitions, while GuessWhat uses automated model generation, filtering, and verification. Likewise, on HotpotQA, the gold supporting facts are used strictly to compute training rewards and are never exposed as inputs to the policy.

Nevertheless, we completely understand your concern regarding reward-supervision parity. In addressing Reviewer UBJd's comment, we conducted an additional set of experiments on HotpotQA that strictly aligns the reward supervision (PVF) across methods to compare our grouping strategy against GiGPO's grouping strategy on an equal footing. We share these results below as well (where Groups D, E, and F are existing results from the paper included for reference):

DatasetGroupMethodAnswer EM (%)Answer F1 (%)Supporting Fact EM (%)Supporting Fact F1 (%)Joint EM (%)Joint F1 (%)
HotpotQAATrajectory-level GRPO + PVF39.5450.9616.3744.6412.3532.32
HotpotQABPVF + GiGPO-style anchor-state grouping40.0551.6417.4246.3313.8633.91
HotpotQACOur turn-level grouping + PVF (w/o OPBR)40.9852.1517.6946.1014.5934.23
HotpotQADFull GiGPO39.5651.4216.7144.7912.7032.35
HotpotQAETrajectory-level GRPO37.2749.0314.9642.1111.4929.88
HotpotQAFHBR-GRPO41.9652.6020.2448.9114.4335.15

Because Groups A, B, and C compare trajectory-level GRPO, GiGPO-style grouping, and HBR-GRPO's grouping under the exact same PVF reward supervision signal, we hope this comparison helps address your concern.

AuthorsOpenReview ↗

Response to the review ( part two )

Unfair S-SFT Initialization We acknowledge that this discrepancy existed in the original main table (Table 1). However, we would like to note that our subsequent horizontal comparisons against other RL algorithms in Tables 4 and 5 already used the exact same S-SFT initialization across all methods, ensuring a fair comparison on Wordle, GuessWhat, and HotpotQA where HBR-GRPO consistently achieves superior performance. Please also allow us to briefly explain our original rationale for Table 1: because the search-aligned S-SFT initialization is itself part of our proposed pipeline, we originally wanted Table 1 to cleanly separate the untouched Playpen baselines from the entirety of the contributions introduced in our paper, and therefore reported the original Playpen SFT initialization for the baselines.

That said, we fully agree that your concern regarding initialization fairness is valid. We have therefore re-trained and re-evaluated all six baseline model families in the main table using S-SFT initialization. The results are as follows:

Wordle:

ModelConfigurationACC (%)QualityPlayed (%)Clemscore
Qwen3-8BS-SFT-only7.746.2250.603.15
Qwen3-8BS-SFT + Standard GRPO8.936.8361.904.23
Gemma-2-9B-ITS-SFT-only8.936.4256.553.63
Gemma-2-9B-ITS-SFT + Standard GRPO8.936.9458.334.05
Mistral-7B-Instruct-v0.3S-SFT-only0.000.0073.210.00
Mistral-7B-Instruct-v0.3S-SFT + Standard GRPO0.000.0070.830.00
Falcon-7B-InstructS-SFT-only0.000.0079.170.00
Falcon-7B-InstructS-SFT + Standard GRPO0.000.0072.020.00
GLM-4-9B-ChatS-SFT-only7.146.6286.315.71
GLM-4-9B-ChatS-SFT + Standard GRPO7.747.1086.316.13
DeepSeek-LLM-7B-ChatS-SFT-only2.383.1358.931.85
DeepSeek-LLM-7B-ChatS-SFT + Standard GRPO0.000.0061.310.00

GuessWhat:

ModelConfigurationACC (%)QualityPlayed (%)Clemscore
Qwen3-8BS-SFT-only35.0035.00100.0035.00
Qwen3-8BS-SFT + Standard GRPO35.0035.00100.0035.00
Gemma-2-9B-ITS-SFT-only50.0031.67100.0031.67
Gemma-2-9B-ITS-SFT + Standard GRPO48.3336.67100.0036.67
Mistral-7B-Instruct-v0.3S-SFT-only21.6721.67100.0021.67
Mistral-7B-Instruct-v0.3S-SFT + Standard GRPO26.6725.00100.0025.00
Falcon-7B-InstructS-SFT-only0.000.0045.000.00
Falcon-7B-InstructS-SFT + Standard GRPO0.000.0048.330.00
GLM-4-9B-ChatS-SFT-only33.3333.33100.0033.33
GLM-4-9B-ChatS-SFT + Standard GRPO31.6731.67100.0031.67
DeepSeek-LLM-7B-ChatS-SFT-only21.6721.67100.0021.67
DeepSeek-LLM-7B-ChatS-SFT + Standard GRPO33.3331.67100.0031.67

Even under identical S-SFT initialization across all models, none of these baselines surpass HBR-GRPO on the primary metric (Clemscore: 8.50 on Wordle and 48.33 on GuessWhat).

AuthorsOpenReview ↗

Response to the review ( part three )

Applying HBR-GRPO to Scenarios Without Enumerable Answers or Reference Action Libraries First, we would like to note that our formulation of HBR-GRPO on HotpotQA already removes the requirement that the answer space must be enumerable: on HotpotQA, our HP reward is defined as the increment in Intersection-over-Union (IoU) between the model's selected evidence and the gold supporting facts, which does not require exhaustively enumerating all possible answers or evidence. Thus, the essential prerequisite for the HP reward is not an enumerable answer space, but rather the ability to construct a reliable heuristic progress evaluation function. In many practical domains, such progress signals can be naturally constructed. For instance, in LLM coding tasks, a viable heuristic progress function is the turn-over-turn increment in the number of passed unit test cases induced by each generated code patch (indeed, to our knowledge, several LLM coding benchmarks—particularly those targeting long-horizon software engineering capabilities such as SWE-CI and SWE-Marathon —already adopt similar progress-based scoring designs).

For scenarios where a pre-built reference action library is unavailable, one potential approach is to sample candidate actions directly from the policy itself and estimate their reference values via short-horizon rollouts (analogous to traditional Bellman action-value estimation) before performing within-group normalization, or alternatively, to query a stronger teacher model to generate reference actions at each visited state. However, we candidly note that these remain preliminary directions for future work; for the regimes where HBR-GRPO has demonstrated reliable empirical gains in the present paper, a reference action library is still required.

Thank you for the detailed responses and the additional experiments. I do not think it is enough to give me the reason to raise my rating. i think my original rating is very reasonable for overall quality. i do not recommend to accept this paper.

Reviewer_UBJdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HBR-GRPO trains interactive reasoning policies by combining task-specific progress rewards, finite-horizon search value, and relative advantages among actions sampled from the same history. Progress is measured through candidate-set reduction in the games and supporting-evidence signals in HotpotQA. The policy is initialized with search-aligned SFT, and branching replay refreshes the training states. Experiments report improvements for Qwen3-8B on Wordle, GuessWhat, and a held-out HotpotQA split.

Critical strengths
  1. The paper makes a useful distinction between gathering information and committing to an answer. The singleton-without-success diagnostics (Appendix E, Table 13) show why candidate reduction alone is insufficient and motivate combining heuristic progress with finite-horizon value.

  2. The reported end-to-end gains are encouraging, with five training seeds for the main game results. In the single-seed HotpotQA experiment, Joint F1 improves from 32.65 for IGPO to 35.15. The disjoint development/test splits and detailed checkpoint records make the evaluation easier to assess.

Critical weaknesses
  1. The checkpoint selection leaves some uncertainty about the contribution of branching replay. Table 7 starts OPBR after 500 fixed-state steps, while Tables 11–12 select checkpoints before any OPBR updates for two Wordle seeds and three GuessWhat seeds. Selecting these checkpoints is reasonable, but their results do not by themselves establish a replay benefit. For example, Wordle seed 20260713 is selected at step 200, and it is unclear whether the PVF+OPBR row in Tables 2 and 4 uses that checkpoint or another run/checkpoint. Clarifying this mapping, together with a matched fixed-state comparison, would make the contribution of replay easier to assess.

  2. A similar question concerns the benefit of sampling fresh actions from the same history. Table 4 changes reward, grouping, and replay together, as the authors acknowledge (lines 418–421), while Tables 2–3 do not ablate grouping. The GiGPO comparison in the single-seed HotpotQA experiment is relevant, but GiGPO groups repeated environment states across trajectories, whereas HBR samples new actions at the same history. A comparison using the same rewards and budget would help separate the benefit of this grouping choice from that of stronger reward supervision.

Questions for authors
  1. Which run/checkpoint produced each seed-20260713 ablation row in Tables 2–4? In particular, does the Wordle PVF+OPBR row use the step-200 checkpoint in Table 11, before the end of the 500-step fixed-state stage? If the existing runs allow it, a fixed-state versus OPBR comparison on one task with the same total budget and selection rule would help clarify the replay benefit.

  2. Is there an existing comparison that separates fresh same-history grouping from trajectory-level credit while keeping PVF rewards, S-SFT, training states, and generation budget fixed? If not, one matched control on a representative task, with uncertainty reported, would address the main remaining question about this component.

  3. In the broader discussion, it would be interesting to compare HBR's external heuristics and reference search with propagating rewards from observed successful states, as in RewardFlow. A short discussion of CoDaPO could also help connect the grouping choice to the question of how confidence and within-group reward variation affect useful updates.

6 comments
AuthorsOpenReview ↗

Response to the review ( part one )

Dear Reviewer,

We sincerely thank you for your time and effort in reviewing our manuscript, for recognizing the contributions of our work, and for raising such insightful and constructive questions. Below, we address each of your comments in detail.

Checkpoint Selection This is a very reasonable concern, and we apologize for not stating this detail explicitly in the Checkpoint Selection section of the appendix. In all ablation experiments involving OPBR components, checkpoint selection is strictly restricted to checkpoints after global step 500—meaning the earliest candidate checkpoint eligible for selection is step 600 (for instance, in the Wordle OPBR + PVF ablation, the selected checkpoint is step 1,250 rather than step 200).

Furthermore, in response to you and Reviewer icJF's question regarding disentangling the individual contributions within OPBR, we conducted eight additional ablation runs isolating the OPBR sub-components. All eight of these newly added experiments strictly follow the same selection rule—selecting the best-performing checkpoint on the development set starting from step 600 onward. We hope this clarification fully resolves your concern.

AuthorsOpenReview ↗

Response to the review ( part two )

Simultaneous Changes to Grouping and Reward Signals in Table 4 First, please allow us to clarify that the primary purpose of Tables 4 and 5 was to provide an end-to-end horizontal comparison between our complete algorithm and existing RL algorithms across these benchmarks. However, we fully agree that disentangling and comparing our grouping strategy and credit-assignment mechanism against those of other RL algorithms is a highly valuable investigation. Given the time and computational constraints of the rebuttal period, we conducted a "cross-method ablation study" between GiGPO and our approach on HotpotQA to separate the effects of credit assignment and action grouping. Disabling OPBR and starting from the exact same S-SFT initialization, we evaluated three controlled configurations: (A) Trajectory-level GRPO + PVF reward, (B) GiGPO's grouping strategy (grouping actions by matching similar historical anchor states across different sampled trajectory branches) + PVF reward, and (C) HBR-GRPO's grouping strategy (sampling multiple actions conditioned on the exact same history) + PVF reward. The results are presented below:

DatasetGroupMethodAnswer EM (%)Answer F1 (%)Supporting Fact EM (%)Supporting Fact F1 (%)Joint EM (%)Joint F1 (%)
HotpotQAATrajectory-level GRPO + PVF39.5450.9616.3744.6412.3532.32
HotpotQABPVF + GiGPO-style anchor-state grouping40.0551.6417.4246.3313.8633.91
HotpotQACOur turn-level grouping + PVF (w/o OPBR)40.9852.1517.6946.1014.5934.23
HotpotQADFull GiGPO39.5651.4216.7144.7912.7032.35
HotpotQAETrajectory-level GRPO37.2749.0314.9642.1111.4929.88
HotpotQAFHBR-GRPO41.9652.6020.2448.9114.4335.15

(Notes: 1. Group B adopts only GiGPO's grouping mechanism; rather than using GiGPO's original credit assignment that combines trajectory-level training signals with Bellman-style step-level rewards, it replaces the credit assignment entirely with HBR-GRPO's heuristic-progress-based PVF credit assignment. 2. Rows D, E, and F are not new runs, but are reproduced from the existing tables in the paper for convenient side-by-side comparison.)

The encouraging takeaway for HBR-GRPO from these results is that comparing Group D with Group B—i.e., original GiGPO versus GiGPO's grouping paired with our credit assignment—shows consistent gains for Group B across all six metrics. A similar pattern holds when comparing Group A and Group E, which partially demonstrates the effectiveness of our heuristic-progress-based step-level credit assignment. At the same time, we candidly acknowledge that when comparing Group B and Group C—holding the credit-assignment mechanism fixed while comparing our same-history grouping against GiGPO's cross-branch anchor-state grouping—our grouping method does not exhibit a pronounced margin, and even trails GiGPO slightly on Supporting Fact F1. Indeed, we find GiGPO's paradigm of first sampling complete trajectories and then grouping actual actions by similar historical states to be very elegant, and we plan to investigate this direction further in future work.

AuthorsOpenReview ↗

Response to the review ( part three )

Disentangling Whether the Contribution of OPBR Stems from Reference Actions Introducing Better States or from On-Policy Transitions Mitigating the Mismatch Between Pre-sampled and Rollout Trajectories Your point is very well taken—nothing addresses this question more directly than a fine-grained ablation study that isolates each individual sub-component. Using turn-level S-SFT initialization and PVF as the reward, while keeping the total training budget fixed at 1,250 steps (500 fixed-state PVF steps followed by OPBR variants starting at step 501), we designed four controlled settings on seed 20260713: (1) adding only states reached by the current policy's actual actions to the training pool, (2) current-policy action states + historical state replay, (3) current-policy action states + reference-action branches, and (4) the full OPBR method. The results are summarized below:

DatasetGroupComponent ConfigurationACC (%)QualityPlayed (%)Clemscore
Wordle1Policy states only (actual actions)11.9012.2054.176.61
Wordle2Historical state replay + Policy states11.3111.4657.146.55
Wordle3Reference branches + Policy states10.7110.6158.936.25
Wordle4Full OPBR12.5010.0968.456.90
GuessWhat1Policy states only (actual actions)51.6749.44100.0049.44
GuessWhat2Historical state replay + Policy states48.3348.33100.0048.33
GuessWhat3Reference branches + Policy states46.6746.67100.0046.67
GuessWhat4Full OPBR51.6751.67100.0051.67

To facilitate comparison, we also summarize the pairwise Clemscore differences across key groups below:

ComparisonControlled Condition and Examined IncrementWordleGuessWhat
Group 2 - Group 1Adding historical state replay without reference branches-0.06-1.11
Group 3 - Group 1Adding reference branches without historical state replay-0.36-2.77
Group 4 - Group 2Adding reference branches in the presence of historical replay+0.35+3.34
Group 4 - Group 3Adding historical state replay in the presence of reference branches+0.65+5.00
Group 4 - Group 1Full OPBR relative to current-policy states only+0.29+2.23

As shown by the results, populating the training pool with states induced by the model's own on-policy actions accounts for the most substantial gain, whereas adding historical state replay or reference-action branches in isolation does not yield a clear standalone improvement; rather, they are beneficial when combined in full OPBR. We have incorporated these new experiments and analyses into the Ablation Study section of the main text.

Reviewer_icJFOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

HBR-GRPO trains multi-turn agents using state-conditioned groups of candidate actions. It combines a heuristic-progress reward, based on reduction of the feasible candidate set, with a finite-horizon search-value reward that favors successful termination. A state-dependent mixture, Progress-Value Fusion (PVF), balances these signals. On-Policy Branching Replay (OPBR) refreshes the training pool using current-policy trajectories, historical states, and audited alternative branch successors. The method uses search-aligned supervised initialization. Experiments with Qwen3-8B report gains on Wordle and GuessWhat, five-seed main-method results.

Critical strengths
  1. The good point of this paper is to combine heuristic progress and finite-horizon search value with same-state action comparisons and branching replay, providing a coherent framework for learning informative actions and timely task completion.

  2. HBR-GRPO improves Qwen3-8B's performance on Wordle, GuessWhat and HotpotQA.

Critical weaknesses
  1. A uniquely determined answer in the environment does not establish that the policy has correctly inferred it. Failure may instead reflect inaccurate state tracking or answer selection. The main-text analysis does not separate continued information gathering from incorrect or invalid answers after the solution becomes identifiable, leaving the proposed stopping-related explanation insufficiently supported.

  2. The gains in experimental results could reflect better states supplied by reference actions rather than the claimed mitigation of state-distribution mismatch. A controlled comparison of current-policy state refresh alone against refresh with reference-action branches would clarify this distinction.

Questions for authors
  1. Could the authors enlarge the text within the figures, especially Figure 1? Several labels are too small to read comfortably at normal viewing size.

  2. Could the authors consistently round the numerical results in Tables 2, 3, and 4 to two decimal places? This would make the tables easier to scan and compare.

2 comments
AuthorsOpenReview ↗

Response to the review ( part one )

Dear Reviewer,

We sincerely thank you for your time and effort in reviewing our manuscript, for recognizing the contributions of our work, and for raising such insightful and constructive questions. Below, we address each of your comments in detail.

Font Size and Formatting Issues We would like to start with these presentation issues. We sincerely apologize for overlooking that the font size in the overview schematic appeared too small when rendered on the PDF page, as well as the inconsistent decimal precision across our tables. These were oversights on our part, and we have promptly corrected both in the revised manuscript: the text in Figure 1 has been enlarged as much as possible while preserving all original information, and all numerical entries in the tables have been standardized to two decimal places.

Characterization of Model Behavior When Only a Single Candidate Answer Remains Prompted by your insightful observation, we inspected the logged model action trajectories on our experimental server. While it is difficult in Wordle to strictly separate whether an action is intended to gather information or to submit a final answer, the boundary between information-gathering queries and final answer submissions is explicit in GuessWhat. Examining the action trajectories trained with the Heuristic Progress (HP)-only reward (seed 20260713, corresponding to our ablation study), we found 22 episodes in which a singleton candidate answer was reached prior to the model's final turn. The breakdown of the model's actions in these 22 episodes is as follows:

Action TypeEpisodesProportion (out of 22 episodes)
Correct answer1359.09%
Incorrect answer627.27%
Invalid output00.00%
Continuing to ask questions313.64%

These data fully support your point. Our original phrasing in the manuscript—attributing the model's failures in this state to "failing to realize that the correct answer has already been uniquely determined" or "failing to learn when to stop gathering information in time"—was imprecise. We have systematically revised these descriptions throughout the paper to more accurate statements such as: "When trained solely with the Heuristic Progress reward, the model makes substantial progress toward narrowing down the final answer, but struggles to finish the task by submitting the correct final guess."

Additionally, we conducted the same statistical breakdown on the trajectories generated by our full method under the same random seed when a singleton candidate answer is reached (28 episodes in total):

Action TypeEpisodesProportion (out of 28 episodes)
Correct answer2071.43%
Incorrect answer725.00%
Invalid output00.00%
Continuing to ask questions13.57%

While results from a single random seed cannot establish a high-confidence statistical claim on their own, the empirical evidence indicates that incorporating the FSV reward in our full method improves task-completion behavior when only one feasible candidate remains, compared to training with the HP reward alone.

AuthorsOpenReview ↗

Response to the review ( part two )

Disentangling Whether the Contribution of OPBR Stems from Reference Actions Introducing Better States or from On-Policy Transitions Mitigating the Mismatch Between Pre-sampled and Rollout Trajectories Your point is very well taken—nothing addresses this question more directly than a fine-grained ablation study that isolates each individual sub-component. Using turn-level S-SFT initialization and PVF as the reward, while keeping the total training budget fixed at 1,250 steps (500 fixed-state PVF steps followed by OPBR variants starting at step 501), we designed four controlled settings on seed 20260713: (1) adding only states reached by the current policy's actual actions to the training pool, (2) current-policy action states + historical state replay, (3) current-policy action states + reference-action branches, and (4) the full OPBR method. The results are summarized below:

DatasetGroupComponent ConfigurationACC (%)QualityPlayed (%)Clemscore
Wordle1Policy states only (actual actions)11.9012.2054.176.61
Wordle2Historical state replay + Policy states11.3111.4657.146.55
Wordle3Reference branches + Policy states10.7110.6158.936.25
Wordle4Full OPBR12.5010.0968.456.90
GuessWhat1Policy states only (actual actions)51.6749.44100.0049.44
GuessWhat2Historical state replay + Policy states48.3348.33100.0048.33
GuessWhat3Reference branches + Policy states46.6746.67100.0046.67
GuessWhat4Full OPBR51.6751.67100.0051.67

To facilitate comparison, we also summarize the pairwise Clemscore differences across key groups below:

ComparisonControlled Condition and Examined IncrementWordleGuessWhat
Group 2 - Group 1Adding historical state replay without reference branches-0.06-1.11
Group 3 - Group 1Adding reference branches without historical state replay-0.36-2.77
Group 4 - Group 2Adding reference branches in the presence of historical replay+0.35+3.34
Group 4 - Group 3Adding historical state replay in the presence of reference branches+0.65+5.00
Group 4 - Group 1Full OPBR relative to current-policy states only+0.29+2.23

As shown by the results, populating the training pool with states induced by the model's own on-policy actions accounts for the most substantial gain, whereas adding historical state replay or reference-action branches in isolation does not yield a clear standalone improvement; rather, they are beneficial when combined in full OPBR. We have incorporated these new experiments and analyses into the Ablation Study section of the main text.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Comment to All Reviewers and the Area Chair

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for your time, thoughtful evaluations, and constructive feedback. Your insightful comments have helped us significantly strengthen both the empirical rigor and the clarity of our manuscript. In response to your suggestions, we have conducted extensive additional experiments and updated the manuscript (with all major revisions highlighted in blue in the revised PDF). Below, we summarize the new experiments, analyses, and revisions added during the rebuttal period:

Fine-Grained Ablation of OPBR Sub-Components (Table 2 & Section 3.5; Reviewer 1 & Reviewer 2):

To disentangle whether OPBR's gains stem from states reached by the current policy's actual actions, historical state replay, or reference-action branches, we added four controlled ablation settings on both Wordle and GuessWhat under identical S-SFT initialization and PVF reward (Policy states only, Policy + replay, Policy + ref. states, and Full OPBR). We also explicitly clarified in Appendix D that checkpoint selection for all OPBR-related ablations (both existing and newly added) is strictly restricted to checkpoints after global step 500 (i.e., starting from step 600 onward).

Fair S-SFT Initialization Across All Six Baseline Model Families (Table 1 & Appendix H; Reviewer 3):

To ensure complete fairness in initialization, we re-trained and evaluated all six open-weight model families (Qwen3-8B, Gemma-2-9B-IT, Mistral-7B-Instruct-v0.3, Falcon-7B-Instruct, GLM-4-9B-Chat, and DeepSeek-LLM-7B-Chat) under both S-SFT-only and S-SFT + Standard GRPO on Wordle and GuessWhat. We updated Table 1 with these aligned S-SFT baselines (while preserving the original Playpen trajectory-level SFT baselines in Appendix H). HBR-GRPO continues to outperform all baselines on the primary metric (Clemscore) across both tasks.

Reward-Aligned Cross-Method Ablation of Credit Assignment vs. Grouping (Appendix J; Reviewer 2 & Reviewer 3):

To separate the effects of action grouping and credit assignment under strictly identical reward supervision, we conducted controlled experiments on HotpotQA (with S-SFT initialization, PVF reward, and without OPBR) comparing: (A) Trajectory-level GRPO + PVF, (B) PVF + GiGPO-style anchor-state grouping, and (C) Our turn-level same-history grouping + PVF.

Empirical Comparison and Discussion with RewardFlow and CoDaPO (Table 5, Section 4, & Appendix I; Reviewer 2):

We implemented and evaluated RewardFlow on HotpotQA under the exact setup of Table 5, and added a detailed discussion comparing its state-graph reward propagation with our heuristic-progress and search-value formulation. We discussed the complementarity between CoDaPO and HBR-GRPO, and compiled detailed training-group diagnostics on GuessWhat (including zero-variance group ratios, distinct action counts U U, reward standard deviation distributions, and their progression across training windows) in Appendix I.

Singleton-State Behavioral Diagnostics and Presentation Refinements (Figure 1, Tables 2–4, Section 2.3, Section 3.7, & Appendix E; Reviewer 1):

We conducted a fine-grained breakdown of model actions when only a single candidate answer remains on GuessWhat (comparing HP-only against full HBR-GRPO), and revised our attribution throughout the paper from "failing to know when to stop gathering information" to the more accurate characterization of failing to finish the task with a valid, correct final guess. We enlarged the text in Figure 1 for better legibility and standardized all numerical entries across Tables 2–4 to two decimal places.

We hope these new results and clarifications thoroughly address your questions, and we remain happy to provide any further information during the discussion phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.