acceptodds
Under review as a conference paper at ICLR 2027

Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards

Abstract

Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choices the policy reconsiders, suggesting where to sample alternative continuations. We introduce Hindsight-Divergence Localization (HDL), which uses hindsight-induced changes in token log-likelihoods to select branch points. HDL generates a small number of complete root trajectories and fills each training group with continuations from the selected positions under the original task context. Each continuation reuses its root prefix and contributes policy updates only through its newly generated suffix, reducing generation cost while focusing additional exploration and learning on decisions after branching. Experiments with three models across math, code, and agent tasks show gains in both rollout efficiency and task performance. Compared with GRPO at matched group sizes and training steps, HDL yields up to a 2.5 reduction in generated tokens and a 1.8 speedup in rollout wall-clock time. Despite this reduced generation budget, HDL improves performance across all three domains, with gains of up to 12.5 points on agent tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning with Verifiable Rewards, Rollout Efficiency, Hindsight-Guided Localization, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_E1jBOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hindsight-Divergence Localization (HDL) to reduce rollout costs in group-relative RL training. After generating complete root trajectories, the policy produces reflections from verifier feedback and measures changes in sampled-token log-likelihoods under hindsight conditioning. Positions with large changes become branch points for additional continuations. Shared prefixes are reused, and continuation losses apply only to newly generated suffixes. Experiments evaluate rollout efficiency and downstream performance across mathematical reasoning, code generation, and ScienceWorld.

Critical strengths

Allocating additional rollouts to intermediate decisions is a reasonable way to reduce redundant generation. The hindsight-based selection rule is straightforward to implement and does not require a separately trained localization model.

The rollout efficiency measurements account for reflection generation and hindsight scoring, rather than reporting only the shorter continuations. This makes the measured wall-clock reductions more informative than generated-token counts alone.

The experiments cover three model backbones and three domains. The controlled comparisons with entropy-based and explicit reflection-based localization provide useful evidence about alternative ways to choose branch points.

Critical weaknesses

The evaluation protocol does not provide a sufficiently reliable basis for the performance claims. Section 4.1 states that checkpoints are selected according to their average scores on the reported benchmarks, without identifying a separate validation set. As written, this appears to use the evaluation benchmarks for model selection. Moreover, the reported standard deviations are calculated across the three selected checkpoints, rather than independent training runs. They therefore do not establish reproducibility or training stability, particularly for the small improvements reported on Math and Code.

The demonstrated contribution beyond existing branching methods is limited. Prefix reuse and branching from intermediate positions are already established approaches; HDL mainly introduces another branch-ranking heuristic. The comparisons with Entropy and Reflection are useful, but random and fixed-depth branching controls are absent. On Qwen3-8B, HDL exceeds Entropy by only 0.14 points on Math and 0.19 points on Code. Stronger evidence is needed to attribute the benefit specifically to hindsight-divergence localization.

The broader performance claims are stronger than the results support. The clearest gains occur on the Qwen models in ScienceWorld, while Math and Code improvements are generally small. On Llama3.1-8B, HDL falls below GRPO on both Math and Code and below DAPO on ScienceWorld. These results support a potentially useful rollout-efficiency trade-off, but do not establish a broadly superior training method across models and domains.

The algorithm is underspecified in ways that affect both interpretation and reproduction. Branch-point eligibility and separation constraints are mentioned without concrete rules, and intermediate environment restoration in ScienceWorld is not explained. The method also pools roots and correlated continuations under one group baseline after selecting branch points using completed outcomes. Sampling suffixes under the original context does not by itself establish that the resulting estimator retains the properties of independently sampled GRPO trajectories. The paper needs a clearer account of this change and its consequences.

Questions for authors

Were checkpoints selected using a separate validation set or the benchmarks reported in Tables 1–3? How do the conclusions change when reporting the final checkpoint or a validation-selected checkpoint?

How many independent training runs were conducted for each configuration? Can you report variation across training seeds rather than across the three highest-scoring checkpoints?

How does HDL compare with random or fixed-depth branching under matched root counts, continuation counts, and rollout budgets? Are the conclusions preserved when controlling for branch depth and suffix length?

How does outcome-dependent branch selection and the shared group baseline affect gradient estimation with correlated continuations? Is the update intended as a modified training objective, or as an estimator of the original policy objective?

What are the exact branch-point eligibility and minimum-separation rules? How are ScienceWorld states restored at intermediate branch points, and are restoration or replay costs included in the reported rollout times?

3 comments

We thank the reviewer for the detailed and rigorous review, and for recognizing that HDL requires no separately trained localization model, that our efficiency measurements include reflection generation and hindsight scoring, and that the controlled comparisons with entropy- and reflection-based localization are informative. We appreciate the careful questions on the evaluation protocol, branching baselines, and algorithmic details, and address each of them below.

W1, Q1, Q2 (evaluation protocol and seeds). We thank the reviewer for raising these points. In the paper, checkpoints were selected on the benchmarks reported in Tables 1–3, without a separate validation set. We agree that the reported standard deviation reflects variation across checkpoints of a single run rather than across training seeds. Multi-seed training is costly in our setting: one seed of the ScienceWorld experiments takes about two days, and one Math or Code run takes nearly two weeks. Within our compute budget, we have re-run the Qwen3-8B ScienceWorld experiments with two additional seeds (seeds 1 and 2; the paper reports seed 0). The table below reports the results of the three runs.

MethodMean ± std
GRPO63.44 ± 3.25
DAPO65.16 ± 3.63
HDL70.70 ± 2.91

To check the effect of checkpoint selection, we also report the mean of the last three evaluated checkpoints instead of the three highest-scoring ones:

DomainModelGRPODAPOHDL
MathQwen3-4B51.1352.4851.44
MathQwen3-8B52.6852.8953.47
MathLlama3.1-8B51.4253.3451.00
CodeQwen3-4B51.6453.3252.22
CodeQwen3-8B55.1954.8055.19
CodeLlama3.1-8B54.6454.0954.33
AgentQwen3-4B56.8959.4866.98
AgentQwen3-8B58.8659.8971.96
AgentLlama3.1-8B68.9075.0171.59

W2, Q3 (comparison with other branching rules). We thank the reviewer for this suggestion. We trained a random branch-point baseline on ScienceWorld with Qwen3-8B. It is identical to HDL, including the number of roots, branch points, and continuations and the constraints on where a branch point may be placed, except that the branch points are drawn uniformly at random among the eligible positions instead of being selected by the hindsight-divergence score. Following the protocol in our paper, we report the mean and standard deviation of the three highest-scoring checkpoints: 64.65 ± 1.11. For the last three checkpoints, the result is 59.73 ± 3.84. HDL and the entropy- and reflection-based localization methods (Table 4) all significantly outperform random branch-point selection.

W3 (scope of the performance claims). On Math and Code, the differences among methods are small, and this includes DAPO, which samples far more tokens. Per training step, DAPO generates 3.4–3.9× as many tokens as HDL on Math and 1.9–2.2× on Code (Figure 2), and its rollouts take 2.6–3.2× and 1.5–1.7× as long. With this budget, DAPO improves over GRPO by only 0.30–1.85 points on Math. HDL stays within 2 points of DAPO in all six Math and Code settings and scores higher in three of them, while generating 47–75% fewer tokens and spending 34–69% less rollout time.

We thank the reviewer for pointing this out and will revise the claim to make it more precise. In addition, to further demonstrate the performance gains of our method, we trained a terminal agent with HDL and GRPO. Starting from Qwen3-8B, both methods are trained on 560 environments from SETA-Env (arXiv:2607.10891) and evaluated on Terminal-Bench 2.0. Following the protocol in our paper, we report the mean and standard deviation of pass@8 over the three checkpoints with the highest scores. HDL reaches 9.36 ± 1.72%, compared with 6.74 ± 1.95% for GRPO, starting from 2.25% for the base model. Using the last three checkpoints gives the same results. This further demonstrates that the gains of HDL extend beyond ScienceWorld to a more challenging agentic setting with long-horizon, multi-turn interaction.

W4, Q5 (branch-point rules and ScienceWorld restoration). Our setting requires the two branch points selected within a root to be at least 64 tokens apart. In ScienceWorld, the state at a branch point is restored by resetting the same task variation and replaying the root's actions up to that point; the simulator is deterministic, so the replay reproduces the root's state. The reported rollout times include this replay.

Q4 (gradient estimation with shared prefixes). Each continuation is sampled from the policy conditioned on the reused prefix, y~≥i∼πθold(⋅∣x,y<i)\tilde{y}{\ge i} \sim \pi{\theta_{\text{old}}}(\cdot \mid x, y_{<i}) (Eq. 3). The prefix is a condition of this sample, not part of it, so the continuation's policy-gradient term involves only the tokens it generated. The prefix tokens were sampled once, by the root, and receive their loss through the root. Applying every trajectory's loss to the shared prefix would update the same sampled decisions once per descendant: a prefix with KK continuations would be counted K+1K+1 times, a multiplicity set by our allocation rather than by the policy. It would also move the prefix according to the outcomes of suffixes that diverge only after it. Because the prefix is identical within a subtree, it cannot account for the differences among the continuations' outcomes; these differences arise from decisions after the branch point, which is where the continuation loss is applied. Continuation returns still enter the root's advantage through the shared group mean, as part of its baseline (as described in section 3.3 in the paper), and the objective remains identical to the GRPO baseline.

Dear Reviewer E1jB,

Thank you again for your detailed and rigorous review, and for your follow-up note pointing out the main concerns, which helped us focus our responses.

In our responses, we have addressed the three main concerns you identified. For the evaluation protocol (Q1), we reported results across three training seeds and with the last three checkpoints. For the contribution of hindsight-divergence localization (Q3), we added a random branch-point baseline under the same setup as HDL. For gradient estimation with shared prefixes (Q4), we explained how the prefix receives credit through the root and why the objective remains identical to the GRPO baseline. We have also added a terminal-agent experiment on Terminal-Bench 2.0.

The discussion period ends in less than two days. We hope to hear whether our responses resolve these concerns, and we are glad to provide further results or clarification if anything remains open.

Thank you for your time and for your engagement with our work.

Reviewer_y4bNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hindsight-Divergence Localization (HDL), a rollout strategy for group-relative RLVR. For each problem, HDL samples M complete “root” trajectories, prompts the policy to reflect on the verifier feedback, and re-scores the root tokens with and without the hindsight context. The positions with the largest absolute log-likelihood change are selected as branch points, and the remaining G − M slots are filled with continuations from these positions under the original context.

Critical strengths
  • The idea is simple and intuitive. Using hindsight divergence instead of entropy to choose branch points is novel.
  • Consistent rollout savings across three domains and three models
  • The analysis in Section 4.4 gives a nice insight into why HDL differs from Entropy.
Critical weaknesses
  • The claim “improves performance across all three domains” is overstated. On Llama3.1-8B, HDL is below GRPO on Math (51.39 vs 51.49) and Code (54.34 vs 54.87), and below DAPO on Agent (71.81 vs 75.35). On Qwen3-4B Math, HDL is also below DAPO. Most gains on Math and Code are within 1 point.

  • The reported std is computed over the top-3 checkpoints of one run, and these checkpoints are selected by the test benchmark score itself. This is not a proper variance estimate, and selecting on the test set inflates all numbers. Multiple training seeds are needed.

  • Lack of a random branch-point baseline. Without it, it’s unclear whether the improvement comes from the hindsight signal or simply from prefix branching. Note that Entropy is very close to HDL on Math (54.02 vs 54.16) and Code (55.37 vs 55.56)

Questions for authors
  • What is the contribution of the reflection r versus the raw feedback f in the hindsight context? Does feedback only work?
  • Why use the absolute value in Eq. 2? Did you try signed divergence (e.g., only decreases for failed roots)?
  • Is there a minimum distance constraint between the two selected positions in HDL? Section 4.4 mentions “branch-point constraints” but they are not specified.
3 comments

We thank the reviewer for the careful reading and for finding the idea simple, intuitive, and novel, the rollout savings consistent across domains and models, and the analysis in Section 4.4 insightful. We appreciate the constructive suggestions and address each of them below.

W1 (scope of the performance claim). On Math and Code, the differences among methods are small, and this includes DAPO, which samples far more tokens. Per training step, DAPO generates 3.4–3.9× as many tokens as HDL on Math and 1.9–2.2× on Code (Figure 2), and its rollouts take 2.6–3.2× and 1.5–1.7× as long. With this budget, DAPO improves over GRPO by only 0.30–1.85 points on Math. HDL stays within 2 points of DAPO in all six Math and Code settings and scores higher in three of them, while generating 47–75% fewer tokens and spending 34–69% less rollout time.

We thank the reviewer for pointing this out and will revise the claim to make it more precise. In addition, to further demonstrate the performance gains of our method, we trained a terminal agent with HDL and GRPO. Starting from Qwen3-8B, both methods are trained on 560 environments from SETA-Env (arXiv:2607.10891) and evaluated on Terminal-Bench 2.0. Following the protocol in our paper, we report the mean and standard deviation of pass@8 over the three checkpoints with the highest scores. HDL reaches 9.36 ± 1.72%, compared with 6.74 ± 1.95% for GRPO, starting from 2.25% for the base model. Using the last three checkpoints gives the same results. This further demonstrates that the gains of HDL extend beyond ScienceWorld to a more challenging agentic setting with long-horizon, multi-turn interaction.

W2 (variance and checkpoint selection). We agree that the reported standard deviation reflects variation across checkpoints of a single run rather than across training seeds. Multi-seed training is costly in our setting: one seed of the ScienceWorld experiments takes about two days, and one Math or Code run takes nearly two weeks. Within our compute budget, we have re-run the Qwen3-8B ScienceWorld experiments with two additional seeds (seeds 1 and 2; the paper reports seed 0). The table below reports the results of the three runs.

MethodMean ± std
GRPO63.44 ± 3.25
DAPO65.16 ± 3.63
HDL70.70 ± 2.91

W3 (random branch-point baseline). We thank the reviewer for this suggestion. We trained a random branch-point baseline on ScienceWorld with Qwen3-8B. It is identical to HDL, including the number of roots, branch points, and continuations and the constraints on where a branch point may be placed, except that the branch points are drawn uniformly at random among the eligible positions instead of being selected by the hindsight-divergence score. Following the protocol in our paper, we report the mean and standard deviation of the three highest-scoring checkpoints: 64.65 ± 1.11. For the last three checkpoints, the result is 59.73 ± 3.84. HDL and the entropy- and reflection-based localization methods (Table 4) all significantly outperform random branch-point selection.

Q1 (reflection versus raw feedback). The hindsight context contains both the verifier feedback and a reflection, h=[f;r]h=[f;r]. We add the reflection because the raw feedback is often too brief to change how the model assesses its earlier decisions. For a correct attempt, the feedback is only the word "Correct." in Math and a count of passed tests in Code, so conditioning on it has little effect. A reflection can be written for both correct and incorrect attempts, and it relates the outcome to the approach the attempt took (Appendix B.1).

Q2 (absolute value). The absolute score measures how strongly hindsight changes support for the sampled token. A decrease can flag a choice to reconsider after failure, which is the common case in failed roots. An increase can flag a choice that hindsight identifies as important to a successful attempt. Sampling alternatives from the same prefix then tests what happens when that choice changes, and the group contrasts the confirmed choice with these alternatives.

An example from the Qwen3-8B ScienceWorld run illustrates this. The task asks the agent to focus on the animal with the longest life span and then on the one with the shortest. A successful root ends with focus on baby dragonfly, and its reflection states that the agent identified "the baby dragonfly as the shortest." Under this hindsight context, the log-likelihood of the token dragon rises by 1.14 nats, and HDL selects it as a branch point. Three of the four continuations keep focus on baby and change the object to rabbit; all three score 0. The fourth repeats dragonfly and scores 1.

To check whether this example reflects a broader pattern, we examined all branch points in successful roots of the same run whose hindsight-divergence score exceeds 1 nat. At points where hindsight increases the likelihood of the sampled token, 14.3% of the continuations end with a lower score than the root, compared with 5.4% at points where hindsight decreases it. Changing a choice that hindsight supports therefore turns a success into a failure more often. Branching at such a point places the successful choice and its failing alternatives in the same group. The gap persists when both kinds of points are compared at the same depth in the trajectory. It widens when the branch point lies on an action argument, as in the example above: 20.2% vs. 6.0%. Increase points are less frequent than decrease points. We will add this analysis and the example to Section 4.4.

Q3 (spacing between branch points). Yes. We require the two branch points selected within each root to be at least 64 tokens apart, so that the two points do not fall within the same decision.

Dear Reviewer y4bN,

Thank you again for your careful reading of our paper and for your constructive suggestions, which have helped us strengthen the experiments.

In our responses, we have compared HDL with DAPO in both accuracy and rollout cost, added a terminal-agent experiment on Terminal-Bench 2.0 and a random branch-point baseline under the same setup as HDL, and clarified the role of the reflection, the absolute divergence score, and the spacing between branch points.

Since the discussion period closes in less than two days, we would appreciate it if you could let us know whether these additions address your concerns, or whether there is anything else we should clarify.

We sincerely appreciate your time and look forward to your reply.

Reviewer_BRx3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hindsight-Divergence Localization (HDL), which selects branching points by comparing the log-probabilities of generated tokens under the original context and a context augmented with feedback and reflection. The method reuses trajectory prefixes and applies policy updates only to newly generated suffixes for continuations, reducing generation costs. Evaluations on math, coding, and interactive tasks demonstrate efficiency gains and performance improvements in some settings.

Critical strengths
  1. Using hindsight feedback to decide where to explore further is an interesting idea that offers a complementary perspective to uncertainty-based branching.
  2. The method is relatively simple, combining the localization signal with prefix reuse. The reported reductions in rollout costs and improvements on interactive tasks suggest practical potential.
Critical weaknesses

Expanding trajectories from only two roots by default may limit exploration of early decisions. The trade-off between efficiency and exploration diversity merits further discussion.

Questions for authors
  1. How do the authors balance prefix reuse and exploration diversity? Are there considerations for choosing the number of roots when tasks require exploring different strategies early in a trajectory?
  2. Could the authors explain the rationale for updating shared prefixes only through the root’s advantage? How do they view the possibility of using returns from multiple branches to assess the value of a prefix?
  3. Could the authors further explain the use of the absolute log-probability difference? When hindsight increases support for the original token, what makes that position valuable for branching?
2 comments

We thank the reviewer for the thoughtful review and for recognizing hindsight feedback as a perspective complementary to uncertainty-based branching, as well as the simplicity and practical potential of the method. We also appreciate the questions on exploration, prefix credit, and the absolute divergence score, which we address below.

Exploration and the number of roots (Weakness, Q1). With G=16G=16 fixed, sampling more roots broadens exploration of early strategies and leaves fewer continuations for each selected decision. Table 5 illustrates this trade-off on Qwen3-8B in ScienceWorld. The two-root configurations with two and one branch points per root scored 71.96 and 71.09, respectively. The four-root configuration scored 67.87 and allocated only one or two continuations per branch point. This comparison jointly changes root coverage and the number of continuations per branch point, and supports our default in this setting. For tasks where different strategies diverge early and are difficult to recover through later branching, a larger root allocation is a reasonable choice. We thank the reviewer for this suggestion and this direction is definitely worth exploring. The allocation could also be adjusted dynamically for each problem, for example by sampling additional roots when the initial roots end with the same outcome or when the highest-scoring branch points fall early in the trajectory. We will expand this discussion in Section 4.5.

Prefix credit (Q2). Each continuation is sampled from the policy conditioned on the reused prefix, y~≥i∼πθold(⋅∣x,y<i)\tilde{y}{\ge i} \sim \pi{\theta_{\text{old}}}(\cdot \mid x, y_{<i}) (Eq. 3). The prefix is a condition of this sample, not part of it, so the continuation's policy-gradient term involves only the tokens it generated. The prefix tokens were sampled once, by the root, and receive their loss through the root. Applying every trajectory's loss to the shared prefix would update the same sampled decisions once per descendant: a prefix with KK continuations would be counted K+1K+1 times, a multiplicity set by our allocation rather than by the policy. It would also move the prefix according to the outcomes of suffixes that diverge only after it. Because the prefix is identical within a subtree, it cannot account for the differences among the continuations' outcomes; these differences arise from decisions after the branch point, which is where the continuation loss is applied. Continuation returns still enter the root's advantage through the shared group mean, as part of its baseline (as described in section 3.3 in the paper), and the objective remains identical to the GRPO baseline. Descendant returns could additionally serve as a value estimate for the prefix, which we view as a complementary direction.

Absolute divergence score (Q3). The absolute score measures how strongly hindsight changes support for the sampled token. A decrease can flag a choice to reconsider after failure, which is the common case in failed roots. An increase can flag a choice that hindsight identifies as important to a successful attempt. Sampling alternatives from the same prefix then tests what happens when that choice changes, and the group contrasts the confirmed choice with these alternatives.

An example from the Qwen3-8B ScienceWorld run illustrates this. The task asks the agent to focus on the animal with the longest life span and then on the one with the shortest. A successful root ends with focus on baby dragonfly, and its reflection states that the agent identified "the baby dragonfly as the shortest." Under this hindsight context, the log-likelihood of the token dragon rises by 1.14 nats, and HDL selects it as a branch point. Three of the four continuations keep focus on baby and change the object to rabbit; all three score 0. The fourth repeats dragonfly and scores 1.

To check whether this example reflects a broader pattern, we examined all branch points in successful roots of the same run whose hindsight-divergence score exceeds 1 nat. At points where hindsight increases the likelihood of the sampled token, 14.3% of the continuations end with a lower score than the root, compared with 5.4% at points where hindsight decreases it. Changing a choice that hindsight supports therefore turns a success into a failure more often. Branching at such a point places the successful choice and its failing alternatives in the same group. The gap persists when both kinds of points are compared at the same depth in the trajectory. It widens when the branch point lies on an action argument, as in the example above: 20.2% vs. 6.0%. Increase points are less frequent than decrease points. We will add this analysis and the example to Section 4.4.

Dear Reviewer BRx3,

Thank you again for your thoughtful review and for recognizing hindsight feedback as a perspective complementary to uncertainty-based branching.

In our responses, we have discussed the trade-off between the number of roots and the number of continuations per branch point (Table 5), explained how the shared prefix receives credit through the root, and clarified the role of the absolute divergence score with an analysis of branch points on ScienceWorld.

As the discussion period ends in less than two days, we would be grateful to know whether our responses have resolved your questions. We would be happy to clarify any remaining points.

Thank you for your time, and we look forward to hearing from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.