- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Hindsight-Divergence Localization (HDL) to reduce rollout costs in group-relative RL training. After generating complete root trajectories, the policy produces reflections from verifier feedback and measures changes in sampled-token log-likelihoods under hindsight conditioning. Positions with large changes become branch points for additional continuations. Shared prefixes are reused, and continuation losses apply only to newly generated suffixes. Experiments evaluate rollout efficiency and downstream performance across mathematical reasoning, code generation, and ScienceWorld.
Critical strengths
Allocating additional rollouts to intermediate decisions is a reasonable way to reduce redundant generation. The hindsight-based selection rule is straightforward to implement and does not require a separately trained localization model.
The rollout efficiency measurements account for reflection generation and hindsight scoring, rather than reporting only the shorter continuations. This makes the measured wall-clock reductions more informative than generated-token counts alone.
The experiments cover three model backbones and three domains. The controlled comparisons with entropy-based and explicit reflection-based localization provide useful evidence about alternative ways to choose branch points.
Critical weaknesses
The evaluation protocol does not provide a sufficiently reliable basis for the performance claims. Section 4.1 states that checkpoints are selected according to their average scores on the reported benchmarks, without identifying a separate validation set. As written, this appears to use the evaluation benchmarks for model selection. Moreover, the reported standard deviations are calculated across the three selected checkpoints, rather than independent training runs. They therefore do not establish reproducibility or training stability, particularly for the small improvements reported on Math and Code.
The demonstrated contribution beyond existing branching methods is limited. Prefix reuse and branching from intermediate positions are already established approaches; HDL mainly introduces another branch-ranking heuristic. The comparisons with Entropy and Reflection are useful, but random and fixed-depth branching controls are absent. On Qwen3-8B, HDL exceeds Entropy by only 0.14 points on Math and 0.19 points on Code. Stronger evidence is needed to attribute the benefit specifically to hindsight-divergence localization.
The broader performance claims are stronger than the results support. The clearest gains occur on the Qwen models in ScienceWorld, while Math and Code improvements are generally small. On Llama3.1-8B, HDL falls below GRPO on both Math and Code and below DAPO on ScienceWorld. These results support a potentially useful rollout-efficiency trade-off, but do not establish a broadly superior training method across models and domains.
The algorithm is underspecified in ways that affect both interpretation and reproduction. Branch-point eligibility and separation constraints are mentioned without concrete rules, and intermediate environment restoration in ScienceWorld is not explained. The method also pools roots and correlated continuations under one group baseline after selecting branch points using completed outcomes. Sampling suffixes under the original context does not by itself establish that the resulting estimator retains the properties of independently sampled GRPO trajectories. The paper needs a clearer account of this change and its consequences.
Questions for authors
Were checkpoints selected using a separate validation set or the benchmarks reported in Tables 1–3? How do the conclusions change when reporting the final checkpoint or a validation-selected checkpoint?
How many independent training runs were conducted for each configuration? Can you report variation across training seeds rather than across the three highest-scoring checkpoints?
How does HDL compare with random or fixed-depth branching under matched root counts, continuation counts, and rollout budgets? Are the conclusions preserved when controlling for branch depth and suffix length?
How does outcome-dependent branch selection and the shared group baseline affect gradient estimation with correlated continuations? Is the update intended as a modified training objective, or as an estimator of the original policy objective?
What are the exact branch-point eligibility and minimum-separation rules? How are ScienceWorld states restored at intermediate branch points, and are restoration or replay costs included in the reported rollout times?
3 comments
W4, Q5 (branch-point rules and ScienceWorld restoration). Our setting requires the two branch points selected within a root to be at least 64 tokens apart. In ScienceWorld, the state at a branch point is restored by resetting the same task variation and replaying the root's actions up to that point; the simulator is deterministic, so the replay reproduces the root's state. The reported rollout times include this replay.
Q4 (gradient estimation with shared prefixes). Each continuation is sampled from the policy conditioned on the reused prefix, (Eq. 3). The prefix is a condition of this sample, not part of it, so the continuation's policy-gradient term involves only the tokens it generated. The prefix tokens were sampled once, by the root, and receive their loss through the root. Applying every trajectory's loss to the shared prefix would update the same sampled decisions once per descendant: a prefix with continuations would be counted times, a multiplicity set by our allocation rather than by the policy. It would also move the prefix according to the outcomes of suffixes that diverge only after it. Because the prefix is identical within a subtree, it cannot account for the differences among the continuations' outcomes; these differences arise from decisions after the branch point, which is where the continuation loss is applied. Continuation returns still enter the root's advantage through the shared group mean, as part of its baseline (as described in section 3.3 in the paper), and the objective remains identical to the GRPO baseline.
Dear Reviewer E1jB,
Thank you again for your detailed and rigorous review, and for your follow-up note pointing out the main concerns, which helped us focus our responses.
In our responses, we have addressed the three main concerns you identified. For the evaluation protocol (Q1), we reported results across three training seeds and with the last three checkpoints. For the contribution of hindsight-divergence localization (Q3), we added a random branch-point baseline under the same setup as HDL. For gradient estimation with shared prefixes (Q4), we explained how the prefix receives credit through the root and why the objective remains identical to the GRPO baseline. We have also added a terminal-agent experiment on Terminal-Bench 2.0.
The discussion period ends in less than two days. We hope to hear whether our responses resolve these concerns, and we are glad to provide further results or clarification if anything remains open.
Thank you for your time and for your engagement with our work.
We thank the reviewer for the detailed and rigorous review, and for recognizing that HDL requires no separately trained localization model, that our efficiency measurements include reflection generation and hindsight scoring, and that the controlled comparisons with entropy- and reflection-based localization are informative. We appreciate the careful questions on the evaluation protocol, branching baselines, and algorithmic details, and address each of them below.
W1, Q1, Q2 (evaluation protocol and seeds). We thank the reviewer for raising these points. In the paper, checkpoints were selected on the benchmarks reported in Tables 1–3, without a separate validation set. We agree that the reported standard deviation reflects variation across checkpoints of a single run rather than across training seeds. Multi-seed training is costly in our setting: one seed of the ScienceWorld experiments takes about two days, and one Math or Code run takes nearly two weeks. Within our compute budget, we have re-run the Qwen3-8B ScienceWorld experiments with two additional seeds (seeds 1 and 2; the paper reports seed 0). The table below reports the results of the three runs.
To check the effect of checkpoint selection, we also report the mean of the last three evaluated checkpoints instead of the three highest-scoring ones:
W2, Q3 (comparison with other branching rules). We thank the reviewer for this suggestion. We trained a random branch-point baseline on ScienceWorld with Qwen3-8B. It is identical to HDL, including the number of roots, branch points, and continuations and the constraints on where a branch point may be placed, except that the branch points are drawn uniformly at random among the eligible positions instead of being selected by the hindsight-divergence score. Following the protocol in our paper, we report the mean and standard deviation of the three highest-scoring checkpoints: 64.65 ± 1.11. For the last three checkpoints, the result is 59.73 ± 3.84. HDL and the entropy- and reflection-based localization methods (Table 4) all significantly outperform random branch-point selection.
W3 (scope of the performance claims). On Math and Code, the differences among methods are small, and this includes DAPO, which samples far more tokens. Per training step, DAPO generates 3.4–3.9× as many tokens as HDL on Math and 1.9–2.2× on Code (Figure 2), and its rollouts take 2.6–3.2× and 1.5–1.7× as long. With this budget, DAPO improves over GRPO by only 0.30–1.85 points on Math. HDL stays within 2 points of DAPO in all six Math and Code settings and scores higher in three of them, while generating 47–75% fewer tokens and spending 34–69% less rollout time.
We thank the reviewer for pointing this out and will revise the claim to make it more precise. In addition, to further demonstrate the performance gains of our method, we trained a terminal agent with HDL and GRPO. Starting from Qwen3-8B, both methods are trained on 560 environments from SETA-Env (arXiv:2607.10891) and evaluated on Terminal-Bench 2.0. Following the protocol in our paper, we report the mean and standard deviation of pass@8 over the three checkpoints with the highest scores. HDL reaches 9.36 ± 1.72%, compared with 6.74 ± 1.95% for GRPO, starting from 2.25% for the base model. Using the last three checkpoints gives the same results. This further demonstrates that the gains of HDL extend beyond ScienceWorld to a more challenging agentic setting with long-horizon, multi-turn interaction.