- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BranchOPD, an on-policy distillation framework that selects informative branching points using student uncertainty and teacher–student disagreement, and resamples student continuations from shared prefixes for token-level teacher supervision. It additionally introduces protocol-boundary repair (PBR) to mitigate repetitive and improperly terminated generations. Experiments across mathematical reasoning and multi-turn agent benchmarks demonstrate improvements over conventional OPD baselines, supported by branching ablations and behavior-coverage analyses.
Critical strengths
-
The paper identifies an important limitation of conventional OPD, where a single trajectory provides limited supervision coverage over alternative continuations. The proposed branching strategy offers a reasonable and intuitive solution to this problem.
-
The authors evaluate BranchOPD across mathematical reasoning and multiple agent benchmarks, including different student sizes and model families. The improvements over PBR-matched OPD baselines provide meaningful empirical evidence for the effectiveness of branching.
-
The paper investigates alternative-action coverage, recovery opportunities, and branching budgets, providing insights into how the method explores different continuations. Shared-prefix reuse also reduces student-generated tokens compared with independent rollouts.
Critical weaknesses
-
PBR addresses protocol-level generation errors, such as repetition and improper termination, which appear largely independent of the trajectory-coverage problem targeted by BranchOPD. The paper does not clearly explain why PBR is particularly necessary for the proposed branching framework rather than serving as a general-purpose enhancement to OPD.
-
PBR alone improves Vanilla OPD from 25.82 to 38.94 on the mathematics benchmark, accounting for a substantial portion of the overall improvement. Although PBR-matched OPD baselines are provided, it seems that the BranchOPD without PBR is absent and PBR-enhanced versions of stronger baselines such as EOPD and SPOT limits the completeness of the component analysis.
-
While BranchOPD achieves substantial improvements on ALFWorld, the gains over the strongest PBR-matched baseline are relatively modest on WebShop and Search-QA. This raises questions about when informative branching provides meaningful advantages and whether its benefits depend strongly on task characteristics or environment structure.
Questions for authors
-
What is the fundamental relationship between PBR and the proposed branching mechanism? Does branching introduce or exacerbate the protocol violations targeted by PBR?
-
Could the authors provide BranchOPD results without PBR and, if feasible, evaluate stronger baselines such as EOPD and SPOT under the same PBR setting? These comparisons would help clarify the independent contributions of branching and protocol repair.
6 comments
Thank the authors for the responses. However, some of my main concerns still remain.
First, the authors explain PBR as a general enhancement to OPD, making its motivation appear disconnected from that of BranchOPD. Moreover, the current results suggest that the performance improvement mainly comes from PBR, yet its contribution has not been sufficiently analyzed. For instance, PBR appears closely related to unlikelihood training, raising concerns about its methodological novelty. Although some relevant studies are cited, the paper lacks direct comparisons with existing approaches such as StableOPD and semantic EOS alignment.
In light of the above concerns, I will retain my rating.
Thank you for the follow-up. We address the remaining concerns below.
1. PBR is motivated by branching.
We designed PBR because of repetition we observed while training BranchOPD. Branches are resampled from prefixes of the main trajectory. Once the main trajectory starts repeating (e.g., continuing after the final answer), its branches inherit the repetitive prefix, and the teacher, conditioned on that prefix, tends to continue the repetition rather than stop (Appendix B.5). Under token-mean aggregation (Eq. 11), these long repetitive suffixes contribute more loss terms than well-terminated ones, so branching increases repetitive supervision in each batch. PBR removes this effect on main and branch trajectories. The same mechanism also exists with a single trajectory, so we apply PBR to every baseline, and comparisons then isolate branching.
2. Branching contributes a substantial gain on its own.
| Setting | Math Avg |
|---|---|
| Vanilla OPD | 25.82 |
| + branching only | 32.87 |
| + PBR only | 38.94 |
| + branching + PBR (BranchOPD) | 44.37 |
- Without PBR, branching adds 7.05 Math points and beats Multi-Rollout OPD (27.84), Entropy-Weighted OPD (29.73), EOPD (30.46), and SPOT (31.28).
- With PBR applied to all methods, branching adds 5.43 (Math), 7.27 (ALFWorld), and 3.62 (agent Avg. SR). On ALFWorld this exceeds the PBR gain (+6.56); on agent Avg. SR it is comparable (+3.86). PBR provides most of the gain only on mathematics.
- The gain holds for other students (Tables 4–5): +6.56 ALFWorld / +4.68 Math (Qwen3-1.7B), +8.76 ALFWorld (Qwen2.5-3B).
- With PBR and the budget fixed, our selector reaches 44.37 vs. 40.11 for random forks (Table 3), so where to branch matters independently of PBR.
3. Analysis of PBR.
Role in training. PBR enters the objective directly, (Eqs. 12–13). For logits at a detected prefix :
- positive:
- negative:
They raise or lower the total probability of with magnitudes and , which vanish once the event is repaired. OPD cannot supply this signal: its gradient acts only along (Eqs. 4, 9–10). Suppose the answer is complete and the student samples EOS, which the teacher also prefers on the repetitive prefix (). Then OPD raises and lowers the EOS probability. PBR adds the missing gradient on the EOS set at exactly these prefixes, so it corrects the training signal rather than acting as a decoding heuristic.
Relation to unlikelihood training. PBR builds on UL (Welleck et al., 2019), and we do not claim a new loss family. It differs in three ways:
- It acts on the total probability of a token set, so equivalent stop IDs form one event.
- It has positive constraints as well; the main mathematics constraint is to stop after a complete answer.
- Constraints are detected on the student's main and branch trajectories, where the teacher signal fails.
Our main methodological contribution is branching; for PBR, the contribution is identifying this OPD failure mode, which branching amplifies, and providing a targeted fix.
Direct comparisons (mathematics, Vanilla OPD base).
| Method | Math Avg |
|---|---|
| Vanilla OPD | 25.82 |
| + repetition penalty | 28.70 |
| + n-gram / line penalty | 30.40 |
| + semantic EOS alignment (Yang et al., 2026b) | 30.67 |
| + overlong-sample filtering | 33.20 |
| + PBR | 38.94 |
The student already uses the teacher's stop tokens (Appendix D.3), yet responses still hit the length cap without PBR (Figure 4), so the failure is not only an EOS-ID mismatch. EOS alignment gains 4.85 points and PBR gains 13.12, because PBR also encourages stopping after a complete answer and suppresses mid-response protocol violations.
Revision plan. We agree that the manuscript presented PBR as a separate add-on. In the revision:
- Motivation: the Introduction will describe the repetition observed in BranchOPD training (inherited repetitive prefixes, teacher continuation, token-mean amplification) as the reason PBR accompanies branching.
- Contributions: the PBR contribution will be rewritten as identifying this failure mode, explaining how branching amplifies it, and fixing it, and will explicitly discuss its relation to unlikelihood training.
- Method/experiments: Section 4.4 will include the branching motivation, and we will add the decomposition table, gradient analysis, and direct comparisons.
A1: Thanks for your comments. We agree that the original manuscript did not clearly explain this relationship. BranchOPD expands the sampling of alternative continuations, while PBR targets protocol-level errors such as repetition after the answer is complete, incorrect termination, and redundant protocol tokens. PBR is a general enhancement to OPD, and it is also an important protocol-control component within BranchOPD, since branching and multi-sampling can give the model more chances to generate and learn repetitive continuations.
PBR helps both methods, and BranchOPD improves over Vanilla OPD even without PBR. Branches inherit their shared prefix. If that prefix already contains repetition or protocol errors, more branches can raise the chance of observing violating continuations. The teacher may also reinforce repetition when conditioned on such prefixes, even though its own rollouts do not repeat (Appendix B.5). PBR therefore acts on both main and branch trajectories, controlling repetition that branching may amplify. With PBR, the fraction of main and branch trajectories containing a protocol violation drops to zero by step 25 on ALFWorld and WebShop and stays near zero (Figure 5), while branches keep high behavior coverage (Appendix C), indicating that protocol repair does not suppress exploration. We will clarify this in the revision.
A2: Thanks for your comments. We agree that the original manuscript lacks BranchOPD without PBR. The EOPD and SPOT numbers in the manuscript were cited from their original papers; we now report our own reproductions with the same training data, number of updates, and evaluation protocol, each with and without PBR:
Without PBR, BranchOPD outperforms SPOT by 1.59 points and EOPD by 2.41 points; with PBR, it outperforms EOPD by 2.54 points. With PBR, BranchOPD and SPOT are comparable (44.37 vs. 44.12, within seed variation; std 0.24, see our response to Reviewer #2), while BranchOPD needs no outcome verifier. PBR improves every method, and the gain from branching is clearest without PBR. We will report both comparisons separately in the revision to show the contributions of branching and protocol repair more clearly.
A3: Thanks for your comments. We agree that the gains differ across tasks. Compared with the strongest PBR baseline, Entropy-Weighted OPD (+PBR), WebShop improves from 41/128 to 43/128 tasks and Search-QA from 48.90% to 49.05%, whereas ALFWorld mean SR improves from 52.18% to 56.54%. We will not describe the WebShop and Search-QA results as significant advantages.
Our behavioral analysis (Tables 14 and 15) suggests that the effect of branching is related to the interaction structure:
The environment step limits are 50, 15, and 4 on ALFWorld, WebShop, and Search-QA (Table 16). Longer interactions give branches more opportunities to change later actions and states. On ALFWorld, branches more often produce new actions and rescue failed main trajectories. On WebShop, both coverage and rescue rate are lower, which is consistent with its smaller gain. On Search-QA, branches show some rescues, but these do not translate into clearly higher final-answer accuracy, so alternative continuations alone do not guarantee task success. Notably, its harm rate R↓ (15.2%, all branches fail although the main trajectory succeeds; Table 15) nearly equals its rescue rate (15.8%), so the net benefit is small.
These observations suggest that the benefit of branching depends on the interaction horizon, the space of alternative actions, environment feedback, and whether alternative continuations can change the final outcome. We will add this analysis and limit our conclusion accordingly.