acceptodds
Under review as a conference paper at ICLR 2027

Branching On-Policy Distillation

Abstract

On-policy distillation (OPD) provides teacher supervision on student-generated trajectories, but a single trajectory realizes only one continuation at each decision, limiting supervision coverage under a finite sampling budget. We propose BranchOPD (Branching On-Policy Distillation), which combines student uncertainty and teacher–student disagreement to select a few decision points, resamples continuations from shared prefixes under the student policy, and provides token-level teacher supervision on both main trajectories and branches. It supports token-level decisions in mathematics and within-turn decisions in agent tasks. We also introduce protocol-boundary repair (PBR), which uses targeted supervision under protocol constraints to mitigate repetition and wasted generation. BranchOPD outperforms Vanilla OPD and Multi-Rollout OPD () across all evaluated benchmarks, with gains over Vanilla OPD of 18.55 percentage points in mean mathematics accuracy over 16 sampled answers (avg@16) and 13.8 points in mean ALFWorld ID/OOD success. With PBR held fixed, prefix reuse reduces student-generated token costs on ALFWorld and WebShop by approximately 19% and 31%, respectively, relative to the seven-trajectory generation reference. Mechanistic analyses and selection ablations show that branching at uncertain decisions expands subsequent behavior coverage and outperforms random selection and turn-initial branching.

Primary area:
reinforcement learning
Keywords:
on-policy distillation, knowledge distillation, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7dNmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BranchOPD, an on-policy distillation framework that selects informative branching points using student uncertainty and teacher–student disagreement, and resamples student continuations from shared prefixes for token-level teacher supervision. It additionally introduces protocol-boundary repair (PBR) to mitigate repetitive and improperly terminated generations. Experiments across mathematical reasoning and multi-turn agent benchmarks demonstrate improvements over conventional OPD baselines, supported by branching ablations and behavior-coverage analyses.

Critical strengths
  1. The paper identifies an important limitation of conventional OPD, where a single trajectory provides limited supervision coverage over alternative continuations. The proposed branching strategy offers a reasonable and intuitive solution to this problem.

  2. The authors evaluate BranchOPD across mathematical reasoning and multiple agent benchmarks, including different student sizes and model families. The improvements over PBR-matched OPD baselines provide meaningful empirical evidence for the effectiveness of branching.

  3. The paper investigates alternative-action coverage, recovery opportunities, and branching budgets, providing insights into how the method explores different continuations. Shared-prefix reuse also reduces student-generated tokens compared with independent rollouts.

Critical weaknesses
  1. PBR addresses protocol-level generation errors, such as repetition and improper termination, which appear largely independent of the trajectory-coverage problem targeted by BranchOPD. The paper does not clearly explain why PBR is particularly necessary for the proposed branching framework rather than serving as a general-purpose enhancement to OPD.

  2. PBR alone improves Vanilla OPD from 25.82 to 38.94 on the mathematics benchmark, accounting for a substantial portion of the overall improvement. Although PBR-matched OPD baselines are provided, it seems that the BranchOPD without PBR is absent and PBR-enhanced versions of stronger baselines such as EOPD and SPOT limits the completeness of the component analysis.

  3. While BranchOPD achieves substantial improvements on ALFWorld, the gains over the strongest PBR-matched baseline are relatively modest on WebShop and Search-QA. This raises questions about when informative branching provides meaningful advantages and whether its benefits depend strongly on task characteristics or environment structure.

Questions for authors
  1. What is the fundamental relationship between PBR and the proposed branching mechanism? Does branching introduce or exacerbate the protocol violations targeted by PBR?

  2. Could the authors provide BranchOPD results without PBR and, if feasible, evaluate stronger baselines such as EOPD and SPOT under the same PBR setting? These comparisons would help clarify the independent contributions of branching and protocol repair.

6 comments

Q1: PBR targets protocol-level errors (repetition, incorrect termination) that seem unrelated to the trajectory-coverage problem of BranchOPD. Why is PBR necessary for branching rather than a generic OPD enhancement? Does branching introduce or exacerbate protocol violations?

A1: Thanks for your comments. We agree that the original manuscript did not clearly explain this relationship. BranchOPD expands the sampling of alternative continuations, while PBR targets protocol-level errors such as repetition after the answer is complete, incorrect termination, and redundant protocol tokens. PBR is a general enhancement to OPD, and it is also an important protocol-control component within BranchOPD, since branching and multi-sampling can give the model more chances to generate and learn repetitive continuations.

Methodsw/o PBR: Math Avg (%)+PBR: Math Avg (%)
Vanilla OPD25.8238.94
BranchOPD32.8744.37

PBR helps both methods, and BranchOPD improves over Vanilla OPD even without PBR. Branches inherit their shared prefix. If that prefix already contains repetition or protocol errors, more branches can raise the chance of observing violating continuations. The teacher may also reinforce repetition when conditioned on such prefixes, even though its own rollouts do not repeat (Appendix B.5). PBR therefore acts on both main and branch trajectories, controlling repetition that branching may amplify. With PBR, the fraction of main and branch trajectories containing a protocol violation drops to zero by step 25 on ALFWorld and WebShop and stays near zero (Figure 5), while branches keep high behavior coverage (Appendix C), indicating that protocol repair does not suppress exploration. We will clarify this in the revision.

Q2: PBR alone lifts Vanilla OPD from 25.82 to 38.94 on mathematics. Can the authors provide BranchOPD without PBR, and EOPD and SPOT under the same PBR setting, to separate the contributions of branching and protocol repair?

A2: Thanks for your comments. We agree that the original manuscript lacks BranchOPD without PBR. The EOPD and SPOT numbers in the manuscript were cited from their original papers; we now report our own reproductions with the same training data, number of updates, and evaluation protocol, each with and without PBR:

Methodsw/o PBR: Math Avg (%)+PBR: Math Avg (%)
Entropy-Weighted OPD29.7340.98
EOPD30.4641.83
SPOT31.2844.12
BranchOPD32.8744.37

Without PBR, BranchOPD outperforms SPOT by 1.59 points and EOPD by 2.41 points; with PBR, it outperforms EOPD by 2.54 points. With PBR, BranchOPD and SPOT are comparable (44.37 vs. 44.12, within seed variation; std 0.24, see our response to Reviewer #2), while BranchOPD needs no outcome verifier. PBR improves every method, and the gain from branching is clearest without PBR. We will report both comparisons separately in the revision to show the contributions of branching and protocol repair more clearly.

Q3: BranchOPD improves notably on ALFWorld, but its gains over the strongest PBR-matched baseline on WebShop and Search-QA are smaller. When does informative branching help, and does its advantage depend on task characteristics or environment structure?

A3: Thanks for your comments. We agree that the gains differ across tasks. Compared with the strongest PBR baseline, Entropy-Weighted OPD (+PBR), WebShop improves from 41/128 to 43/128 tasks and Search-QA from 48.90% to 49.05%, whereas ALFWorld mean SR improves from 52.18% to 56.54%. We will not describe the WebShop and Search-QA results as significant advantages.

Our behavioral analysis (Tables 14 and 15) suggests that the effect of branching is related to the interaction structure:

TasksAction Disagreement (EBR)New-Action Coverage (NSR)Rescue Rate R↑R_\uparrow
ALFWorld74.2%55.5%14.3%
WebShop47.7%37.5%5.0%
Search-QA——15.8%

The environment step limits are 50, 15, and 4 on ALFWorld, WebShop, and Search-QA (Table 16). Longer interactions give branches more opportunities to change later actions and states. On ALFWorld, branches more often produce new actions and rescue failed main trajectories. On WebShop, both coverage and rescue rate are lower, which is consistent with its smaller gain. On Search-QA, branches show some rescues, but these do not translate into clearly higher final-answer accuracy, so alternative continuations alone do not guarantee task success. Notably, its harm rate R↓R_\downarrow (15.2%, all branches fail although the main trajectory succeeds; Table 15) nearly equals its rescue rate (15.8%), so the net benefit is small.

These observations suggest that the benefit of branching depends on the interaction horizon, the space of alternative actions, environment feedback, and whether alternative continuations can change the final outcome. We will add this analysis and limit our conclusion accordingly.

Thank the authors for the responses. However, some of my main concerns still remain.

First, the authors explain PBR as a general enhancement to OPD, making its motivation appear disconnected from that of BranchOPD. Moreover, the current results suggest that the performance improvement mainly comes from PBR, yet its contribution has not been sufficiently analyzed. For instance, PBR appears closely related to unlikelihood training, raising concerns about its methodological novelty. Although some relevant studies are cited, the paper lacks direct comparisons with existing approaches such as StableOPD and semantic EOS alignment.

In light of the above concerns, I will retain my rating.

Thank you for the follow-up. We address the remaining concerns below.

1. PBR is motivated by branching.

We designed PBR because of repetition we observed while training BranchOPD. Branches are resampled from prefixes of the main trajectory. Once the main trajectory starts repeating (e.g., continuing after the final answer), its branches inherit the repetitive prefix, and the teacher, conditioned on that prefix, tends to continue the repetition rather than stop (Appendix B.5). Under token-mean aggregation (Eq. 11), these long repetitive suffixes contribute more loss terms than well-terminated ones, so branching increases repetitive supervision in each batch. PBR removes this effect on main and branch trajectories. The same mechanism also exists with a single trajectory, so we apply PBR to every baseline, and comparisons then isolate branching.

2. Branching contributes a substantial gain on its own.

SettingMath Avg
Vanilla OPD25.82
+ branching only32.87
+ PBR only38.94
+ branching + PBR (BranchOPD)44.37
  • Without PBR, branching adds 7.05 Math points and beats Multi-Rollout OPD (27.84), Entropy-Weighted OPD (29.73), EOPD (30.46), and SPOT (31.28).
  • With PBR applied to all methods, branching adds 5.43 (Math), 7.27 (ALFWorld), and 3.62 (agent Avg. SR). On ALFWorld this exceeds the PBR gain (+6.56); on agent Avg. SR it is comparable (+3.86). PBR provides most of the gain only on mathematics.
  • The gain holds for other students (Tables 4–5): +6.56 ALFWorld / +4.68 Math (Qwen3-1.7B), +8.76 ALFWorld (Qwen2.5-3B).
  • With PBR and the budget fixed, our selector reaches 44.37 vs. 40.11 for random forks (Table 3), so where to branch matters independently of PBR.

3. Analysis of PBR.

Role in training. PBR enters the objective directly, Ltrain=Ltree+λ mean ℓprotoL_{\text{train}} = L_{\text{tree}} + \lambda\,\text{mean}\,\ell_{\text{proto}} (Eqs. 12–13). For logits zvz_v at a detected prefix ss:

  • positive: ∂[−log⁡pθ(C∣s)]/∂zv=πv−1[v∈C] πv/pθ(C∣s)\partial[-\log p_\theta(C\mid s)]/\partial z_v = \pi_v - \mathbb{1}[v\in C]\,\pi_v/p_\theta(C\mid s)
  • negative: ∂[−log⁡(1−pθ(C∣s))]/∂zv=πv−1[v∉C] πv/(1−pθ(C∣s))\partial[-\log(1-p_\theta(C\mid s))]/\partial z_v = \pi_v - \mathbb{1}[v\notin C]\,\pi_v/(1-p_\theta(C\mid s))

They raise or lower the total probability of CC with magnitudes 1−pθ(C∣s)1-p_\theta(C\mid s) and pθ(C∣s)p_\theta(C\mid s), which vanish once the event is repaired. OPD cannot supply this signal: its gradient acts only along ∇log⁡πθ(at∣st)\nabla\log\pi_\theta(a_t\mid s_t) (Eqs. 4, 9–10). Suppose the answer is complete and the student samples at≠a_t\neq EOS, which the teacher also prefers on the repetitive prefix (Δlog⁡pt>0\Delta\log p_t>0). Then OPD raises πθ(at∣st)\pi_\theta(a_t\mid s_t) and lowers the EOS probability. PBR adds the missing gradient on the EOS set at exactly these prefixes, so it corrects the training signal rather than acting as a decoding heuristic.

Relation to unlikelihood training. PBR builds on UL (Welleck et al., 2019), and we do not claim a new loss family. It differs in three ways:

  1. It acts on the total probability of a token set, so equivalent stop IDs form one event.
  2. It has positive constraints as well; the main mathematics constraint is to stop after a complete answer.
  3. Constraints are detected on the student's main and branch trajectories, where the teacher signal fails.

Our main methodological contribution is branching; for PBR, the contribution is identifying this OPD failure mode, which branching amplifies, and providing a targeted fix.

Direct comparisons (mathematics, Vanilla OPD base).

MethodMath Avg
Vanilla OPD25.82
+ repetition penalty28.70
+ n-gram / line penalty30.40
+ semantic EOS alignment (Yang et al., 2026b)30.67
+ overlong-sample filtering33.20
+ PBR38.94

The student already uses the teacher's stop tokens (Appendix D.3), yet responses still hit the length cap without PBR (Figure 4), so the failure is not only an EOS-ID mismatch. EOS alignment gains 4.85 points and PBR gains 13.12, because PBR also encourages stopping after a complete answer and suppresses mid-response protocol violations.

Revision plan. We agree that the manuscript presented PBR as a separate add-on. In the revision:

  • Motivation: the Introduction will describe the repetition observed in BranchOPD training (inherited repetitive prefixes, teacher continuation, token-mean amplification) as the reason PBR accompanies branching.
  • Contributions: the PBR contribution will be rewritten as identifying this failure mode, explaining how branching amplifies it, and fixing it, and will explicitly discuss its relation to unlikelihood training.
  • Method/experiments: Section 4.4 will include the branching motivation, and we will add the decomposition table, gradient analysis, and direct comparisons.
Reviewer_oohWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. BranchOPD first samples a student trajectory, then selects fork points using student uncertainty and teacher-student disagreement. The student samples new continuations from those prefixes. The teacher supervises tokens on both the main trajectory and the branches, and shared prefixes enter the loss once. The default uses three fork points and two continuations per fork.
  2. The paper also introduces protocol-boundary repair (PBR) to address stopping and stray format tags. Experiments cover mathematics, ALFWorld, WebShop, and Search-QA, with comparisons against independent rollouts and several fork-selection methods. Agents still use one trajectory per task at evaluation.
Critical strengths
  1. The method addresses a clear limitation: a single student trajectory provides no supervision on alternative continuations. Reusing prefixes to collect more student behavior is a reasonable improvement, and training does not require task-correctness rewards.
  2. Gains remain when both methods use PBR. Compared with seven independent rollouts, mathematics rises from 39.09 to 44.37 and the ALFWorld ID/OOD mean rises from 48.88 to 56.54. WebShop improves from 31.25 to 33.59 and Search-QA from 48.60 to 49.05 (Tables 1, 2, and 11). Format repair therefore cannot explain all of the improvement. Mathematics avg@16 is mean accuracy over 16 answers per problem.
  3. The internal controls test random forks, entropy or disagreement alone, turn-initial forks, branch counts, and forced tokens (Tables 3 and 6–10). Tables 4–5 add Qwen3-1.7B and Qwen2.5-3B, while the appendices examine action changes, successful continuations after failed main trajectories, and replay costs. The models remain within the Qwen family, so broader applicability is still untested.
Critical weaknesses
  1. The novelty mainly comes from combining existing ideas, and the closest comparisons do not separate PBR from branching. BPO already uses branching with task returns, while SPOT combines teacher-proposed tokens with student continuations. BranchOPD differs by letting the student sample branches and supervising every new continuation token. These works are already cited by the paper. In Table 1, PBR raises Vanilla OPD from 25.82 to 38.94, and BranchOPD then reaches 44.37. The 13.12- and 5.43-point differences are not causal contribution estimates, but show why the two changes need separate tests. Appendix D.2 says EOPD and SPOT do not use PBR, and the main table lacks BranchOPD without PBR. The results therefore do not yet isolate an advantage of branching over these methods. Related work: BPO (https://arxiv.org/abs/2607.14171); SPOT (https://arxiv.org/abs/2608.04419).
  2. Result stability and the mechanism explanation need stronger support. Against Entropy-Weighted OPD with PBR, WebShop improves only from 41/128 to 43/128 solved tasks, and Search-QA from 48.90 to 49.05. The manuscript reports no independent training seeds, repeated results, or confidence intervals. Sixteen answers to each of 30 AIME problems do not create 480 independent problems. Tables 8 and 12 also contain settings where forced branching wins, so the default is not uniformly better. Proposition 1 supports a conditional sampling property at fixed prefixes; it does not guarantee an unchanged overall training objective or better final performance. The appendices already acknowledge this limit.
  3. Fewer generated tokens do not establish lower total training cost. Figure 3 constructs independent-rollout cost by multiplying single-rollout length by the rollout count; mathematics and Search-QA costs in Table 11 are estimated from fork positions. The reported 19.1% and 31.2% savings on ALFWorld and WebShop cover student generation only, excluding teacher scoring, training computation, and replay. The authors acknowledge these limits. Total time or GPU-hours and comparisons at equal total budget are still needed to judge practical efficiency.
  4. The paper leaves some reproduction details unclear. Appendix D gives the main training and branching settings, but D.1 only states that teacher queries return top-64 log probabilities and that log probabilities are floored at −10. It does not explain how a sampled token outside the top-64 is handled. The mathematics eligibility rules and PBR weight are also not fully specified. Table 16 marks the Search-QA training sample count as NR, while D.4 and E.4 do not identify the sources, size, and split of the mixed validation set, or the retrieval backend and index version. These details affect whether others can reconstruct the same experimental settings from the paper.
Questions for authors
  1. Can you report SPOT + PBR, EOPD + PBR, and BranchOPD without PBR using the same data, training updates, and evaluation protocol, with budgets stated for each method? If only one comparison is feasible, please prioritize SPOT. For PBR, Appendix D.3 already aligns decoding stop sets, but can you also compare grouping equivalent EOS tokens into one stopping event to test the additional benefit? Relevant work is When EOS Tokens Disagree (https://arxiv.org/abs/2609.20511).
  2. Can you provide training and evaluation seeds, checkpoint and hyperparameter selection rules, per-item predictions, and paired confidence intervals? Keep the 16 mathematical answers grouped by problem and preserve task pairing for agents; distinguish evaluation uncertainty from training variation. Where feasible, add independent training runs on a key task. At equal budget, compare action coverage and successful continuations for the default selector, random forks, and independent rollouts, and explain the forced-branch wins in Tables 8 and 12.
  3. Can you directly measure student generation, teacher scoring, training computation, and environment restoration for seven independent rollouts and BranchOPD? Please report total time or GPU-hours and actual generated and supervised token counts. Separate measurements from estimates, and compare performance at equal generation budget or equal total training budget.
  4. Can you clarify the settings in Appendix D that affect reproduction: how probabilities are obtained for sampled tokens outside the teacher top-64, whether top-20 probabilities are renormalized for entropy, how the 128-token window is positioned, and the mathematics eligibility rules and PBR weight? For Search-QA, please give the sources, sizes, and splits of the training and mixed validation sets, checks for training–test overlap, and the retrieval backend and index version. If these details are already included in the submission materials, please point to their specific location.
4 comments

Q1: Under the same data, updates, and evaluation, report SPOT + PBR, EOPD + PBR, and BranchOPD without PBR. Does grouping equivalent EOS tokens into one stop event match PBR?

A1: Thanks for your comments. The EOPD and SPOT numbers in the manuscript were cited from their original papers. We now report our own reproductions with the same data, updates, and evaluation, with and without PBR:

Methodsw/o PBR: Math Avg (%)+PBR: Math Avg (%)
Entropy-Weighted OPD29.7340.98
EOPD30.4641.83
SPOT31.2844.12
BranchOPD32.8744.37

Without PBR, BranchOPD outperforms SPOT/EOPD by 1.59/2.41 points. With PBR, it is comparable to SPOT (44.37 vs. 44.12, within seed std 0.24) without an outcome verifier.

MethodsMath Avg (%)
Vanilla OPD25.82
Vanilla OPD + STOP aggregation30.67
Vanilla OPD (+PBR)38.94

Grouping equivalent stop tokens helps (+4.85), but PBR gains more (+13.12) as it also suppresses mid-response violations.

Q2: Report seeds, checkpoint selection, and statistical uncertainty; compare the default selector, random forks, and independent rollouts under the same budget; and explain why forced branching wins in Tables 8 and 12.

A2: Thanks for your comments. We trained BranchOPD with two additional seeds (seed 42 is the original run):

TasksSeed 42Seed 123Seed 2025Mean ± Std
Math Avg (%)44.3744.1244.6044.36 ± 0.24
WebShop SR (%)33.5932.8134.3833.59 ± 0.78
Search-QA SR (%)49.0549.1248.9149.03 ± 0.11

All runs use pre-fixed step counts (Table 16) and the final checkpoint. Our evaluation follows existing protocols: 16 answers per mathematics problem are aggregated per problem (EOPD), and each agent benchmark is trained and evaluated separately (ATOD, TCOD).

The default selector, random forks, and independent rollouts (Multi-Rollout OPD, n = 7) share the same budget: seven trajectories per prompt, identical length limits, training steps, and evaluation (Tables 3 and 16). The default selector outperforms random forks (ALFWorld 57.86/55.22 vs. 50.71/46.27; Math 44.37 vs. 40.11) and independent rollouts (Tables 1 and 2); Tables 14 and 15 report its action coverage and rescue rate.

We agree that Tables 8 and 12 show the default selector is not always best (e.g., forced turn-initial + RB: 54.48 vs. 52.90 for resampling, Table 12), so we will not claim it dominates in all settings. We will also clarify that Proposition 1 is a conditional sampling property, not a guarantee of better performance.

Q3: Fewer generated tokens do not imply lower training cost. Please measure full costs and compare performance under an equal budget.

A3: Thanks for your comments. We measured full training costs on ALFWorld (64 prompts, 200 steps, 8 student + 8 teacher GPUs, seven trajectories per prompt); all values are measured:

Measured Items7 Independent RolloutsBranchOPD
Generated = supervised = teacher-scored tokens / prompt19,69115,924
Teacher forward passes / prompt77
Environment steps / prompt228.9231.3
Time / step (generation, log-prob, update)749 s (550, 40, 152)1,058 s (678, 72, 305)
Total wall-clock / GPU-hours41.6 h / 66658.8 h / 941
ALFWorld mean SR (%)48.8856.54

Environment restoration adds negligible time. BranchOPD generates 19.1% fewer tokens, but its longer update raises end-to-end cost from 666 to 941 GPU-hours. Under the same seven-trajectory budget, it improves mean SR by 7.66 points. In the revision, we will separate generation cost from total cost and will not present token savings as training-cost savings.

Q4: Clarify the teacher top-64 handling, top-20 entropy, the 128-token window, mathematics eligibility rules, PBR weights, and the Search-QA data and retrieval setup.

A4: Thanks for your comments. We will add to Appendix D:

  • Teacher probabilities. The teacher runs a forward pass over the full student sequence and returns its log-probability of each sampled token, log⁡πT(at∣st)\log\pi_T(a_t\mid s_t), even when ata_t is outside its top-64.
  • Entropy. Top-20 probabilities are renormalized, qi=pi/∑jpjq_i=p_i/\sum_j p_j, giving a truncated entropy H=−∑iqilog⁡qiH=-\sum_i q_i\log q_i.
  • Disagreement window. For position tt, the window is [t,t+128)[t,t+128), truncated at the response end and averaged over the tokens it contains.
  • Mathematics eligibility. We skip the first response token, special tokens (incl. EOS), whitespace, punctuation, and non-mathematical symbols, and require at least two valid top-kk candidates, entropy ≥ 0.5 nats, and ≥ 8 remaining generation tokens.
  • PBR weights. Mathematics: 0.05 on EOS after a complete answer, no cap. Agent tasks: 0.01 for suppressing excess closing-thought tags, cap 0.02.
  • Search-QA. Training uses Search-R1 processed data (169,615 examples: 79,168 NQ, 90,447 HotpotQA). Validation is a separate mixed set of 2,048 examples (1,024 each from NQ and HotpotQA), with no ID overlap with training. Retrieval: E5-base-v2 + FAISS over Wiki-18 (top-3).

Thank you for the matched PBR controls, multi-seed results, and measured training costs. These additions resolve or substantially reduce most of my earlier concerns. I have two remaining questions:

  1. Baseline results. The original EOPD/SPOT scores were 37.97/38.43, while the new reproductions achieve 30.46/31.28. Please distinguish cited results from reproductions and explain the main differences in experimental settings.
  2. Search-QA reporting. With one evaluation per example and equally weighted accuracy over 2,048 examples, 49.05% cannot arise from an integer count of correct answers. Please clarify the effective sample count, correct-answer count, and any additional averaging. I provisionally retain 2/4 (weak reject), while acknowledging the improvements and remaining open to revising my score after these points are clarified.

Thank you for the follow-up and for the careful check of the Search-QA numbers. We address both points below.

1. Cited vs. reproduced EOPD/SPOT results.

The EOPD and SPOT scores in Table 1 of the manuscript (37.97 / 38.43) are cited from the original papers (Jin et al., 2026; Qu et al., 2026). The rebuttal scores (30.46 / 31.28 without PBR, 41.83 / 44.12 with PBR) are our reproductions under the same setting as BranchOPD and all other methods in Table 1. The main differences are:

SettingEOPD / SPOT (cited)Our reproduction (BranchOPD setting)
Training dataDAPO-Math-14k (14,116 examples)DAPO-Math-17K (17,398 examples)
Training budget2 epochs2 epochs
Learning rate3×10−63\times10^{-6}1×10−61\times10^{-6}
Training length limitresponse 4,096 tokens; SPOT prompt 1,024prompt 2,048, response 16,384 tokens
Evaluation metricAvg@8 and Pass@8avg@16
Samples per problem816
Sampling parameterstemperature 1.0, top-p 0.8temperature 0.6, top-p 0.95, top-k 20
Evaluation lengthup to 8,192 tokensprompt 2,048, response 16,384 tokens

Because the training data, learning rate, length limits, and evaluation protocol all differ, the cited numbers cannot be compared directly with ours. Beyond these configuration differences, we believe the main cause of the lower reproduced scores is the training response budget: 16,384 tokens in our setting versus 4,096 in EOPD. A longer budget aggravates repetition. Once the student starts repeating, the teacher, conditioned on the repetitive prefix, also favors continuing the repetition and assigns higher scores to the repeated tokens (Appendix B.5). Each such token contributes a small gradient that encourages repetition, and a longer budget lets many more of these tokens enter training. The student then generates more long repetitive samples, which degrades training. Consistent with this, every OPD variant without PBR scores only 25.82–31.28 in our setting (Vanilla, Multi-Rollout, Entropy-Weighted, EOPD, SPOT), with responses running to the length cap (Figure 4). Once PBR suppresses this repetition, EOPD and SPOT reach 41.83 / 44.12.

In the revision, Table 1 will report only our matched reproductions, with and without PBR. The cited numbers will move to a separately marked row or footnote that lists their settings. We will also replace the comparison "BranchOPD outperforms EOPD and SPOT by 6.40 and 5.94", which mixes cited and reproduced settings, with matched margins: +2.41 / +1.59 without PBR and +2.54 / +0.25 with PBR.

2. Search-QA reporting.

Thank you for catching this. Each Search-QA score is the average of three evaluation passes of the same final checkpoint over all 2,048 questions, giving 3 × 2,048 = 6,144 trajectories, each scored by exact match and weighted equally. For example, BranchOPD answers 3,014 of 6,144 correctly (about 1,004.7 per pass), i.e., 49.056%, reported as 49.05 in Table 2.

Dear Reviewer,

Thank you again for your time and thoughtful comments on our paper. We have posted a detailed response to your concerns. As the discussion period is coming to an end, we would greatly appreciate it if you could take a look and let us know whether our response has addressed your concerns. We are happy to provide any further clarification if needed.

Best regards, Authors

Reviewer_eunnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BranchOPD, which expands on-policy distillation through selective branching from student-generated trajectories. Fork points are selected using student uncertainty and teacher–student disagreement, and the student samples additional continuations from these shared prefixes. Teacher supervision is applied to the main trajectory and the branches, with shared-prefix loss counted only once. The paper also introduces protocol-boundary repair (PBR) to address termination and formatting issues. Experiments on mathematical reasoning and agent tasks show improvements over vanilla and multi-rollout OPD, including comparisons where PBR is enabled for all methods.

Critical strengths
  1. A useful approach to allocating additional sampling in distillation. Branching from selected student states is a reasonable way to explore alternative continuations while avoiding repeated generation of shared prefixes. The method addresses which additional trajectories to collect, complementing approaches that modify supervision on existing trajectories. Its application to both mathematical reasoning and interactive agent tasks adds practical value.

  2. The experiments provide evidence for benefits beyond protocol repair alone. The paper includes PBR-enhanced vanilla and multi-rollout baselines, as well as random, entropy-only, disagreement-only, and turn-initial branching controls. In mathematics, BranchOPD improves over Vanilla OPD + PBR and seven-trajectory Multi-Rollout OPD + PBR by 5.43 and 5.28 points, respectively. These comparisons support a meaningful contribution from selective branching rather than attributing the entire improvement to PBR.

Critical weaknesses
  1. The comparison with vanilla OPD does not fully separate branching from the allocation of the total trajectory budget. Multi-Rollout OPD and BranchOPD already use matched nominal trajectory counts per prompt, which is an appropriate control. However, Vanilla OPD uses only one trajectory per prompt. An additional comparison in which Vanilla OPD + PBR performs more rounds of single-trajectory sampling to reach the same total trajectory count would clarify whether branching is preferable to spending that budget on more frequent online updates. The number of optimizer updates and total training cost should also be reported, since these differ between budget-allocation strategies.

  2. PBR has a substantial effect that merits further explanation, and its capped objective needs clarification. PBR alone raises mathematics accuracy from 25.82 to 38.94. The termination and repetition analyses support its usefulness, but do not fully explain how much of this gain reflects improved answer completion/extraction versus improved reasoning. In addition, Eq. (13) permits a hard cap of the form Ψ(u)=min⁡(u,cPBR)\Psi(u)=\min(u,c_{\mathrm{PBR}}). If implemented literally, the PBR gradient becomes zero above the cap. This behavior deserves an explicit explanation, particularly when protocol errors are severe. These are clarification and validation concerns rather than reasons to dismiss the demonstrated branching gains.

Questions for authors
  1. Could you compare Vanilla OPD + PBR, Multi-Rollout OPD + PBR, and BranchOPD under the same total number of collected trajectories? In particular, could Vanilla OPD use additional rounds of single-trajectory sampling and updates to match the trajectory budget of the other two methods? Please report optimizer-update counts and training costs alongside performance.

  2. PBR alone provides a large accuracy improvement. How much of this improvement comes from better termination and answer extraction, rather than improved reasoning? Is the underlying failure mode specific to the teacher–student template or termination-token configuration?

  3. Is the PBR cap implemented directly as min⁡(u,cPBR)\min(u,c_{\mathrm{PBR}}), so that its gradient vanishes above the threshold? If so, is this intentional, how often is the cap activated during training, and what happens without it?

  4. In Table 7, increasing the number of continuations per fork from k=2k=2 to k=3k=3 increases the sampling budget but slightly decreases all three reported performance metrics. Could you explain this observation and clarify whether the differences fall within run-to-run variation?

  5. As an optional extension, a simple late-position-first branching baseline, or a curriculum that gradually moves fork points from later to earlier positions, would be interesting.

1 comment

Q1: Compare Vanilla OPD + PBR under a matched total trajectory budget, and report optimizer updates, training cost, and performance.

A1: Thanks for your comments. We added seven-round online Vanilla OPD (+PBR): each round samples one trajectory per prompt and performs one optimizer update, so each prompt receives seven trajectories in total. This matches Multi-Rollout OPD (n = 7) and BranchOPD (one main trajectory + six branches). These two methods put all seven trajectories into one PPO minibatch and perform one update (ppo_epochs=1, micro-batch gradient accumulation), so the online variant uses 7× as many updates. Update counts follow the rollout batches per task (Table 16).

MethodsUpdates (Math/ALF/Web/SQA)Math AvgALFWorld ID/OOD (Mean)WebShopSearch-QAAvg. SR
Vanilla OPD (+PBR)270/200/420/60038.9449.29/49.25 (49.27)30.4748.5842.77
Multi-Rollout OPD (n = 7, +PBR)270/200/420/60039.0950.00/47.76 (48.88)31.2548.6042.91
Seven-round online Vanilla OPD (+PBR)1890/1400/2940/420039.9450.29/50.25 (50.27)31.4748.5843.44
BranchOPD270/200/420/60044.3757.86/55.22 (56.54)33.5949.0546.39

With 7× as many updates, the online variant improves over Vanilla OPD (+PBR) by 1.00 Math Avg and 0.67 Avg. SR, and BranchOPD achieves the best results on all benchmarks. Like Multi-Rollout OPD, the online variant generates seven full trajectories per prompt, whereas BranchOPD reuses shared prefixes and generates 19.1%/31.2% fewer student tokens on ALFWorld/WebShop (Table 11). We will add this comparison to the revision.

Q2: How much of the PBR gain comes from termination rather than reasoning? Is the failure mode template- or EOS-specific? Does the cap in Eq. (13) zero the gradient, how often is it triggered, and what happens without it?

A2: Thanks for your comments.

Source of the gain. PBR targets repetition in OPD training: continuing after the final answer and emitting redundant protocol tokens. Encouraging EOS after a complete answer stabilizes response length and keeps responses from exhausting the generation budget (Figure 4). We believe the gain mainly comes from the model reliably producing and keeping its final answer, while more stable responses may also help reasoning. Generic remedies that only discourage repetition or drop overlong samples reach 28.70–30.40 and 33.20 Math Avg, versus 38.94 for PBR (Table 13).

Generality. For mathematics, teacher and student share the stop tokens [151645, 151643] (Appendix D.3), yet responses still reach the length cap without PBR (Figure 4). On ALFWorld, where both share one template, Vanilla OPD raises excess leading </think> markers per response from the teacher's 1.64/1.50 (ID/OOD) to 1.94/1.91, and PBR reduces them to zero (Appendix B.5). This suggests the failure mode is not specific to one template or EOS setting. We will also discuss related work on EOS mismatch (Yang et al., 2026b).

Cap gradient. We thank the reviewer for catching an error in our description. The cap is implemented as Ψ(u)=u⋅min⁡(1,cPBR/sg(u))\Psi(u)=u\cdot\min(1, c_{\mathrm{PBR}}/\mathrm{sg}(u)). Its forward value is min⁡(u,cPBR)\min(u,c_{\mathrm{PBR}}), but above the cap the gradient is scaled by cPBR/uc_{\mathrm{PBR}}/u, not zeroed. The cap is intentional: it prevents the PBR gradient from dominating the OPD gradient. We will correct Section 4.4 and Appendix A.4.

Trigger frequency. The cap (cPBR=0.02c_{\mathrm{PBR}}=0.02) is used only for agent tasks. It changed the loss in 2 of 200 ALFWorld steps (steps 1–2; the step-1 uncapped term of 0.0248 became 0.0178 after per-micro-batch capping). It never took effect on WebShop (max 0.00057) or Search-QA (max 0.0031). Removing the cap changes the mean ALFWorld SR from 49.27 to 49.24.

Q3: In Table 7, why does k=3k=3 perform slightly worse than k=2k=2?

A3: Thanks for your comments. From k=2k=2 to k=3k=3, ALFWorld ID/OOD SR and Math Avg change from 57.86/55.22/44.37 to 57.14/54.48/43.91. With the number of forks fixed, extra continuations give diminishing marginal coverage (Eq. 8, Appendix A.3), and they may duplicate existing continuations or add low-value supervision. The benefit of extra sampling may therefore saturate, with a slight decline at k=3k=3.

Q4: Late-position-first branching or a late-to-early position curriculum.

A4: Thanks for your suggestion. We added both on mathematics:

Math Fork-Position StrategiesMath Avg (%)
Random fork positions40.11
Late-position-first forking41.80
w/o mathematics-aware eligibility42.03
Late-to-early curriculum42.50
BranchOPD (entropy + disagreement)44.37

The late-to-early curriculum is a useful alternative and outperforms late-position-first forking, suggesting that gradually widening the fork range helps. Selecting forks by uncertainty and disagreement achieves the highest score (44.37). We will add these results to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.