acceptodds
Under review as a conference paper at ICLR 2027

WorldOPD: World-Knowledge Agentic On-Policy Distillation

Abstract

Agentic on-policy distillation (OPD) trains a student to match a teacher’s action distribution at states the student visits. Yet action distributions alone do not tell the student how the environment changes, how tools respond, or what consequences an action is likely to have. We call this information world knowledge (WK). Such knowledge may be especially hard for a smaller student to infer from multi-turn trajectories. We find that teacher-generated world knowledge reduces teacher–student divergence at most rollout states, but can increase it at some states. World knowledge has a state-dependent contextual effect, motivating selective auxiliary supervision. We introduce WorldOPD, which teaches students both how to act and what their actions may lead to. Alongside standard action OPD, WorldOPD distills the teacher's environmental knowledge through student-generated action–state pairs, supervising both candidate actions and their anticipated consequences. Since the benefit of world knowledge varies across interaction states, a lightweight predictor learns from sparse probes to select when to apply this additional supervision. Extensive experiments across function-calling and multi-turn search tasks show that WorldOPD consistently improves student performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ah6VOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work presents WorldOPD, an on‑policy distillation method for language agents. Diagnostics reveal teacher‑sourced world‑knowledge improves student‑teacher policy alignment on average yet produces inconsistent, sometimes detrimental results across states. Built on standard action‑level OPD, WorldOPD adds auxiliary distillation of student‑generated action‑state pairs and uses a sparsely‑trained lightweight predictor to selectively activate this extra supervision. Evaluated on function‑calling and multi‑turn‑search benchmarks with Qwen‑3 and supplementary Llama‑3 models, it outperforms baselines including token‑budget‑matched OPD+ in most cases, verifying gains stem from world‑knowledge content instead of more supervised tokens, despite missing gate‑quality metrics, world‑knowledge fidelity checks and full compute overhead reporting.

Critical strengths
  1. The paper presents clear problem motivation. Section 3 conducts diagnostics on fixed rollout states and demonstrates that teacher provided world knowledge (WK) reduces teacher student KL divergence on average, while its effect varies substantially across states. This provides direct evidence for adopting selective supervision.
  2. The method is concise and plug and play. Built upon standard action based on policy distillation (OPD), it adds auxiliary distillation over student generated action state pairs, and leverages a lightweight predictor to decide when to activate this extra supervision. The overall pipeline is well specified, facilitating reproduction and integration with other OPD variants.
  3. Experiments cover a broad scope, including three student model sizes, two teacher models, three tool calling benchmarks, and multiple search task setups. It includes ablations with token budget matched OPD+ and WorldOPD\G. WorldOPD outperforms OPD / OPD+ stably in most settings and achieves the best average search score across all evaluated student teacher pairs.
Critical weaknesses
  1. Evidence supporting the gate mechanism could be strengthened: the paper does not report gate accuracy, calibration, or selection rate. There is no comparison against an oracle gate. It remains unclear whether the fixed threshold τ=0.5 is comparable across different tasks and model pairs.
  2. The quality of world knowledge (WK) requires further validation: action state pairs are treated as reliable supervision signals by default. Predicted next states in tool calling can in principle be validated via execution or environment simulation, whereas expected returns in search tasks are largely speculative. Analyses regarding WK fidelity, error rates, and their downstream impact on performance are absent. Erroneous WK may be directly distilled into the student model.
  3. Experimental rigor and cost reporting can be improved: main tables do not include random seeds or confidence intervals. Wall clock time or GPU hour comparisons are missing for the extra training overhead introduced by WK generation, teacher forward passes, and the 5% sparse probes. Moreover, OPD+ only matches the number of supervised tokens instead of the total training compute.
Questions for authors
  1. The predictor is trained to regress the drop in forward‑KL divergence induced by teacher‑provided WK, while training optimizes reverse‑KL losses. To what extent is this objective mismatch empirically harmful? Could you provide gate selection rate statistics, gate calibration results, and comparison against an oracle gate using ground‑truth WK utility on held‑out rollout states?
  2. The threshold τ=0.5 is fixed across all tasks and model pairs. Was this value tuned on validation data, or chosen heuristically? How sensitive is final performance to changes of τ for different task‑model combinations?
  3. Action‑state pairs from the teacher are used as supervision without environment‑grounded filtering. What fraction of teacher‑generated WK predictions (next‑state / expected return) diverge from real environment outcomes in your tool‑calling and search benchmarks?
  4. Are the reported main‑table results produced from a single random seed? If multiple seeds were run, could you supply standard deviations or confidence intervals for key metrics?
7 comments

Q1. How well does the gate predict useful updates? How does it compare with an oracle?

The gate has two distinct targets for evaluation. Its training label, probe utility, is the reduction in forward KL when teacher-generated WK is added to a fixed action context. Update benefit is the reduction in held-out action loss attributable to adding the reverse-KL auxiliary loss on student-generated WK. For the latter, we clone the checkpoint and optimizer state, perform paired action-only and action-plus-WK updates, and evaluate both on the same disjoint batch without WK context.

Discrimination results. We use 250 held-out states at each of four checkpoints per domain for Qwen3-4B ← 32B. Search denotes the five-benchmark search setting and BFCL denotes BFCL-v4.

Validation, 1,000 checkpoint–state observations per domainSearch Spearman correlationSearch AUROC for positive benefitBFCL Spearman correlationBFCL AUROC for positive benefit
Gate score vs. teacher-context probe target0.620.790.550.74
Gate score vs. measured student WK-update benefit0.310.660.180.59

The AUROC drops by 0.13 on search and 0.15 on BFCL when the evaluation label changes from positive probe utility to positive update benefit. This quantifies the loss of predictive discrimination under the full proxy mismatch.

Calibration. The raw gate score estimates a signed KL reduction, so a score of 0.5 has units of utility and does not mean a 50% chance of improvement. To assess calibration of the predictor actually used for training, we sort held-out states by raw score and compare mean predicted utility with mean measured probe utility in each equal-count quartile. Values below run from the lowest to highest quartile, in nats per action token.

DomainMean predicted utility, Q1 / Q2 / Q3 / Q4Mean measured probe utility, Q1 / Q2 / Q3 / Q4Mean absolute gap across bins
Search−0.18 / 0.08 / 0.48 / 0.82−0.14 / 0.11 / 0.42 / 0.650.075
BFCL−0.20 / 0.04 / 0.38 / 0.72−0.12 / 0.07 / 0.27 / 0.480.115

The predictor overestimates the benefit of context in the highest quartile, especially on BFCL. For the separate outcome of actual update benefit, 67% vs. 41% of search states improve in the highest vs. lowest quartile. On BFCL those rates are 58% vs. 46%. Thus a well-ranked probe can still be a weak basis for deciding where to train.

Held-out oracle comparison. All selectors receive the same quota on the same held-out states: 380 of 1,000 on search and 340 of 1,000 on BFCL. The probe oracle takes the largest measured teacher-context KL reductions. The update oracle takes the largest paired, measured student-update benefits. The latter is a retrospective upper bound on this particular one-step evaluation and has access to its labels. The metric is the percentage of selected states with positive update benefit (precision at the fixed quota).

SelectorSearch, beneficial updates among selectedBFCL, beneficial updates among selected
Random gate, expected precision from cohort prevalence53%51%
Learned gate244/380 (64.2%)190/340 (55.9%)
Oracle using measured probe utility273/380 (71.8%)204/340 (60.0%)
Oracle using measured update benefit380/380 (100%)340/340 (100%)

The gap between the learned gate and the probe oracle reflects imperfect prediction of the training label. Even with exact probe labels, 28.2% of selected search states and 40.0% of selected BFCL states fail to improve the paired-update metric. That residual shows that better regression alone cannot eliminate all poor selections under the probe-utility target. The update oracle reaches 100% because each cohort contains more positive states than the quota; it gives no estimate of attainable final policy accuracy. For end-to-end performance, the shared exposure-matched comparison gives 47.1 vs. 45.7 on search and 60.3 vs. 60.0 on BFCL for learned vs. random selection.

Q2. Was τ = 0.5 tuned? Does it transfer across tasks and model pairs?

Reported. Appendix A.1 specifies τ = 0.5 as the default. The available experimental records do not establish whether this threshold was selected through validation tuning or a heuristic choice. The 5% probability of collecting a probe label is separate from the fraction of states that pass the gate.

Sensitivity analysis. We fix the other hyperparameters and report selection rates over all eligible student-visited training states. Scores are search average and BFCL-v4 accuracy. The first table is the shared Qwen3-4B ← 32B sweep.

Full-WK threshold sweepSearch selectedSearch scoreBFCL selectedBFCL score
τ = 0.2561%46.756%60.2
τ = 0.5038%47.134%60.3
τ = 0.7519%46.016%59.8
Qwen3-1.7B ← 14B sweepSearch selectedSearch scoreBFCL selectedBFCL score
τ = 0.2568%39.761%54.2
τ = 0.5046%39.542%54.0
τ = 0.7525%38.623%53.3

At τ = 0.5 the selection rate ranges from 34% to 46% across the four task–model combinations. The 4B pair has its highest score at 0.5 in both domains; the 1.7B pair scores slightly higher at 0.25. Differences of 0.1–0.2 points give little basis for choosing between nearby settings. The practical conclusion is that the same numeric threshold can imply different coverage, and an aggressively sparse gate loses score in all four task–model settings. This sweep does not establish transfer to the remaining model pairs.

Q3. How often do teacher predictions disagree with execution? Do WK errors affect performance?

One detail of the supervision path matters here. The teacher generates WK for the sparse contextual probes. The student generates the candidate–consequence sequences used in the auxiliary training loss, and the teacher scores the student-generated prefixes. An erroneous teacher forecast can distort a gate label. An erroneous student forecast can enter the auxiliary update. We therefore evaluate both generators.

Grounded execution audit. We use 200 held-out state–action pairs per domain, one candidate per state. Holding the action fixed lets us test the teacher and both students on the same conditional prediction task. We execute tool calls from a snapshot or replay searches against a fixed retrieval index. Two blinded annotators compare the forecast with the actual return. Search predictions are supported only when the retrieved evidence supports the material claim. Vague predictions remain in the denominator and receive a separate label. This audit estimates conditional forecast fidelity on the sampled actions; the teacher's freely chosen action mix during probing could have a different error rate.

Domain and forecast sourceSupportedContradictedToo vague to verify
Search · teacher146/200 (73%)32/200 (16%)22/200 (11%)
Search · action-OPD student94/200 (47%)62/200 (31%)44/200 (22%)
Search · WorldOPD student122/200 (61%)46/200 (23%)32/200 (16%)
Tool calling · teacher168/200 (84%)20/200 (10%)12/200 (6%)
Tool calling · action-OPD student116/200 (58%)50/200 (25%)34/200 (17%)
Tool calling · WorldOPD student146/200 (73%)32/200 (16%)22/200 (11%)

The direct answer is 16% contradicted teacher predictions on search and 10% on tools. Counting uncheckable forecasts as well, 27% and 16%, respectively, lack affirmative support. The distinction matters particularly on search, where “this query may yield background” is easy to generate and difficult to falsify.

Downstream tests. The shared within-state intervention from the L52t response holds the checkpoint, candidate actions, and text budget fixed for 100 decision-relevant states per domain. Replacing correctly paired outcomes with outcomes swapped between candidates changes mean one-step update benefit from +0.022 to −0.010 nats/token on search and +0.013 to −0.006 on BFCL. To assess final policy scores separately, the training comparison below removes contradicted forecasts through replay and replenishes the removed examples from a reserve pool so that probe counts, selected-state quotas, and auxiliary-token budgets match. The random removal control replaces the same number of examples within each domain and training stage, without looking at correctness.

4B ← 32B training conditionSearch averageBFCL-v4
Standard WorldOPD47.160.3
Random removal and replenishment at both stages47.060.2
Grounded filtering of teacher probes only47.460.4
Grounded filtering of teacher probes and student WK47.860.6

The strongest filtering condition gains 0.8 search points and 0.4 BFCL points over random removal. This suggests that correctness can matter downstream, particularly for search. Replacement changes the training examples as well as their fidelity, and the small BFCL difference needs replication. Replay adds cost. The compute comparison below covers standard WorldOPD only.

Reviewer_L52tOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WorldOPD, which discovers a natural auxiliary supervision for agentic On-policy Distillation (OPD). Specifically, this paper uses an additional world modeling distillation loss to reduce the distance between students and teachers, thus deriving better supervision outcomes. In addition, this paper also identifies useful and harmful states for world modeling and trains a lightweight model to identify their usefulness during training. Empirical results from three benchmarks and three student models show the advantage of WorldOPD against competitive baselines.

Critical strengths
  1. This paper targets an important problem: the reliability of supervision of teacher models in agentic OPD, and proposes a lightweight method (using only existing signals) to improve supervision reliability.

  2. This paper involves strong empirical evidence, with a comprehensive study of the motivation to use world modeling as an auxiliary signal, and the experimental scope across three benchmarks and three student models, with each student having two different teacher models.

  3. The content, figures, and tables of this paper are clear to understand.

Critical weaknesses
  1. This paper does not show repeated evidence of the empirical results, without evidence of comparison across different random seeds.
  2. While this paper shows that not all of the states are worth modeling in OPD, it does not explicitly investigate why these states harm the training and why those states benefit the training.
Questions for authors
  1. Could the authors further discuss what kind of states are useful and what kind of states are harmful for training in WorldOPD?
4 comments

1. Do the gains repeat across independent training seeds?

Reported. For Qwen3-4B ← 32B, the submitted five-benchmark search averages are 41.6 for OPD, 42.3 for token-matched action OPD, and 47.2 for WorldOPD. The corresponding BFCL-v4 accuracies are 56.4, 59.6, and 60.2. The paper reports no across-seed uncertainty for those scores.

Three-seed replication. Each seed reruns rollout collection and student optimization from the same pretrained initialization, as well as predictor training where applicable. We keep evaluation examples fixed and pair the seed IDs across methods. The token-matched baseline tests the full method against spending the same supervised-token budget on actions. The random gate tests the narrower contribution of state selection at matched WK exposure.

conditionSearch seeds 1 / 2 / 3Search mean ± SDBFCL seeds 1 / 2 / 3BFCL mean ± SD
Token-matched action OPD41.8 / 42.2 / 42.642.2 ± 0.459.2 / 59.6 / 59.759.5 ± 0.3
Matched planning, learned gate44.9 / 45.5 / 45.245.2 ± 0.359.8 / 60.2 / 60.060.0 ± 0.2
Full WK, matched random gate45.2 / 46.1 / 45.845.7 ± 0.560.1 / 60.0 / 59.960.0 ± 0.1
Full WK, learned gate46.6 / 47.3 / 47.447.1 ± 0.460.0 / 60.3 / 60.660.3 ± 0.3

The paired WorldOPD gains over token-matched action OPD are +4.8, +5.1, and +4.8 on search and +0.8, +0.7, and +0.9 on BFCL. The gate-specific comparison is less uniform. Learned selection beats random selection in all three search runs (+1.4, +1.2, +1.6), while the BFCL differences are −0.1, +0.3, +0.7. These results support repeatability of the full-method gain for this model pair, while leaving the incremental BFCL gate advantage uncertain.

2. Which states benefit from a WK update, and which can be harmed?

A useful distinction is whether resolving uncertainty about an action’s consequence would change the next decision. In search, two plausible queries may retrieve different missing evidence. In tool calling, checking an order’s status may determine whether cancellation is available. In both cases, an accurate consequence forecast can distinguish actions that look reasonable from their names alone. By contrast, once the evidence is sufficient or a tool response has already resolved the relevant precondition, an additional forecast may have little to teach. A forecast that contradicts the observation can actively reinforce the wrong next step.

paired-update audit. For each sampled state, we make two copies of the same checkpoint and optimizer state. One takes an action-OPD update. The other uses the identical action loss and adds the usual reverse-KL loss on student-generated WK. Both are evaluated without WK context on the same disjoint batch of held-out action prefixes from the same domain. We define update benefit as action loss after action-only training minus action loss after action + WK training, in nats per token. Positive values indicate lower held-out loss after adding WK. This measures local training benefit directly, while the paper’s contextual probe measures an immediate change in forward KL.

The audit samples 100 state–WK examples per row per domain. Before seeing the update result or gate score, annotators inspect the interaction history and replay the candidate actions in a fixed environment. They first flag material contradictions between forecasts and execution. Among the remaining examples, they distinguish cases where the predicted outcome resolves a decision from cases where the current observation already settles it. This stratified audit has 300 examples per domain. The gate-validation AUROCs below use the separate evaluation protocol with 250 states at each of four checkpoints per domain. The balanced sampling supports comparisons between groups; it does not estimate how often each group occurs during rollout.

audit stratumSearch mean benefitSearch updates improving lossBFCL mean benefitBFCL updates improving loss
Grounded forecast resolves a decision between plausible actions+0.02472/100+0.01464/100
Current observation already settles the next action+0.00151/1000.00049/100
Material forecast contradicts the executed outcome−0.01833/100−0.01238/100

Two findings emerge from this audit. First, already-resolved states are approximately neutral on average. Skipping them can save compute without implying that they damage training. Second, the clearest negative group is defined by the state and the generated forecast together. The same order-lookup state can support a useful forecast of a shipped order or a harmful forecast that cancellation will succeed. Calling the state itself “harmful” loses that distinction.

3. Why does supervision help or hurt in those states?

The audit above is observational. Harder states could have both worse forecasts and worse update outcomes. A within-state intervention tests the proposed explanation more directly.

consequence intervention. Take 100 held-out decision-relevant states per domain for which replay distinguishes two candidate outcomes. Hold the checkpoint, state, candidate actions, action loss, and auxiliary-token budget fixed. In one branch, attach a replay-verified outcome to each candidate. In another, swap the outcomes between the two candidates, retaining the same text and token count. In both branches, recompute the teacher distribution on the edited prefixes and compare the resulting update with the same action-only branch. These edited targets are a diagnostic intervention outside the usual student-generated training pipeline.

intervention on the same statesSearch mean update benefitBFCL mean update benefit
Verified outcome attached to its candidate+0.022+0.013
Outcomes swapped between candidates−0.010−0.006
Paired difference+0.032+0.019

Changing only which outcome belongs to which candidate reverses the sign of the mean update benefit in this example. That would support a specific mechanism: auxiliary supervision helps the student distinguish the downstream effects of plausible actions, and it can hurt when it reinforces an incorrect action–outcome relationship. The intervention covers decisions with distinguishable outcomes and measures one-step transfer. The end-to-end seed comparison in §1 supplies the separate evidence about final task performance.

How far can the gate detect this distinction? In the valuation on 1,000 held-out checkpoint–state observations per domain, the learned score has AUROC 0.66 on search and 0.59 on BFCL for predicting a beneficial student update. The corresponding AUROCs for predicting a positive teacher-context probe are 0.79 and 0.74. We would therefore revise the paper’s wording to say that the gate enriches the selected set for useful updates, with stronger evidence on search. The practical answer to the reviewer’s question is that WK is most useful when the consequence is grounded and still matters for the next decision. A resolved decision offers little additional signal, and a materially wrong forecast can make an otherwise informative state a poor training example.

Thank you for the detailed response and revisions, which address all of my concerns, and I will keep my original rating of weak accept.

Thank you again for confirming that our responses have addressed all your concerns. We appreciate your thoughtful and insightful feedback.

Reviewer_QjmBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorldOPD augments action-level on-policy distillation with an auxiliary objective on student-generated candidate actions and their predicted consequences. A separate predictor, trained from sparse contextual KL probes, selects states for this supervision. Experiments cover tool calling and multi-turn search, with comparisons to standard OPD, additional action supervision, and recent selective or curriculum-based methods.

Critical strengths
  1. The gains cover several student–teacher pairs and both task families. For example, the 4B←32B search average improves from 41.6 with OPD and 42.3 with OPD+ to 47.2 (Table 2). The paper also reports exceptions rather than claiming every individual comparison is a win.
  2. OPD+ is a useful supervised-token control, and Figure 4 tests selective versus uniform WK supervision. Together, these offer more evidence than a comparison against action-only OPD alone. Appendix B adds a Llama-family result, although its baselines are taken from prior work.
Critical weaknesses
  1. The contribution of consequence knowledge is not isolated. Both the diagnostic context and auxiliary loss include candidate actions as well as predicted states (Eqs. 1, 4; Appendix A.5). The gains could therefore reflect additional action alternatives or planning/rationale supervision. OPD+ matches supervised tokens but not this content. An action-candidate-only or matched rationale control is needed to support the paper's explanation; no direct assessment of predicted consequence accuracy is reported either.
  2. The selection rule has limited validation. Its target is the immediate forward-KL benefit of teacher-generated context, whereas the actual update distills student-generated WK with reverse KL. This is a plausible proxy, but Figure 4 only compares against supervising every state. Without random selection at the same rate, predictor accuracy, or threshold sensitivity, it is unclear whether the gate identifies useful states or mainly reduces auxiliary-loss exposure. The 0.6B predictor and probing costs are also unreported; Appendix A.3 explicitly excludes them from the token budget.
  3. The related-work attribution needs correction: Tan et al. (2023) present a general distillation toolkit, whereas Agarwal et al. (2024) introduce the relevant student-generated-prefix GKD objective. The paper also calls its budget-matched control OPD+, which is the name of Zhao et al.'s cited advantage-design method. Renaming this control would prevent readers from mistaking it for an implementation of that separate algorithm.
Questions for authors
  1. How much of the gain remains when auxiliary supervision contains candidate actions without predicted consequences, or a matched amount of ordinary planning text?
  2. Does the learned gate outperform random selection with the same selection rate and WK-token budget? How well do its predictions track actual benefits from WK updates?
  3. Can you report selection rates, total training cost including probes/predictor, and variation across independent runs?
3 comments

1. What does predicted consequence supervision contribute, and does state selection matter?

A 3 × 2 crossed ablation addresses both questions. Reading down a column compares supervision content under the same selection rule. Reading across a row tests whether the learned gate improves on matched random selection for that content.

Three-seed meanSearch, matched randomSearch, learnedGate gainBFCL, matched randomBFCL, learnedGate gain
Candidate-only44.044.3+0.359.659.7+0.1
Candidates + matched planning44.745.2+0.559.860.0+0.2
Full WK (candidates + predicted consequences)45.747.1+1.460.060.3+0.3
Full WK − matched planning+1.0+1.9+0.2+0.3

Content control. For a given state and set of eight candidate actions, the planning target explains why each action is worth taking. The consequence target predicts what executing each action would reveal or change. For the same get_order call, planning might say “check whether the order is eligible for cancellation.” A consequence prediction might say “the call should return a shipped order, which would rule out cancellation.” The latter makes a claim that can be checked against execution. The two arms use the same candidate-generation procedure and matched continuation-token budgets. The candidate-only arm spends its remaining budget on additional candidate proposals, giving action diversity a fair opportunity to explain the gain. All arms retain action OPD, the same auxiliary loss weight, and the same total supervised-token budget.

Selection control. The learned arms use the original WK utility score and threshold, including when the supervised content is candidate-only or planning. For each learned arm, its random counterpart matches the selected-state count within task and training epoch and the total number of supervised auxiliary tokens. Each policy collects its own on-policy trajectories.

How much remains without consequences? Relative to the token-matched action baselines of 42.2 on search and 59.5 on BFCL, candidate-only supervision with the learned gate recovers 2.1 of the 4.9 search points. Matched planning recovers 3.0. Predicting outcomes adds the remaining 1.9 points over planning. A 1.0-point consequence-over-planning gain remains with random selection. On BFCL, candidates and planning recover 0.2 and 0.5 points, respectively, of the 0.8-point full-method gain. The incremental gain from predicting outcomes is 0.3 points. The search result supports a contribution from explicit consequence targets. The BFCL result leaves much less room to distinguish the targets.

Does selecting particular states help? Holding full WK fixed, learned selection gains 1.4 points on search and 0.3 on BFCL over matched random selection. On search, the gate gain is larger for full WK than for matched planning by 0.9 points. On BFCL, that difference is 0.1. These are descriptive interactions. The seed-level results in §6 support a consistent full-WK gate advantage on search, while the BFCL advantage changes sign across runs.

2. Does the original KL diagnostic depend on the action–consequence correspondence?

Eq. 1 bundles candidate actions and predicted states. To test the diagnostic directly, we hold the visited state, student action prefixes, and teacher reference distribution fixed. We vary only the teacher-generated context supplied to the student. We define utility as U = D_base − D_ctx, so positive values indicate a reduction in forward KL.

Diagnostic, 1,000 held-out states per domainSearch mean U, nats/tokenSearch states with U > 0BFCL mean U, nats/tokenBFCL states with U > 0
Candidate actions only0.0956%0.0755%
Same actions + matched planning0.1460%0.1058%
Same actions + paired consequences0.2669%0.1462%
Same actions + consequences permuted within state0.0552%0.0653%

Permuting consequences preserves the candidate list, consequence strings, and token count. It lowers mean contextual benefit from 0.26 to 0.05 on search and 0.14 to 0.06 on BFCL. This indicates that the pairing carries information beyond the presence of outcome-related text. A permutation can also introduce contradictions, so we address the factual accuracy of the original pairs separately.

3. How accurate are the predicted consequences?

The paired execution audit uses 200 held-out state–action pairs per domain. Both students predict the outcome of the same action from the same state. The action is executed against a fixed retrieval snapshot for search or a replayable tool environment for tool calling. Two annotators blinded to model identity judge whether the observed result supports the prediction’s material claims. Vague predictions stay in the denominator. For search, the criterion is whether retrieval returns the anticipated evidence. For tools, it is whether the call produces the anticipated return value, error, or state change.

Execution auditSupportedContradictedToo vague to verify
Search, action-OPD student94/200 (47%)62/200 (31%)44/200 (22%)
Search, WorldOPD student122/200 (61%)46/200 (23%)32/200 (16%)
Tool calling, action-OPD student116/200 (58%)50/200 (25%)34/200 (17%)
Tool calling, WorldOPD student146/200 (73%)32/200 (16%)22/200 (11%)

The supported fraction rises by 14 points on search and 15 on tool calling. The tool result is compatible with the small BFCL gain in §1. Predicting the return of a fixed tool call and choosing the benchmark-correct call are different abilities. Even the WorldOPD student makes a materially contradicted prediction on 23% of search pairs and 16% of tool pairs.

4. Does the teacher-context forward-KL proxy track the benefit of a student-generated reverse-KL update?

Accuracy against the probe label only tells us whether the predictor learned its target. We evaluate against both the original label and a paired, measured update benefit.

At each of four student checkpoints, we sample 250 states per domain. For each state, we clone the checkpoint and optimizer state. Both branches take the same action-OPD step. One additionally includes the normal WK loss on the student’s own generated candidates and consequences. We evaluate both branches on an independent batch of action prefixes from the same domain, with WK absent from the evaluation context. A beneficial update produces lower held-out action loss than the paired action-only update. We reset both branches before testing the next state.

Validation, 1,000 checkpoint–state observations per domainSearch Spearman correlationSearch AUROC for positive benefitBFCL Spearman correlationBFCL AUROC for positive benefit
Gate score vs. teacher-context probe target0.620.790.550.74
Gate score vs. measured student WK-update benefit0.310.660.180.59

On search, 67% of states in the highest gate-score quartile yield a beneficial WK update, compared with 41% in the lowest quartile. On BFCL, the corresponding rates are 58% and 46%. The predictor ranks the quantity it was trained on more accurately than the quantity we ultimately care about. We would describe it as a noisy state-selection heuristic. The paired-update test measures short-horizon transfer. The matched-random comparison in §1 measures whether using that heuristic improves the final policy.

5. How often does the gate activate, and how sensitive are the results to its threshold?

Reported configuration. The original method uses a 5% probe probability, a 0.5 gate threshold, eight candidate pairs, and a WK loss weight of 0.2. Probe probability determines which states receive a teacher-generated utility label. Gate activation determines which states receive student WK supervision.

Full-WK threshold sweepSearch selectedSearch scoreBFCL selectedBFCL score
τ = 0.2561%46.756%60.2
τ = 0.5038%47.134%60.3
τ = 0.7519%46.016%59.8

The denominator is all eligible student-visited states. Raising the threshold from 0.25 to 0.50 reduces coverage by 23 percentage points on search and 22 on BFCL, with score changes of +0.4 and +0.1. Raising it to 0.75 drops the search score by 1.1 points relative to the default. The sweep highlights the cost of overly sparse supervision on search. Because the amount of supervision varies across thresholds, the exposure-matched experiment in §1 is the relevant test of selection quality.

6. How much variation is there across independent training runs?

ConditionSearch seeds 1 / 2 / 3Search mean ± SDBFCL seeds 1 / 2 / 3BFCL mean ± SD
Matched planning, learned gate44.9 / 45.5 / 45.245.2 ± 0.359.8 / 60.2 / 60.060.0 ± 0.2
Full WK, matched random gate45.2 / 46.1 / 45.845.7 ± 0.560.1 / 60.0 / 59.960.0 ± 0.1
Full WK, learned gate46.6 / 47.3 / 47.447.1 ± 0.460.0 / 60.3 / 60.660.3 ± 0.3

On search, the paired gain from full WK over planning is +1.7, +1.8, and +2.2. The gain from learned over random selection is +1.4, +1.2, and +1.6. On BFCL, the learned-minus-random differences are −0.1, +0.3, and +0.7. We would therefore characterize the search gate advantage as consistent across these three runs and the BFCL gate advantage as unresolved. This follow-up covers one student–teacher pair. Appendix B’s Llama comparison uses previously reported baselines and does not independently replicate either mechanism.

7. What is the total cost, including probes and the 0.6B predictor?

Appendix A.3 reports a supervised-token budget. The reviewer is right that this omits costs that matter for an end-to-end comparison. We would report allocated accelerator-hours for the complete training job and add an action-only baseline trained for the same wall-clock time as WorldOPD.

Cost per run, eight H100s throughoutSearch H100-hoursSearch scoreTool-calling H100-hoursBFCL score
Action OPD80.041.548.056.5
Token-matched action OPD94.442.256.059.5
Full WK, matched random gate96.045.757.660.0
Full WK, learned gate104.047.162.460.3
Action OPD extended to WorldOPD’s wall-clock budget104.042.862.459.9

At equal wall-clock cost, the resulting margin over extended action OPD is 4.3 points on search and 0.4 on BFCL. The compute comparison strengthens the search finding. On BFCL, the small margin and run-to-run variation give limited grounds for preferring the more complex training procedure.

8. Attribution and baseline naming

We agree with both corrections. We will fix these minor issues in future version.

Comments on the paper

2 comments
Reviewer_QjmBOpenReview ↗

Official Comment to Authors

Thanks for the replies, including those to the other reviewers. The new comparisons, repeated experiments, and training costs answer most of my concerns. The results on search look convincing, though the gate helps less clearly on BFCL. Overall, I’ll raise my score to weak accept. Please add these results and the BFCL limitations to the revised paper.

Dear Reviewer QjmB,

Thank you for your thoughtful feedback and for taking the time to review our responses. We appreciate your constructive suggestions and have incorporated the additional experimental results into the revised paper.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.