acceptodds
Under review as a conference paper at ICLR 2027

Distilling Actions from Black-Box Teachers: Action-Posterior On-Policy Distillation for Agentic Tasks

Abstract

On-policy distillation (OPD) trains a language model with teacher supervision on states induced by the model itself. Standard OPD relies on the teacher's next-token logits, which provide a rich supervision signal when available. However, many of the strongest models are proprietary black boxes that expose only sampled text, preventing them from serving as OPD teachers. Token-level supervision is also poorly matched to interactive environments, where task outcomes are determined by executable actions: different responses may induce the same action, while similar responses may lead to different state transitions. We therefore argue that black-box distillation should recover the teacher's distribution over environment-defined decisions rather than imitate its surface text. We introduce Action-Posterior On-Policy Distillation (AP-OPD), a reward-free method that pushes repeated teacher samples through the environment's parser or tool schema, aggregates decision-equivalent responses, and estimates an action posterior at each student-visited state. Because these posteriors concentrate rapidly at most states but remain diffuse at a few decision points, a consensus-gated sampler stops when responses agree and allocates additional queries to ambiguous states. We derive the query-optimal oracle allocation for a prescribed estimation accuracy, whose plug-in form defines the sampler, and train the student model by matching the estimated posterior with action-level cross-entropy. Across ALFWorld, ScienceWorld, and -bench, AP-OPD improves over single-sample on-policy imitation by relative on ALFWorld and over white-box OPD by , while matching the ALFWorld success of ten-sample querying with fewer teacher tokens. Using the black-box DeepSeek-V4-Flash teacher further improves a Qwen3-4B student by on ALFWorld and on ScienceWorld relative to Qwen3-32B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, On Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VWn7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a method to distill models through actions alone. The objective is intuitive and makes sense I quite like it actually, its just a simple way to modify the data for sft basically with an extra kl which may or may not be necessary.

Critical strengths

The method seems simple.

Critical weaknesses

The paper is unreadable. Why are there so many proofs, do we really need to be writting this much theory for the sake of theory. I really don't undersand the purpose of them...

Section 4.2 starts with a theorem, why? Not sure the theorem isn't motivated, serves no purpose is never referenced again in the text, I've parsed through that section a few times and still have no clue what the authors are trying to communicate. I can say the same for section 4.3 what is the purpose of these sections?

The results are fine I guess, apart that the reasoninig of the model's is turned off for some reason, i get you train the models not to reason so its fair for a baseline, but most models benefit from this so...

I'm overall significantly dissapointed with the presentation which requires significant work in presenting the method, motivating the idea, not bloating with random math and convincing me this works.

You're also missing quite an important baseline :

Privileged Information Distillation (https://arxiv.org/pdf/2610.19820)

This paper is not cited, does exactly what you are claiming to do in this paper and proposed self-distillation which you use as a baseline.

Questions for authors

See weaknesses

8 comments

Thank you for taking the time to review our paper and for raising questions about the role of the analysis, reasoning-enabled inference, and related work. We have revised the presentation to make the connection between the theory and algorithm explicit, added thinking-enabled evaluations, and expanded the related-work discussion. We address each point below and explain why the theoretical results are central to the method.

W1

Each subsection of Section 4 supports a central component of AP-OPD:

SectionRole in the method
4.1Relates canonical-string loss to action-distribution KL divergence and identifies the aggregation gap.
4.2Bounds posterior estimation error from finite black-box samples in terms of executable action classes.
4.3Derives disagreement-based query allocation and bounds the probability of a unanimous pilot at an uncertain state.

The analysis connects the action-level objective, finite-sample posterior error, and disagreement-driven query allocation within one framework. This connection provides the statistical basis for the loss and sampler used by AP-OPD. We consider such analysis an important part of a methodological paper: it explains the algorithm beyond its empirical performance. Describing it as “theory for the sake of theory” overlooks these roles. The proofs are in Appendix F; the revised Section 4 states the purpose of each result before presenting it.

W2

Sections 4.2–4.3 provide the statistical basis for the sampler in Section 3.3. Theorem 2 quantifies estimation error at a fixed sample count, while Lemma 3 and Proposition 4 motivate allocating additional queries using pilot disagreement. Section 4.2 also explicitly connects adaptive stopping to estimation bias, which Appendix F.6 analyzes. We have made these links explicit in the introductory text of both subsections.

W3

The main comparison uses the same non-thinking setting for all students. We additionally enable thinking at inference for OPD n=1, ROPD, and AP-OPD, with a maximum of 4,096 generated tokens per environment step, including reasoning and the final action. Only the final action is executed. AP-OPD remains the strongest of the three on both benchmarks.

MethodALFWorld: thinking off (%)ALFWorld: thinking on (%)ScienceWorld: thinking off (score)ScienceWorld: thinking on (score)
OPD n=174.9 ± 1.175.454.1 ± 2.755.2
ROPD82.0983.755.0556.3
AP-OPD84.3 ± 2.285.864.8 ± 2.265.1

With thinking enabled, AP-OPD reaches 85.8% ALFWorld success and 65.1 ScienceWorld score, compared with 83.7%/56.3 for ROPD and 75.4%/55.2 for OPD n=1. The performance ordering is preserved with thinking enabled. These results are now in Section 5.5, Table 4. Entries with ± are the original three-training-seed results; the added evaluations are single-seed results.

W4

The central contribution is a decision-level learning target for black-box agent distillation. AP-OPD maps teacher responses to executable actions, retains their relative frequencies as a soft target, and directs additional teacher queries to states with disagreement. The analysis explains these choices, and the experiments test each one.

In particular, Table 2 holds the five teacher samples fixed: using the full canonical action distribution improves over surface-string supervision by 5.0 points on ALFWorld and 4.3 on ScienceWorld, and over majority voting by 2.7 and 2.3 points. At approximately 700M teacher tokens, AP-OPD achieves 84.3% ALFWorld success versus 75.1% for cost-matched OPD n=1 (revised Table 6). These comparisons isolate the value of the target and query allocation.

We have strengthened the links between the method and Section 4, added the baseline and thinking results in Section 5.5, and added downstream pilot and temperature analyses in Appendices D.6–D.7. The revised manuscript marks additions and changes in blue.

W5

Thank you for highlighting this related work. The title and URL in the comment identify two different papers, so we address both.

Privileged Information Distillation for Language Models studies training-time privileged information. Its π-Distill method jointly trains a privileged-information-conditioned teacher and an unconditioned student with shared parameters and reward/KL objectives; the paper also studies OPSD. AP-OPD instead estimates action posteriors from an external black-box teacher's sampled responses, without environment rewards or additional privileged information.

Our experimental question is how effectively a student can learn from sampled teacher responses under these access conditions. Directly adopting PID's full training formulation would introduce additional supervision and a different teacher–student relationship. Such a comparison would evaluate a broader training setup, while our controlled comparisons examine the contribution of action-posterior supervision. We therefore include single-sample OPD, action-level SFT, OmniOPD, GAD, and ROPD, and discuss PID as related work.

The linked paper, ActionOPD: Distilling Decisions, Not Tokens, uses action-agreement rewards and policy optimization. Its first public version appeared on October 14, 2026, after the September 25 submission deadline. We have added it as concurrent work and clarified the distinction from posterior cross-entropy and disagreement-based query allocation.

DimensionAP-OPDPID / π-DistillActionOPD
SupervisionAction-posterior cross-entropyTask rewards and KL objectives with training-time privileged informationTeacher-action agreement rewards
TeacherExternal black-box teacherPrivileged-information-conditioned teacher sharing student parametersExternal teacher; separate Self-ActionOPD extension
Sampling and optimizationDisagreement-driven sampling and posterior fittingJoint teacher–student trainingFixed teacher sampling and PPO-style updates

For context, ROPD, Section 3.1 and OmniOPD, Section 5.1 also do not include PID among their experimental baselines. ActionOPD explicitly discusses PID in Section 4, while its distillation comparison in Section 3.3 evaluates KD, OPD, and ActionOPD. These examples provide context for baseline selection; our comparison scope follows the supervision available in our problem setting.

Dear Reviewer VWn7,

Thank you for your time and efforts in evaluating our work. As the discussion window is near closing, we would like to check if our responses and revisions have adequately resolved your concerns. We would be very grateful for any additional feedback you might have, as your expertise is vital to enhancing the quality of our submission.

Best regards,
Authors

Reviewer_9evpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies black-box on-policy distillation for interactive tasks and proposes Action-Posterior On-Policy Distillation (AP-OPD). Instead of imitating sampled text directly, AP-OPD repeatedly queries the teacher at states visited by the student, maps the responses to executable actions through the environment interface, and estimates a distribution over actions. The student is then trained to match this action distribution with an action-level cross-entropy objective. To reduce query cost, the method uses a consensus-gated sampler that stops early when the sampled actions agree and allocates more queries to states with higher action disagreement. Experiments show that the proposed approach consistently improves student performance over the compared distillation baselines while using teacher queries more efficiently.

Critical strengths
  1. The paper gives a clear way to use black-box teachers in interactive tasks by distilling distributions over executable actions rather than surface text. This is well matched to environments where different responses can lead to the same action.
  2. The experimental evaluation is strong and includes useful ablations. The paper separates the effects of repeated sampling, action-level aggregation, and distributional targets, and also shows that the adaptive sampler can reduce teacher-query cost while keeping similar performance.
Critical weaknesses
  1. AP-OPD works best when the environment provides a clear action interface that can reliably group different teacher responses into the same decision. This is true for the benchmarks used here, but it is less clear how well the method would work when the action space is less structured or the equivalence between responses is ambiguous.
  2. The adaptive sampler is well motivated, and the paper includes useful cost and budget-matched comparisons. However, the main experiments use one fixed set of sampling hyperparameters, so it is still unclear how sensitive the final performance and query cost are to these choices.
Questions for authors
  1. AP-OPD outperforms white-box logit-based OPD in the reported experiments, even though the latter has access to the teacher’s token probabilities. Could the authors discuss why action-posterior supervision works better in this setting? It would be helpful to understand whether this mainly comes from the action-level target or from repeated teacher sampling.
  2. Could the authors discuss how sensitive the estimated action posterior is to the teacher’s decoding configuration, especially the sampling temperature? Since AP-OPD relies on repeated teacher samples to form the supervision target, changes in the teacher’s sampling behavior may affect the learned distribution.
4 comments

Thank you for the thoughtful questions about action equivalence, adaptive sampling, white-box comparisons, and teacher temperature. We appreciate your assessment that the adaptive sampler is well motivated and that the cost and budget-matched comparisons are useful. We have added downstream pilot-size and temperature experiments, and explain below how they complement the controlled target ablations and posterior diagnostics.

W1

AP-OPD uses the execution interface to define action equivalence. For command environments, this means parsed executable commands; for tool environments, it means tool names and normalized arguments. Appendix B.1 also specifies identity mapping for free-form responses, so distinct responses remain distinct when no reliable equivalence rule is available.

InterfaceTreatmentLocation
CommandsExtract executable commands; normalize known aliases and formatting.§5.1 / Appendix B.1
Tool callsIdentify actions by tool name and normalized arguments.§5.1 / Appendix B.1
Free-form textIdentity mapping; no semantic merging of distinct responses.Appendix B.1

Our evaluation includes tool use as well as command environments. On τ²-bench telecom-solo, AP-OPD improves over OPD n=1 with both teachers:

Method (Qwen3-4B student)Qwen3-32B teacher: pass₁ (%)DeepSeek-V4-Flash teacher: pass₁ (%)
SFT43.5 ± 2.267.4 ± 4.3
OPD n=154.3 ± 4.376.1 ± 4.3
OmniOPD58.7 ± 4.378.3 ± 3.8
AP-OPD65.2 ± 3.884.8 ± 2.2

The gain is 10.9 points with Qwen3-32B and 8.7 points with DeepSeek-V4-Flash. The useful degree of aggregation depends on the interface: a command grammar or tool schema provides a concrete equivalence rule; identity mapping reduces the target to a distribution over distinct responses. We clarify this scope in Section 6 and Appendix B.1.

W2

We added downstream evaluations for pilot sizes 2, 3, and 4 using the main stopping rule, the same training rounds, and the unchanged cap of 18. The results are now in Appendix D.6, Table 10.

PilotALFWorld success (%)ScienceWorld score (/100)ALFWorld teacher tokens / run (M)
279.962.1576.1
384.3 ± 2.264.8 ± 2.2698.7
485.064.5853.2

Pilot three improves over pilot two by 4.4 percentage points on ALFWorld and 2.7 score points on ScienceWorld. Increasing the pilot from three to four changes these results by +0.7 and −0.3 points while increasing ALFWorld teacher-token cost by 22.1%. The larger pilot therefore yields no consistent performance gain across the two environments. Pilot three provides the better observed performance–cost trade-off. The pilot-three results are the main three-training-seed measurements; the added pilot-two and pilot-four results are single-seed measurements.

The original replay analysis in Appendix D.5 complements this downstream test. With pilot three, modal-action changes relative to the ten-sample posterior are only 0.2%–0.4%, while teacher-token savings are 19.7%–32.3% across the three benchmarks (revised Table 9). The replay pool contains ten samples per state; the training cap is 18. The two experiments respectively measure posterior fidelity and final-policy performance.

Q1

Our interpretation is that action-posterior supervision better matches the decision executed by the agent. Several strings can induce the same action; AP-OPD combines their probability mass and retains the relative frequencies of different actions. Proposition 1 connects canonical-string loss to action-level KL. With the same Qwen3-32B teacher and Qwen3-4B student, the main results are:

MethodALFWorld success (%)ScienceWorld mean score (/100)
OPD n=149.8 ± 0.923.4 ± 2.4
White-box OPD-logit56.7 ± 0.725.8 ± 1.9
AP-OPD67.4 ± 1.729.4 ± 1.6

Complementary evidence comes from the DeepSeek-V4-Flash target ablation in Table 2, which holds the same five responses fixed across its n=5 rows:

Supervision targetALFWorld success (%)ScienceWorld score (/100)
Canonical action, n=174.9 ± 1.154.1 ± 2.7
All surface strings, n=577.1 ± 0.758.2 ± 1.9
Canonical majority action, n=579.4 ± 1.360.2 ± 1.6
Canonical full distribution, n=582.1 ± 1.362.5 ± 2.3

Repeated sampling improves the action label through majority voting. Retaining the full action distribution improves further: at n=5, it exceeds majority voting by 2.7/2.3 points and surface-string supervision by 5.0/4.3 points. This ablation supports the contributions of both repeated sampling and action-level distributional supervision, and provides evidence for our interpretation of the white-box comparison.

The teacher-budget-matched comparison in revised Table 6 further separates the gain from total query expenditure:

MethodTeacher tokens (M)ALFWorld success (%)
OPD n=1, teacher-cost matched710.375.1 ± 1.7
Canonical distribution, fixed n=5706.282.1 ± 1.3
AP-OPD, adaptive sampling698.784.3 ± 2.2

At approximately the same teacher-token cost, AP-OPD improves over single-sample supervision by 9.2 points. These experiments support the value of action-posterior supervision and adaptive allocation; more teacher calls alone do not explain the improvement.

Q2

Temperature changes the action posterior, while final student performance varies modestly over the tested range of 0.7–1.3. We added both posterior diagnostics and downstream evaluations (Appendix D.7, Tables 11–12).

Teacher temperatureTeacher ALFWorld success (%)Teacher ScienceWorld scoreAP-OPD ALFWorld success (%)AP-OPD ScienceWorld score
0.783.665.482.863.7
1.083.666.184.364.8
1.382.066.383.665.4

These DeepSeek-V4-Flash experiments hold other training settings and student evaluation decoding fixed. Student performance spans only 1.5 percentage points on ALFWorld and 1.7 score points on ScienceWorld, with no temperature consistently best across both environments.

A separate Qwen3-32B diagnostic uses 6,400 responses across matched states and an independent temperature-1.0 replicate:

EnvironmentModal-action overlap: 0.7 vs 1.0Modal-action overlap: 1.3 vs 1.0Excess JS at T=1.3 (bits; 95% CI)
ALFWorld24/2524/250.0229 [0.0102, 0.0359]
ScienceWorld22/2220/220.0070 [−0.0052, 0.0183]

Excess JS subtracts same-temperature sampling divergence; modal overlap counts states sharing a most frequent valid action. ALFWorld shows a distributional shift beyond sampling variation, although the modal action usually remains unchanged; ScienceWorld's interval includes zero.

Together, these observations support stable task performance under moderate temperature changes, not an invariant posterior.

Thank you for your response and the additional experiments, which address most of my concerns. Overall, I think the paper deserves a weak acceptance, so I will keep my rating at 3.

Dear Reviewer 9evp,

Thank you for your thoughtful feedback and for taking the time to review our responses and additional experiments. We appreciate your constructive suggestions, which have helped us improve the paper.

Best regards,
Authors

Reviewer_nDsmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of black-box on-policy distillation (OPD) where token-level logits from teacher is unvailable. The paper proposes Action-Posterior On-Policy Distillation (AP-OPD) to estimates the teacher's probability distribution over executable actions by aggregating multiple sampled responses, and introduces a consensus-gated sampling strategy to allocate teacher queries efficiently based on action disagreement. Experiments on ALFWorld, ScienceWorld, and τ2\tau^2-bench demonstrate improvements over existing on-policy distillation baselines while reducing teacher tokens.

Critical strengths
  • Black-box OPD is a timely research topic
  • Distilling actions rather than token-level outputs is well-motivated as different textual responses may correspond to the same action
Critical weaknesses
  • The baselines and experiments are limited. The paper mainly compares against SFT, vanilla OPD, and OmniOPD, omitting other relevant black-box OPD methods such as GAD (Ye et al. 2025).
  • In addition to black-box OPD baselines, the paper should also consider to evaluate against action-level SFT and distillation for a fairer comparison and better establish the benefits of action level OPD.
  • The method assumes that teacher responses can be reliably mapped to canonical executable actions. This limits its applicability to environments with well-defined action spaces and parsers.
Questions for authors

See weakness

2 comments

Thank you for your careful review and for suggesting stronger black-box and action-level baselines. These suggestions helped us expand the empirical comparison and clarify the distinction between teacher-trajectory imitation, single-sample on-policy distillation, and action-posterior supervision. We have added GAD, ROPD, and action-level SFT results and clarified the scope of canonicalization in the revised manuscript.

W1

We added GAD, ROPD, and action-level SFT with the Qwen3-4B student and DeepSeek-V4-Flash teacher. ROPD also uses DeepSeek-V4-Flash as its rubricator and verifier. The expanded comparison is now in Section 5.5, Table 4.

MethodALFWorld success (%)ScienceWorld mean score (/100)
Action-SFT80.6053.90
OPD n=1 (paper)74.9 ± 1.154.1 ± 2.7
ROPD82.0955.05
GAD34.3319.34
AP-OPD (paper)84.3 ± 2.264.8 ± 2.2

AP-OPD achieves the highest performance in this comparison on both benchmarks. Relative to ROPD, the gain is 2.21 percentage points on ALFWorld and 9.75 score points on ScienceWorld; relative to action-SFT, it is 3.70 and 10.90 points. The original OPD n=1 and AP-OPD rows report mean ± standard deviation over three training seeds; the added baselines report single-seed results.

GAD provides little improvement over the base student in our setting. A related pattern is reported in ROPD, Appendix E, where GAD yields smaller gains than the other on-policy baselines. ROPD identifies early discriminator saturation as a possible explanation: distinguishing teacher from student responses can be easier than providing useful reward differences among student responses. Our results show that adversarial supervision is less effective than action-posterior supervision in the evaluated agent environments.

W2

The original SFT baseline learns to generate a complete teacher trajectory as one output sequence. Action-level SFT instead splits each trajectory into stepwise examples: the model conditions on the preceding interaction summary and current state to predict the next executable action. We retain valid actions from both successful and unsuccessful teacher episodes. OPD n=1 uses the same action-level form of supervision, but queries the teacher at student-visited states. These baselines distinguish full-trajectory generation, stepwise action imitation, and on-policy action distillation.

MethodALFWorld success (%)ScienceWorld mean score (/100)
SFT (paper)73.4 ± 1.952.4 ± 2.3
Action-level SFT (added)80.6053.90
OPD n=1 (paper)74.9 ± 1.154.1 ± 2.7
AP-OPD (paper)84.3 ± 2.264.8 ± 2.2

To isolate the effect of the supervision target, Table 2 uses the same five cached teacher samples for surface-string supervision, canonical majority voting, and the full canonical action distribution:

Supervision targetALFWorld success (%)ScienceWorld score (/100)
Canonical action, n=174.9 ± 1.154.1 ± 2.7
All surface strings, n=577.1 ± 0.758.2 ± 1.9
Canonical majority action, n=579.4 ± 1.360.2 ± 1.6
Canonical full distribution, n=582.1 ± 1.362.5 ± 2.3

At fixed n=5, retaining the full action distribution improves over majority voting by 2.7/2.3 points and over surface-string supervision by 5.0/4.3 points. Thus the gain comes from both aggregating decision-equivalent responses and preserving the teacher's uncertainty over actions.

W3

AP-OPD uses the execution interface to define action equivalence. For command environments, this means parsed executable commands; for tool environments, it means tool names and normalized arguments. Appendix B.1 also specifies identity mapping for free-form responses, so distinct responses remain distinct when no reliable equivalence rule is available.

InterfaceTreatmentLocation
CommandsExtract executable commands; normalize known aliases and formatting.§5.1 / Appendix B.1
Tool callsIdentify actions by tool name and normalized arguments.§5.1 / Appendix B.1
Free-form textIdentity mapping; no semantic merging of distinct responses.Appendix B.1

Our evaluation includes tool use as well as command environments. On τ²-bench telecom-solo, AP-OPD improves over OPD n=1 with both teachers:

Method (Qwen3-4B student)Qwen3-32B teacher: pass₁ (%)DeepSeek-V4-Flash teacher: pass₁ (%)
SFT43.5 ± 2.267.4 ± 4.3
OPD n=154.3 ± 4.376.1 ± 4.3
OmniOPD58.7 ± 4.378.3 ± 3.8
AP-OPD65.2 ± 3.884.8 ± 2.2

The gain is 10.9 points with Qwen3-32B and 8.7 points with DeepSeek-V4-Flash. The useful degree of aggregation depends on the interface: a command grammar or tool schema provides a concrete equivalence rule; identity mapping reduces the target to a distribution over distinct responses. We clarify this scope in Sec 6 and Appendix B.

Dear Reviewer nDsm

Thank you for your time and efforts in evaluating our work. As the discussion window is near closing, we would like to check if our responses and revisions have adequately resolved your concerns. We would be very grateful for any additional feedback you might have, as your expertise is vital to enhancing the quality of our submission.

Best regards, Authors

Comments on the paper

1 comment

Summary of Rebuttal

Dear Chairs,

We sincerely thank the reviewers for their feedback. All issues raised by Reviewers 9evp and nDsm have been resolved through additional experiments and clarifications; we are awaiting Reviewer nDsm’s feedback. Reviewer VWn7 also acknowledged the additional experiments, but remains concerned about presentation and similarity to related methods, which we have responded to through manuscript revisions, detailed comparisons, and an additional adapted-baseline experiment.

Consensus Among Reviewers

  • Motivation for action-level distillation: Reviewers nDsm and 9evp recognize the value of distilling executable actions rather than surface text when different responses lead to the same decision. Reviewer VWn7 also describes the objective as “intuitive and makes sense” and states, “I quite like it actually.”

  • Empirical effectiveness: Reviewers nDsm and 9evp both note improvements over existing distillation baselines. Reviewer 9evp further highlights the strong evaluation and useful ablations separating repeated sampling, action aggregation, and distributional targets. Reviewer VWn7 also acknowledges the results and appreciates the additional experiments.

  • Teacher-query efficiency: Reviewers nDsm and 9evp both recognize the reported reduction in teacher usage. Reviewer 9evp specifically notes that adaptive sampling reduces query cost while preserving similar performance and describes the sampler as “well motivated.”

Summary of Revisions and Clarifications

  • Motivation and role of the theory (VWn7): We explain the practical purpose of each result before presenting it. Section 4.1 connects string-level training to action-distribution KL; Section 4.2 characterizes finite-sample posterior estimation; Section 4.3 motivates the disagreement-based sampler in Section 3.3. The revised explanation of the aggregation gap makes the implication of Proposition 1 explicit.

  • Additional baselines (nDsm): We added action-level SFT, GAD, and ROPD. AP-OPD achieves the highest reported performance on both benchmarks, exceeding ROPD by 2.21 percentage points on ALFWorld and 9.75 score points on ScienceWorld, and action-level SFT by 3.70 and 10.90 points (Section 5.5).

  • Controlled comparisons (nDsm, 9evp): With the same five teacher responses, the full action distribution improves over majority voting by 2.7/2.3 points and surface-string supervision by 5.0/4.3 points. At approximately 700M teacher tokens, AP-OPD achieves 84.3% ALFWorld success versus 75.1% for cost-matched OPD n=1 (Tables 2 and 6).

  • Thinking-enabled inference (VWn7): AP-OPD reaches 85.8% ALFWorld success and 65.1 ScienceWorld score, compared with 83.7%/56.3 for ROPD and 75.4%/55.2 for OPD n=1. The performance ordering is preserved.

  • Sampling sensitivity (9evp): Increasing the pilot from three to four changes performance by +0.7/−0.3 points while increasing ALFWorld teacher-token cost by 22.1%. Across teacher temperatures 0.7–1.3, student performance spans 1.5/1.7 points. These experiments support pilot three's observed performance–cost trade-off and modest downstream sensitivity to temperature (Appendices D.6–D.7).

  • Related work and applicability (VWn7, nDsm, 9evp): We clarify differences in supervision, sampling policy, and learning target from the suggested related methods. We also distinguish trajectory SFT, action-level SFT, and on-policy distillation, and explain canonicalization for command, tool, and free-form interfaces.

The added experiments use one seed; the main AP-OPD and OPD n=1 results report three-training-seed means and standard deviations. All manuscript revisions are marked in blue. We hope the Chairs will find the additional evidence and clarifications helpful for the final assessment.

Thank you again for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.