- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a method to distill models through actions alone. The objective is intuitive and makes sense I quite like it actually, its just a simple way to modify the data for sft basically with an extra kl which may or may not be necessary.
Critical strengths
The method seems simple.
Critical weaknesses
The paper is unreadable. Why are there so many proofs, do we really need to be writting this much theory for the sake of theory. I really don't undersand the purpose of them...
Section 4.2 starts with a theorem, why? Not sure the theorem isn't motivated, serves no purpose is never referenced again in the text, I've parsed through that section a few times and still have no clue what the authors are trying to communicate. I can say the same for section 4.3 what is the purpose of these sections?
The results are fine I guess, apart that the reasoninig of the model's is turned off for some reason, i get you train the models not to reason so its fair for a baseline, but most models benefit from this so...
I'm overall significantly dissapointed with the presentation which requires significant work in presenting the method, motivating the idea, not bloating with random math and convincing me this works.
You're also missing quite an important baseline :
Privileged Information Distillation (https://arxiv.org/pdf/2610.19820)
This paper is not cited, does exactly what you are claiming to do in this paper and proposed self-distillation which you use as a baseline.
Questions for authors
See weaknesses
8 comments
W5
Thank you for highlighting this related work. The title and URL in the comment identify two different papers, so we address both.
Privileged Information Distillation for Language Models studies training-time privileged information. Its π-Distill method jointly trains a privileged-information-conditioned teacher and an unconditioned student with shared parameters and reward/KL objectives; the paper also studies OPSD. AP-OPD instead estimates action posteriors from an external black-box teacher's sampled responses, without environment rewards or additional privileged information.
Our experimental question is how effectively a student can learn from sampled teacher responses under these access conditions. Directly adopting PID's full training formulation would introduce additional supervision and a different teacher–student relationship. Such a comparison would evaluate a broader training setup, while our controlled comparisons examine the contribution of action-posterior supervision. We therefore include single-sample OPD, action-level SFT, OmniOPD, GAD, and ROPD, and discuss PID as related work.
The linked paper, ActionOPD: Distilling Decisions, Not Tokens, uses action-agreement rewards and policy optimization. Its first public version appeared on October 14, 2026, after the September 25 submission deadline. We have added it as concurrent work and clarified the distinction from posterior cross-entropy and disagreement-based query allocation.
| Dimension | AP-OPD | PID / π-Distill | ActionOPD |
|---|---|---|---|
| Supervision | Action-posterior cross-entropy | Task rewards and KL objectives with training-time privileged information | Teacher-action agreement rewards |
| Teacher | External black-box teacher | Privileged-information-conditioned teacher sharing student parameters | External teacher; separate Self-ActionOPD extension |
| Sampling and optimization | Disagreement-driven sampling and posterior fitting | Joint teacher–student training | Fixed teacher sampling and PPO-style updates |
For context, ROPD, Section 3.1 and OmniOPD, Section 5.1 also do not include PID among their experimental baselines. ActionOPD explicitly discusses PID in Section 4, while its distillation comparison in Section 3.3 evaluates KD, OPD, and ActionOPD. These examples provide context for baseline selection; our comparison scope follows the supervision available in our problem setting.
Dear Reviewer VWn7,
Thank you for your time and efforts in evaluating our work. As the discussion window is near closing, we would like to check if our responses and revisions have adequately resolved your concerns. We would be very grateful for any additional feedback you might have, as your expertise is vital to enhancing the quality of our submission.
Best regards,
Authors
Thank you for taking the time to review our paper and for raising questions about the role of the analysis, reasoning-enabled inference, and related work. We have revised the presentation to make the connection between the theory and algorithm explicit, added thinking-enabled evaluations, and expanded the related-work discussion. We address each point below and explain why the theoretical results are central to the method.
W1
Each subsection of Section 4 supports a central component of AP-OPD:
The analysis connects the action-level objective, finite-sample posterior error, and disagreement-driven query allocation within one framework. This connection provides the statistical basis for the loss and sampler used by AP-OPD. We consider such analysis an important part of a methodological paper: it explains the algorithm beyond its empirical performance. Describing it as “theory for the sake of theory” overlooks these roles. The proofs are in Appendix F; the revised Section 4 states the purpose of each result before presenting it.
W2
Sections 4.2–4.3 provide the statistical basis for the sampler in Section 3.3. Theorem 2 quantifies estimation error at a fixed sample count, while Lemma 3 and Proposition 4 motivate allocating additional queries using pilot disagreement. Section 4.2 also explicitly connects adaptive stopping to estimation bias, which Appendix F.6 analyzes. We have made these links explicit in the introductory text of both subsections.
W3
The main comparison uses the same non-thinking setting for all students. We additionally enable thinking at inference for OPD n=1, ROPD, and AP-OPD, with a maximum of 4,096 generated tokens per environment step, including reasoning and the final action. Only the final action is executed. AP-OPD remains the strongest of the three on both benchmarks.
With thinking enabled, AP-OPD reaches 85.8% ALFWorld success and 65.1 ScienceWorld score, compared with 83.7%/56.3 for ROPD and 75.4%/55.2 for OPD n=1. The performance ordering is preserved with thinking enabled. These results are now in Section 5.5, Table 4. Entries with ± are the original three-training-seed results; the added evaluations are single-seed results.
W4
The central contribution is a decision-level learning target for black-box agent distillation. AP-OPD maps teacher responses to executable actions, retains their relative frequencies as a soft target, and directs additional teacher queries to states with disagreement. The analysis explains these choices, and the experiments test each one.
In particular, Table 2 holds the five teacher samples fixed: using the full canonical action distribution improves over surface-string supervision by 5.0 points on ALFWorld and 4.3 on ScienceWorld, and over majority voting by 2.7 and 2.3 points. At approximately 700M teacher tokens, AP-OPD achieves 84.3% ALFWorld success versus 75.1% for cost-matched OPD n=1 (revised Table 6). These comparisons isolate the value of the target and query allocation.
We have strengthened the links between the method and Section 4, added the baseline and thinking results in Section 5.5, and added downstream pilot and temperature analyses in Appendices D.6–D.7. The revised manuscript marks additions and changes in blue.