acceptodds
Under review as a conference paper at ICLR 2027

When to Trust Your Teacher: Uncertainty-Aware On-Policy Distillation for Embodied Policies

Abstract

On-policy distillation mitigates compounding errors by aligning supervision with the student-induced state distribution, but its effectiveness is limited by a mismatch between this distribution and the teacher's reliable support. We study how to effectively leverage teacher policies for on-policy learning of flow-based robotic policies. We formulate OPD as policy optimization in a teacher-induced reward MDP and derive an uncertainty-penalized objective that, under an admissible uncertainty estimate, lower-bounds the return in the optimal-policy-induced MDP. Our analysis shows that optimizing this pessimistic objective yields a policy that is near-optimal among policies with bounded cumulative reward uncertainty, revealing a trade-off between the scope and tightness of the guarantee. We propose Uncertainty-Aware On-Policy Distillation (UA-OPD), a framework that accounts for potentially unreliable teacher supervision through uncertainty penalization. We instantiate UA-OPD for flow-based robotic policies with a self-consistency uncertainty estimator, combining soft penalization with uncertainty-based trajectory truncation. Experiments on LIBERO and RoboTwin 2.0 show that UA-OPD achieves a 1.5 improvement in sample efficiency and higher average peak success rates than OPD, while surpassing the teacher on most evaluated suites and tasks. Its benefits extend to out-of-distribution generalization and one-step policy distillation. These findings suggest that embodied OPD benefits from adapting the use of teacher supervision to its reliability.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Intelligence, On-Policy Distillation, Uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LwpBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies an important limitation of on-policy distillation: although querying a teacher on student-induced states addresses distribution mismatch for the student, the teacher itself may be unreliable on those states. The proposed UA-OPD method estimates teacher uncertainty by comparing action chunks generated from the same initial noise using different denoising budgets. This estimate guides both trajectory truncation and an uncertainty-weighted log-probability penalty. Experiments on LIBERO and RoboTwin 2.0 report improvements over vanilla OPD.

Critical strengths
  • Well-motivated problem. The distinction between covering the student’s state distribution and obtaining reliable supervision on that distribution is important. A teacher’s strong performance from standard initial states does not establish its ability to recover from states produced by a weaker student.
  • Simple and practical uncertainty estimator. The proposed estimator reuses a frozen teacher and does not require separately trained models or an auxiliary uncertainty network.
Critical weaknesses
  1. When does Theorem 4.2 provide an informative guarantee for a suboptimal teacher?

Proposition 4.1 establishes optimality preservation under exact imitation of an optimal policy, whereas the practical teacher is suboptimal. Moreover, the discrepancy ∣log⁡πT−log⁡π⋆∣\left|\log\pi_T-\log\pi^\star\right|measures disagreement with a particular optimal policy, which need not correspond to task-level suboptimality.

Defining L⋆(π):=LOPDπ⋆(π)=−ηM(π)L_\star(\pi):=\mathcal L_{\mathrm{OPD}}^{\pi^\star}(\pi)=-\eta_M(\pi), Theorem 4.2 can be rewritten as

\le \inf_{\pi\in\Pi_\epsilon}L_\star(\pi)+2\epsilon+\delta.$$ An informative guarantee therefore requires **both a strong comparator and a sufficiently small uncertainty budget**; the theorem does not establish that these conditions hold simultaneously. In particular, admissibility implies, for a realizable teacher with finite divergence, $$\epsilon_u(\pi_T)\ge\overline{\mathbb E}_{\rho_P^{\pi_T}}\left[\log\frac{\pi_T(a\mid s)}{\pi^\star(a\mid s)}\right]=L*(\pi_T).$$ Consequently, whenever $\pi_T\in\Pi_\epsilon$, the error term alone satisfies $2\epsilon+\delta\ge 2L_\star(\pi_T)+\delta$. Even with zero comparator loss, this bound cannot certify improvement over the teacher. Choosing a smaller budget may help, but then the existence of strong comparators needs justification. **Can the authors identify conditions under which the bound is non-vacuous, and clarify how an approximate ideal-OPD guarantee translates into actual environment performance?** 2. **Why should discretization self-consistency reliably identify harmful teacher supervision?** Fixing the initial noise controls variation from independent random inputs, but does not guarantee that different discretizations preserve the same action mode. Conversely, numerical agreement does not establish correctness: both discretizations share the same potentially inaccurate teacher. The paper appropriately describes its estimator as measuring discretization sensitivity. Writing the numerical output as $A^{(k)}=A_T^{(\infty)}+e^{(k)}$, where $A_T^{(\infty)}$ denotes the exact teacher-flow solution, gives $$A^{(5)}-A^{(10)}=e^{(5)}-e^{(10)}.$$ The shared teacher output cancels, so this difference need not detect errors in the teacher’s underlying decision rule. Likewise, discretization-induced mode changes need not imply harmful supervision. The authors acknowledge that the score is not a verified reward-error upper bound. **What evidence establishes its calibration beyond correlation, its robustness to denoising-budget choices, and its ability to distinguish unreliable supervision from harmless numerical or behavioral variation?**
Questions for authors
  1. How is Eq. (14) derived from Eq. (12)? In particular, how does the current-chunk uncertainty-weighted log-probability update assign credit to actions that affect future uncertainty?

  2. Can the authors report multi-seed variability and final-checkpoint performance, rather than only peak success rates?

  3. Do the efficiency gains persist after counting discarded trajectories, additional teacher evaluations, and wall-clock time? Would matched-retention random truncation or per-chunk filtering achieve similar improvements?

Reviewer_J7H2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem that, in on-policy distillation (OPD) of flow-based VLA policies, teacher supervision becomes unreliable when the student visits states that are out-of-distribution for the teacher. The authors formulate OPD as policy optimization in a teacher-induced reward MDP, where the teacher's log-likelihood serves as the reward. Building on this, they propose a pessimistic objective (UA-OPD) that subtracts teacher uncertainty from the reward. If the uncertainty upper-bounds the reward error, this objective yields a lower bound on the ideal OPD return and a near-optimality guarantee (Theorem 4.2).

In practice, the teacher is run from the same initial noise with 5 and 10 denoising steps, and the discrepancy between the resulting action chunks is used as the uncertainty UU. UU is used for soft uncertainty penalization (Eq. 14) and for trajectory truncation (Eq. 13), which discards all chunks from the first time UU exceeds a threshold. Experiments with π0.5 on all four LIBERO suites and four RoboTwin 2.0 tasks show improvements over OPD in sample efficiency and peak success rates. The paper also compares the methods under out-of-distribution environments and in one-step distillation.

Critical strengths
  • Problem setting of teacher reliability, supported by analysis The question of how far to trust teacher supervision in OPD is natural and important in embodied settings. In the teacher-takeover evaluation (Table 1), the teacher itself is more likely to fail from states with higher UU, with success-rate gaps of 20–34 points on RoboTwin 2.0. UA-OPD requires no additional model training. It reaches high success rates with fewer samples than OPD, and on LIBERO Long, the most challenging suite, it raises the peak success rate from 93.0% to 98.0%.
Critical weaknesses
  • Gap between the theoretical guarantee and the implementation Theorem 4.2 assumes that uu upper-bounds the difference between the log-likelihoods of the teacher and the optimal policy. The authors acknowledge that the implemented UU is an empirical estimate (Section 4.3), and Section 5.1 reports a rank correlation with the reward discrepancy, but this does not show that UU provides such an upper bound. In addition, trajectory truncation lies outside the objective. Furthermore, if the implementation follows the description, I believe the soft penalty in Eq. (14) is not the gradient of the penalty in Eq. (12). UtU_t is determined by the state sts_t and the initial noise, and does not depend on the transition sampled at that step. Therefore, by the score-function identity, the expectation of λUt∇log⁡pS,t,i\lambda U_t \nabla \log p_{S,t,i} is zero. Correctly differentiating Eq. (12) would instead weight the score by the discounted future uncertainty ∑t′>tγt′−tUt′\sum_{t'>t} \gamma^{t'-t} U_{t'}. The ablation in Figure 5 is limited to a single task without reported seed variance, so the performance gains may come mainly from truncation. In that case, the connection to the pessimistic objective in Section 4.1 would be weaker than presented.

  • Limited range of teacher quality All teachers used in the experiments are strong, with average success rates of 96.9% on LIBERO and 93.75% on RoboTwin 2.0. Teacher unreliability therefore arises mainly when the student drifts out of distribution, and settings where the teacher itself is of low quality are not examined. Section 5.1 includes an analysis with a lower-performing teacher, but it is used only to examine the correlation between uncertainty and reward discrepancy; distillation results with that teacher are not reported. One would expect the benefit of UA-OPD to grow as the teacher becomes weaker. On the other hand, truncation reduces the amount of usable data, which could instead put UA-OPD at a disadvantage. Without such results, it is difficult to judge under which conditions the proposed method is effective.

Questions for authors
  • On the penalty in Eq. (14) If there is an implementation reason why the expected gradient of the penalty term is nonzero (e.g., performing multiple updates on the same rollouts, which makes them off-policy), please explain. Beyond the Handover Block ablation in Figure 5, could you also provide a comparison that removes only the penalty, across multiple tasks and multiple seeds?

  • On teacher quality Could you prepare teachers with different performance levels (e.g., by reducing the number of SFT demonstrations) and show how the gap between OPD and UA-OPD changes as a function of the teacher's success rate? Please also report the fraction of samples excluded by truncation in each case.

Reviewer_LhC5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on on-policy distillation(OPD) of flow-matching robot policies where the frozen teacher is used to label the rollouts of the student policy. The paper casts reverse-KL OPD as entropy-regularized RL with reward logπTlog \pi_T. Under an admissible uncertainty the paper shows that penalized reward lower-bounds the ideal OPD return. Teacher uncertainty is measured as the normalized disagreement b/w action chunks generated from the same noise with 10 and with 5 denoising steps. Experiments on LIBERO and RobotTwin 2.0 show better peak success rate than OPD.

Critical strengths
  1. On-policy distillation is an important topic and addressing the uncertainty of the teacher on the student's rollouts is an important problem.

  2. The uncertainty estimator is cheap and training-free: it only reruns the frozen teacher with two solver budgets. The teacher-takeover protocol (Table 1, Fig. 2a) directly tests whether the teacher can still complete the task from a student-induced state. On RoboTwin, success differs signifincantly between low- and high-uncertainty states.

Critical weaknesses
  1. Admissibility assumption is clearly violated by the paper's own analysis in Fig. 2b. Uncertainty seems to be always underestimating the reward gap.

  2. Uncertainty score measured in the paper doesn't necessarily reflect teacher reliability or confidence. Disagreement between 5- and 10-step solutions measures solver error, which grows with the curvature of the flow trajectory. Curvature also arises at in-distribution states where the teacher has several valid actions (aleatoric uncertainty).

  3. Poor empirical evaluation: The paper only uses a single seed for most of its analysis which doesn't clearly establishes rigorousness of the results. The paper also doesn't compare against Speculative KD [1] which also tackles the same problem.

[1] W. Xu, R. Han, Z. Wang, L. T. Le, D. Madeka, L. Li, W. Y. Wang, R. Agarwal, C.-Y. Lee, T. Pfister. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling. ICLR 2025.

Questions for authors
  1. How does UA-OPD compare to random, fixed-time-step, and teacher–student-disagreement truncation at the same fraction of retained chunks? What fraction of chunks is truncated per task during training?

  2. What budget did PPO receive? How does UA-OPD compare with PPO at πRL's budget [1] (which reports much higher performance for PPO than this paper), in environment steps and GPU-hours, counting the teacher's additional demonstrations and training?

  3. Can you report your results on more seeds? Also, Both teachers are near ceiling (93.75% on RoboTwin, 96.9% on LIBERO), so the setting offers little opportunity for teacher unreliability to matter. Please evaluate UA-OPD with weaker teachers on the existing tasks, for example the lower-ceiling teacher already used in Fig. 2b or teachers trained on fewer demonstrations, and on RoboTwin 2.0 tasks where the teacher is clearly imperfect.

[1] K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, X. Li, et al. πRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Models. arXiv:2510.25889, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.