- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies an important limitation of on-policy distillation: although querying a teacher on student-induced states addresses distribution mismatch for the student, the teacher itself may be unreliable on those states. The proposed UA-OPD method estimates teacher uncertainty by comparing action chunks generated from the same initial noise using different denoising budgets. This estimate guides both trajectory truncation and an uncertainty-weighted log-probability penalty. Experiments on LIBERO and RoboTwin 2.0 report improvements over vanilla OPD.
Critical strengths
- Well-motivated problem. The distinction between covering the student’s state distribution and obtaining reliable supervision on that distribution is important. A teacher’s strong performance from standard initial states does not establish its ability to recover from states produced by a weaker student.
- Simple and practical uncertainty estimator. The proposed estimator reuses a frozen teacher and does not require separately trained models or an auxiliary uncertainty network.
Critical weaknesses
- When does Theorem 4.2 provide an informative guarantee for a suboptimal teacher?
Proposition 4.1 establishes optimality preservation under exact imitation of an optimal policy, whereas the practical teacher is suboptimal. Moreover, the discrepancy measures disagreement with a particular optimal policy, which need not correspond to task-level suboptimality.
Defining , Theorem 4.2 can be rewritten as
\le \inf_{\pi\in\Pi_\epsilon}L_\star(\pi)+2\epsilon+\delta.$$ An informative guarantee therefore requires **both a strong comparator and a sufficiently small uncertainty budget**; the theorem does not establish that these conditions hold simultaneously. In particular, admissibility implies, for a realizable teacher with finite divergence, $$\epsilon_u(\pi_T)\ge\overline{\mathbb E}_{\rho_P^{\pi_T}}\left[\log\frac{\pi_T(a\mid s)}{\pi^\star(a\mid s)}\right]=L*(\pi_T).$$ Consequently, whenever $\pi_T\in\Pi_\epsilon$, the error term alone satisfies $2\epsilon+\delta\ge 2L_\star(\pi_T)+\delta$. Even with zero comparator loss, this bound cannot certify improvement over the teacher. Choosing a smaller budget may help, but then the existence of strong comparators needs justification. **Can the authors identify conditions under which the bound is non-vacuous, and clarify how an approximate ideal-OPD guarantee translates into actual environment performance?** 2. **Why should discretization self-consistency reliably identify harmful teacher supervision?** Fixing the initial noise controls variation from independent random inputs, but does not guarantee that different discretizations preserve the same action mode. Conversely, numerical agreement does not establish correctness: both discretizations share the same potentially inaccurate teacher. The paper appropriately describes its estimator as measuring discretization sensitivity. Writing the numerical output as $A^{(k)}=A_T^{(\infty)}+e^{(k)}$, where $A_T^{(\infty)}$ denotes the exact teacher-flow solution, gives $$A^{(5)}-A^{(10)}=e^{(5)}-e^{(10)}.$$ The shared teacher output cancels, so this difference need not detect errors in the teacher’s underlying decision rule. Likewise, discretization-induced mode changes need not imply harmful supervision. The authors acknowledge that the score is not a verified reward-error upper bound. **What evidence establishes its calibration beyond correlation, its robustness to denoising-budget choices, and its ability to distinguish unreliable supervision from harmless numerical or behavioral variation?**Questions for authors
-
How is Eq. (14) derived from Eq. (12)? In particular, how does the current-chunk uncertainty-weighted log-probability update assign credit to actions that affect future uncertainty?
-
Can the authors report multi-seed variability and final-checkpoint performance, rather than only peak success rates?
-
Do the efficiency gains persist after counting discarded trajectories, additional teacher evaluations, and wall-clock time? Would matched-retention random truncation or per-chunk filtering achieve similar improvements?