acceptodds
Under review as a conference paper at ICLR 2027

ID2P: Future-Guided Policy Learning via IDM-to-Policy Distillation

Abstract

Future observations reveal the consequences of executed actions, motivating existing methods to leverage them for robot policy learning. However, prior work primarily uses future observations for representation-level supervision, while direct action-level future supervision remains less explored. We propose IDM-to-Policy Distillation (ID2P), a training framework that combines action-level and representation-level future supervision for flow-based policies. ID2P first trains an inverse dynamics model (IDM) conditioned on current and future observations using the same architecture as the policy. The trained IDM then provides action supervision by mixing its predicted action velocity with the expert action velocity from expert trajectories, while its action expert hidden states provide complementary representation supervision for the policy. Under an ideal IDM, we show that the mixed action velocity objective preserves the expected gradient of expert-only supervision while achieving no greater target variance. ID2P requires neither future observations nor explicit future state prediction at inference time, preserving the policy architecture and inference cost. Extensive simulation and real-world experiments demonstrate that ID2P consistently improves task success rates across flow-based policies of different architectures and scales, including SmolVLA (0.45B) and (3B).

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Flow-based Policies, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gwphOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors proposed ID2P for training flow-based robot policies, which contains an inverse dynamics model (IDM) using current and future observations. The frozen IDM then teaches a policy through action-velocity supervision and hidden-feature matching, while the policy only needs current observations at test time. The paper reports higher average success rates on LIBERO, LIBERO-Plus, RoboTwin, and real robot tasks.

Critical strengths
  1. Future observations are used only during training. The final policy does not need the IDM or extra future prediction at test time.
  2. The paper tests both SmolVLA and pi0.5. It includes simulation, real robot tasks, component ablations, and a comparison with longer baseline training.
  3. The paper shows that an ideal IDM can reduce action-target variance without changing the expected training gradient.
Critical weaknesses
  1. The two distillation losses are not explained well enough. Table 4 shows that representation-only distillation performs better than action-only distillation on LIBERO-Plus. The paper does not show why this happens or what each loss changes in the policy's actions. The ablations tell us which settings score higher, but not why.

  2. The role of future images is not fully separated from the role of an extra trained teacher. The shared-teacher ablation and training-budget comparison are useful. However, they do not compare ID2P with an independently trained teacher that sees only current observations. It remains unclear how much of the gain comes specifically from future information.

  3. The policy analysis is too limited. Appendix Figure 4 shows example robot executions, but not direct comparisons with the baseline from the same starting state. There is little analysis of changed actions or failures. For example, Pack Gift drops from 80.0% to 73.3% in Table 3. The paper does not explain why ID2P helps some tasks but hurts others.

Questions for authors
  1. Could you plot the measured action-target variance and gradient variance during training? Please compare expert-only, IDM-only, and mixed supervision. Does lower variance lead to better results?
  2. Could you plot IDM prediction and policy action information over training?
  3. Table 4 shows a larger gain from representation distillation than from action distillation on LIBERO-Plus. Could you show training curves for both losses and their combination? When do the two signals help each other or conflict?
  4. Please visualize their predicted actions and full executed trajectories. Include successes and failures, especially Pack Gift, and report uncertainty for the real robot results.
13 comments
AuthorsOpenReview ↗

W2: Compare ID2P with an independently trained teacher that sees only current observations.

The results are shown below.

TeacherFuture inputStudent settingα\alphaλRD\lambda_{\mathrm{RD}}LIBEROLIBERO-PlusAvg.
——Expert-only (baseline)0095.5±0.495.5^{\pm 0.4}39.9±2.639.9^{\pm 2.6}67.7±1.467.7^{\pm 1.4}
NFT✗Mixed0.5095.5±0.595.5^{\pm 0.5}40.5±3.540.5^{\pm 3.5}68.0±2.068.0^{\pm 2.0}
FT✓Mixed0.5095.9±0.595.9^{\pm 0.5}40.3±3.140.3^{\pm 3.1}68.1±1.468.1^{\pm 1.4}
NFT✗Expert-only + RD0294.8±0.594.8^{\pm 0.5}39.7±5.139.7^{\pm 5.1}67.2±2.767.2^{\pm 2.7}
FT✓Expert-only + RD0296.2±0.596.2^{\pm 0.5}45.5±6.145.5^{\pm 6.1}70.9±2.870.9^{\pm 2.8}
NFT✗Mixed + RD0.5295.4±0.695.4^{\pm 0.6}40.8±5.140.8^{\pm 5.1}68.1±2.868.1^{\pm 2.8}
FT✓Mixed + RD (Ours)0.5296.5±0.2\mathbf{96.5}^{\pm 0.2}46.4±4.6\mathbf{46.4}^{\pm 4.6}71.4±2.2\mathbf{71.4}^{\pm 2.2}

The above results are from Table 4 of our paper and report success rates with mean±std across three training seeds. The notation is defined as follows:

  • NFT (no future teacher): A policy trained with expert-only action supervision and no future image inputs. It sees only current observations.
  • FT (future teacher): An IDM trained with expert-only action supervision and future image inputs. NFT and FT differ only in that FT receives future images while NFT does not. All other settings, including the architecture and training recipe, are the same.
  • Expert-only: Uses expert-only action supervision (α=0\alpha=0, λRD=0\lambda_{\mathrm{RD}}=0).
  • Mixed: Uses both expert and teacher action supervision (α=0.5\alpha=0.5, λRD=0\lambda_{\mathrm{RD}}=0).
  • + RD: Adds representation distillation (λRD=2\lambda_{\mathrm{RD}}=2).
  • Expert-only + RD: Combines expert action supervision with representation distillation (α=0\alpha=0, λRD=2\lambda_{\mathrm{RD}}=2).
  • Mixed + RD: Combines expert and teacher action supervision with representation distillation (α=0.5\alpha=0.5, λRD=2\lambda_{\mathrm{RD}}=2).
  • Future input: Marks teacher access to future images.
  • Action distillation: Introduces action supervision from the teacher.
  • Representation distillation: Introduces representation supervision from the teacher.

Conclusion: Compared with excluding future observations from teacher inputs, including them substantially benefits representation distillation, while providing a modest benefit to action distillation. Future observations also benefit joint action and representation distillation. These results indicate that ID2P's gains primarily come from future information rather than current-only self-distillation.

Please refer to Sec. 5.4 (Q3) and App. B.3 of our paper for details.

AuthorsOpenReview ↗

Q1: Could you plot the measured action-target variance and gradient variance during training? Does lower variance lead to better results?

Directly estimating the action-target variances VPolicy\mathcal V_{\mathrm{Policy}} and VAD\mathcal V_{\mathrm{AD}} is difficult. Their computation requires sampling from the conditional distributions pτ(At∣Atτ,ot)p_\tau(A_t\mid A_t^\tau,o_t) and pτ(At,Ft∣Atτ,ot)p_\tau(A_t,F_t\mid A_t^\tau,o_t), respectively (Eqs. (3) and (8) of our paper). However, each observation oto_t in the robot demonstration datasets corresponds to only one expert action chunk AtA_t, making these conditional distributions difficult to construct. Therefore, we did not measure action-target variance.

To measure gradient variance, we train an expert-only SmolVLA policy on LIBERO and save checkpoints at different training steps. At each checkpoint, we compute the gradient variance for the following action velocity targets:

  • Expert-only: Expert action velocities (α=0\alpha=0, λRD=0\lambda_{\mathrm{RD}}=0).
  • Mixed w/ FT: A mixture of expert and frozen IDM action velocities (α=0.5\alpha=0.5, λRD=0\lambda_{\mathrm{RD}}=0).
  • Teacher-only w/ FT: Frozen IDM action velocities only (α=1\alpha=1, λRD=0\lambda_{\mathrm{RD}}=0).
  • Mixed w/ NFT: A mixture of expert and frozen policy teacher action velocities (α=0.5\alpha=0.5, λRD=0\lambda_{\mathrm{RD}}=0).
  • Teacher-only w/ NFT: Frozen policy teacher action velocities only (α=1\alpha=1, λRD=0\lambda_{\mathrm{RD}}=0).

FT (future teacher) receives future images, whereas NFT (no future teacher) does not.

The following table reports gradient variance Tr⁡Cov⁡^(g)\operatorname{Tr}\widehat{\operatorname{Cov}}(g) (×105\times 10^5) from Figure 8(a) of our paper.

Training stepsExpert-onlyMixed w/ FTTeacher-only w/ FTMixed w/ NFTTeacher-only w/ NFT
0756.6752.2750.8751.7750.1
5k18.618.218.218.118.1
10k10.910.310.310.09.9
15k8.47.57.77.17.0
20k6.65.46.04.95.2
25k5.64.45.33.74.5
30k5.44.25.23.64.4

To explain this observation, we have added a theorem characterizing the gradient variance reduction property of action distillation using a frozen ideal IDM relative to expert-only supervision (Theorem 3 of our paper).

Conclusion: Both FT settings have lower gradient variance than Expert-only, providing empirical support for the variance nonincrease predicted by Theorem 3 with a learned IDM. NFT yields even lower gradient variance than FT at the same mixing weight.

Furthermore, we define the following metrics: Student-to-teacher error measures MSE between the student's and teacher's predicted action velocities. Teacher-to-expert error measures MSE between the teacher's predicted action velocities and expert action velocities.

Under Mixed, we observe lower student-to-teacher error with NFT than with FT (17.3×10−317.3\times 10^{-3} vs. 24.6×10−324.6\times 10^{-3}, Figure 6(e)), indicating that the student fits NFT more closely. However, NFT has higher teacher-to-expert error than FT (67.1×10−367.1\times 10^{-3} vs. 61.2×10−361.2\times 10^{-3}, Figure 6(f)), indicating a less accurate teacher.

Together, these results suggest that NFT enables the student to fit a less accurate teacher more stably, which does not guarantee higher success rates. With FT, the student fits a more accurate teacher while still having lower gradient variance than Expert-only.

We therefore compare the success rates of policies trained with each of the five action velocity targets above:

Action targetLIBEROΔ\DeltaLIBERO-PlusΔ\Delta
Expert-only (baseline)95.5±0.495.5^{\pm 0.4}0.039.9±2.639.9^{\pm 2.6}0.0
Mixed w/ FT95.9±0.5\mathbf{95.9}^{\pm 0.5}+0.440.3±3.140.3^{\pm 3.1}+0.4
Teacher-only w/ FT95.7±0.495.7^{\pm 0.4}+0.240.4±5.140.4^{\pm 5.1}+0.5
Mixed w/ NFT95.5±0.595.5^{\pm 0.5}0.040.5±3.540.5^{\pm 3.5}+0.6
Teacher-only w/ NFT95.5±0.795.5^{\pm 0.7}0.041.6±4.1\mathbf{41.6}^{\pm 4.1}+1.8

Δ\Delta denotes the success rate gain over the expert-only baseline.

Conclusion: Lower gradient variance does not necessarily lead to higher success rates. Although FT achieves less gradient variance reduction than NFT, both FT variants improve success rates over the expert-only baseline on both benchmarks, whereas NFT variants show almost no improvement on LIBERO.

Please refer to Sec. 5.4 (Q9) and App. B.9 of our paper for details.

In App. B.10 (Figure 9), we also train five SmolVLA policies on LIBERO from the same initialization, each using its own supervision for 30k steps. We measure mini-batch gradient variance and per-sample loss variance at each policy's own trained parameters, rather than at a shared set of parameters, using 128 fixed batches (global batch size 128). All four distillation variants have lower values than Expert-only in late training, showing that lower variance is also observed during policy optimization. However, lower variance does not necessarily lead to higher success rates.

AuthorsOpenReview ↗

W1/Q3: Analysis of action distillation, representation distillation, and joint distillation.

(1) Could you show training curves for the action distillation loss, the representation distillation loss, and their combination?

Figure 6 shows the training curves for the following ID2P configurations on SmolVLA trained on LIBERO, represented by the blue curves labeled Trained IDM:

  • Action distillation alone: Mixed (α=0.5\alpha=0.5, λRD=0\lambda_{\mathrm{RD}}=0), solid lines.
  • Representation distillation alone: Expert + RD (α=0\alpha=0, λRD=2\lambda_{\mathrm{RD}}=2), dashed lines.
  • Joint distillation: Mixed + RD (α=0.5\alpha=0.5, λRD=2\lambda_{\mathrm{RD}}=2), dash-dot lines.

Figures 6(a), 6(b), and 6(c) report action distillation loss, representation distillation loss, and total loss, respectively.

(2) When do the two signals help each other or conflict?

With the trained IDM, we observe that action distillation and representation distillation help each other, with no clear conflicts between the two signals.

Mixed + RD has lower action distillation loss and student-to-teacher error than Mixed (Figure 6(a,e)), and slightly lower representation distillation loss than Expert + RD (Figure 6(b)). These reductions provide important empirical evidence that action distillation helps representation distillation better align student and teacher representations, while representation distillation helps action distillation better fit the teacher's action velocities. Mixed + RD also achieves the highest success rates (Table 4), further validating the complementary nature of the two supervision signals.

(3) Why does representation distillation alone (Expert + RD) perform better than action distillation alone (Mixed)?

We hypothesize that two factors explain this difference.

  1. Action distillation directly changes the action velocity target, so inaccurate teacher predictions can directly affect action learning. Representation distillation does not modify action velocity targets and instead provides softer guidance through hidden states, influencing action learning indirectly.

  2. Representation distillation supervises all action expert layers, allowing future information to guide representations at multiple depths. This may provide a richer and more diverse learning signal than supervising the final action velocity alone.

Student-to-expert error measures MSE between the student's predicted action velocities and expert action velocity targets. Mixed has higher student-to-expert error than Expert + RD (71.4×10−371.4\times 10^{-3} vs. 64.8×10−364.8\times 10^{-3}), providing a clue to its lower success rates. However, Expert-only has lower student-to-expert error than Mixed (67.1×10−367.1\times 10^{-3} vs. 71.4×10−371.4\times 10^{-3}), yet Mixed performs better on both benchmarks (Table 4). Therefore, even a large student-to-expert error is only an indirect indicator of success rates and cannot perfectly predict their ranking.

Reviewer_oKQLOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes ID2P, a two-stage distillation method for improving flow-based VLAs. It first trains an inverse dynamics model (IDM) using additional future observations 16 steps ahead. A new policy is then trained with two losses: action distillation, which mixes the IDM's predicted velocity into the flow-matching target, and representation distillation, which aligns the policy's hidden features with those of the IDM. The method requires no changes at inference time. The paper also provides theoretical results showing that, under an ideal IDM, the mixed objective is equivalent to standard flow matching up to a constant while reducing target variance. Experiments with SmolVLA and π0.5 show improvements on LIBERO (+1.5/+0.5 points), LIBERO-Plus (+9.3/+2.2), and RoboTwin (+0.2 to +2.8). On four real-robot tasks, success improves from 55.0% to 64.2%.

Critical strengths
  1. The method is simple, requires no architecture changes or extra inference cost, and works with both SmolVLA and π0.5, though it needs future observations and additional training.
  2. The evaluation is fairly extensive, covering two backbones across three benchmarks with thousands of test episodes, as well as real bimanual robot experiments.
  3. The improvements hold up against longer-trained baselines (71.3 vs. 67.4 in Table 8), with particularly clear gains on SmolVLA LIBERO-Plus (45.9% vs. 38.7%/39.1%).
  4. The ablations are thorough, with detailed results across different benchmarks and task categories in Appendix A.
Critical weaknesses
  1. The main benefit of future observations is not clearly established, since the gains could also come from distillation itself; a teacher trained without future frames would be an important baseline.
  2. At a matched training budget, action distillation alone offers almost no improvement (39.0% vs. 38.7% on LIBERO-Plus), while most of the gains come from representation distillation, which lacks theoretical analysis.
  3. The theory relies on an ideal IDM and mainly explains variance reduction, but neither the actual target variance nor the IDM's accuracy is measured, making its practical relevance unclear.
  4. No training seeds or variance are reported, making it difficult to judge the reliability of the gains, especially since some improvements are quite small and vary across task categories.
  5. The comparison with PFD is incomplete, and the shared-parameter teacher performs surprisingly poorly, raising questions about whether the baseline was implemented properly.
  6. Several design choices are insufficiently explored, including the future-frame offset and representation loss weight, while the connection to REPA-style alignment and some implementation details are missing.
Questions for authors
  1. What happens with the same two-stage pipeline when the teacher is trained without the future image, for action distillation, representation distillation and both? A clearly worse no-future teacher would raise my score.
  2. In Tables 4 and 8, action distillation alone (96.0 / 39.0) is on par with the epoch-matched 60k-step baseline (96.1 / 38.7), while representation distillation alone reaches 96.4 / 43.3, all at 28 epochs. How do you reconcile this with the claim in Sec. 5.4 that action-level and representation-level supervision "each independently improve policy performance"?
  3. Can you report mean ± std over at least three training seeds for LIBERO, LIBERO-Plus and RoboTwin, at least for SmolVLA? Please include the action-distillation-only and representation-distillation-only variants and the 60k-step baseline.
  4. How accurate is the IDM on held-out demonstrations, and how closely does it fit the training demonstrations? Can you measure the per-sample loss variance or stochastic-gradient variance under the expert target versus the mixed target, with the trained IDM and with a no-future teacher?
  5. How do results change with other future offsets (e.g., 8 or 32) and with λ_RD?
  6. How was the shared-policy teacher in Table 4 trained, and does it see the future image? Why does it collapse with action distillation alone (57.8 vs. 95.1 on LIBERO)? Can you compare with PFD under identical settings, or explain why this teacher is a faithful proxy for PFD?
  7. For the real robot, how many demonstrations per task, what success criteria, what randomization and what trial order did you use? How was the ImageWAM row obtained? Please give confidence intervals or a significance test for the per-task and average results.
  8. Where do the external baseline numbers in Tables 1–2 come from (reported or re-run), and with what training data?
18 comments
AuthorsOpenReview ↗

W1/Q1: What happens with the same two-stage pipeline when the teacher is trained without the future image?

A frozen teacher without future images can offer modest gains over the expert-only baseline in some settings, while a teacher with future images achieves higher average performance across the two benchmarks. The results are shown below.

TeacherFuture inputStudent settingα\alphaλRD\lambda_{\mathrm{RD}}LIBEROLIBERO-PlusAvg.
——Expert-only (baseline)0095.5±0.495.5^{\pm 0.4}39.9±2.639.9^{\pm 2.6}67.7±1.467.7^{\pm 1.4}
NFT✗Mixed0.5095.5±0.595.5^{\pm 0.5}40.5±3.540.5^{\pm 3.5}68.0±2.068.0^{\pm 2.0}
FT✓Mixed0.5095.9±0.595.9^{\pm 0.5}40.3±3.140.3^{\pm 3.1}68.1±1.468.1^{\pm 1.4}
NFT✗Expert-only + RD0294.8±0.594.8^{\pm 0.5}39.7±5.139.7^{\pm 5.1}67.2±2.767.2^{\pm 2.7}
FT✓Expert-only + RD0296.2±0.596.2^{\pm 0.5}45.5±6.145.5^{\pm 6.1}70.9±2.870.9^{\pm 2.8}
NFT✗Mixed + RD0.5295.4±0.695.4^{\pm 0.6}40.8±5.140.8^{\pm 5.1}68.1±2.868.1^{\pm 2.8}
FT✓Mixed + RD (Ours)0.5296.5±0.2\mathbf{96.5}^{\pm 0.2}46.4±4.6\mathbf{46.4}^{\pm 4.6}71.4±2.2\mathbf{71.4}^{\pm 2.2}

The above results are from Table 4 of our paper and report success rates with mean±std across three training seeds. The notation is defined as follows:

  • NFT (no future teacher): A policy trained with expert-only action supervision and no future image inputs.
  • FT (future teacher): An IDM trained with expert-only action supervision and future image inputs. NFT and FT differ only in that FT receives future images while NFT does not. All other settings, including the architecture and training recipe, are the same.
  • Expert-only: Uses expert-only action supervision (α=0\alpha=0, λRD=0\lambda_{\mathrm{RD}}=0).
  • Mixed: Uses both expert and teacher action supervision (α=0.5\alpha=0.5, λRD=0\lambda_{\mathrm{RD}}=0).
  • + RD: Adds representation distillation (λRD=2\lambda_{\mathrm{RD}}=2).
  • Expert-only + RD: Combines expert action supervision with representation distillation (α=0\alpha=0, λRD=2\lambda_{\mathrm{RD}}=2).
  • Mixed + RD: Combines expert and teacher action supervision with representation distillation (α=0.5\alpha=0.5, λRD=2\lambda_{\mathrm{RD}}=2).
  • Future input: Marks teacher access to future images.
  • Action distillation: Introduces action supervision from the teacher.
  • Representation distillation: Introduces representation supervision from the teacher.

Conclusion: Compared with excluding future observations from teacher inputs, including them substantially benefits representation distillation, while providing a modest benefit to action distillation. Future observations also benefit joint action and representation distillation. These results indicate that ID2P's gains primarily come from future information rather than current-only self-distillation.

Please refer to Sec. 5.4 (Q3) and App. B.3 of our paper for details.

AuthorsOpenReview ↗

W2: Analysis of action distillation.

The following table reports average metrics for expert-only baseline (Expert-only), action distillation alone (Mixed), representation distillation alone (Expert + RD), and their combination (Mixed + RD) over the final 500 training steps.

Student-to-expert error measures MSE between the student's predicted action velocities and expert action velocities. Student-to-teacher error measures MSE between the student's and teacher's predicted action velocities. Representation distillation loss measures the average cosine distance between projected student representations and teacher representations across all action expert layers. All distillation settings use a trained IDM teacher. “—” denotes an inactive metric.

SettingStudent-to-expert error (×10−3\times 10^{-3})Student-to-teacher error (×10−3\times 10^{-3})Representation distillation loss (×10−3\times 10^{-3})
Expert-only67.1——
Mixed71.424.6—
Expert + RD64.8—57.7
Mixed + RD70.220.556.1

We answer two questions:

(1) Why does action distillation alone (Mixed) underperform representation distillation alone (Expert + RD)?

We hypothesize that two factors explain this difference.

  1. Action distillation directly changes the action velocity target, so inaccurate teacher predictions can directly affect action learning. Representation distillation does not modify action velocity targets and instead provides softer guidance through hidden states, influencing action learning indirectly.

  2. Representation distillation supervises all action expert layers, allowing future information to guide representations at multiple depths. This may provide a richer and more diverse learning signal than supervising the final action velocity alone.

Mixed has higher student-to-expert error than Expert + RD (71.4×10−371.4\times 10^{-3} vs. 64.8×10−364.8\times 10^{-3}), providing a clue to its lower success rates. However, Expert-only has lower student-to-expert error than Mixed (67.1×10−367.1\times 10^{-3} vs. 71.4×10−371.4\times 10^{-3}), yet Mixed performs better on both benchmarks (Table 4). Therefore, even a large student-to-expert error is only an indirect indicator of success rates and cannot perfectly predict their ranking.

(2) What role does action distillation play in joint distillation?

Adding action distillation to Expert + RD reduces representation distillation loss from 57.7×10−357.7\times 10^{-3} to 56.1×10−356.1\times 10^{-3}, suggesting that action distillation helps representation alignment. Mixed + RD improves LIBERO success from 96.2% to 96.5% and LIBERO-Plus success from 45.5% to 46.4% compared with Expert + RD (Table 4).

Representation distillation also helps action distillation: Mixed + RD achieves lower student-to-teacher error than Mixed (20.5×10−320.5\times 10^{-3} vs. 24.6×10−324.6\times 10^{-3}). These results suggest that action distillation and representation distillation are complementary.

Please refer to Figure 6 of our paper for details.

AuthorsOpenReview ↗

Q2: Independent contributions of action-level and representation-level supervision.

Action distillation alone (Mixed, 30k policy training steps) improves success over the 30k expert-only baseline on both LIBERO (95.9% vs. 95.5%) and LIBERO-Plus (40.3% vs. 39.9%). Compared with the 60k expert-only baseline, Mixed still improves LIBERO (95.9% vs. 95.5%) but is 0.5 percentage points lower on LIBERO-Plus (40.3% vs. 40.8%), as reported in Tables 4 and 8. We have therefore clarified in Sec. 5.4 (Q1) that the independent gains from action distillation are relative to the 30k baseline.

Representation distillation alone (Expert + RD) achieves 96.2% on LIBERO and 45.5% on LIBERO-Plus, yielding larger independent gains than action distillation. Nevertheless, adding action distillation to representation distillation further improves LIBERO from 96.2% to 96.5% and LIBERO-Plus from 45.5% to 46.4% (Table 4). These results support the additional contribution of action distillation when combined with representation distillation.

Reviewer_uUPiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ID2P first trains a flow-based inverse dynamics model with current and future observations. A separately initialized policy is then trained using a mixture of expert and frozen-IDM velocity targets, together with action-expert representation distillation. Future images and the teacher are training-only. The paper proves gradient equivalence and target-variance reduction for ideal-IDM action distillation, and evaluates SmolVLA and pi0.5 on LIBERO, LIBERO-Plus, RoboTwin, and four Astribot manipulation tasks.

Critical strengths
  1. The strongest evidence is not the near-saturated LIBERO score but the robustness improvement and its controls. SmolVLA improves from 36.6% to 45.9% on LIBERO-Plus. The 60k-step baseline reaches 38.7% at the same aggregate epoch count, and the 80k-step baseline reaches 39.1% at comparable reported GPU hours (Table 8). These comparisons directly address the concern that teacher training merely adds compute. Improvements also extend to pi0.5 and real manipulation, whose average rises from 55.0% to 64.2%, although Pack Gift regresses.
  2. The ablations support complementary contributions from the two losses. On SmolVLA, action distillation alone gives 39.0% on LIBERO-Plus, representation distillation gives 43.3%, and their combination gives 45.9% (Table 4). The paper also tests teacher initialization, representation targets, and mixing endpoints. The ideal-teacher proofs are explicit about their assumption, and retaining the original policy architecture avoids the inference-time cost of generating future observations.
Critical weaknesses
  1. The evidence does not yet distinguish future-specific supervision from generic teacher denoising and representation regularization. Theorem 2 follows a conditional-variance decomposition: its reduction is proportional to residual target variance after conditioning on the future. If that future already determines the expert velocity, the residual is zero and the stated variance-reduction benefit vanishes. Thus, the theorem does not establish that a more informative future necessarily produces a larger reduction. A separately trained current-observation-only teacher is an important missing control. Moreover, the guarantee concerns the ideal action-distillation objective, not the imperfect learned teacher or the additional representation loss in Eq. (12). The observed control gains remain valid, but the proposed variance explanation is not empirically verified.
  2. he smaller gains and the shared-teacher comparison need stronger qualification. Independent-training uncertainty is not reported, while pi0.5 improves by only 0.5 points on LIBERO and SmolVLA by 0.7 points on the RoboTwin average. Thirty real trials per task also leave considerable uncertainty around modest differences. The shared-policy teacher collapses from 95.1%/36.6% to 37.1%/10.6% when both losses are used, but the submission does not sufficiently specify its optimization and privileged-input training to explain this failure. Since this is an ID2P-internal proxy rather than a full PFD reproduction, it supports the chosen implementation more directly than a broad superiority claim over that prior approach.
Questions for authors
  1. Does an equally trained, frozen teacher without future images retain the gains under the same action and representation distillation losses? Please distinguish the benefit of future access from that of adding a trained teacher. Available held-out teacher-error or target-variance measurements would help connect the theory to practice.
  2. Can the authors provide available independent-training results for the smaller gains and the joint-versus-representation-only ablation? Please also specify the shared teacher's losses, gradient stops, parameter updates, and future-input exposure so that its severe degradation can be interpreted.
  3. How much of the result depends on teacher quality or future offset beyond the fixed 16-step setting? A focused existing sensitivity result, or a narrower explanation acknowledging this dependence, would strengthen the claim without requiring a new large-scale benchmark.
9 comments
AuthorsOpenReview ↗

W1/Q1: Does an equally trained, frozen teacher without future images retain the gains under the same action and representation distillation losses?

A frozen teacher without future images can offer modest gains over the expert-only baseline in some settings, while a teacher with future images achieves higher average performance across the two benchmarks. The results are shown below.

TeacherFuture inputStudent settingα\alphaλRD\lambda_{\mathrm{RD}}LIBEROLIBERO-PlusAvg.
——Expert-only (baseline)0095.5±0.495.5^{\pm 0.4}39.9±2.639.9^{\pm 2.6}67.7±1.467.7^{\pm 1.4}
NFT✗Mixed0.5095.5±0.595.5^{\pm 0.5}40.5±3.540.5^{\pm 3.5}68.0±2.068.0^{\pm 2.0}
FT✓Mixed0.5095.9±0.595.9^{\pm 0.5}40.3±3.140.3^{\pm 3.1}68.1±1.468.1^{\pm 1.4}
NFT✗Expert-only + RD0294.8±0.594.8^{\pm 0.5}39.7±5.139.7^{\pm 5.1}67.2±2.767.2^{\pm 2.7}
FT✓Expert-only + RD0296.2±0.596.2^{\pm 0.5}45.5±6.145.5^{\pm 6.1}70.9±2.870.9^{\pm 2.8}
NFT✗Mixed + RD0.5295.4±0.695.4^{\pm 0.6}40.8±5.140.8^{\pm 5.1}68.1±2.868.1^{\pm 2.8}
FT✓Mixed + RD (Ours)0.5296.5±0.2\mathbf{96.5}^{\pm 0.2}46.4±4.6\mathbf{46.4}^{\pm 4.6}71.4±2.2\mathbf{71.4}^{\pm 2.2}

The above results are from Table 4 of our paper and report success rates with mean±std across three training seeds. The notation is defined as follows:

  • NFT (no future teacher): A policy trained with expert-only action supervision and no future image inputs.
  • FT (future teacher): An IDM trained with expert-only action supervision and future image inputs. NFT and FT differ only in that FT receives future images while NFT does not. All other settings, including the architecture and training recipe, are the same.
  • Expert-only: Uses expert-only action supervision (α=0\alpha=0, λRD=0\lambda_{\mathrm{RD}}=0).
  • Mixed: Uses both expert and teacher action supervision (α=0.5\alpha=0.5, λRD=0\lambda_{\mathrm{RD}}=0).
  • + RD: Adds representation distillation (λRD=2\lambda_{\mathrm{RD}}=2).
  • Expert-only + RD: Combines expert action supervision with representation distillation (α=0\alpha=0, λRD=2\lambda_{\mathrm{RD}}=2).
  • Mixed + RD: Combines expert and teacher action supervision with representation distillation (α=0.5\alpha=0.5, λRD=2\lambda_{\mathrm{RD}}=2).
  • Future input: Marks teacher access to future images.
  • Action distillation: Introduces action supervision from the teacher.
  • Representation distillation: Introduces representation supervision from the teacher.

Conclusion: Compared with excluding future observations from teacher inputs, including them substantially benefits representation distillation, while providing a modest benefit to action distillation. Future observations also benefit joint action and representation distillation. These results indicate that ID2P's gains primarily come from future information rather than current-only self-distillation.

Please refer to Sec. 5.4 (Q3) and App. B.3 of our paper for details.

AuthorsOpenReview ↗

W1: Missing empirical validation of variance reduction.

Directly estimating the target variances VPolicy\mathcal V_{\mathrm{Policy}} and VAD\mathcal V_{\mathrm{AD}} is difficult. Their computation requires sampling from the conditional distributions pτ(At∣Atτ,ot)p_\tau(A_t\mid A_t^\tau,o_t) and pτ(At,Ft∣Atτ,ot)p_\tau(A_t,F_t\mid A_t^\tau,o_t), respectively (Eqs. (3) and (8) of our paper). However, each observation oto_t in the robot demonstration datasets corresponds to only one expert action chunk AtA_t, making these conditional distributions difficult to construct.

To analyze the effect of action distillation on policy optimization, we have added a theory of gradient variance reduction (Theorem 3 of our paper).

Specifically, we train an expert-only SmolVLA policy on LIBERO and save checkpoints at different training steps. At each checkpoint, we compute the gradient variance for the following action velocity targets:

  • Expert-only: Expert action velocities (α=0\alpha=0, λRD=0\lambda_{\mathrm{RD}}=0).
  • Mixed w/ FT: A mixture of expert and frozen IDM action velocities (α=0.5\alpha=0.5, λRD=0\lambda_{\mathrm{RD}}=0).
  • Teacher-only w/ FT: Frozen IDM action velocities only (α=1\alpha=1, λRD=0\lambda_{\mathrm{RD}}=0).
  • Mixed w/ NFT: A mixture of expert and frozen policy teacher action velocities (α=0.5\alpha=0.5, λRD=0\lambda_{\mathrm{RD}}=0).
  • Teacher-only w/ NFT: Frozen policy teacher action velocities only (α=1\alpha=1, λRD=0\lambda_{\mathrm{RD}}=0).

FT (future teacher) receives future images, whereas NFT (no future teacher) does not.

The results ( Tr⁡Cov⁡^(g)\operatorname{Tr}\widehat{\operatorname{Cov}}(g) (×105\times 10^5) ) are shown below.

Training stepsExpert-onlyMixed w/ FTTeacher-only w/ FTMixed w/ NFTTeacher-only w/ NFT
0756.6752.2750.8751.7750.1
5k18.618.218.218.118.1
10k10.910.310.310.09.9
15k8.47.57.77.17.0
20k6.65.46.04.95.2
25k5.64.45.33.74.5
30k5.44.25.23.64.4

Conclusion: Both FT settings have lower gradient variance than Expert-only, providing empirical support for the variance nonincrease predicted by Theorem 3 with a learned IDM. NFT yields even lower gradient variance than FT at the same mixing weight.

Furthermore, we define the following metrics: Student-to-teacher error measures MSE between the student's and teacher's predicted action velocities. Teacher-to-expert error measures MSE between the teacher's predicted action velocities and expert action velocities.

Under Mixed, we observe lower student-to-teacher error with NFT than with FT (17.3×10−317.3\times 10^{-3} vs. 24.6×10−324.6\times 10^{-3}, Figure 6(e)), indicating that the student fits NFT more closely. However, NFT has higher teacher-to-expert error than FT (67.1×10−367.1\times 10^{-3} vs. 61.2×10−361.2\times 10^{-3}, Figure 6(f)), indicating a less accurate teacher.

Together, these results suggest that NFT enables the student to fit a less accurate teacher more stably, which does not guarantee higher success rates. With FT, the student fits a more accurate teacher while still having lower gradient variance than Expert-only.

We therefore compare the success rates of policies trained with each of the five action velocity targets above:

Action targetLIBEROΔ\DeltaLIBERO-PlusΔ\Delta
Expert-only (baseline)95.5±0.495.5^{\pm 0.4}0.039.9±2.639.9^{\pm 2.6}0.0
Mixed w/ FT95.9±0.5\mathbf{95.9}^{\pm 0.5}+0.440.3±3.140.3^{\pm 3.1}+0.4
Teacher-only w/ FT95.7±0.495.7^{\pm 0.4}+0.240.4±5.140.4^{\pm 5.1}+0.5
Mixed w/ NFT95.5±0.595.5^{\pm 0.5}0.040.5±3.540.5^{\pm 3.5}+0.6
Teacher-only w/ NFT95.5±0.795.5^{\pm 0.7}0.041.6±4.1\mathbf{41.6}^{\pm 4.1}+1.8

Δ\Delta denotes the success rate gain over the expert-only baseline.

Conclusion: Lower gradient variance does not necessarily lead to higher success rates. Although FT achieves less gradient variance reduction than NFT, both FT variants improve success rates over the expert-only baseline on both benchmarks, whereas NFT variants show almost no improvement on LIBERO.

Please refer to Sec. 5.4 (Q9) and App. B.9 of our paper for details.

In App. B.10 (Figure 9), we also train five SmolVLA policies on LIBERO from the same initialization, each using its own supervision for 30k steps. We measure mini-batch gradient variance and per-sample loss variance at each policy's own trained parameters, rather than at a shared set of parameters, using 128 fixed batches (global batch size 128). All four distillation variants have lower values than Expert-only in late training, showing that lower variance is also observed during policy optimization. However, lower variance does not necessarily lead to higher success rates.

AuthorsOpenReview ↗

W1: Interpretation of the variance reduction guarantee.

Indeed, a more informative future does not necessarily produce a larger variance reduction. Theorem 2 guarantees variance nonincrease, including the zero-residual case. For α∈(0,2)\alpha\in(0,2), the reduction is strict whenever the conditional residual variance is positive. In practice, it may be rare for current observations, future images, and noisy actions to uniquely determine the expert actions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers

We thank all reviewers for their time and constructive feedback. Your comments and suggestions have helped us improve the clarity and rigor of the manuscript. We appreciate your careful consideration of our work.

AuthorsOpenReview ↗

Summary for all reviewers and the Area Chairs

We thank all reviewers and the Area Chairs for their time and constructive feedback. We appreciate the reviewers' recognition of three strengths of ID2P:

  • Simple and efficient method. "The method is simple, requires no architecture changes or extra inference cost" (Reviewer oKQL). "retaining the original policy architecture avoids the inference-time cost of generating future observations." (Reviewer uUPi). "The final policy does not need the IDM or extra future prediction at test time." (Reviewer gwph).

  • Extensive experimental evaluation. "The evaluation is fairly extensive" and "The ablations are thorough" (Reviewer oKQL). "The ablations support complementary contributions from the two losses." (Reviewer uUPi).

  • Theoretical analysis. "The paper shows that an ideal IDM can reduce action-target variance without changing the expected training gradient." (Reviewer gwph).

Our revisions and responses clarify six main points:

  1. Independent and joint gains. With a future teacher (FT, i.e., IDM) on SmolVLA, action distillation (AD), representation distillation (RD), and AD+RD all improve mean success over the 30k expert-only baseline on both benchmarks. AD+RD performs best among these configurations (Table 4), with training curves also supporting their complementarity (Figure 6).

  2. FT versus NFT. FT outperforms a no future teacher (NFT, i.e., policy) on both benchmarks under representation and joint distillation, with comparable performance under action distillation alone (Table 4).

  3. Theory and empirical validation. Theorems 1–3 compare action distillation using an ideal FT with expert-only supervision (not FT versus NFT). With a learned FT, we observe lower gradient variance than expert-only supervision at the same student parameters (Figures 5 and 8(a)), providing empirical support for Theorem 3. FT-trained policies also achieve higher mean success than expert-only on LIBERO and LIBERO-Plus (Table 9).

  4. Teacher accuracy. FT has lower action and velocity MSE than NFT on both training and held-out data, demonstrating improved teacher accuracy with future observations (Figures 4 and 7).

  5. Multiple training seeds. We report success rates as mean±std over three training seeds (1000, 2000, and 3000) for the main SmolVLA and π0.5\pi_{0.5} comparisons and key ablations on LIBERO and LIBERO-Plus, covering 19 of 25 settings. Per-seed results are provided in App. C.

  6. Parameter sensitivity. All tested future offsets {8,16,32}\lbrace 8,16,32\rbrace and representation distillation weights λRD∈{1,2,5}\lambda_{\mathrm{RD}}\in\lbrace 1,2,5\rbrace outperform expert-only on both benchmarks, with small differences across the tested weights (App. B.11–B.12).

We hope this summary helps the reviewers and Area Chair assess the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.