- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors proposed ID2P for training flow-based robot policies, which contains an inverse dynamics model (IDM) using current and future observations. The frozen IDM then teaches a policy through action-velocity supervision and hidden-feature matching, while the policy only needs current observations at test time. The paper reports higher average success rates on LIBERO, LIBERO-Plus, RoboTwin, and real robot tasks.
Critical strengths
- Future observations are used only during training. The final policy does not need the IDM or extra future prediction at test time.
- The paper tests both SmolVLA and pi0.5. It includes simulation, real robot tasks, component ablations, and a comparison with longer baseline training.
- The paper shows that an ideal IDM can reduce action-target variance without changing the expected training gradient.
Critical weaknesses
-
The two distillation losses are not explained well enough. Table 4 shows that representation-only distillation performs better than action-only distillation on LIBERO-Plus. The paper does not show why this happens or what each loss changes in the policy's actions. The ablations tell us which settings score higher, but not why.
-
The role of future images is not fully separated from the role of an extra trained teacher. The shared-teacher ablation and training-budget comparison are useful. However, they do not compare ID2P with an independently trained teacher that sees only current observations. It remains unclear how much of the gain comes specifically from future information.
-
The policy analysis is too limited. Appendix Figure 4 shows example robot executions, but not direct comparisons with the baseline from the same starting state. There is little analysis of changed actions or failures. For example, Pack Gift drops from 80.0% to 73.3% in Table 3. The paper does not explain why ID2P helps some tasks but hurts others.
Questions for authors
- Could you plot the measured action-target variance and gradient variance during training? Please compare expert-only, IDM-only, and mixed supervision. Does lower variance lead to better results?
- Could you plot IDM prediction and policy action information over training?
- Table 4 shows a larger gain from representation distillation than from action distillation on LIBERO-Plus. Could you show training curves for both losses and their combination? When do the two signals help each other or conflict?
- Please visualize their predicted actions and full executed trajectories. Include successes and failures, especially Pack Gift, and report uncertainty for the real robot results.
13 comments
Q1: Could you plot the measured action-target variance and gradient variance during training? Does lower variance lead to better results?
Directly estimating the action-target variances and is difficult. Their computation requires sampling from the conditional distributions and , respectively (Eqs. (3) and (8) of our paper). However, each observation in the robot demonstration datasets corresponds to only one expert action chunk , making these conditional distributions difficult to construct. Therefore, we did not measure action-target variance.
To measure gradient variance, we train an expert-only SmolVLA policy on LIBERO and save checkpoints at different training steps. At each checkpoint, we compute the gradient variance for the following action velocity targets:
- Expert-only: Expert action velocities (, ).
- Mixed w/ FT: A mixture of expert and frozen IDM action velocities (, ).
- Teacher-only w/ FT: Frozen IDM action velocities only (, ).
- Mixed w/ NFT: A mixture of expert and frozen policy teacher action velocities (, ).
- Teacher-only w/ NFT: Frozen policy teacher action velocities only (, ).
FT (future teacher) receives future images, whereas NFT (no future teacher) does not.
The following table reports gradient variance () from Figure 8(a) of our paper.
| Training steps | Expert-only | Mixed w/ FT | Teacher-only w/ FT | Mixed w/ NFT | Teacher-only w/ NFT |
|---|---|---|---|---|---|
| 0 | 756.6 | 752.2 | 750.8 | 751.7 | 750.1 |
| 5k | 18.6 | 18.2 | 18.2 | 18.1 | 18.1 |
| 10k | 10.9 | 10.3 | 10.3 | 10.0 | 9.9 |
| 15k | 8.4 | 7.5 | 7.7 | 7.1 | 7.0 |
| 20k | 6.6 | 5.4 | 6.0 | 4.9 | 5.2 |
| 25k | 5.6 | 4.4 | 5.3 | 3.7 | 4.5 |
| 30k | 5.4 | 4.2 | 5.2 | 3.6 | 4.4 |
To explain this observation, we have added a theorem characterizing the gradient variance reduction property of action distillation using a frozen ideal IDM relative to expert-only supervision (Theorem 3 of our paper).
Conclusion: Both FT settings have lower gradient variance than Expert-only, providing empirical support for the variance nonincrease predicted by Theorem 3 with a learned IDM. NFT yields even lower gradient variance than FT at the same mixing weight.
Furthermore, we define the following metrics: Student-to-teacher error measures MSE between the student's and teacher's predicted action velocities. Teacher-to-expert error measures MSE between the teacher's predicted action velocities and expert action velocities.
Under Mixed, we observe lower student-to-teacher error with NFT than with FT ( vs. , Figure 6(e)), indicating that the student fits NFT more closely. However, NFT has higher teacher-to-expert error than FT ( vs. , Figure 6(f)), indicating a less accurate teacher.
Together, these results suggest that NFT enables the student to fit a less accurate teacher more stably, which does not guarantee higher success rates. With FT, the student fits a more accurate teacher while still having lower gradient variance than Expert-only.
We therefore compare the success rates of policies trained with each of the five action velocity targets above:
| Action target | LIBERO | LIBERO-Plus | ||
|---|---|---|---|---|
| Expert-only (baseline) | 0.0 | 0.0 | ||
| Mixed w/ FT | +0.4 | +0.4 | ||
| Teacher-only w/ FT | +0.2 | +0.5 | ||
| Mixed w/ NFT | 0.0 | +0.6 | ||
| Teacher-only w/ NFT | 0.0 | +1.8 |
denotes the success rate gain over the expert-only baseline.
Conclusion: Lower gradient variance does not necessarily lead to higher success rates. Although FT achieves less gradient variance reduction than NFT, both FT variants improve success rates over the expert-only baseline on both benchmarks, whereas NFT variants show almost no improvement on LIBERO.
Please refer to Sec. 5.4 (Q9) and App. B.9 of our paper for details.
In App. B.10 (Figure 9), we also train five SmolVLA policies on LIBERO from the same initialization, each using its own supervision for 30k steps. We measure mini-batch gradient variance and per-sample loss variance at each policy's own trained parameters, rather than at a shared set of parameters, using 128 fixed batches (global batch size 128). All four distillation variants have lower values than Expert-only in late training, showing that lower variance is also observed during policy optimization. However, lower variance does not necessarily lead to higher success rates.
W1/Q3: Analysis of action distillation, representation distillation, and joint distillation.
(1) Could you show training curves for the action distillation loss, the representation distillation loss, and their combination?
Figure 6 shows the training curves for the following ID2P configurations on SmolVLA trained on LIBERO, represented by the blue curves labeled Trained IDM:
- Action distillation alone: Mixed (, ), solid lines.
- Representation distillation alone: Expert + RD (, ), dashed lines.
- Joint distillation: Mixed + RD (, ), dash-dot lines.
Figures 6(a), 6(b), and 6(c) report action distillation loss, representation distillation loss, and total loss, respectively.
(2) When do the two signals help each other or conflict?
With the trained IDM, we observe that action distillation and representation distillation help each other, with no clear conflicts between the two signals.
Mixed + RD has lower action distillation loss and student-to-teacher error than Mixed (Figure 6(a,e)), and slightly lower representation distillation loss than Expert + RD (Figure 6(b)). These reductions provide important empirical evidence that action distillation helps representation distillation better align student and teacher representations, while representation distillation helps action distillation better fit the teacher's action velocities. Mixed + RD also achieves the highest success rates (Table 4), further validating the complementary nature of the two supervision signals.
(3) Why does representation distillation alone (Expert + RD) perform better than action distillation alone (Mixed)?
We hypothesize that two factors explain this difference.
-
Action distillation directly changes the action velocity target, so inaccurate teacher predictions can directly affect action learning. Representation distillation does not modify action velocity targets and instead provides softer guidance through hidden states, influencing action learning indirectly.
-
Representation distillation supervises all action expert layers, allowing future information to guide representations at multiple depths. This may provide a richer and more diverse learning signal than supervising the final action velocity alone.
Student-to-expert error measures MSE between the student's predicted action velocities and expert action velocity targets. Mixed has higher student-to-expert error than Expert + RD ( vs. ), providing a clue to its lower success rates. However, Expert-only has lower student-to-expert error than Mixed ( vs. ), yet Mixed performs better on both benchmarks (Table 4). Therefore, even a large student-to-expert error is only an indirect indicator of success rates and cannot perfectly predict their ranking.
W2: Compare ID2P with an independently trained teacher that sees only current observations.
The results are shown below.
The above results are from Table 4 of our paper and report success rates with mean±std across three training seeds. The notation is defined as follows:
Conclusion: Compared with excluding future observations from teacher inputs, including them substantially benefits representation distillation, while providing a modest benefit to action distillation. Future observations also benefit joint action and representation distillation. These results indicate that ID2P's gains primarily come from future information rather than current-only self-distillation.
Please refer to Sec. 5.4 (Q3) and App. B.3 of our paper for details.