- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposed TILDE and AutoDAgger where they use an off-the-shelf reward function to monitor progress of a student VLA, and when it gets stuck, uses a teacher VLA that was trained on large scale action data to guide the student VLA. The experiments focuses on distilling this large scale action data pertaining into the student VLA. Experiments cover real world tasks, cross architecture transfer, and comparison with HG-DAgger. It shows TILDE achieves better success rate compared to other distillation baselines and needs less teacher supervision compared to HG-Dagger.
Critical strengths
- Idea is very straightforward and figure 1 shows a motivation.
- The monitoring reward function is an off-the-shelf module so it doesn't need further fine-tuning or extra components.
- Shows good performance on real world tasks and shows scaling capabilities.
Critical weaknesses
- Distilling from a teacher is a very common idea (RPD, VLA-OPD) so the real contribution is the auto monitoring loop. But there is no ablation on the reward model choices. This also feels like a very limited contribution.
- The current formulation only works for quasi-static tasks and would be hard to implement on more dynamic tasks or tasks where progress is hard to judge (e.g., tasks that requires waiting where the robot does nothing while waiting).
- More in the questions section below
Questions for authors
- How does the quality of the monitoring reward function impact the distillation process? I imagine as long as it picks sufficient coverage from the student's distribution then it would suffice? What if we replace the entire procedure of AutoDAgger to be randomly sampling from the student's distribution or have the teacher randomly intervene?
- For figure 1(d), both yellow and red lines are generated by the same teacher VLA? the current color coding is a little confusing.
- The notation in equation 3 is confusing. After we sample the observations do we assign a score to each of the observation or only one score as a whole? How are the observations sampled? at random, equally spaced? why not just always evaluate the last K_t frames to get the latest progress?
- For the comparisons in Table 4, how much of the efficiency improvements comes from the fact that humans have longer reaction times / potentially suboptimal performance than teacher VLAs?
- One of the claim is "large-scale action pretraining can be transferred", but is there any experiment shows the action pretraining knowledge outside of the demonstrated task or other task variations? Right now the experiments are only strong enough to conclude that "large-scale action pretraining knowledge related to this task can be transferred".
- One of the motivation is to avoid large scale action data pertaining, but the teacher with the action data pertaining still outperforms the trained student by a considerable margin. Additionally, large scale action data pertaining typically uses SFT without the need of on-policy rollouts, so it can scale very well as compute grows. How does the actual training time compare? How much time would the proposed TILDE and AutoDAgger actually save at the cost of the performance degradation?