- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper trains small students (Long-CLIP or SigLIP2) offline on successful LIBERO rollouts of a large VLA teacher (π0.5 or UniVLA). Rules and a VLM judge find the phases where student failures happen, and doubling the loss on those phases raises success in all 14 configurations.
Critical strengths
-
The weighting gain is small but consistent, and it is reported honestly. Success rises in all 14 paired configurations (125 episodes rescued, 68 broken), with an exact McNemar p-value for each.
-
The recipe is cheap, well described, and works with two teachers and two student backbones. Unweighted students are within 3 points of their teacher in 7 of 8 Spatial and Object cells, at 4.1 to 6.2 USD per student.
Critical weaknesses
-
The new idea is to let student failures choose which phases get more weight. But the diagnosis picks grasping and placing in 12 of 14 configurations, where a fixed grasping-and-placing rule would give the same weights. So the 14 gains mostly show that weighting these phases helps, not that the diagnosis is needed, which is the main reason for my rating.
-
The diagnosis differs from the fixed rule only on Spatial, and the test there is small. On LIBERO-10, weighting transporting and operating also helps (166 vs. 163, 176 vs. 173), so part of the gain may not depend on phase choice.
-
Apart from the diagnosis, the weighting is close to earlier work. Keyframe-focused imitation (Wen et al. [1]) and StaKe (Xu et al. [2]) shape training around action changepoints and gripper events, which seem to mark grasping and placing. The replay labels beat gripper-command labels in a model-based audit (94% vs. 74%), but it is not shown that this changes student success.
-
Training on teacher rollouts is a known form of behaviour cloning, so the value the teacher adds needs a direct check. The paper does not train the same student on the LIBERO human demonstrations that both teachers used for fine-tuning. The Octo, SmolVLA, and VLA-OS rows in Table 1 use other models and their own protocols, so they do not settle this.
-
The evidence is narrow for the claimed generality. All results are on LIBERO with one student per suite, and unweighted students already reach 91.0 to 99.5% on three of the four suites. The labelling and diagnosis also need simulator replay and recorded simulator state, so use on real robots, the setting the conclusion targets, is not shown.
-
The speed gain of the SigLIP2 student, which comes closest to the teachers, is not clear. Its 172 ms is per forward pass and it runs at every step, while π0.5 (431 ms) outputs action chunks and may run less often. Time per executed action, and a quantised-teacher baseline, would help judge the efficiency claim.
References:
[1] Chuan Wen et al. Keyframe-focused visual imitation learning. ICML, 2021.
[2] Yuan Xu et al. Improving vision-language-action model fine-tuning with structured stage and keyframe supervision. arXiv preprint arXiv:2606.26801, 2026.
Questions for authors
Questions:
- How does the same student do when trained with the same recipe on the LIBERO human demonstrations? A clear gap in favour of teacher data would support the distillation claim.
- Which initial states did the teachers use to collect data, and do they overlap with the 200 evaluation episodes? Large overlap could make the students look closer to their teachers and could lower my score.
- Of the rescued episodes, how many had failed in the selected phases, as the paper's explanation would predict? For the π0.5 Long-CLIP student on Goal, at least 3 of the 16 rescued episodes had failed in other phases (Table 6, Figure 4).
Suggestions:
- Report the gain on held-out initial states, since the same 200 episodes are used to pick the phases and to measure the gain.
- The diagnosis check would be stronger with more than one human rater and a second checker from a different model family than the resolver.
2 comments
Thanks for the extra experiments. The human-demo control answers my first question. I’m keeping my score at 2 because weighting still shows no consistent benefit on held-out states. Across the four retrained pairs, on their own training states, 62 failures became successes and 70 successes became failures. Gripper-command labels also worked just as well as replay labels.
Given the large overlap, I put more weight on the disjoint-state results, where the base student trails its teacher by 8-10 points. The student trained during rebuttal draws level largely because of task 8, where the teacher scored only 26/50 despite succeeding on at least 77% of the task-8 placements during data collection. I’d like to see that comparison repeated under MuJoCo 3.8.0, where the teacher reaches 471/500 overall. Since latency only drops from 20 to 16 ms per action, lower memory use seems to be the clearest practical gain.
Response to Reviewer L83b
Thank you for the thorough review and the concrete suggestions. We added the human-demonstration control (new Tables R1 and R4), an evaluation on initial states that are not in the training data, with two seeds (new Section 5), the gain on held-out states (new Table R2), the two labellings compared by student success (new Table R3), time per executed action (Old Table 3), and a second human annotator (new Appendix I.5). Teacher rollouts are clearly better than human demonstrations, and our student is level with its teacher on initial states it never trained on. Changes made for you are in green, and changes that several reviewers asked for are in purple.
W1. The new idea is to let student failures choose … the main reason for my rating.
A fixed rule already assumes the result of the diagnosis: that grasping and placing are the phases to weight is what the diagnosis found, with 184 of the 217 student failures in these two phases, more than five times all other phases combined (Old Table 6). These two phases take only a quarter of the training frames (new Table R7). Without the diagnosis we could not tell which phases to weight, or by how much. We added this description to new Appendix I.7.
W2. The diagnosis differs from the fixed rule only on Spatial … may not depend on phase choice.
These differences are 3 to 4 of 200 episodes. With repeated seeds on LIBERO-10 one configuration varies by up to 14 episodes (new Appendix I.2), so they are within seed-to-seed variation, and Section 4.5 now states this.
W3. Apart from the diagnosis, the weighting is close to earlier work … changes student success.
Thank you for pointing to these two papers. Section 2 now discusses Keyframe-Focused and StaKe as the closest work and states the differences: our labels come from physical events recovered by replay, accepted on 94% of audited moments against 74% for gripper-command labels, and the phases to weight are chosen from the student's failures. new Table R3 compares student success under the two labellings.
W4. Training on teacher rollouts is a known form of behaviour cloning … do not settle this.
This check is now in the paper; please see Q1.
W5. The evidence is narrow for the claimed generality … is not shown.
The Limitations in Section 6 state this scope: LIBERO in simulation, with labelling that needs simulator state. Within it, new Section 5 adds a harder test on initial states that are not in the training data, with two seeds. The student itself needs no simulator state; real robots are future work and not a claim of this paper.
W6. The speed gain of the SigLIP2 student … judge the efficiency claim.
We would like to clarify that our contribution is a distillation framework and not a speed record: it shows that a 189M student can do the work of a 3.3B teacher on these tasks, so the large model is often not needed. Old Table 3 now reports time per executed action; timed on one GPU, our student takes 16 ms per action versus 20 ms for the teacher (new Appendix I.6). A quantised teacher was not evaluated.
Q1. How does the same student do … would support the distillation claim.
The same student reaches 383 of 500 with the LIBERO human demonstrations and 456 with teacher rollouts (new Table R4), and new Table R1 shows the same gap for the base student. This clear gap supports the distillation claim.
Q2. Which initial states did the teachers use … could lower my score.
We added an evaluation in which the teacher's success trajectories are collected on placements that we sampled ourselves with the LIBERO code, and the student is tested on the official LIBERO initial states, which are not in the training data (new Section 5). After the refinements described in new Appendix I.6, our 189M student reaches 456 and 453 of 500 with two seeds, versus 451 for its 3.3B teacher (new Table R4). The 200-episode protocol of Old Table 1 uses the benchmark's own initial states for both (new Appendix I.1). The student matches its teacher on initial states it never trained on.
Q3. Of the rescued episodes, how many had failed in the selected phases …?
In the retrained pairs, 53 of the 62 rescued episodes had failed in grasping or placing, in line with the share of failures in these two phases. new Section 5 reports this.
Suggestion 1. Report the gain on held-out initial states … to measure the gain.
This is new Table R2 in new Appendix I.2, with two seeds.
Suggestion 2. The diagnosis check would be stronger … than the resolver.
We added a second human annotator; the agreement is in new Appendix I.5. Section 4.5 notes that the model checker is from the same family as the resolver.