- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SALT, a test-time adaptation method for VLA models facing unexpected visual disruptions. It uses the unexecuted actions from the previous action chunk as self-supervision to update the policy online. The method is evaluated on two VLA backbones across several LIBERO benchmarks, along with real-robot experiments.
Critical strengths
-
The main idea is simple and interesting. Reusing leftover actions for online adaptation avoids the need for additional demonstrations or external feedback.
-
The experiments show consistent improvements across SmolVLA and GR00T N1.7, with limited impact on nominal performance.
-
The ablations are fairly comprehensive and help explain the importance of temporal alignment and sequential updates.
Critical weaknesses
-
The comparison with BID is useful, but it does not fully answer this question. A simpler baseline would be to directly combine the previous leftover actions with the current prediction, without updating the model. It would be interesting to see whether SALT still offers a clear advantage. Also, the Self-training baseline supervises all H actions, while SALT only uses the L overlapping actions. Using the same supervision mask would make the comparison fairer.
-
While task progress improves, the actual success rates remain quite low. Based on Table 6, SALT succeeds in only 47 out of 240 trials, including just 1 out of 80 on the Stack task. Another concern is the computational cost. SALT requires synchronous gradient updates, and the reported inference latency increases considerably. It would help to report the actual execution delays and task completion time on the real robot.
-
The adaptation gate can also trigger during normal execution. Although average clean performance remains similar, some episodes might become worse after unnecessary adaptation. A more detailed analysis of these cases, particularly around grasping or object contact, would be helpful.
Questions for authors
-
How does SALT compare with simple action blending or temporal ensembling without parameter updates?
-
Can the authors provide more details on real-world failure cases and actual execution latency?
-
When the gate activates incorrectly, does adaptation ever turn a successful episode into a failure?
3 comments
Author Response [2/3]
Weakness 3) The adaptation gate can also trigger during normal execution. Although average clean performance remains similar, some episodes might become worse after unnecessary adaptation. A more detailed analysis of these cases, particularly around grasping or object contact, would be helpful.
We analyzed every clean episode in which the gate activated and compared it with the same episode under Frozen. Using contact information from MuJoCo [r1], we define contact as the gripper touching a task-relevant object and grasping as both finger pads touching the same object, following robosuite’s grasp check [r2].
| Backbone | Gate activated | Both succeed | Both fail | Success → failure | Failure → success |
|---|---|---|---|---|---|
| SmolVLA | 53/200 (26.5%) | 40 | 9 | 3 | 1 |
| GR00T N1.7 | 25/200 (12.5%) | 18 | 4 | 1 | 2 |
-
Changes in episode outcomes. Unnecessary activation changed the outcome in only 4 of 53 SmolVLA episodes and 3 of 25 GR00T episodes, and the changes went in both directions (3 worse and 1 better on SmolVLA, 1 worse and 2 better on GR00T).
-
How the degraded episodes failed. None of the degraded episodes failed at the moment of activation or on an object being held. On SmolVLA, SALT released the first object at the same replan as Frozen, and the episode diverged later, at the second object. On GR00T, the gate was activated at the first replan, before any contact. We observed no immediate grasp/contact failures at gate activation in these episodes.
-
Activation during grasping or contact. Activation rarely coincides with contact/grasp onset, about half as often as expected under random timing, and none of these episodes degraded. In the 18 SmolVLA episodes where the gate activated while an object was held, we observed no drops of that object.
Backbone At contact/grasp onset Random timing Degraded While holding an object (dropped) SmolVLA 7 (13.2%) 24.0% 0 18 (0) GR00T N1.7 2 (8.0%) 23.6% 0 0
[r1] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
[r2] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, et al. robosuite: A modular simulation framework and benchmark for robot learning. arXiv:2009.12293, 2020.
Question 1) How does SALT compare with simple action blending or temporal ensembling without parameter updates?
Following the reviewer’s suggestion, we have evaluated temporal ensembling, which blends overlapping action predictions without parameter updates. We have also evaluated direct execution of leftover actions. Both use the same gate as SALT. SALT outperformed temporal ensembling and leftover execution on both backbones.
| Method | SmolVLA (%) | GR00T (%) |
|---|---|---|
| Temporal ensembling | 48.8 | 62.9 |
| Leftover execution | 47.2 | 62.0 |
| SALT | 53.2 | 66.0 |
These results support the benefit of learning from leftovers beyond ensembling or directly executing action predictions. We have added these comparisons to Section 6.2 and Appendix G.3.
(Continued in the next comment)
Author Response [3/3]
Question 2) Can the authors provide more details on real-world failure cases and actual execution latency?
-
Real-world failure cases. Common real-world failures involved precise placement, such as placing the block inside the drawer or stacking it on another block, corresponding to the stages illustrated in
Figure 12. Under mesh camera occlusion, Frozen often failed to grasp the gray cat, whose visibility was reduced by the mesh (Figure 5), whereas SALT completed more trials (5/20 vs. 1/20,Table 7). -
Actual execution latency. For execution latency, we have analyzed real-robot logs. Including execution and communication, the median replan interval was 1.70 s on SALT update replans, compared with 1.24 s for Frozen. Across each method’s successful trials, median completion times were 62.3 s versus 60.7 s on Two Cats and 31.8 s versus 27.6 s on Drawer (SALT versus Frozen). To reduce this overhead, we have optimized the implementation with identical computation results, reducing update-replan compute latency from 529 ms to 124 ms for SmolVLA and from 735 ms to 226 ms for GR00T in simulation benchmarks on an H200. We have added the latency measurements and implementation optimizations to
Appendix F.1.
Question 3) When the gate activates incorrectly, does adaptation ever turn a successful episode into a failure?
Yes. Across 400 clean episodes, four Frozen successes became failures under SALT, while three failures became successes. We provide the detailed episode-level analysis, including activation around grasping and contact, in our response to Weakness 3.
In summary, SALT outperformed simpler baselines without parameter updates and self-training with the same supervision mask, supporting the benefit of learning from leftover targets. On the real robot, SALT more than doubled the binary success of Frozen, and we have reduced update-replan compute latency from 529–735 ms to 124–226 ms. Finally, unnecessary activation during normal execution rarely changed outcomes, with changes in both directions, and rarely coincided with contact/grasp onset, and none of those episodes degraded. We sincerely hope that our responses could properly address all your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.
Best regards,
Authors
Author Response [1/3]
Dear Reviewer oeHJ,
Thank you for your thoughtful comments and constructive suggestions! We appreciate your recognition of SALT’s simple idea of reusing leftover actions without additional demonstrations or external feedback, the consistent improvements across SmolVLA and GR00T N1.7 with limited impact on nominal performance, and the comprehensive ablations on temporal alignment and sequential updates. Your feedback has helped us strengthen our baseline comparisons, report real-robot execution latency and failure cases, and analyze unnecessary adaptation around grasping and object contact. We have also uploaded a revised manuscript, with all changes highlighted in blue. Below, we address each weakness (W1–W3) and question (Q1–Q3).
Thank you for these suggestions. With these baselines evaluated under the same gate, SALT still offers a clear advantage over both simpler baselines without parameter updates and self-training with the same supervision mask, showing that its gains come from learning from leftover targets. The table below summarizes the success rates.
For self-training, we have additionally matched the supervision mask to SALT’s by supervising only the L overlapping actions. We have added the comparisons with leftover execution and temporal ensembling, along with their implementation details, to
Section 6.2andAppendix G.3.We appreciate the reviewer’s concern about the low success rates, particularly on Stack. However, our results indicate that these rates partly reflect the difficulty of the long-horizon tasks, and that SALT still substantially improved binary success over the baselines. Even under clean conditions, Frozen achieved 80%, 50%, and 20% success on Two Cats, Drawer, and Stack, respectively (
Appendix E). Under corruption, SALT succeeded in 47 out of 240 trials, more than doubling the 21 of Frozen and exceeding the 32 of BID (Table 7). To give a clearer picture, we have added the binary success rates alongside task progress (Section 5.3). The actual execution delays and task completion time on the real robot are detailed in our response toQuestion 2.(Continued in the next comment)