- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes MAIL-Bench, a benchmark for evaluating VLA models under visual interruption. It creates scenarios where different camera inputs are interrupted at multiple stages of successful reference trajectories.
The paper also proposes MINT, which trains VLA policies to handle missing visual inputs and temporarily provides predicted images when cameras fail. It uses optical-flow extrapolation when the wrist camera is unavailable and an action-conditioned world model when the third-person camera is unavailable.
Experiments on π0.5 and GR00T N1.5 show that MINT improves task success under camera interruption compared to the original models. The paper also conducts real-robot experiments to demonstrate the effectiveness of the proposed approach.
Critical strengths
- The paper identifies an important yet relatively underexplored scenario in robotics, where camera observations may become unavailable during task execution, and addresses this problem in a novel way.
- By providing both a benchmark and methodological baselines that can serve as foundations for future research, the paper has the potential to have a meaningful impact on the research community.
- The proposed method consistently improves performance over the baseline models across different settings, suggesting that the methodology is robust and stable.
Critical weaknesses
-
The evaluation is limited to wrist and third-person views, and there is insufficient validation for settings involving other types of camera views.
-
The benchmark only considers interruption points at 30%, 45%, and 60% of the task execution time. It would be valuable to investigate how the results change at other interruption points, particularly at extreme cases such as very early or very late interruptions.
-
There is a lack of comparison between MINT and prior work on learning with missing modalities. Currently, the main evidence for MINT appears to be that it improves the performance of the baseline models, but this alone does not sufficiently establish the necessity of MINT. In particular, I would like to see what happens when existing missing-modality recovery techniques are used to replace the missing camera inputs. If such approaches achieve comparable performance, the motivation and necessity of MINT would need to be reconsidered. Therefore, I consider this a critical omission.
I believe the authors should demonstrate the necessity of MINT through direct comparisons with relevant existing methods. If these comparisons show that MINT provides clear improvements over existing approaches, I would be willing to increase my score.
Questions for authors
-
What happens if the action-conditioned world model is used when the wrist camera is missing, and optical-flow extrapolation is used when the third-person camera is missing? I am curious why the authors chose different models for the two types of camera views.
-
I question whether generating the missing visual input is necessary in the first place. In particular, it is unclear how much computational overhead is introduced by the visual generation process and whether the resulting computational cost is practical for real-time robotic task execution. The authors should report the computational cost and latency of visual prediction and discuss whether it is feasible for real-time deployment.