- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new benchmark and a VLA model. The authors address the occlusion problem in the robot manipulation task. Given the LIBERO benchmark, the authors newly curate the data to have occlusion cases. The name of this curated dataset is LIBERO-occ.
As a next step, the authors newly train a VLA model. Especially, to deal with the occlusion problem at the exocentric viewpoint, the model "imagines" the top-view image. Technically speaking, the imagination is not something new and the authors clearly said so. Nonetheless, introducing the top-view imagination makes sense to me.
Critical strengths
S-1 Good motivation
It is quite important to deal with occlusion cases for the robot manipulation task. I pretty much like the motivation of this work. Moreover, I do agree that existing benchmarks, especially LIBERO, LIBERO-pro, LIBERO-plus, etc, did not deal with the occlusion cases. However, in the real-world demo, this is quite serious issues.
S-2 Good dataset curation
The curated dataset is simple, but it is good enough to demonstrate the results of existing VLA models that do not consider this occlusion problem during training.
Critical weaknesses
W-1 Unfair comparison?
It is a trick issue. Usually in LIBERO-plus and LIBERO-pro that also extends the LIBERO benchmark with the specific purposes, lots of recent VLA models trained with the original LIBERO and evaluate the models with the LIBERO-plus and LIBERO-pro benchmarks. So, it can be understood as "zero-shot" evaluation. This is a common practice in this field. Here, the problem comes. In the table 2, I wonder whether the authors also newly train or newly finetune the related studies (UniVLA, OpenVLA, ... PI0.5) with LIBERO-occ that the authors newly curated. I deeply check the paper, but the authors did not clearly provide the information about this. It is highly important from my side.
If the proposed method VIM is the only method finetuned with LIBERO-occ, it is totally unfair comparisons.
Moreover, following the conventions in this field, I wonder the performances of VIM that is purely trained with LIBERO benchmark. I truly understand that the LIBERO benchmark does not provide top-view images, but it is quite difficult to judge whether the imagination learning from LIBERO-occ is truly effective or not. At this moment, I cannot verify this issue through the paper.
W-2 Lack of information about generalization test.
The section 4.2.4 is about the generalization test. The authors claim that they evaluate the proposed model in different benchmark having occlusion (I guess). But it is hard to follow what it means. From Line 345 to Line 356, I cannot understand any single sentences at all. What existing datasets did you use for the generalization test? LIBERO-plus? Then, how did you sample evaluation episodes having occlusion cases? I also read the supplementary material, but this part is totally out of my understanding.
Since the authors propose a new method so the authors should evaluate the proposed method at least more than 2 benchmarks in my opinion. In that perspective, this highly confusing and hidden descriptions about the generalization test are not qualified as a ICLR submission.
I hope authors to revise this section in manuscript and the section in the supp entirely. This is quite an important issue as a reviewer.
Questions for authors
Overall, I enjoyed reading this paper. I also previously had a same idea and motivation of this task and the authors exactly address the problem through this submission. I have no doubts on the novelty of this paper. Honestly, the proposed method is highly naive but both the proposed dataset and the proposed method are well aligned with the motivation. I pretty much like this consistency.
Please refer to the weakness section and I hope to see the rebuttal soon.
No rebuttal?
Do the authors plan to upload the rebuttal?