- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Positional Blind Spots (PBS), a targeted-data collection framework for improving the robustness of VLAs to task-irrelevant distractors. PBS follows a two-stage pipeline: (i) discretize the state-space into cells for which to place the task-irrelevant distractor, roll-out the policy many times in each cell, average performance in each cell, then select the top K cells based on a LLR score comparing failure rate relative to the rest of the workspace. The K cells represent the PBS, where the authors then (ii) collect demos in the PBS, fine-tune with LoRA, and re-evaluate, showing improved performance. They test on different VLA backbones and conduct this study in LIBERO and VLA-Arena.
Critical strengths
To me, the main strength of the paper is the controlled study characterizing VLA policy sensitivity to a distractor's position in the workspace. By holding all other variables constant, the authors demonstrate that VLAs are particularly sensitive to specific locations in the environment, rather than the presence of the distractor at any random region. This matters because it shows that average success rates can hide localized failure modes, which is useful for policy evaluation.
Critical weaknesses
There are two critical weakness that I can find.
-
Impractical: performing this exhaustive grid-sweep is prohibitively expensive in the real-world. For example, in their experiments, they require 980 policy executions to determine the seven PBS regions, which is incredibly time-consuming for evaluation. Moreover, while simulation provides a test-bed and allows for reproducible experiments, there are no real-world evaluation, so its difficult to assess how the method performs in practice.
-
Missing baselines: a strong baseline that the authors do not compare to is doing un-targeted data collection with distractors placed at random, with an equal demo budget This is much more aligned with how practitioners would perform fine-tuning in real life, and is a critical baseline to compare to. As it stands now, there is no observed benefit purely due to the search process of PBS relative to a baseline without it.
Questions for authors
- Can you add the baseline method I mentioned above: collect the same number of demos as the PBS method, but with the distractor placed uniformly at random within the environment, and fine-tune with identical LoRA parameters? Again, this is how practitioners would perform teleoperation for fine-tuning, and without clear benefits over this baseline, its hard to argue for the expensive data collection scheme to identify PBS in the first place.