- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses feedback(human designed rules) from a fixed VLA to keep adjusting scenes and tasks. It combines scene generation, execution checks, and feedback-guided changes into an evaluation framework(In fact, LLM guided env. generation is normal). The experiments evaluate several VLAs. Exp. how that guided changes find harder cases more effectively.
Critical strengths
- The pipeline is complete. It reduces the manual work to build tests(from results they find harder cases than random changes).
2.The evaluation covers a useful range of dimensions.Vision, language, deployment settings, long-horizon tasks, and safety, human-designed, offering several ways to test models.
Critical weaknesses
- The novelty is limited. Scene generation(LLM-driven,see robotwin series and so on) and feedback-driven testing(last year someone has lifted) have both been explored(everywhere in github). This feels more like an integration of existing ideas.
2.The added value is unclear. The method finds more failures, more test, more data, but what can actually learn from them is less clear. It seems less useful, isn't it? (in today's multiple, much more huge dataset); If you focus on model's limitation and improve it, why not RL? If your work valuable, show me.
3.The search baseline is weak. Human designed, how much the detailed process feedback actually helps unclear. Can you compare with the video method or llm-driven method or something to show your advantage?
4.IF you persist on your work, the exp. is not enough.How well these tests reflect model capabilities?(five root tasks is not enough);the failure labels do not necessarily explain the causes; different tasks or source model maybe have an another conclusion(your work is built around particular models).
Questions for authors
See Critical Weaknesses Please.
Dear authors,
thank you for providing thorough ablations and explanations. I keep my Weak acceptance decision for the paper.