- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VISTA uses a coding meta-agent to improve an embodied agent’s harness while keeping the underlying models fixed. It combines active simulator diagnosis, code revision, and a DAG of previous harnesses. Experiments show large improvements on LIBERO-Pro development instances and smaller gains on RoboCasa365.
Critical strengths
-
Active diagnosis is a useful idea for failures that recorded rollouts cannot explain well. Figure 6 gives a concrete example: correcting the lift reference and release behavior improves actual execution.
-
The gains cover all 40 LIBERO-Pro tasks in Table A2. This supports the usefulness of harness adaptation beyond a few selected examples, although these are development results.
Critical weaknesses
-
Appendix C.1 reports results on the same 16-cell development matrices used for harness selection. Fresh episodes on these instances do not establish generalization to held-out seeds. This is my main concern about the near-perfect LIBERO-Pro results.
-
The RoboCasa improvement is unclear. Table 2 gives an initial Seen score of 18.4%, while Table A3 reports 59/256, or 23.0%. The latter implies only about a 2.7-point gain to 25.8%, rather than the claimed 7.8 points. This materially changes the evidence from the second benchmark.
Questions for authors
-
Evaluation clarification: Do the final frozen harnesses retain their gains on seeds excluded from all development and selection?
-
Results clarification: Which RoboCasa scores are correct, and how should Tables 2/A3 and Section 4.2 be reconciled?
-
Budget clarification: How does the 110% token cap in Section 4.3 allow 94.2M passive versus 42.2M active tokens in round 1? What are the total simulator rollout counts, including diagnostic interactions, for both methods?