- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents COPE, a framework for in-context adaption of VLAs. It aims to address prior work limitations that do not work well on complex, multi-step tasks due to improper action precondition resolution or incorrect subtask ordering. The proposed method first organizes relevant demonstration fragments for the subtasks on the global level, then adapt the organized sequence to the execution. In the first part, it uses a symbolic analyzer to validate the preconditions and the results of the demonstration fragment. In the latter part, a closed-loop grounding mechanism guides the trajectory composer to generate valid actions that can be connected together. The authors performed experiments on EB-ALFRED and EB-Habitat against 4 baseline methods. The idea is innovative, but the evaluations need further work.
Critical strengths
-
The idea in Section 4.1 to treat long horizon demonstrations as procedural fragments followed by analysis and ordering is innovative.
-
Enforcing subtask validity using symbolic analysis avoids the typical hallucinations in pure VLMs.
-
The ablation results in Table.4 supports the effectiveness of the proposed fragment reuse and symbolic analysis mechanisms.
-
The paper is well very clear.
Critical weaknesses
-
In Section 5.1, it is not necessary to compare against standard VLM planning. Having them in the table is ok for reference purposes, but they do not suggest that the proposed method is advantageous because those standard VLMs have less human-tuned reasoning prior in them. Secondly, when comparing against the other training-based/free methods, the authors should fix the same VLM backbone. Otherwise, it would be hard to tell if the difference is due to the VLM or due to the proposed innovations.
-
The authors should provide a time complexity analysis regarding the steps described in Section 4.1. The fragment selection, for example, does it have to be re-run as the task executes? And for how often?
-
Eq.2 contains an unexplained complex function . Is there a bijective mapping from the demonstration task to the target task?
-
In Table.4, it is recommended to also ablate the visual verifier, which can isolate the contribution of the visual grounding.
Questions for authors
-
Line 148: how is the rule set R defined? Does it come from human annotation or simply just simulator state flags?
-
Line 207: is there an underlying assumption that a relevant fragment can always be found? What happens if one cannot be found?
-
Line 210: it is recommended to use another variable to denote demonstration id. could lead to confusions since i and j are usually used pairwise.
-
Line 244: how does the VLM guarantee the correctness of the fragment ordering?
-
Could the authors give examples of the failure cases and explain them? Are they due to mismatched demonstration, or any other issues.