- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VINE, a hierarchical learning framework that uses both success and failure demonstrations to increase its robustness. Following standard hierarchical planning setup, this system is composed of a high-level reasoning System 2 followed by a low-level continuous controller System 1. System 2 uses a pretrained visual language model to do a Monte Carlo Tree Search over a 2D scene graph. It optimizes the plan (defined as a series of subgoals and their transitions) using a first-exit value function paired with asymmetric expectile loss to avoid overestimation. System 1 uses flow matching policy that is trained only on successful demonstrations to execute the selected subgoals. The authors evaluated this model on simulations (MuJoCo, SIMPLER) and real manipulators and observed better OOD generalization compared to non-hierarchical baselines. The authors claim that failure demonstration improves test-time evaluation of the subgoal feasibilities, which then leads to this better performance.
Note: the reviewer is familiar with VLAs but not SMDP.
Critical strengths
- Using failure data to improve first-exit reach-avoid probability and use the value for high-level plan search is innovative.
- In Section 4.3, the low-level System 1 is trained with perturbed data so that it can tolerate upstream errors. This design follows standard practice.
- Ablation study validates the effectiveness of using failure data.
Critical weaknesses
-
In Section 3.2, if I understand correctly, the authors make an assumption that all failures are terminal states. This can make the formulation of success v.s. failure into a simple binary value, but could also lead to the framework being unable to handle error recoveries. It is recommended for the authors to discuss how the model handles recoverable situations.
-
In Section E, by using 2D bounding boxes to form the scene graph, the model might ignore 3D geometry and 3D dynamics, which are key to manipulators.
-
The proof in Section 3.2 and the appendix needs clarification.
-
The text descriptions at Line 196 do not match Fig.1. For example, where are the option generator, the learned world model, and the failure-aware value?
Questions for authors
-
In hierarchical VLAs, the low-level policy typically uses a different (usually simpler) architecture than the high-level one, so as to ensure higher inference frequency. However, in the proposed system, both are based on the same Vision-language backbone. Does this mean that, at inference time, they take turns executing? From Section 4.1, by saying “latency is a one-time planning cost”, do the authors mean System 2 is only run once?
-
How is the flow-matching controller aligned with the high-level planner? When is the high-level termination predictor queried? What if termination happens while the controller is executing?
-
Line 157: if the agent enters an avoid set, but the option does not terminate until later at a safe node, will the return <-> indicator relation still hold? The option does not terminate instantly in this case.
-
Line 794: if you use termination-on-arrival, at t=0 the equation reduces to , which means it checks the termination at the start state, right? Could the authors clarify?
3 comments
Official Comment by Authors
Dear Reviewer HpsV,
Thank you again for your careful reading and constructive feedback. We would like to follow up on our responses above, which address the definition of terminal failures and the failure set, the corrected option kernel in App. B.1, the correspondence between Section 4.1 and Fig. 1, and the scheduling and termination of the two systems.
As the discussion period draws to a close, we would be grateful to hear whether these responses have addressed your concerns. If any questions remain, we would be happy to provide further clarification.
Best regards, The Authors
Thanks to the authors for the clarifications. Though I incline towards keeping the current score due to that defining the failure set as terminating timeout label (i.e. does not reach in the time window) in the offline dataset rather than a physical state set reduces the SMDP to simplified binary prediction on truncated trajectories. Secondly, system 2 takes 27.4s (Tab.9), this makes the system difficult to resolve initial errors like those from occlusions (unanswered in Q1).
Response to Reviewer HpsV
W1. Terminal failures and recoverable situations
Failures are terminal in our formulation, in the sense of a task outcome within the prescribed time window, not physical irreversibility. A transition is labeled a failure when it does not reach its target in time, and the episode ends there; the dataset was not collected to include failure-and-recovery trajectories. A situation that needs a different plan is handled before execution, by ranking the corresponding branch low. Runtime motor recovery is outside the scope of this work. We have added this discussion to Section 3.2 and App. K.
W2. 2D scene graphs omit 3D geometry and dynamics
The scene graph is an abstraction for high-level decisions (object identities and coarse spatial relations), not the representation used for control. System 1 generates actions from the current wrist and third-person images, conditioned on the selected subgoal and target graph, so 3D geometry and contact enter through the images. The value also accounts for physical details that the graph omits when those details produced failures in the training data, as in the clock example of Section 5.4. Details that did not cause failures in the data can still affect plan ranking, which App. K lists as a limitation.
W3 / Q3. Entering the avoid set inside an option and terminating later at a safe node
In our formulation, the failure set is absorbing because a failure label ends the episode. A transition is labeled a failure when it does not reach its target within the prescribed time window, and no trajectory in our data continues after a failure label. A transient execution error inside an option, after which the target is still reached within the time window, is therefore not a failure-set event. Under this definition the scenario described by the reviewer does not arise, and the return–indicator identity holds for the decision-epoch process of App. B.2.
Absorption follows from episode termination, not from physical irreversibility. We have replaced the phrase "irrecoverable manipulation state" in Section 3.2 with this definition and defined both sets at decision epochs, so that Section 3.2 and App. B.2 describe the same process.
W4. Text at Line 196 does not match Fig. 1
We thank the reviewer for pointing this out. We have revised the caption of Fig. 1 and Section 4.1 so that each term is mapped to the component shown in the figure:
Q1. Shared backbone and inference scheduling
Yes, System 2 runs only once, and the two systems do not alternate in the default configuration. System 2 builds the tree before the first action (27.4 ± 6.0 s at w = 3, Table 9), and only System 1 runs afterwards, at about 0.14 s per action chunk. Both systems use the same frozen language block with different LoRA adapters and heads. System 1 is fast enough because it generates a chunk of actions per forward pass and does not decode text. The 20 Hz figure is the rate at which actions are executed, not the rate of model inference. We have stated this in Section 4.1.
Q2. Alignment of the controller with the planner, and termination
The planner hands System 1 the selected edge and the predicted target graph of that edge; these condition both the action expert and the done expert. The termination predictor is System 1's done expert, not a part of System 2.
The done probability is predicted in the same forward pass that generates an action chunk and is held while the chunk is executed. When it exceeds the threshold, and a minimum number of steps has elapsed for the subgoal, the controller advances to the next edge, and later chunks are generated under that edge and its target graph. If a subgoal completes while a chunk is being executed, the remaining actions of that chunk are still executed, and the switch takes effect from the next chunk. Task success is checked by the environment before every control action. We have aligned Algorithm 2 with this description.
Q4. Termination-on-arrival at Line 794
We thank the reviewer for checking this expression. With the row-stochastic convention P(s, s′) = P_πe(s′ | s) and B = diag(β_e), the corrected expression is
PoeS=∑t≥0(P(I−B))tPB
An initiated option first takes a transition and then checks termination at the state it reaches, so the t = 0 term is P·B: one transition followed by a check at the arrival state. This corrects the analytical expression in App. B.1 and changes neither the evaluated controller nor any reported number.