- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a hierarchical VLA framework to improve instruction following. A Qwen-VL planner produces structured manipulation plans augmented with continuous segmentation and depth tokens, which condition a π0.5-based executor. Auxiliary reconstruction encourages the executor to retain the plan’s geometric information, while a progress predictor triggers replanning. The paper reports improvements on LIBERO-Plus, RoboTwin 2.0, target-replacement evaluation, and a small real-world benchmark.
Critical strengths
- The paper addresses an important limitation of current VLAs: successful manipulation does not necessarily imply sensitivity to language instructions. The target-replacement evaluation is particularly relevant, with a reported improvement from 55% to 74% over π0.5.
- The proposed combination is technically plausible and reports improvements across several settings, including LIBERO-Plus, bimanual manipulation, and real-world distribution shifts. Training the executor with generated rather than ground-truth plans is also a sensible design choice.
Critical weaknesses
- The contribution of the plan interface is unclear. The system adds a planner, geometric supervision, reconstruction, and replanning, but Table 4 does not separate their effects well. Reconstruction also does not guarantee that the action head uses the plan.
- Important training and evaluation details are missing. The paper needs to explain the supervision sources, data splits, and training budgets. The target-replacement protocol and Table 1’s aggregation are also unclear.
Questions for authors
- How are structured plans and segmentation/depth labels generated for simulation and real-world data? Please specify the datasets, quantities, splits, and training budgets for both stages, including whether replacement-target instructions appear in planner supervision.
- How does the full model compare with the same planner and executor using a text-only plan interface, while retaining reconstruction and replanning? Results on target replacement, together with interventions that change or remove plan information while holding observations fixed, would help establish which components actually drive instruction following.
- Please provide the target-replacement task/instruction pairs, success criteria, rollout counts, and seeds. Please also explain Table 1’s aggregation: using the task counts published in LIBERO-Plus Appendix C.2, Table 7, I obtain approximately 81.96 for π0 and 89.85 for the proposed method, rather than 82.94 and 89.72. Were different task subsets or aggregation rules used?
2 comments
Thanks for the clarifications. Some of my concerns have been addressed, but I still think the text-only baseline is worth testing. Since the executor receives updated visual observations, an unchanged text plan does not necessarily prevent it from adapting. Without this comparison, it is hard to tell how much the geometric interface actually contributes.I will keep my rating unchanged as 2 (Weak Rejection).
Response to Reviewer aL7Q
Response to Question 1: In simulation, ground-truth masks are fetched directly from the simulator API, and structured plans are annotated manually. For real-world data, masks and depth labels are annotated using DepthAnything, GroundingDINO and Segment Anything. Simulation and real-world models are trained independently on their respective datasets with no mixed training.
Response to Question 2: We considered the proposed text-only baseline while keeping reconstruction and replanning unchanged. However, this comparison is not meaningful. When execution stalls at a sub-task, only object positions and visual tokens change, while the text plan content stays the same. Replanning cannot yield useful updates in this case. We will also clarify our novelty as the structured geometric interface that binds planner semantics to geometric masks for valid replanning, and remove the unsupported “mathematically enforces” statement in Section 3.3.
Response to Question 3: For target-replacement, an example instruction pair is “put the bowl on the plate” and “put the wine bottle on the plate”. We follow the official LIBERO success criteria: an episode is successful if all goal predicates are satisfied, verified by the simulator internal state. We evaluate 10 tasks with 10 rollouts per task. Regarding Table 1 aggregation, this was indeed our mistake. After checking our code, we found that we computed the simple arithmetic mean. We will fix this, and this issue does not affect our core results.