- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper propose that spatiotemporal control references, such as keypoints, flows, orientations, and keyposes, can be viewed as partial specifications of the same temporal Cartesian action space. With the unified masked action space (MAP), they view the action generation as an inpainting problem. To support this formulation, they also introduce a progress-guided alignment module and a two-stage training strategy combining random and structured masks. Experiments on ManiSkill, LIBERO-10, and real-world robotic manipulation tasks demonstrate the potential of the proposed framework for controllable action generation.
Critical strengths
Unified and flexible framework for controllable action generation. It is a good idea to unify different motion representation methods from the perspective of inpainting, which allows for intervention in the generation of trajectories and guidance during reasoning. And corresponding experiments results shown in Figure 5 and so on prove its effectiveness.
Meaningful experiments with real-world demonstrations. The authors carried out extensive experiments. The real-world experiment demonstrated novel scenarios, including placing the cup at the specific position on the plate and controlling the pouring behavior.
Critical weaknesses
Insufficiently competitive and controlled baselines. The simulation experiments primarily compare against baselines from early works, including DP, DiT, and STC. Meanwhile, LIBERO has become a largely saturated benchmark, but the proposed method achieves only 64.67% on LIBERO-10, substantially below the performance of numerous recent advanced policies. The supplementary experiments report substantially higher LIBERO-10 performance with 1,000 demonstrations, but corresponding baseline results under the same setting are missing. Moreover, the unconditioned baselines do not receive the action references available to IP, making it difficult to isolate the contribution of masked action reconstruction.
Insufficient validation of progress-guided alignment and temporal controllability. The paper does not clearly explain how the human can give reasonable progress values, nor does it investigate the consequences of significant progress estimation errors. Such errors could lead to misaligned constraints and potentially affect task execution.
Questions for authors
- Could the authors provide comparisons with more competitive policies to better demonstrate the effectiveness of the proposed method?
- How robust is the proposed method to inaccurate progress assignments or significant progress estimation errors, and how do these errors affect constraint alignment and task success? And since user-specified keypoints or other references may provide cues about the corresponding task stages, why not allow the policy to automatically infer their alignment?
2 comments
Response to Reviewer H2iY (part2)
We continue our response below.
Question 2
We would like to address the two questions the reviewer raises as follows:
-
The reviewer raises concerns about inaccurate progress assignments and significant progress estimation errors, which could lead to misaligned constraints and potentially affect task execution. In order to demonstrate the robustness of IP to these errors , we conducted two additional experiments: perturbing reference-progress annotations and perturbing progress module estimated progress values.
-
First, we test inaccurate progress annotations with Gaussian noise while preserving spatial keypoints and their order on PokeCube, PickCube and LiftPegUpright in ManiSkill. Across noise levels from σ=0.02 to σ=0.10, the three tasks retain their clean-reference success rates with only marginal fluctuations, demonstrating strong robustness against temperal level reference error (See Supplementary §3.6 for more experimental details).
-
Second, we add independent Gaussian noise to the progress module's prediction at each action-chunk inference, leaving the references unchanged. Across noise levels from σ=0.02 to σ=0.10, the policy only shows minor degree of decrease in performance which is reasonable due to disturbed prediction of the current task stage which lead to noisy guidance signals. The complete task-wise results are showed in Supplementary §3.6.
-
-
Regarding the reviewer's question on why not allow the policy to automatically infer the temporal alignment of references:
-
First, we would like to argue that our work aims to enable both spatial and temporal control, so reference timing is part of the user’s intended specification. For example, when executing the task of putting sauce and food on the center of a plate, a robot can dispense the sauce first underneath the food or place the food first with sauce on top. Both sequences are feasible, but they produce different results according to the user’s preference. The spatial targets alone do not specify which result is desired. Explicit progress labels let users or high-level planners specify their temporal intents and ensure the task to be excuted as planned.
-
Second, with the rapid development of foundation models, LLMs or VLMs today can easily specify the temporal progress label of references. To demonstrate this, we use an Astra agent to estimate keypoints and task progress from the simulator’s observation, without using any demonstrations. With these noisy references, the IP achieves 78% on PickCube and 92% on LiftPegUpright without additional fine-tuning, providing evidence that our policy could generalize to independently generated, noisy, or imperfect references (See Supplementary §3.6 for more experimental details).
-
Other than these two main questions, regarding the reviewer's concern about missing baseline results with 1,000 LIBERO-10 demonstrations: our policy achieved 100% success versus DiT’s 94%, an improvement of 6 percentage points.(See Supplementary §3.4 for updated experiment results)
Thank you again for your time and feedback! We would love to hear your further insights on our comment.
Response to Reviewer H2iY (part1)
We would like to first thank the reviewer for your careful reading and insightful feedbacks. Please refer to the latest versions of our manuscript and supplementary material when reading our responses. All section, figure, and table references in our responses refer to these updated versions. And we would like to address your two main concerns as follows:
Question 1
We understand that the reviewer might be concerned that our policy does not achieve state-of-the-art performance on certain benchmarks and did not compete with large robotics models such as VLAs or WAMs, but we would like to argue from the following perspectives:
Our research objective is not to achieve state-of-the-art performance or substantially improve task success rates. Our method extends existing generative policies with a new capability: beyond simply completing tasks, they can now be precisely controlled in both space and time.
Not every study needs to build on the most advanced policy framework, particularly research on fundamental capabilities in robot learning. Many excellent studies this year use Diffusion Policy or small transformer as backbones and baselines, including Chronos Policy [3], STACK [4], and Do You Know Where Your Camera Is? [5],which received the ICRA 2026 Best Paper Award on Robot Learning.
The purpose of baselines is to provide fair comparisons that assess changes in task performance and the emergence of new capabilities introduced by a method. We rigorously match our method and the baselines in backbone architecture, parameter count, and training dataset. Our base models include U-Nets and Transformers of different sittings, representing the fundamental architectures underlying mainstream policy models. These experiments comprehensively validate the controllability our method brings to these classic, foundational visuomotor policies.
Our policy is designed as a unified low-level controller that can accept heterogeneous references from different high-level planners. The success rate of our policy is therefore dependant on and positively correlated with the quality of the references provided by high-level planners. A fair comparison would be to keep the high-level planner and its generated references fixed, replace the low-level controller in an existing hierarchical framework with our policy, and compare task success rates under matched training and evaluation conditions. We select 3D HAMSTER (IROS 2026) as a recent hierarchical baseline and replaced its low-level trajectory-conditioned 3DFA controller with our IP, keeping the high-level planner fixed. On six Colosseum tasks—close box, open drawer, close laptop lid, slide block to target, basketball in hoop, and reach and drag—our policy increased the average success rate from 61.33% to 86.00% under matched training and evaluation conditions. These results demonstrate that IP improves execution of the same high-level guidance, directly addressing the reviewer’s request for comparisons with more competitive policies (see Supplementary §3.9, Table 15)