acceptodds
Under review as a conference paper at ICLR 2027

Robotic Spatiotemporal Editing: A Unified Interface for Controllable Action Generation via Masked Action Reconstruction

Abstract

Generative policies can synthesize robot behaviors, yet controlling and editing them through diverse spatiotemporal references remains challenging. In the real world, human intent requires robot behaviors that satisfy constraints on where, when, and how actions are executed. Our key insight is that spatiotemporal control references, such as keypoints, flows, orientations, and keyposes, can be viewed as partial specifications of the same temporal Cartesian action space. We introduce the MAP (Masked Action Space) interface to unify these action references and formulate controllable manipulation as an action inpainting problem. Our Inpainting Policy (IP) reconstructs missing action dimensions of the end-effector, combining progress-guided alignment with multi-mask training to support unseen action-constraint combinations. We further adopt a training-free RePaint variant that iteratively inpaints the action during denoising to reduce control error. On ManiSkill and LIBERO-10, our policy achieves average task success rates of 92.67% and 64.67%, respectively.Real-world experiments demonstrate spatiotemporal control with average positional and rotational errors of 44.3 mm and 7°, respectively, showing that our policy can use diverse references to guide executable robot behaviors.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Controllable Action Generation, Robot Leaning, Human Robot Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_H2iYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose that spatiotemporal control references, such as keypoints, flows, orientations, and keyposes, can be viewed as partial specifications of the same temporal Cartesian action space. With the unified masked action space (MAP), they view the action generation as an inpainting problem. To support this formulation, they also introduce a progress-guided alignment module and a two-stage training strategy combining random and structured masks. Experiments on ManiSkill, LIBERO-10, and real-world robotic manipulation tasks demonstrate the potential of the proposed framework for controllable action generation.

Critical strengths

Unified and flexible framework for controllable action generation. It is a good idea to unify different motion representation methods from the perspective of inpainting, which allows for intervention in the generation of trajectories and guidance during reasoning. And corresponding experiments results shown in Figure 5 and so on prove its effectiveness.

Meaningful experiments with real-world demonstrations. The authors carried out extensive experiments. The real-world experiment demonstrated novel scenarios, including placing the cup at the specific position on the plate and controlling the pouring behavior.

Critical weaknesses

Insufficiently competitive and controlled baselines. The simulation experiments primarily compare against baselines from early works, including DP, DiT, and STC. Meanwhile, LIBERO has become a largely saturated benchmark, but the proposed method achieves only 64.67% on LIBERO-10, substantially below the performance of numerous recent advanced policies. The supplementary experiments report substantially higher LIBERO-10 performance with 1,000 demonstrations, but corresponding baseline results under the same setting are missing. Moreover, the unconditioned baselines do not receive the action references available to IP, making it difficult to isolate the contribution of masked action reconstruction.

Insufficient validation of progress-guided alignment and temporal controllability. The paper does not clearly explain how the human can give reasonable progress values, nor does it investigate the consequences of significant progress estimation errors. Such errors could lead to misaligned constraints and potentially affect task execution.

Questions for authors
  1. Could the authors provide comparisons with more competitive policies to better demonstrate the effectiveness of the proposed method?
  2. How robust is the proposed method to inaccurate progress assignments or significant progress estimation errors, and how do these errors affect constraint alignment and task success? And since user-specified keypoints or other references may provide cues about the corresponding task stages, why not allow the policy to automatically infer their alignment?
2 comments
AuthorsOpenReview ↗

Response to Reviewer H2iY (part1)

We would like to first thank the reviewer for your careful reading and insightful feedbacks. Please refer to the latest versions of our manuscript and supplementary material when reading our responses. All section, figure, and table references in our responses refer to these updated versions. And we would like to address your two main concerns as follows:

Question 1

We understand that the reviewer might be concerned that our policy does not achieve state-of-the-art performance on certain benchmarks and did not compete with large robotics models such as VLAs or WAMs, but we would like to argue from the following perspectives:

  1. Our research objective is not to achieve state-of-the-art performance or substantially improve task success rates. Our method extends existing generative policies with a new capability: beyond simply completing tasks, they can now be precisely controlled in both space and time.

  2. Not every study needs to build on the most advanced policy framework, particularly research on fundamental capabilities in robot learning. Many excellent studies this year use Diffusion Policy or small transformer as backbones and baselines, including Chronos Policy [3], STACK [4], and Do You Know Where Your Camera Is? [5],which received the ICRA 2026 Best Paper Award on Robot Learning.

  3. The purpose of baselines is to provide fair comparisons that assess changes in task performance and the emergence of new capabilities introduced by a method. We rigorously match our method and the baselines in backbone architecture, parameter count, and training dataset. Our base models include U-Nets and Transformers of different sittings, representing the fundamental architectures underlying mainstream policy models. These experiments comprehensively validate the controllability our method brings to these classic, foundational visuomotor policies.

  4. Our policy is designed as a unified low-level controller that can accept heterogeneous references from different high-level planners. The success rate of our policy is therefore dependant on and positively correlated with the quality of the references provided by high-level planners. A fair comparison would be to keep the high-level planner and its generated references fixed, replace the low-level controller in an existing hierarchical framework with our policy, and compare task success rates under matched training and evaluation conditions. We select 3D HAMSTER (IROS 2026) as a recent hierarchical baseline and replaced its low-level trajectory-conditioned 3DFA controller with our IP, keeping the high-level planner fixed. On six Colosseum tasks—close box, open drawer, close laptop lid, slide block to target, basketball in hoop, and reach and drag—our policy increased the average success rate from 61.33% to 86.00% under matched training and evaluation conditions. These results demonstrate that IP improves execution of the same high-level guidance, directly addressing the reviewer’s request for comparisons with more competitive policies (see Supplementary §3.9, Table 15)

AuthorsOpenReview ↗

Response to Reviewer H2iY (part2)

We continue our response below.

Question 2

We would like to address the two questions the reviewer raises as follows:

  1. The reviewer raises concerns about inaccurate progress assignments and significant progress estimation errors, which could lead to misaligned constraints and potentially affect task execution. In order to demonstrate the robustness of IP to these errors , we conducted two additional experiments: perturbing reference-progress annotations and perturbing progress module estimated progress values.

    • First, we test inaccurate progress annotations with Gaussian noise while preserving spatial keypoints and their order on PokeCube, PickCube and LiftPegUpright in ManiSkill. Across noise levels from σ=0.02 to σ=0.10, the three tasks retain their clean-reference success rates with only marginal fluctuations, demonstrating strong robustness against temperal level reference error (See Supplementary §3.6 for more experimental details).

    • Second, we add independent Gaussian noise to the progress module's prediction at each action-chunk inference, leaving the references unchanged. Across noise levels from σ=0.02 to σ=0.10, the policy only shows minor degree of decrease in performance which is reasonable due to disturbed prediction of the current task stage which lead to noisy guidance signals. The complete task-wise results are showed in Supplementary §3.6.

  2. Regarding the reviewer's question on why not allow the policy to automatically infer the temporal alignment of references:

    • First, we would like to argue that our work aims to enable both spatial and temporal control, so reference timing is part of the user’s intended specification. For example, when executing the task of putting sauce and food on the center of a plate, a robot can dispense the sauce first underneath the food or place the food first with sauce on top. Both sequences are feasible, but they produce different results according to the user’s preference. The spatial targets alone do not specify which result is desired. Explicit progress labels let users or high-level planners specify their temporal intents and ensure the task to be excuted as planned.

    • Second, with the rapid development of foundation models, LLMs or VLMs today can easily specify the temporal progress label of references. To demonstrate this, we use an Astra agent to estimate keypoints and task progress from the simulator’s observation, without using any demonstrations. With these noisy references, the IP achieves 78% on PickCube and 92% on LiftPegUpright without additional fine-tuning, providing evidence that our policy could generalize to independently generated, noisy, or imperfect references (See Supplementary §3.6 for more experimental details).

Other than these two main questions, regarding the reviewer's concern about missing baseline results with 1,000 LIBERO-10 demonstrations: our policy achieved 100% success versus DiT’s 94%, an improvement of 6 percentage points.(See Supplementary §3.4 for updated experiment results)

Thank you again for your time and feedback! We would love to hear your further insights on our comment.

Reviewer_RKi3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies controllable robot action generation through a unified interface for heterogeneous spatiotemporal references. The proposed Masked Action Space (MAP) represents positions, trajectories, orientations, poses, and gripper states as partially specified entries in a shared Cartesian action representation. An Inpainting Policy reconstructs the remaining action components, using progress-guided reference alignment and training with multiple mask families. Training examples can be constructed by masking existing demonstrations, and the paper also considers RePaint-style reference replacement during inference.

The evaluation covers three ManiSkill tasks, ten LIBERO tasks, and two real-world manipulation tasks. It includes comparisons across reference types, a held-out combination of known constraint types, and supplementary studies of several implementation choices. The results provide encouraging evidence for a unified editing interface, although the breadth and conclusiveness of the validation remain concerns.

Critical strengths
  1. A clear and useful formulation of heterogeneous control inputs, with a practical way to reuse demonstrations.

I find the central idea appealing: different control references can be treated as partial specifications of the same underlying action trajectory. This gives the paper a clear problem formulation and a coherent interface for combining positional, orientational, and gripper-related constraints. The representation also makes specified and unspecified action components explicit, which is useful for both human interpretation and policy conditioning.

A consequential practical advantage is that training examples can be generated by masking existing demonstrations, without collecting separate control-reference annotations. This makes the proposed formulation relatively accessible to existing imitation-learning pipelines and increases its potential usefulness.

  1. Concrete evidence that the interface can support combinations of references beyond those explicitly used during training.

The paper provides experimental support for the idea. In particular, the held-out Mix0 composition combines known reference types and achieves an 80.56% success rate without additional fine-tuning (main paper, Figure 5; supplementary Section 2.4). This is encouraging evidence that a shared representation can support compositional control.

The supplementary material also strengthens the implementation evidence. For example, the progress-source comparison uses the same policy checkpoint and evaluation data and reports an increase in success rate from 67% to 83% when learned progress replaces rollout-timestep alignment (supplementary Table 10). Together with the simulation and real-robot demonstrations, these results make the proposed direction interesting and worthy of further investigation.

Critical weaknesses
  1. The evaluation does not yet establish how reliably the proposed implementation realizes the breadth of the editing interface.

Although the paper evaluates 15 tasks overall, the evidence for general editing capability covers a narrower range of conditions. The explicitly held-out composition is Mix0, which retains dense 2D positional guidance throughout the trajectory. The physical experiments involve two task-specific policies. These results support feasibility, but leave substantial uncertainty about performance across other combinations, reference placements, and levels of sparsity or imprecision.

The reported failure with a single late-stage reference is particularly relevant: task success falls below 30%, and the supplement explains that such a reference may remain outside the local conditioning window until late in execution (main Section 4.3; supplementary Section 2.6). This suggests that the practical effectiveness of the interface depends strongly on how references are distributed across task stages. The operating conditions under which the method provides reliable editing are not yet characterized sufficiently.

Furthermore, the control-error metric matches each reference to its nearest trajectory point, regardless of the requested progress or ordering. As the supplement acknowledges, it measures reference proximity rather than temporal alignment (supplementary Section 2.3). The progress-alignment ablation is useful, but it does not directly establish that constraints are satisfied at the intended stage or in the intended order. These limitations make it difficult to assess the full spatiotemporal-control claim.

  1. The comparative experiments only partially isolate the advantage of the proposed learning and conditioning design.

The comparisons with reference-free DP and DiT provide useful context, but improvements over these policies combine the effect of receiving additional guidance with the effect of the proposed method. I appreciate that STC is a relevant reference-conditioned baseline and that the paper reports matched backbone and training settings. Nevertheless, the comparison with this particular flattened-conditioning implementation leaves uncertainty about which aspects of masked reconstruction, reference encoding, and training account for the observed advantage.

The supplement provides several useful component studies, including controlled progress-source and window-length comparisons. However, some other studies do not isolate the component being examined: the MAP encoder comparison varies training budgets, batch sizes, and evaluation counts, while the progress-module configuration comparison changes architecture, observation history, and checkpoint selection together (supplementary Tables 6 and 8). These results give limited evidence for the individual design choices.

The real-world comparisons also require clarification. The supplementary aggregate IP results pool multiple reference configurations and checkpoints and explicitly do not describe one fixed policy configuration. More concerningly, supplementary Tables 4–5 state that the pouring DP success labels remain unverified. This limits the weight that can be placed on that particular comparison. Overall, the experimental evidence leaves me less confident in the demonstrated advantage and robustness of the current implementation than in the conceptual value of the interface.

Questions for authors

The following questions concern evidence that could materially change my assessment.

  1. Information and protocol matching. Do STC and IP receive the same visible reference values, visibility masks, and progress-aligned reference windows, under comparable mask distributions, training schedules, and checkpoint-selection procedures? Please also clarify which checkpoint is used for IP-RePaint: main Section 4.2 describes applying it to baseline DP/DiT policies, whereas the supplement describes it as a variant of IP.

  2. Evidence for the implementation choices. Beyond the controlled progress-source and window-length studies, what evidence isolates the benefit of the MAP encoding and reconstruction design under matched training and evaluation budgets? Can the authors distinguish the contribution of these choices from differences in conditioning representation and training configuration?

  3. Compositional and temporal control. How does performance vary across held-out combinations beyond Mix0 and across reference sparsity and placement? Can the existing rollouts be evaluated with a progress-aligned or order-sensitive constraint metric, so that successful task completion can be distinguished from satisfying the requested constraints at the intended stages?

  4. Real-world evaluation. Please specify the reference configurations, policy checkpoints, trial counts, success criteria, and uncertainty for the main real-world comparisons. How do these settings relate to the pooled supplementary results, and can the pouring DP outcomes marked as unverified in supplementary Tables 4–5 be verified?

2 comments

We thank the reviewer for your careful reading and insightful feedbacks. We have updated the supplementary material.

Question 1

The reviewer raises an important question regarding the matching of conditioning information and experimental protocols between STC and IP.

As described in Supplementary Sections 2.1, STC and IP share the same backbone, training schedules and configuration, datasets, random seeds, and checkpoint-selection procedures. They also receive equivalent visible reference values under equivalent visibility masks and mask distributions. The key difference lies in how this information is represented: STC flattens the full spatiotemporal references into conditioning features, whereas IP explicitly encodes visibility masks and uses progress-aligned local MAP windows.

RePaint is a training-free inpainting procedure that replaces known action entries with noise-level-matched reference values at each denoising step. It uses exactly the same checkpoint as DP/DiT, differing only in this inference-time operation. The wording describing RePaint as an IP variant was ambiguous in the previous version. We have clarified this description in the revised Supplementary Section 2.1.

Question 2

To better isolate the contributions of Progress-guide MAP window and reconstruction design, we conducted additional ablation experiments under matched training and evaluation protocols.

  1. Progress-guide MAP Window. By holding the reconstruction objective fixed to mask-weighted MSE, we compare MAP window with STC-style flattened conditioning. MAP conditioning improves Top-5 success rate from 17.11% to 80.56% (+63.45 percentage points). This result supports the effectiveness of the overall MAP window in representing spatiotemporal references. It preserves explicit visibility information and temporal structure. Progress alignment further helps the policy use references at the appropriate task stages.

  2. Reconstruction Design. By keeping progress-guide MAP window fixed and varying only the reconstruction objective, we compare standard MSE with mask-weighted MSE. The weighted objective improves Top-3 success rate from 76.67% to 80.56% (+3.89 percentage points). This demonstrates the positive contribution of mask-weighted reconstruction. Mask-weighted MSE encourages the model to focus more on predicting the masked regions.

Detailed results are provided in Supplementary 3.10.

We further investigate the effects of MAP Window Encoder and feature dimension under matched training and evaluation protocols.

  1. MAP Window Encoder. By keeping the feature dimension fixed at 32, we compare 1DConv and 2DConv under the same training and evaluation configuration. The current Top-5 success rates are 83.4% for 1DConv and 89.4 % for 2DConv(+6.0 percentage points). This result imply that 2DConv processes MAP conditions as image-like representations.This may help preserve spatiotemporal information in the features.

  2. Feature Dimension. We further compare 2DConv encoders with feature dimensions of 32, 64, and 128 while keeping the remaining settings fixed. Their Top-3 success rates are 89.4%, 79.8%, and 90.2%, respectively. This reflects the positive influence of a large dimension, which provides greater capacity to retain detailed MAP information.

Above experimental details are provided in Supplementary 3.1.

We also examine the effect of the training schedule under a matched evaluation protocol.

Joint vs. Two-stage Training. Under the matched checkpoint-evaluation protocol, joint and two-stage training achieve Top-3 success rates of 57.33% and 64.67%, respectively. This result indicates that the two-stage strategy remains effective, supporting the main paper's approach of structured joint training. Additional details are included in Supplementary 3.4.

Finally, we conduct additional ablation experiments to isolate the effects of different progress module configurations.

Timestep vs Progress. Our comparative experiments show that task progress is more suitable than timesteps as the temporal coordinate for controllable action generation. In Supplementary §3.5, Table 9, we compare physical rollout timestep and observation-based progress as different temporal guidance source. With the same PickCube checkpoint and 100 episodes, it improves SR from 67% to 83% and CE from 0.03983 to 0.02912 over normalized rollout time (Supplementary §3.5, Table 9).

Progress module Ablation. We separately evaluate observation-history length, frame-sampling gap, model capacity, and checkpoint-selection criteria while keeping the downstream IP checkpoint fixed. Results show that longer observation histories and larger progress module architectures do not consistently improve downstream performance, while checkpoint selection can have a significant impact. Detailed experimental results and configurations are provided in Supplementary Section 3.3.

Question 3

First, the reviewer raises a question about different mix type, sparsity and placement.We evaluated two additional held-out compositions, Mix1 and Mix2, alongside Mix0 using the same 125k checkpoint and 50 LIBERO-10 trajectories. Their success rates are 83.0%, 83.0%, and 85.0%, respectively, with no statistically significant differences.

Reference Sparsity and Placement. Mix1 increases sparsity by reducing 2D-flow references ratio from 100%(Mix0) to 50% of timesteps,while Mix2 keeps the same 2D-flow ratio as Mix1 and replaces one full-pose and three position anchors with two orientation anchors. Performance remains comparable across these configurations, suggesting that variations in reference sparsity have limited impact on performance. For all mask patterns, including Mix0, reference placements are randomly sampled along the temporal axis, ensuring that evaluation covers diverse temporal placements rather than being restricted to fixed reference locations.

Second, the reviewer asks whether the existing rollouts can be evaluated with a progress-aligned or order-sensitive constraint metric, so that successful task completion can be distinguished from satisfying the requested constraints at the intended stages. We address the progress-aligned evaluation aspect with a new metric and report it alongside task success.

We introduce BiCE (Bidimensional Control Error), which adds temporal progress error at the spatially closest action to spatial error at the temporally closest action:

|q_{\arg\min_t D_\ell(t)}-p_\ell| +D_\ell\!\left(\arg\min_t|q_t-p_\ell|\right) \right].$$ Here, K counts reference constraints, p_{ℓ} is specified progress, and qₜ is estimated execution progress. D_{ℓ}(t) measures action-reference distance on visible coordinates and both terms have equal weight. Across three ManiSkill tasks and five reference configurations, we evaluate STC, IP, and IP-RePaint with 50 episodes per group (2,250 total). Both IP and IP-RePaint achieve significantly lower mean CE and BiCE than STC. These results support stronger spatial and progress-aware control, with stronger reference enforcement from IP-RePaint (see Supplementary §3.7, Table 13). ## Question 4 ### Q4. Real-world Evaluation The main-text results in Section 4.3 (Figure 6) focus on controlling specific spatial target regions and pouring targets, whereas the pooled supplementary results aggregate evaluations with varying reference types, numbers of reference points, and checkpoints. For cup arrangement, we evaluate **2D and 3D keypoint keypose references** using IP checkpoints at 15k training steps; for pouring, we evaluate **orientation** references using the 15k step checkpoint. The corresponding DP baselines use 100k and 50k checkpoints, respectively. We use **different numbers of reference points, 4, 8 and 16 to represent the different sparsity level.** Task success criteria is assessed by whether the cup is put in the plate, or whether the robot pours water into the tube. For the main-text comparisons, the estimated trial counts are Cup IP: 28/30 (93.3%), Cup DP: 12/20 (60.0%), Pouring IP: 29/30 (approximately 96.7%), and Pouring DP: 15/20 (75.0%), the corresponding 95% Wilson confidence intervals are [78.7%, 98.2%], [38.7%, 78.1%], [83.3%, 99.4%], and [53.1%, 88.8%], respectively. We will clarify the evaluation counts and success criteria in the revised supplement. The “unverified” label for pouring DP was retained from an earlier draft; we have since verified the reported success rate and will correct it.
Reviewer_BiKVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified interface for controllable robot action generation through masked action reconstruction. The proposed Masked Action Space (MAP) represents different spatiotemporal references, including keypoints, trajectories, orientations, and poses, as partially specified Cartesian action sequences. An Inpainting Policy reconstructs the missing actions using diffusion, with progress-guided alignment and multi-mask training to handle different reference types and their combinations. The authors also explore RePaint-based inference for stronger constraint enforcement.

Experiments on ManiSkill, LIBERO-10, and real-world manipulation tasks demonstrate reference-conditioned action generation, with improvements in task success and constraint-following accuracy.

Critical strengths
  1. Interesting and intuitive problem formulation. I like the idea of treating different spatiotemporal references as partial specifications of the same action space. It provides a simple and potentially useful interface for controlling and editing robot behaviors.

  2. Unified control across different reference types. The masked action representation supports keypoints, trajectories, orientations, and poses within a single policy, avoiding the need to design separate conditioning mechanisms for each reference type.

  3. Generalization through multi-mask training. Training with structured and random masks allows the policy to handle different constraint combinations. The results on unseen mask combinations are encouraging and suggest potential for more flexible action generation.

  4. Real-world validation and practical applicability. The paper evaluates the method on both simulation benchmarks and real-robot tasks, demonstrating user-guided control of manipulation behaviors. The real-world results provide useful evidence of the framework's potential beyond simulation.

Critical weaknesses
  1. Practical problem formulation. I like the idea of a unified control interface, but the paper assumes that different references can be converted into Cartesian action constraints. In practice, translating keypoints, object trajectories, or human instructions into executable robot actions is not straightforward. I would like to better understand how this framework handles that gap.

  2. Methodological novelty. Masked reconstruction and diffusion inpainting are already established techniques. The proposed combination is interesting, but the paper could better clarify what is fundamentally new compared with existing action-inpainting and constrained generation methods.

  3. Generalization to realistic references. Most simulation references are constructed from ground-truth demonstrations, where the constraints are naturally consistent with successful actions. It would be more convincing to evaluate references generated independently by humans or other models, especially when they are noisy or imperfect.

  4. Questionable progress-based constraint alignment. I find the use of normalized task progress to specify when an action should happen insufficiently motivated. Task progress is not always clearly defined, especially when different subtasks take varying amounts of time or execution deviates from the demonstration. It is unclear whether a constraint assigned to a particular progress value corresponds to the intended manipulation stage. The Control Error metric also ignores the specified progress when measuring constraint satisfaction, leaving this temporal alignment largely unverified.

  5. Understanding constraint adherence. The paper shows that stronger adherence to reference constraints does not always translate into better task execution. It would be useful to further analyze when enforcing constraints helps, when it interferes with the original policy, and how the framework balances the two objectives.

Questions for authors
  1. How practical is the proposed interface when converting keypoints, object trajectories, or human instructions into valid Cartesian action constraints is itself a nontrivial problem? How does the method handle references that are not already aligned with robot actions?

  2. Given that masked reconstruction and diffusion inpainting are well-established, what is the main methodological novelty beyond combining existing techniques into a shared action representation?

  3. Most simulation references are extracted from successful demonstrations, making them naturally consistent with the target actions. How well does the method generalize to independently generated, noisy, or imperfect references? Can the authors provide evidence beyond reconstructing partially observed demonstrations?

  4. Why is normalized task progress a reliable way to specify when an action should occur? Progress does not necessarily correspond to meaningful manipulation events, especially when execution deviates from demonstrations. Also, why does the Control Error metric ignore the intended progress and search for the closest point anywhere in the trajectory?

  5. How does the method balance constraint satisfaction with task success? Stronger constraint enforcement can reduce task performance, so what prevents the model from satisfying individual references while producing an unsuccessful trajectory?

I find the overall intuition promising but remain unconvinced by the practical formulation and validation. I am open to raising my rating if the authors convincingly address these concerns, particularly the realism of the references and the evaluation of temporal controllability.

5 comments
AuthorsOpenReview ↗

Response to Reviewer BiKV (part1)

We first thank the reviewer for your careful reading and insightful feedbacks. Please refer to the latest versions of our manuscript and supplementary material when reading our responses. All section, figure, and table references in our responses refer to these updated versions. And we would like to adress your five main concerns as follows:

Question 1

This is a very good question and directly relates to our ongoing project (see Figure 1). We are not restricted to a particular input method, as our goal is to develop a unified interface at the policy level. There are two main pathways for providing control references to the MAP interface:

  1. Model-generated inputs within a hierarchical framework. Control inputs can come from spatiotemporal signals generated by language-conditioned models. For example, a video world model can generate videos or subgoal images based on user prompts. With the image space aligned with the robot’s action space, optical flow estimation can be used to extract 2D/3D end-effector motion trajectories from generated 2D/3D videos, while subgoal images can provide x–y spatial information and corresponding temporal coordinates for the overall action. Object trajectories can be extracted from third-person camera observations using optical flow–based tracking tools.
  2. Human-provided inputs. This first requires aligning the third-person camera’s coordinate frame with the coordinate frame used for robot end-effector control. Users can then provide inputs in two ways. The first is through a desktop GUI, where users draw sketches over third-person observations. These sketches serve as image inputs from which action constraints, such as keypoints or flow trajectories, can be extracted. Sketches can then be converted into waypoints aligned with the robot’s workspace coordinate frame; see “Robotic visual instruction” “Vocal Sandbox”, and “L2D2” for related approaches. The second is through teleoperation tools, such as VR interfaces or arm-based teleoperation systems, which allow users to provide 3D position references or full 6-DoF pose or orietation references. We are also developing a 3D desktop GUI tools for human–robot interaction, which reconstructs the robot’s 3D workspace from camera-captured point clouds. Within this interface, users can select constraint types and action dimensions through mouse clicks, dragging, and numerical input, and add these constraints to the rollout timeline of one existing demonstration. Some aspects of this interface are illustrated in Figure 6. In our experiments, we primarily used two methods to provide control inputs: a teleoperation system and our custom-built interactive interface.

Question 2

In regard to the novelty of out work, please refer to our revised Related Work section, which more clearly explains how our approach differs from prior work and highlights its novelty. In summary:

  1. To the best of our knowledge, in robot manipulation, we are the first to investigate a unified method for controllable action generation in both space and time dimension. Compared with other constrained generation methods, current designs typically rely on one or a few modality-specific action policies.
  2. In robotic manipulation, existing methods primarily use inpainting to accelerate asynchronous inference in action or improve visual generalization through observation-space inpainting. In contrast, we perform inpainting in the Cartesian action space to enable controllable action generation in both space and time.
  3. We are the first to study this problem in robotic manipulation. Our method incorporates several design choices tailored to the challenges posed by unified controllable action generation, such as a progress-guided MAP window to improve the effectiveness of policy conditioning. Please refer to the Method section for further details on these and other novel components. In summary, our method is the first to enable precise control in both space and time while unifying all spatiotemporal control signals within a shared representation for action generation.
AuthorsOpenReview ↗

Response to Reviewer BiKV (part2)

We continue our response below.

Question 3

The reviewer points out that demonstration-derived references alone could be too ideal for evaluation. In order to demonstrate the performance of IP under independently generated, noisy, or imperfect references, we conducted two additional experiments: perturbing reference-progress annotations and replacing demonstration-derived keypoints with independently constructed references.

  1. We test inaccurate progress annotations with Gaussian noise while preserving spatial keypoints and their order on PokeCube, PickCube and LiftPegUpright in ManiSkill. Across noise levels from σ=0.02 to σ=0.10, the three tasks retain their clean-reference success rates with only marginal fluctuations, demonstrating strong robustness against temperal level reference error (See Supplementary §3.6 for more experimental details).

  2. We use an Astra agent to estimate keypoints and task progress from the simulator’s observation, without using any demonstrations. With these noisy references, the IP achieves 78% on PickCube and 92% on LiftPegUpright without additional fine-tuning, providing evidence that our policy could generalize to independently generated, noisy, or imperfect references (See Supplementary §3.6 for more experimental details).

Question 4

Firstly, you raise a good point that CE measures spatial proximity but ignores the progress. We address this limitation with a new metric.

We introduce BiCE(Bidimensional Control Error), which adds temporal progress error at the spatially closest action to spatial error at the temporally closest action:

|q_{\arg\min_t D_\ell(t)}-p_\ell| +D_\ell\!\left(\arg\min_t|q_t-p_\ell|\right) \right].$$ Here, $K$ counts reference constraints, $p_\ell$ is specified progress, and $q_t$ is estimated execution progress. $D_\ell(t)$ measures action-reference distance on visible coordinates and both terms have equal weight. Across three ManiSkill tasks and five reference configurations, we evaluate STC, IP, and IP-RePaint with 50 episodes per group (2,250 total). Both IP and IP-RePaint achieve significantly lower mean CE and BiCE than STC. These results support stronger spatial and progress-aware control, with stronger reference enforcement from IP-RePaint (**See Supplementary §3.7, Table 13**). Secondly, progress is a continuous value. Its advantage over timesteps is that it is unaffected by the robot’s execution speed: task progress is predicted directly from the current observation. As shown in **Figure 6**, users can identify meaningful manipulation events from the visual rollout images of a single demonstration and add spatiotemporal constraints at the corresponding progress values. Related works on progress estimation like [ReWiND [1]](https://arxiv.org/html/2505.10911v2) and [SARM [2]](https://arxiv.org/abs/2509.25358) show that advanced progress estimation models can remain robust even in multi-stage or long-horizon manipulation tasks and can align different trajectories at the same stage of task execution. Our **comparative experiments** show that task progress is more suitable than timesteps as the temporal coordinate for controllable action generation. **In Supplementary §3.5, Table 9**, we compare physical rollout timestep and observation-based progress as different temporal guidance source. With the same PickCube checkpoint and 100 episodes, it improves SR from 67% to 83% and CE from 0.03983 to 0.02912 over normalized rollout time.
AuthorsOpenReview ↗

Response to Reviewer BiKV (part3)

We continue our response below.

Question 5

I would like to answer the following two questions that the reviewer raises: (1) we provide a practical way to balance success rate and control accuracy. (2) we explain why stronger constraint adherence can trade off against task success.

(1)First, within the two methods raised in our work IP-vanilla provides a balanced solution for controlability and task capability, while IP-RePaint offers a training-free option with higher control accuracy. To further make this balance adjustable, we introduce a guidance-strength parameter λ∈[0,1]\lambda\in[0,1] that controls reference injection after each denoising step. Let A~(k−1)\widetilde A^{(k-1)} be the policy proposal, MtchunkM_t^{\mathrm{chunk}} the constraint mask, and Rt(k−1)R_t^{(k-1)} the reference at the corresponding noise level. We update the action chunk as

A(k−1)=A~(k−1)+λMtchunk⊙(Rt(k−1)−A~(k−1)).A^{(k-1)}=\widetilde A^{(k-1)}+\lambda M_t^{\mathrm{chunk}}\odot\left(R_t^{(k-1)}-\widetilde A^{(k-1)}\right).

Here, λ=0\lambda=0 recovers IP, λ=1\lambda=1 applies hard RePaint to the same IP checkpoint. We evaluate a sweep of λ\lambda from 0 to 1 on PickCube and LiftPegUpright, comparing SR, CE, and BiCE to assess task completion together with spatial and progress-aware constraint adherence (Supplementary §3.8, Table 14). As shown in the results, increasing λ\lambda lowers both CE and BiCE on both tasks, while the highest observed SR occurs at intermediate strengths. Accordingly, λ\lambda should be treated as a task-specific hyperparameter: an appropriate choice can preserve task success while reducing both CE and BiCE, as demonstrated on the two evaluated tasks.

(2)Second, we believe that following a reference does not always lead to task success for the following three reasons: 1) A reference guides only part of the action, while task success requires position, orientation, gripper state, and contact to work together in the current environment. 2) Execution deviations: The robot may deviate from the motion expected by the high-level planner before reaching the reference waypoint, making that waypoint unsuitable for the current situation. Overall, forcing constraints can limit the policy’s ability to adjust its actions based on current observations. This may explain why stronger reference following can come at the cost of task success.

Thank you again for your time and feedback. We would love to hear your further insights on our comment.

Comments on the paper

1 comment

As we near the end of the discussion period for this submission, please check whether the authors have answered questions or responded to critiques. If they have, we encourage the reviewers to leave a comment for the authors. If reviewers have raised new points, a gentle nudge to the authors to respond would also be appreciated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.