acceptodds
Under review as a conference paper at ICLR 2027

Transferring Relative Motion Into Action for Dynamic-Centric Manipulation

Abstract

Vision-language-action (VLA) models perform well in static manipulation but remain challenged by independently moving targets. As objects continue to move during inference, generated actions can become misaligned with the scene at execution time. Existing approaches address this perception-to-execution gap through execution scheduling or scene modeling, but often leave observed target dynamics weakly coupled to action generation. We propose Motion Into Action (MIA), a framework that couples object-centric perception with dynamics-conditioned action generation. MIA maps task-relevant optical-flow history to a motion-induced latent action that parameterizes a motion-aligned Gaussian source for diffusion, guiding the action head toward executable actions in fewer denoising steps. Context-aware asynchronous execution further overlaps inference and execution while promoting temporal consistency across action chunks. Across five DOMINO tasks, achieves an average success rate of 41.6%, relatively outperforming the strongest baseline by 55.22%. On two tasks, it maintains performance when denoising is reduced from 20 to 5 steps. Across five real-world tasks, MIA achieves an average success rate of 88%, with qualitative examples illustrating transfer to faster motion, occlusion, and moving distractors beyond the demonstration conditions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dynamic Object Manipulation, Perception-Action Coupling, VLA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J2dCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Motion Into Action (MIA) for manipulating independently moving objects. MIA uses task-relevant optical-flow history and robot state to construct a motion-induced latent action, which parameterizes a conditional Gaussian source for diffusion-based action generation. The action expert refines this source under vision-language, proprioceptive, and recent-action conditioning, while asynchronous execution overlaps inference with ongoing control. Evaluation covers five DOMINO tasks and five real-world conditions across two task families. The reported results include 41.6% average simulation success, 88% average real-world success, and maintained performance with five denoising steps on two tasks.

Critical strengths
  1. A coherent approach to incorporating target motion into action generation. The paper uses observed object motion to guide action generation by learning a motion-based source distribution. It also combines this with asynchronous execution to generate smoother actions. This design allows the model to use motion information directly during action generation without predicting future scenes.

  2. On the five evaluated DOMINO tasks, MIA improves average success from 26.8% for VLASH to 41.6%. In the real-world evaluation, it achieves 88% average success versus 71% for VLASH. The tests with faster motion, temporary occlusion, and moving distractors are particularly valuable because these conditions are absent from the training demonstrations and require no condition-specific fine-tuning. These results provide meaningful evidence of the complete system's practical utility.

Critical weaknesses
  1. Although Table 2 includes historical-flow ablations, it is unclear how motion features are incorporated without the proposed flow prior. A controlled comparison using standard motion conditioning methods, such as cross-attention or AdaLN, while retaining a standard Gaussian source would better demonstrate the advantage of the proposed design. Furthermore, Figure 3 lacks an NFE comparison using the same backbone with and without motion-conditioned source initialization. Therefore, it remains unclear whether the improved few-step performance is specifically due to the proposed motion prior.

  2. The model architecture is not described in sufficient detail. Although the paper presents the overall pipeline, it provides limited details about the architectures of key components, including the vision-language encoder, motion encoder, and action expert. As a result, the complete model architecture and its implementation remain unclear.

Questions for authors
  1. Could the authors compare MIA with directly injecting the same motion features into the action expert through cross-attention or AdaLN, while using a standard Gaussian source?

  2. Could the authors provide an NFE ablation using the same backbone with and without motion-conditioned source initialization, under matched training and execution settings? This would help determine whether the proposed motion prior is responsible for maintaining performance with fewer denoising steps.

  3. Could the authors provide more details about the overall architecture of MIA, including the vision-language backbone, motion encoder, and action expert?

2 comments
AuthorsOpenReview ↗

W1 - More specific ablation study

Thank you for the constructive review. Your three questions map directly to the experiments below: a same-backbone comparison of motion-injection routes (Q1), a matched NFE ablation with and without the motion-conditioned source (Q2), and the full architecture (Q3).

We use the following variants. All four share the same backbone, demonstrations, training schedule, and asynchronous execution; V1–V3 also share the motion encoder and the motion latent z_act, and differ only in how z_act enters action generation.

VariantSourceMotion route
V0isotropic N(0, I)none (MIA's backbone with an isotropic source)
V1isotropic N(0, I)z_act as a cross-attention KV token in the action expert
V2isotropic N(0, I)z_act via AdaLN scale/shift in the action expert
V3 (ours)learned motion-aligned Gaussianz_act + state to source

W1 + Q1 - Direct motion conditioning versus motion-aligned source

V1 and V2 are the configuration you requested: the same motion features injected into the action expert through cross-attention or AdaLN, with a standard Gaussian source.

Variantadjust_bottle K20 SR ↑adjust_bottle K5 SR ↑handover_mic K20 SR ↑handover_mic K5 SR ↑
V020/50 (40%)12/50 (24%)19/50 (38%)10/50 (20%)
V125/50 (50%)17/50 (34%)21/50 (42%)12/50 (24%)
V227/50 (54%)18/50 (36%)22/50 (44%)11/50 (22%)
V3 (ours)30/50 (60%)32/50 (64%)27/50 (54%)31/50 (62%)

At K20, injecting z_act into the action expert improves over V0 (+5–7 successes on adjust_bottle and +2–3 on handover_mic), so the motion features are informative in either route. Using them to construct the source (V3) gives the highest success on both tasks, and the gap widens at fewer steps (Section 2).

W1 + Q2 - NFE ablation with and without the motion-conditioned source

Under matched backbone, training, and execution settings, the isotropic-source variants lose much of their success when K drops from 20 to 5, whereas V3 does not:

TaskV0 K20→K5V1 K20→K5V2 K20→K5V3 (ours) K20→K5
adjust_bottle20→1225→1727→1830→32
handover_mic19→1021→1222→1127→31

Because V1 and V2 receive the same z_act but still degrade, the few-step robustness comes specifically from the motion-conditioned source, not from the backbone or from motion conditioning alone. Across the full range of step counts, V3 remains stable down to K2 and fails only with one step:

Task SRK20K15K10K5K2K1
adjust_bottle30/50(60%)32/50(64%)31/50(62%)32/50(64%)32/50(64%)15/50(30%)
handover_mic27/50(54%)28/50(56%)27/50(54%)31/50(62%)30/50(60%)7/50(14%)

Differences from K20 to K2 are within the sampling variability of 50 episodes. A refinement analysis supports the mechanism: at both K20 and K5, the refined chunk stays within an RMS of about 0.086 per coordinate of the source mean μ in the normalized action space, so a few steps suffice to complete it. Fewer steps also matter in practice: on handover_mic, reducing K from 20 to 5 lowers the full request latency from 70.71 to 49.62 ms on an RTX 4090.

AuthorsOpenReview ↗

W2 + Q3 - Architecture

MIA has three components. The vision-language backbone encodes the multi-view images and the instruction into action-query features h_t. The motion encoder maps head-camera optical-flow history and the robot state to the motion latent z_act and the source parameters (μ_t, Σ_t). The action expert refines a sample from this source in K DDIM steps, conditioned on h_t, the state s_t, and the committed-action context c_t.

Vision-language backbone.

ItemSpecification
ModelQwen3-VL-4B-Instruct (hidden width 2560, BF16), initialized from pretrained weights and fully fine-tuned (no LoRA)
Visual inputHead, left-wrist, and right-wrist cameras; current images at 224×224 plus four historical images per view (offsets −16, −12, −8, −4) at 64×64
OutputFinal hidden states of 16 action-query tokens, h_t ∈ ℝ^{16×2560}

Motion encoder.

ItemSpecification
Optical flowFarneback on head-camera frames at 64×64; endpoints [−16, −12, −8, −4, 0] give four intervals; displacement normalized by image size and interval duration, clipped to ±4
Flow encoder ΦShared CNN (channels 2→32→64→128; kernel/stride 5/2, 3/2, 3/2; GroupNorm, SiLU; 4×4 pooling); four intervals give 64 flow tokens of width 128
Object / robot queriesObject query from the mean of h_t; one learnable 128-D robot query; four-head cross-attention over the flow tokens (Eq. 5)
Relation module R / latent mapper GR: MLP 256→256→128 on the concatenated readouts; G: MLP 142→128→128 on the motion latent and the 14-D state
Source mapper PMLP 142→512→512 with a 32-D horizon-position embedding, then 544→28 per position, giving μ_t = tanh(·) and σ_t = 0.05 + 1.45·sigmoid(·) for the 14 action dimensions (diagonal Σ_t)

Action expert.

ItemSpecification
Denoiser DResidual MLP, width 2048, three blocks, applied per horizon position; conditioned on h_t, s_t, and c_t
Context encoder C_θM = 8 committed commands with time offsets; 15→128 projection, CLS token, two pre-norm Transformer layers (4 heads, FFN width 512)
Diffusion and samplingα_γ = cos(πγ/2), σ_γ = sin(πγ/2) with γ ~ U[0, 1) in training; deterministic DDIM on a uniform grid at inference, one denoiser call per step (K = NFE)
State / actions (Sim)14-D state (6 joints + gripper per arm); 16 × 14 absolute joint/gripper targets normalized to [−1, 1]

Training, sampling, and evaluation.

SettingValue
Data50 demonstrations per task
Budget20,000 updates, global batch 8 (4 × H100, batch 2 per GPU), seed 42
OptimizerAdamW, β = (0.9, 0.95), weight decay 1e−8, gradient clipping 1.0
Learning rate1e−5 (VLM) and 1e−4 (heads); 1,000 warmup steps, then cosine decay
ExecutionChunk length 16, 8-action prefix, inference triggered after the 4th action, 50 Hz control
BaselinesTrained on the same 50 demonstrations per task and evaluated with the identical protocol (the same 50 test episodes and fixed evaluation seed)
Real worldRealMan RX75 arm; 150 demonstrations per task family; all policies trained on the same demonstrations and evaluated with 20 rollouts per condition
Reviewer_G4C6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce Motion into Action (MIA) for manipulating moving objects. The proposed method uses recent optical flow to estimate task-relevant motion and predict a starting distribution for diffusion-based action generation. Then it refiness this initial prediction into actions for the robot. The main contribution of this paper is using observed object motion to shape the starting point of action generation. MIA reports 41.6% average simulation success, compared with 26.8% for VLASH, and 88% real-world success, compared with 71%.

Critical strengths

Strengths:

  • I believe that the motion based initialization is well motivated and supported by the paper ablations. On the other hand , starting action generation from a motion-informed prediction directly addresses the difficulty of reacting to moving targets. With motion history and asynchronous execution enabled, adding the learned prior increases average success as it is shown in Table 2.

  • The paper shows that the evaluation includes meaningful real-world transfer, all compared policies use the same demonstrations from two training conditions. MIA then performs well under faster motion, temporary occlusion, and moving distractors without additional training, which I believe strengthen the evidence that the method can handle changes beyond its demonstration settings.

Critical weaknesses

Weaknesses:

  • The claimed speed advantage is not directly measured in the paper. The paper shows that MIA can use fewer denoising steps, but I think that a report of total inference time would be useful. Optical-flow estimation and motion processing add computation that could affect the savings. Since reducing the delay between observation and action is a central motivation, the paper needs these comparisons that include the entire pipeline in my opinion.

  • Missing implementation details make the comparisons difficult to assess. The paper does not specify the underlying vision-language model and action backbone, optical-flow estimator, training settings, or how the predicted covariance is implemented. There is also an apparent reporting mismatch: Figure 3 shows PUMA near 50% success on Adjust Bottle, while Table 1 reports 24%, sorry if I missed something here, but these gaps make it difficult determine whether the comparisons are consistent.

Questions for authors

Please refer to weaknesses. I would be happy to raise my score if my doubts are solved.

2 comments
AuthorsOpenReview ↗

W1 - Inference time

Thank you for the careful review and the offer to reconsider the score. We address total inference time, implementation details, and the PUMA discrepancy below.

W1 - Inference time

We time the full request, from decoded RGB/state in memory to the decoded 16×14 action chunk, on one NVIDIA RTX 4090 over 30 fixed demonstration windows per task and step count. The timing includes optical-flow computation and all motion processing.

Task / modelK20 mean / P95 (ms)
adjust_bottle / VLASH112.34 / 115.14
adjust_bottle / MIA64.85 / 66.57
handover_mic / VLASH117.49 / 121.97
handover_mic / MIA70.71 / 75.86

Latency breakdown on handover_mic (ms).

ComponentK20K5
Current RGB preprocessing3.783.83
Optical-flow computation (4 pairs, 64×64)3.553.40
Other history and request preparation3.062.96
Motion encoder and source mapper2.311.24
VLM inference and denoising55.3835.85
Transport, action decoding, and remaining overhead2.632.34
Total70.7149.62

At K20, MIA is 1.73× and 1.66× faster than VLASH on adjust_bottle and handover_mic. The overhead of optical flow and motion processing is small: the explicit motion branch (optical flow, motion encoder, and source mapper) costs 5.86 ms at K20 and 4.64 ms at K5, about 8–9% of the request. This is well below the 21.09 ms saved by reducing K from 20 to 5 (70.71 → 49.62 ms), most of which comes from VLM inference and denoising (55.38 → 35.85 ms).

Effect on handoff. With an 8-action prefix and inference triggered after the 4th action, τ_rem = 4Δt = 80 ms at 50 Hz. By Eq. (11), τ_wait = 0 whenever τ_gen < 80 ms. MIA meets this budget even at K20 (P95 ≤ 75.86 ms on both tasks), so inference is fully hidden behind execution; K5 adds about 30 ms of headroom.

AuthorsOpenReview ↗

W2 - Implementation details and PUMA in Figure 3 versus Table 1

W2 - Implementation details

MIA has three components. The vision-language backbone encodes the multi-view images and the instruction into action-query features h_t. The motion encoder maps head-camera optical-flow history and the robot state to the motion latent z_act and the source parameters (μ_t, Σ_t). The action expert refines a sample from this source in K DDIM steps, conditioned on h_t, the state s_t, and the committed-action context c_t.

Vision-language backbone.

ItemSpecification
ModelQwen3-VL-4B-Instruct (hidden width 2560, BF16), initialized from pretrained weights and fully fine-tuned (no LoRA)
Visual inputHead, left-wrist, and right-wrist cameras; current images at 224×224 plus four historical images per view (offsets −16, −12, −8, −4) at 64×64
OutputFinal hidden states of 16 action-query tokens, h_t ∈ ℝ^{16×2560}

Motion encoder.

ItemSpecification
Optical flowFarneback on head-camera frames at 64×64; endpoints [−16, −12, −8, −4, 0] give four intervals; displacement normalized by image size and interval duration, clipped to ±4
Flow encoder ΦShared CNN (channels 2→32→64→128; kernel/stride 5/2, 3/2, 3/2; GroupNorm, SiLU; 4×4 pooling); four intervals give 64 flow tokens of width 128
Object / robot queriesObject query from the mean of h_t; one learnable 128-D robot query; four-head cross-attention over the flow tokens (Eq. 5)
Relation module R / latent mapper GR: MLP 256→256→128 on the concatenated readouts; G: MLP 142→128→128 on the motion latent and the 14-D state
Source mapper PMLP 142→512→512 with a 32-D horizon-position embedding, then 544→28 per position, giving μ_t = tanh(·) and σ_t = 0.05 + 1.45·sigmoid(·) for the 14 action dimensions (diagonal Σ_t)

Action expert.

ItemSpecification
Denoiser DResidual MLP, width 2048, three blocks, applied per horizon position; conditioned on h_t, s_t, and c_t
Context encoder C_θM = 8 committed commands with time offsets; 15→128 projection, CLS token, two pre-norm Transformer layers (4 heads, FFN width 512)
Diffusion and samplingα_γ = cos(πγ/2), σ_γ = sin(πγ/2) with γ ~ U[0, 1) in training; deterministic DDIM on a uniform grid at inference, one denoiser call per step (K = NFE)
State / actions (Sim)14-D state (6 joints + gripper per arm); 16 × 14 absolute joint/gripper targets normalized to [−1, 1]

Training, sampling, and evaluation.

SettingValue
Data50 demonstrations per task
Budget20,000 updates, global batch 8 (4 × H100, batch 2 per GPU), seed 42
OptimizerAdamW, β = (0.9, 0.95), weight decay 1e−8, gradient clipping 1.0
Learning rate1e−5 (VLM) and 1e−4 (heads); 1,000 warmup steps, then cosine decay
ExecutionChunk length 16, 8-action prefix, inference triggered after the 4th action, 50 Hz control
BaselinesTrained on the same 50 demonstrations per task and evaluated with the identical protocol (the same 50 test episodes and fixed evaluation seed)
Real worldRealMan RX75 arm; 150 demonstrations per task family; all policies trained on the same demonstrations and evaluated with 20 rollouts per condition

W2 - PUMA in Figure 3 versus Table 1

Thank you for catching this, and we apologize for the confusion. The two entries come from different PUMA configurations. Table 1 reports the original PUMA baseline, whose action head regresses actions with an MLP and therefore has no denoising steps. To study the effect of the number of denoising steps in Figure 3, we replaced only this MLP head with a diffusion action head, keeping the rest of PUMA and the evaluation protocol unchanged. The two configurations therefore differ in success rate: on adjust_bottle, the diffusion variant reaches 26/50 (52%) at K20, compared with 24% for the original baseline in Table 1, and still remains below MIA (60%). The same distinction applies to handover_mic. The revised Figure 3 labels this variant as “PUMA(diffusion)”, and Section 4.5 states the modification.

Reviewer_Jz6eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies manipulation of moving objects with VLA-style policies. The authors identify a perception-to-execution gap: an action chunk is generated from the observation at time t, but the object has moved by the time the chunk is executed.

The proposed method, MIA, conditions action generation on object and scene motion in three ways:

  1. Motion encoding. A history of optical flow from a fixed head camera is tokenized. A task-conditioned object query and a learnable robot query cross-attend to these tokens. A relation module fuses the two readouts into a motion latent, which is mapped to a motion-induced action latent.

  2. Motion-aligned source distribution. The action latent and proprioception predict a Gaussian source N(μ_t, Σ_t) over the action chunk, trained with Gaussian NLL against demonstrations. Few-step deterministic DDIM starts from a sample of this source instead of isotropic noise. Motion enters only through the source. The denoiser is conditioned on vision-language features, state, and execution context.

  3. Context-aware asynchronous execution. The robot executes a "committed prefix" while the next chunk is generated. A context encoder summarizes the last M committed commands.

Results:

  • Simulation (5 DOMINO tasks): 41.6% average success rate vs. 26.8% for VLASH, the strongest baseline.
  • Real world (Conveyor and Rolling Bottle): 88% average vs. 71% for VLASH and 66% for PUMA, including zero-shot variants with a faster conveyor, occlusion, and spatial reasoning.
Critical strengths
  1. Important, under-explored problem. Large VLAs struggle with dynamic objects because of inference latency. Framing this as a perception-to-execution gap is clear and well motivated.

  2. Novel angle. Injecting motion through the source distribution of the diffusion process, rather than only as a conditioning signal, is conceptually interesting. It lets few-step sampling start near a motion-consistent solution. Figures 1 and 2 contrast the paradigms clearly.

  3. Practical system design. Pairing a learned source prior with asynchronous execution and a committed-action context is a reasonable, deployable combination.

  4. Consistent empirical gains in simulation and the real world, more robustness when denoising steps are reduced, and encouraging zero-shot transfer to a faster conveyor and to occlusion.

Critical weaknesses

W1. Motion affects only the source, and a single Gaussian limits capacity. Section 3.3 states that "motion enters through source construction," so the denoiser never sees the motion latent. All motion information must therefore pass through (μ, Σ) of a single unimodal Gaussian. This raises concerns about multimodal responses: when the correct response to motion has several modes (e.g., intercept now vs. wait), a unimodal Gaussian can only cover them with a large Σ, which might erode the benefit.

W2. Key ablations are missing. Table 2 does not isolate that adding the motion condition to change source noise, directly as the denoiser condition with isotropic source. I would like to see:

  • Motion latent fed to the denoiser as conditioning, with an isotropic source.
  • Historical flow (HF) alone drops performance from 26% to 15%. The paper does not explain how HF is used without the flow prior, or why it hurts.

W3. The asynchronous handoff mechanism is unclear.

Section 3.4 mentions a "committed prefix," says the new chunk starts "at the prefix's handoff once generation is complete," and gives τ_wait = max(0, τ_gen − τ_rem). It does not explain:

  • How the prefix length is chosen: fixed, or adapted to the measured τ_gen.

  • When a new inference call is triggered.

  • How the new chunk is time-aligned. Does it predict actions from t or from t + τ_gen? Are the first actions skipped, as in RTC or VLASH?

  • What the robot does when τ_wait > 0: idle, or hold the last action.

  • What the training-time "handoff convention" is, and how delays and c_t are simulated during training.

W4. No latency measurements. Latency is the paper's main motivation, yet it reports no wall-clock τ_gen, control frequency, or end-to-end latency compared with baselines. Computing optical flow over L frames plus the motion encoder adds its own perception latency, which is part of the gap the paper aims to close.

W5. Fixed-camera assumption: can MIA handle camera changes?

W6. Five-step DDIM gives the best result, which is counterintuitive and unexplained.

Questions for authors
  • What exactly is the robot query? Is it a single learnable vector shared across all inputs? Is there any supervision that makes it attend to the robot? How are bimanual tasks handled? Can you provide an ablation and attention visualizations?

  • How is Σ_t parameterized, and how is positive definiteness enforced? What is its dimension, and do you observe variance collapse?

  • Since motion only affects the source, have you tried conditioning the denoiser on the motion latent directly? How does a mixture or otherwise multimodal source compare? How much of μ survives after K DDIM steps?

  • What is the observation O_t (frames, views, resolution)? Which flow estimator is used, with what L and frame spacing?

  • What does the state s_t contain?

6 comments
AuthorsOpenReview ↗

W1 + Q2 + Q3 — Motion path, source parameterization, and source capacity

We thank the reviewer for the detailed questions about source construction, motion conditioning, and execution. MIA's central contribution is to translate observed motion into the starting point of action generation, alongside the vision-language conditioning of the denoiser. Below we clarify the implementation and provide the requested controlled comparisons and wall-clock timings.

We use the following variant definitions throughout this response.

VariantSourceExplicit motion route
V0isotropic N(0,I)none, MIA's backbone with an isotropic source.
V1isotropic N(0,I)z_act as a single cross-attention KV token in the denoiser
V2isotropic N(0,I)z_act via denoiser AdaLN scale/shift
V3 (ours)learned motion-aligned Gaussianz_act + state to source

W1 + Q2 + Q3 — Motion path, source parameterization, and source capacity

Where motion enters (W1). In V3, the flow-derived latent z_act parameterizes the source. The source is not, however, the only channel through which motion reaches the action expert: at every refinement step, the denoiser is also conditioned on vision-language features of the current and historical multi-view RGB frames, the robot state, and the committed-action context. The explicit motion latent therefore determines where generation starts, while the denoiser can still correct the action from the observation. V1 and V2 test the alternative of feeding z_act directly to the denoiser with an isotropic source; under the same backbone, both remain below V3 and drop sharply at K5 (W2).

Unimodal source and multimodal responses (W1). A unimodal source does not by itself restrict the output to a single mode: standard diffusion policies also start from the unimodal N(0, I) and rely on the conditional denoiser to shape the action distribution. In addition, MIA replans every 8 actions from a new observation, so a choice such as "intercept now or wait" is made across successive chunks rather than committed within a single chunk. The mixture experiment below finds no benefit from multimodal sources on our tasks, which suggests that the demonstrated responses are close to unimodal per state.

Covariance parameterization and variance collapse (Q2). Σ_t in Eq. (6) is a d_A × d_A positive-definite matrix with d_A = H·d_a = 16 × 14 = 224. We instantiate it in diagonal form, which keeps the source mapper and the NLL in Eq. (7) lightweight:

μ_t = tanh(r_μ), σ_t = 0.05 + 1.45·sigmoid(r_σ), Σ_t = diag(vec(σ_t²)),

where r_μ, r_σ ∈ ℝ^{16×14} are predicted by the source mapper P from (z_act, s_t), and actions are normalized to [−1, 1] to match the range of tanh. Because σ_t is predicted separately for every time step and action dimension, Σ_t sets both the overall scale of the source and how that scale is distributed across the chunk: depending on the observed motion and robot state, the source can be tight on some steps and joints and broad on others. The lower bound σ_t ≥ 0.05 makes Σ_t positive definite by construction and prevents its variance from collapsing to zero, and the upper bound of 1.5 keeps the source comparable in scale to the isotropic N(0, I).

Mixture sources (Q3). On adjust_bottle, we compare the single Gaussian with 2- and 4-component Gaussian mixtures under the same protocol:

SourceK20 SR ↑K5 SR ↑
Single Gaussian (ours)30/50 (60%)32/50 (64%)
2 Gaussians30/50 (60%)30/50 (60%)
4 Gaussians21/50 (42%)19/50 (38%)

The 2-component mixture matches the single Gaussian at K20 and is slightly lower at K5, while the 4-component mixture is lower at both step counts. The learned mixture weights also concentrate on a single component: the median effective number of components per state is 1.002. On these tasks, a single Gaussian therefore provides the same initialization quality with fewer parameters.

How much of μ survives refinement (Q3). We compare K20 and K5 on 24 observation states drawn from 12 held-out episodes.

QuantityK20K5
‖Â − μ‖₂1.2795 [1.0243, 1.5547]1.2784 [1.0236, 1.5531]
Pooled output/μ correlation0.9905 [0.9870, 0.9938]0.9906 [0.9870, 0.9938]

The refined chunk stays close to μ: a deviation of ‖Â − μ‖₂ ≈ 1.28 over 224 coordinates corresponds to an RMS of about 0.086 per coordinate in the normalized [−1, 1] action space, and the pooled correlation exceeds 0.99. The deviation is nearly identical at K20 and K5, so most of the refinement is completed within five steps. The denoiser thus applies a small correction around the motion-aligned mean rather than overriding it. This correction is still necessary, since one-step generation fails (15/50 and 7/50 at K1; W6).

AuthorsOpenReview ↗

W2 + Q3 — Motion in the denoiser versus motion in the source

Controlled comparison of motion routes. The four variants share the same backbone, demonstrations, training schedule, and asynchronous execution; V1–V3 also share the motion encoder and z_act. They differ only in how z_act enters generation. V1/V2 correspond to the configuration requested by the reviewer: z_act conditions the denoiser (via cross-attention or AdaLN) with an isotropic source.

Variantadjust_bottle K20 SR ↑adjust_bottle K5 SR ↑handover_mic K20 SR ↑handover_mic K5 SR ↑
V020/50 (40%)12/50 (24%)19/50 (38%)10/50 (20%)
V125/50 (50%)17/50 (34%)21/50 (42%)12/50 (24%)
V227/50 (54%)18/50 (36%)22/50 (44%)11/50 (22%)
V3 (ours)30/50 (60%)32/50 (64%)27/50 (54%)31/50 (62%)

At K20, conditioning the denoiser on z_act improves over V0, and V3 is highest on both tasks. The separation is clearest at K5: all isotropic-source variants drop sharply (e.g., V2: 27→18/50 and 22→11/50), whereas V3 holds (30→32/50 and 27→31/50). Since the backbone and motion latent are shared, this few-step robustness comes from building the source from motion, not from the backbone or from motion conditioning alone. Consistent with W1+Q3, the refined chunk stays close to μ, so a few steps suffice.

How HF-only uses flow (W2). HF-only (Table 2, Set 2) has no motion encoder or learned source: the four 64×64 Farneback flow fields (head camera, offsets −16 to 0) are color-coded, overlaid on the head-camera RGB images, and fed to the VLM. We attribute the drop from 26% to 15% to two factors: the overlay entangles arm motion with target motion, and it alters the RGB appearance the pretrained VLM relies on. MIA instead encodes flow separately and uses object and robot queries to separate target motion from self-induced motion. Consistently, the same flow helps once compressed into z_act (V1/V2 vs. V0), and removing the robot query drops MIA to 15/50, below V0 (results in Q1).

AuthorsOpenReview ↗

W3 — Asynchronous handoff mechanism

  1. Each action chunk contains 16 predicted actions. We execute only the first 8, using a fixed prefix length of 8 policy steps.

  2. The inference trigger is a fixed hyperparameter specifying how many actions of the current chunk are executed before starting inference for the next chunk. We evaluate values from 1 to 7 and obtain the best performance at 4.

  3. Following VLASH, the next chunk is conditioned on the estimated state after action 8. It takes over after the first 8 actions of the current chunk, and the unused tail is discarded.

  4. When τ_wait > 0, the robot maintains the target of the previous chunk's last executed action (action 8) until a new chunk is generated.

  5. During training, we sample d uniformly from {0,…,8} to simulate the interval between observation t and handoff e = t + d. RGB/language/flow remain at t; the recorded state and the 16-action target chunk are taken at e. The context c_t contains the eight demonstration actions preceding e.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful and constructive feedback. We are encouraged that the reviewers found the motion-informed source novel, well motivated, and coherent (Jz6e, G4C6, J2dC), the problem important (Jz6e), and the zero-shot real-world transfer to faster motion, occlusion, and moving distractors meaningful (Jz6e, G4C6, J2dC).

The reviews raised three shared concerns, which we address with new experiments and details:

  1. Is the gain specific to the motion-aligned source? (Jz6e, J2dC) We compare four variants that share the same backbone, data, training, and asynchronous execution: an isotropic source without motion (V0), the same motion latent injected into the action expert via cross-attention (V1) or AdaLN (V2), and MIA. Direct conditioning helps at K20, but all isotropic-source variants drop sharply at K5 (e.g., 27→18/50 and 22→11/50 for AdaLN), whereas MIA holds (30→32/50 and 27→31/50). MIA remains stable down to K2 and fails only at K1.

  2. End-to-end latency including optical flow (Jz6e, G4C6). On an RTX 4090, the motion branch (optical flow, motion encoder, and source mapper) costs 4.6–5.9 ms, about 8–9% of a request and well below the 21 ms saved by reducing K from 20 to 5. MIA's request latency (P95 ≤ 75.86 ms at K20) fits within the 80 ms handoff budget at 50 Hz, so inference is fully hidden behind execution.

  3. Implementation details (G4C6, J2dC, Jz6e). We report the full architecture (Qwen3-VL-4B-Instruct backbone, Farneback flow, CNN motion encoder with object and robot queries, diagonal Gaussian source mapper, and residual-MLP action expert), the training settings, and the asynchronous handoff protocol.

We also address reviewer-specific questions on camera-pose sensitivity, source capacity and covariance, and the robot query (Jz6e), and on the PUMA discrepancy between Figure 3 and Table 1, which arises from replacing PUMA's MLP head with a diffusion head to study denoising steps (G4C6).

Revisions. Section 3.3 clarifies the two paths through which motion reaches the action expert; Section 4.1 adds implementation details; Section 4.4 and Fig. 6 add the camera-pose sensitivity study; and Section 4.5 states the PUMA modification used in Figure 3.

We hope these additions resolve the reviewers' concerns and are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.