- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents Motion Into Action (MIA) for manipulating independently moving objects. MIA uses task-relevant optical-flow history and robot state to construct a motion-induced latent action, which parameterizes a conditional Gaussian source for diffusion-based action generation. The action expert refines this source under vision-language, proprioceptive, and recent-action conditioning, while asynchronous execution overlaps inference with ongoing control. Evaluation covers five DOMINO tasks and five real-world conditions across two task families. The reported results include 41.6% average simulation success, 88% average real-world success, and maintained performance with five denoising steps on two tasks.
Critical strengths
-
A coherent approach to incorporating target motion into action generation. The paper uses observed object motion to guide action generation by learning a motion-based source distribution. It also combines this with asynchronous execution to generate smoother actions. This design allows the model to use motion information directly during action generation without predicting future scenes.
-
On the five evaluated DOMINO tasks, MIA improves average success from 26.8% for VLASH to 41.6%. In the real-world evaluation, it achieves 88% average success versus 71% for VLASH. The tests with faster motion, temporary occlusion, and moving distractors are particularly valuable because these conditions are absent from the training demonstrations and require no condition-specific fine-tuning. These results provide meaningful evidence of the complete system's practical utility.
Critical weaknesses
-
Although Table 2 includes historical-flow ablations, it is unclear how motion features are incorporated without the proposed flow prior. A controlled comparison using standard motion conditioning methods, such as cross-attention or AdaLN, while retaining a standard Gaussian source would better demonstrate the advantage of the proposed design. Furthermore, Figure 3 lacks an NFE comparison using the same backbone with and without motion-conditioned source initialization. Therefore, it remains unclear whether the improved few-step performance is specifically due to the proposed motion prior.
-
The model architecture is not described in sufficient detail. Although the paper presents the overall pipeline, it provides limited details about the architectures of key components, including the vision-language encoder, motion encoder, and action expert. As a result, the complete model architecture and its implementation remain unclear.
Questions for authors
-
Could the authors compare MIA with directly injecting the same motion features into the action expert through cross-attention or AdaLN, while using a standard Gaussian source?
-
Could the authors provide an NFE ablation using the same backbone with and without motion-conditioned source initialization, under matched training and execution settings? This would help determine whether the proposed motion prior is responsible for maintaining performance with fewer denoising steps.
-
Could the authors provide more details about the overall architecture of MIA, including the vision-language backbone, motion encoder, and action expert?
2 comments
W2 + Q3 - Architecture
MIA has three components. The vision-language backbone encodes the multi-view images and the instruction into action-query features h_t. The motion encoder maps head-camera optical-flow history and the robot state to the motion latent z_act and the source parameters (μ_t, Σ_t). The action expert refines a sample from this source in K DDIM steps, conditioned on h_t, the state s_t, and the committed-action context c_t.
Vision-language backbone.
| Item | Specification |
|---|---|
| Model | Qwen3-VL-4B-Instruct (hidden width 2560, BF16), initialized from pretrained weights and fully fine-tuned (no LoRA) |
| Visual input | Head, left-wrist, and right-wrist cameras; current images at 224×224 plus four historical images per view (offsets −16, −12, −8, −4) at 64×64 |
| Output | Final hidden states of 16 action-query tokens, h_t ∈ ℝ^{16×2560} |
Motion encoder.
| Item | Specification |
|---|---|
| Optical flow | Farneback on head-camera frames at 64×64; endpoints [−16, −12, −8, −4, 0] give four intervals; displacement normalized by image size and interval duration, clipped to ±4 |
| Flow encoder Φ | Shared CNN (channels 2→32→64→128; kernel/stride 5/2, 3/2, 3/2; GroupNorm, SiLU; 4×4 pooling); four intervals give 64 flow tokens of width 128 |
| Object / robot queries | Object query from the mean of h_t; one learnable 128-D robot query; four-head cross-attention over the flow tokens (Eq. 5) |
| Relation module R / latent mapper G | R: MLP 256→256→128 on the concatenated readouts; G: MLP 142→128→128 on the motion latent and the 14-D state |
| Source mapper P | MLP 142→512→512 with a 32-D horizon-position embedding, then 544→28 per position, giving μ_t = tanh(·) and σ_t = 0.05 + 1.45·sigmoid(·) for the 14 action dimensions (diagonal Σ_t) |
Action expert.
| Item | Specification |
|---|---|
| Denoiser D | Residual MLP, width 2048, three blocks, applied per horizon position; conditioned on h_t, s_t, and c_t |
| Context encoder C_θ | M = 8 committed commands with time offsets; 15→128 projection, CLS token, two pre-norm Transformer layers (4 heads, FFN width 512) |
| Diffusion and sampling | α_γ = cos(πγ/2), σ_γ = sin(πγ/2) with γ ~ U[0, 1) in training; deterministic DDIM on a uniform grid at inference, one denoiser call per step (K = NFE) |
| State / actions (Sim) | 14-D state (6 joints + gripper per arm); 16 × 14 absolute joint/gripper targets normalized to [−1, 1] |
Training, sampling, and evaluation.
| Setting | Value |
|---|---|
| Data | 50 demonstrations per task |
| Budget | 20,000 updates, global batch 8 (4 × H100, batch 2 per GPU), seed 42 |
| Optimizer | AdamW, β = (0.9, 0.95), weight decay 1e−8, gradient clipping 1.0 |
| Learning rate | 1e−5 (VLM) and 1e−4 (heads); 1,000 warmup steps, then cosine decay |
| Execution | Chunk length 16, 8-action prefix, inference triggered after the 4th action, 50 Hz control |
| Baselines | Trained on the same 50 demonstrations per task and evaluated with the identical protocol (the same 50 test episodes and fixed evaluation seed) |
| Real world | RealMan RX75 arm; 150 demonstrations per task family; all policies trained on the same demonstrations and evaluated with 20 rollouts per condition |
W1 - More specific ablation study
Thank you for the constructive review. Your three questions map directly to the experiments below: a same-backbone comparison of motion-injection routes (Q1), a matched NFE ablation with and without the motion-conditioned source (Q2), and the full architecture (Q3).
We use the following variants. All four share the same backbone, demonstrations, training schedule, and asynchronous execution; V1–V3 also share the motion encoder and the motion latent z_act, and differ only in how z_act enters action generation.
W1 + Q1 - Direct motion conditioning versus motion-aligned source
V1 and V2 are the configuration you requested: the same motion features injected into the action expert through cross-attention or AdaLN, with a standard Gaussian source.
At K20, injecting z_act into the action expert improves over V0 (+5–7 successes on adjust_bottle and +2–3 on handover_mic), so the motion features are informative in either route. Using them to construct the source (V3) gives the highest success on both tasks, and the gap widens at fewer steps (Section 2).
W1 + Q2 - NFE ablation with and without the motion-conditioned source
Under matched backbone, training, and execution settings, the isotropic-source variants lose much of their success when K drops from 20 to 5, whereas V3 does not:
Because V1 and V2 receive the same z_act but still degrade, the few-step robustness comes specifically from the motion-conditioned source, not from the backbone or from motion conditioning alone. Across the full range of step counts, V3 remains stable down to K2 and fails only with one step:
Differences from K20 to K2 are within the sampling variability of 50 episodes. A refinement analysis supports the mechanism: at both K20 and K5, the refined chunk stays within an RMS of about 0.086 per coordinate of the source mean μ in the normalized action space, so a few steps suffice to complete it. Fewer steps also matter in practice: on handover_mic, reducing K from 20 to 5 lowers the full request latency from 70.71 to 49.62 ms on an RTX 4090.