acceptodds
Under review as a conference paper at ICLR 2027

ARGaM: Action-Recoverable Gaussian Motion for Robotic Manipulation

Abstract

Gaussian world models predict future scene evolution in explicit 3D representations, offering geometric cues for robotic manipulation. However, visually accurate predictions alone do not ensure that the underlying motion can reliably guide action trajectories. To address this, we propose ARGaM, which learns Gaussian dynamics autoregressively with temporally aligned visual and action supervision, and refines directly regressed trajectories using a separately recovered geometric motion prior. The dynamics model predicts successive Gaussian states and a coarse action trajectory from shared temporal features, supervised by future-image reconstruction and demonstrated actions across the prediction horizon. An additional motion loss supervises selected gripper Gaussian displacements using demonstration-derived targets, bringing explicit motion constraints into the predicted geometry. Geometric registration then produces a motion prior from the predicted Gaussian correspondences. A residual diffusion model combines this prior, its recovery confidence, the coarse trajectory, and rendered future visual context to predict corrections to the coarse actions. On nine RLBench tasks, ARGaM achieves a mean closed-loop success rate of 72.7%, improving over the strongest evaluated baseline (60.4%) by 12.3 percentage points. On six real-robot tasks, ARGaM succeeds in 48 of 60 trials (80.0%), compared with 35 of 60 (58.3%) for the same baseline, an improvement of 21.7 percentage points.

Primary area:
applications to robotics, autonomy, planning
Keywords:
4D World Model, Gaussian Splatting, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dbWZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ARGaM, which autoregressively predicts future Gaussian scene states and coarse action trajectories, with demonstration-derived supervision of gripper-region Gaussian displacements. Weighted rigid registration recovers a geometric motion prior from predicted Gaussian correspondences. A residual diffusion model then combines this prior, its recovery reliability, the coarse actions, and predicted visual context to refine the trajectory. Evaluations cover nine RLBench tasks and six real-robot tasks, with ablations examining motion supervision, temporal modeling, and geometric motion guidance.

Critical strengths
  1. A useful connection between geometric motion supervision and action generation. Demonstrated actions directly constrain the local Gaussian displacements used for motion recovery, making predicted geometry informative for action refinement.
  2. Reported gains and ablations provide positive evidence. Simulation and real-robot tables show substantial improvements over baselines, while ablations support local motion supervision and geometric guidance.
Critical weaknesses
  1. Section 4.2 contains inconsistent simulation results. The “Closed-loop manipulation” paragraph describes Table 1 using results matching the real-world Table 3, contradicting the abstract and simulation Table 1. The correct results and the cause of this mismatch require clarification.
  2. Geometric recovery and reliability modulation are insufficiently specified. The gripper-mask source, registration-weight computation, and reliability-based fusion mechanism remain unclear, limiting assessment of the core pipeline’s completeness and reproducibility.
Questions for authors
  1. Please explain the discrepancy between Section 4.2, the abstract, and Table 1, and verify the correct correspondence between simulation and real-robot results against the experimental records. Do any other results contain similar mismatches?
  2. How are gripper masks obtained during training and deployment, registration weights computed, and recovery reliability used to modulate GMP’s contribution to action refinement? Do these steps rely only on information available at deployment?
3 comments
AuthorsOpenReview ↗

Response to Reviewer dbWZ

We thank the reviewer for recognizing the value of our geometric motion supervision and its empirical benefits. For clarity, we have organized and numbered the concerns below. We have uploaded the revised manuscript and hope to address the concerns raised.

Weaknesses:

  1. Q1. 1). Section 4.2 contains inconsistent simulation results.
  2. Q2. 1). The gripper-mask source, 2). registration-weight computation, and 3). reliability-based fusion mechanism remain unclear.

Questions

  1. Q1. 1). explain the discrepancy between Section 4.2, the abstract, and Table 1, and verify the correct correspondence between simulation and real-robot results against the experimental records. 2). Do any other results contain similar mismatches?
  2. Q2. 1). How are gripper masks obtained during training and deployment, 2). registration weights computed, 3). and recovery reliability used to modulate GMP’s contribution to action refinement? Q3. Do these steps rely only on information available at deployment?

A1. 1). Thank you for identifying this discrepancy. The first two sentences of Section 4.2 mistakenly reported the mean success rate and percentage-point differences from the real-robot experiments in Table 3, instead of the RLBench simulation results in Table 1. The abstract and Table 1 report the correct simulation results. ARGaM achieves a mean success rate of 72.7%, compared with 60.4% for GAF, the strongest baseline, yielding a 12.3 percentage-point improvement.

We have corrected Section 4.2 as follows:

Table 1 reports closed-loop manipulation success rates on nine RLBench tasks. ARGaM achieves the highest mean success rate of 72.7%, outperforming ACT, ManiGaussian, Act3D, and GAF by 27.9, 22.6, 19.6, and 12.3 percentage points, respectively. Compared with GAF, the strongest baseline, ARGaM achieves higher success on eight of nine tasks, with the largest improvement on Close Fridge (42% to 88%). On Lift Lid Up, both methods perform near the ceiling, with ARGaM and GAF achieving 98% and 100%, respectively. These results demonstrate ARGaM's effectiveness across the evaluated manipulation tasks.

2). We also rechecked the numerical results throughout the manuscript and found no other numerical mismatches.

A2. To explain the gripper selection, registration, and reliability-guided refinement clearly, we have added Algorithm 1 and its accompanying explanation to Section 3.3 of the revised manuscript.

1).Gripper mask. Algorithm 1 (Lines 1–3) selects the MM Gaussian centroids closest to the initial gripper center. During training, the current demonstrated gripper pose is transformed into the Gaussian reference frame. At deployment, the same procedure uses the current gripper pose. Future demonstrated gripper poses are used only to construct AGM displacement targets.

2). Registration weights. Algorithm 1 (Line 4) computes normalized distance-based weights:

wi=miexp⁡(−di/s)∑jmjexp⁡(−dj/s),s=max⁡(min⁡i∈Idi,10−4),w_i=\frac{m_i\exp(-d_i/s)} {\sum_jm_j\exp(-d_j/s)}, \qquad s=\max\left(\min_{i\in I}d_i,10^{-4}\right),

where did_i is the distance to the initial gripper center. Closer Gaussians receive higher weights. These weights are reused for weighted Kabsch alignment across predicted frames.

3). Reliability-guided refinement. Algorithm 1 (Lines 8 and 11–14) computes four recovery-quality descriptors: normalized registration residual (rtr_t), effective point support (utu_t), covariance singular-value ratio (gtg_t), and adjacent-frame motion discrepancy (qtq_t). Together with cumulative, step-wise, and remaining motion, they form the 22-D GMP token.

As described in Section 3.4, the token is encoded into chc_h and fused with the action embedding aha_h through a learnable gate:

\operatorname{sigmoid} \left(\operatorname{MLP}([a_h,c_h])\right),$$ $$a'_h=a_h+\alpha_h\cdot c_h.$$ This gate adaptively regulates the geometric prior's contribution according to the action context and encoded recovery quality. The fused feature is used to predict residual corrections to the coarse action trajectory. We have also added the explicit mathematical definitions of the four reliability descriptors and the complete pipeline pseudocode to the supplementary material. We will release the full source code to facilitate reproducibility. **A3.** Yes. All operations rely exclusively on information available at deployment, including: Current stereo images and camera calibration. Current gripper pose in the Gaussian reference frame. Future Gaussian states predicted by the world dynamics model. The model-predicted coarse action trajectory. The selection mask, registration weights, motion transformations, and reliability descriptors are computed from these inputs and predicted Gaussian states. Future demonstrated gripper poses and ground-truth future actions are used only for training supervision, never during deployment.

Dear Reviewer dbWZ,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

To clarify how ARGaM selects the gripper region and recovers the geometric motion prior, we include the pseudocode below.

Thank you for your time and effort in reviewing our paper.

 
 


Algorithm 1 Gripper region selection and geometric motion prior recovery


Input: G0\mathcal G_0, G^1:T\widehat{\mathcal G}_{1:T}, gripper pose X0X_0 in the Gaussian frame, point count MM, horizon TT.

Output: Selection mask mm and 22-D GMP token sequence M1:TGMP\mathcal M_{1:T}^{\mathrm{GMP}}.


1. μi0←Centroids⁡(G0),pgrip0←GripperCenter⁡(X0)\\{\boldsymbol\mu_i^0\\}\gets\operatorname{Centroids}(\mathcal G_0),\quad \mathbf p_{\mathrm{grip}}^0\gets\operatorname{GripperCenter}(X_0)

2. di←∥μi0−pgrip0∥2(∀i),I←SmallestIndices⁡(d,M)d_i\gets\lVert\boldsymbol\mu_i^0-\mathbf p_{\mathrm{grip}}^0\rVert_2\\ (\forall i),\quad I\gets\operatorname{SmallestIndices}(d,M)

3. mi←1i∈I,s←max⁡(min⁡i∈Idi,10−4)m_i\gets\mathbf{1}_{\\{i\in I\\}},\quad s\gets\max\bigl(\min_{i\in I}d_i,10^{-4}\bigr)

4. wi←mie−di/s/∑jmje−dj/sw_i\gets m_i e^{-d_i/s}\big/\sum_jm_je^{-d_j/s}

5. P0←μi0i∈I,ΔT0→0GMP←I4P_0\gets\\{\boldsymbol\mu_i^0\\}_{i\in I},\quad \Delta\mathbf T_{0\to0}^{\mathrm{GMP}}\gets\mathbf I_4

6. for t=1,…,T do\textbf{for }t=1,\ldots,T\textbf{ do}

7. Pt←μ^iti∈I,ΔT0→tGMP←WeightedKabsch⁡(P0,Pt,w)\quad P_t\gets\\{\hat{\boldsymbol\mu}_i^t\\}_{i\in I},\quad \Delta\mathbf T_{0\to t}^{\mathrm{GMP}}\gets\operatorname{WeightedKabsch}(P_0,P_t,w)

8. (rt,ut,gt)←FitQuality⁡(P0,Pt,w,ΔT0→tGMP)\quad (r_t,u_t,g_t)\gets\operatorname{FitQuality}(P_0,P_t,w,\Delta\mathbf T_{0\to t}^{\mathrm{GMP}})

9. end for\textbf{end for}

10. for t=1,…,T do\textbf{for }t=1,\ldots,T\textbf{ do}

11. ΔTstep,t←(ΔT0→t−1GMP)−1ΔT0→tGMP,ΔTremain,t←(ΔT0→tGMP)−1ΔT0→TGMP\quad \Delta\mathbf T_{\mathrm{step},t}\gets (\Delta\mathbf T_{0\to t-1}^{\mathrm{GMP}})^{-1}\Delta\mathbf T_{0\to t}^{\mathrm{GMP}},\quad \Delta\mathbf T_{\mathrm{remain},t}\gets (\Delta\mathbf T_{0\to t}^{\mathrm{GMP}})^{-1}\Delta\mathbf T_{0\to T}^{\mathrm{GMP}}

12. ΔTdirect,t←WeightedKabsch⁡(Pt−1,Pt,w),qt←Discrepancy⁡(ΔTdirect,t,ΔTstep,t)\quad \Delta\mathbf T_{\mathrm{direct},t}\gets\operatorname{WeightedKabsch}(P_{t-1},P_t,w),\quad q_t\gets\operatorname{Discrepancy}(\Delta\mathbf T_{\mathrm{direct},t},\Delta\mathbf T_{\mathrm{step},t})

13. (ξtcum,ξtstep,ξtremain)←PolicyFrame6D⁡(ΔT0→tGMP,ΔTstep,t,ΔTremain,t)\quad (\xi_t^{\mathrm{cum}},\xi_t^{\mathrm{step}},\xi_t^{\mathrm{remain}})\gets \operatorname{PolicyFrame6D}(\Delta\mathbf T_{0\to t}^{\mathrm{GMP}}, \Delta\mathbf T_{\mathrm{step},t},\Delta\mathbf T_{\mathrm{remain},t})

14. ftGMP←Concat⁡(ξtcum,ξtstep,ξtremain,rt,ut,gt,qt)\quad \mathbf f_t^{\mathrm{GMP}}\gets\operatorname{Concat}( \xi_t^{\mathrm{cum}},\xi_t^{\mathrm{step}},\xi_t^{\mathrm{remain}},r_t,u_t,g_t,q_t)

15. end for\textbf{end for}

16. return m,M1:TGMP=ftGMPt=1T\textbf{return }m,\\ \mathcal M_{1:T}^{\mathrm{GMP}}=\\{\mathbf f_t^{\mathrm{GMP}}\\}_{t=1}^{T}


Thank you for correcting Section 4.2 and clarifying that the discrepancy came from reporting the real-robot results in the simulation paragraph. The added pseudocode and implementation details clarify how the gripper region is selected, how registration weights are computed, and how recovery reliability informs action refinement using only information available at deployment. These revisions address my two main concerns, so I am raising my rating from 2 to 3 (Weak acceptance).

Reviewer_zVPBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ARGaM, a Gaussian world-modeling framework for robotic manipulation that aims to make predicted scene motion useful for action generation. The method jointly predicts future Gaussian states and a coarse action trajectory, using visual reconstruction, action regression, and demonstration-derived gripper displacement supervision. It then recovers a geometric motion prior through reliability-weighted rigid registration and refines the coarse trajectory using residual diffusion conditioned on this prior, its reliability, and rendered future observations. Experiments report a mean success rate of 72.7% on nine RLBench tasks and 80.0% on six real-world tasks, compared with 60.4% and 58.3% for GAF, respectively.

Critical strengths
  1. The paper addresses a meaningful gap between visual prediction quality and reliable action generation. Persistent Gaussian identities do not necessarily imply physically accurate motion, making explicit action supervision a well-motivated approach to improving the usefulness of predicted geometry for control.

  2. The action-generation framework is well designed. Rather than directly executing geometrically recovered trajectories, it uses a regressed trajectory as initialization and conditions residual diffusion refinement on geometric motion and its reliability. This design leverages geometric information while retaining the flexibility to correct reconstruction and registration errors.

  3. The simulation and real-world results provide encouraging empirical evidence. ARGaM outperforms GAF on eight of nine simulation tasks, increasing mean success from 60.4% to 72.7%. Even excluding Close Fridge, which shows the largest gain, the average improvement remains eight percentage points. Real-world successes also increase from 35/60 to 48/60. All five rendering metrics improve across the four evaluated tasks, indicating that the control gains do not come at the expense of the reported visual prediction quality.

Critical weaknesses
  1. Table 5 reports mean success rates of 47.3%, 58.9%, 62.1%, and 72.7% as AGD, Init-A, and GMP are progressively added to the baseline. In contrast, the text accompanying Table 6 states that the action head, GMP conditioning, and residual refiner are held fixed while AGD and AGM are varied, yet reports the same success rates of 47.3%, 58.9%, and 72.7%. The recurrence of these values under apparently different configurations makes it difficult to attribute the improvements to individual components. Please specify the architecture and supervision used in each row and clarify the relationship between the two ablation studies.

  2. Section 4.1 refers readers to an appendix for hyperparameters, image resolutions, and execution details, but the provided PDF ends after the references and does not include an appendix. Moreover, the correspondence weights witw_i^t, geometric reliability descriptors, and reliability modulation mechanism are not defined precisely enough to support reproduction. Please provide the referenced appendix and clarify how the correspondence weights and reliability descriptors are computed, as well as how reliability modulates the geometric prior during action refinement.

Questions for authors

Please refer to the "Critical Weaknesses" for specific issues. If these can be resolved, I will raise the score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer zVPB

We thank the reviewer for your time and effort to review the manuscript. For clarity, we have reorganized the comments, with our point-by-point responses provided below. We have uploaded the revised manuscript and hope these responses address your concerns.

Weaknesses:

  1. Q1. specify the architecture and supervision used in each row and Q2. clarify the relationship between the two ablation studies.

  2. Q3. provide the referenced appendix and Q4. clarify how the 1) correspondence weights and 2) reliability descriptors are computed, Q5. how reliability modulates the geometric prior ...

A1. Thank you for the insightful comment. The original AGM row in Table 6 corresponds to the full GMP configuration in Table 5. We have adjusted AGM to GMP and added an Init-A row to Table 6 as follows in the revised manuscript to ensure consistent configuration naming across both tables.

Table 6. Action accuracy and task success.

MethodCoarse (mm) ↓Refined (mm) ↓Success (%) ↑
Base3.1132.27247.3
+AGD2.7942.05358.9
+Init-A2.7941.96762.1
+GMP2.6801.83272.7

A2. Tables 5 and 6 provide complementary evaluations of the same ablation. Table 5 focuses on task success, while Table 6 additionally reports coarse and refined action errors. AGD improves success from 47.3% to 58.9%, while Init-A further raises it to 62.1% by providing the coarse action for refinement. The full configuration additionally enables AGM supervision to align Gaussian motion with demonstrated gripper movements, allowing GMP to guide action refinement and achieving 72.7% success. Correspondingly, coarse and refined errors decrease to 2.680 and 1.832 mm. AGM supervision is disabled in the preceding configurations.

A3. We initially provided Supplementary Materials.pdf and have further revised it to include the complete hyperparameter settings and implementation details. We have also appended the supplementary material to the revised manuscript for convenient reference.

HyperparameterValue
Input resolution / prediction horizon128×128128\times128 / 8 steps
Batch size per GPU8
Stage 1 training80k iterations; LR: 2×10−52\times10^{-5}; visual feature LR multiplier: 0.1
Stage 2 training50k iterations; AdamW; LR: 10−410^{-4}; weight decay: 10−210^{-2}
Loss weightsλDSSIM=0.05\lambda_{\mathrm{DSSIM}}=0.05; λAGM=0.01\lambda_{\mathrm{AGM}}=0.01; λinit=0.01\lambda_{\mathrm{init}}=0.01
Temporal supervisionSteps: t+2,t+4,t+6,t+8t+2,t+4,t+6,t+8; weights: 0.4, 0.3, 0.2, 0.1
Minimum registration weight scale10−410^{-4}
Diffusion steps (training / inference)50 / 3 (DDIM)

A4. 1). Correspondences and registration weights. Given the space constraints, we include the core gripper-region selection and motion-recovery procedure as Algorithm 1 in Section 3.3 of the revised main paper. The complete pipeline pseudocode and detailed descriptor definitions are provided in the supplementary material. As shown in Algorithm 1, we select the MM Gaussian centroids closest to the initial gripper center and retain their indices across predicted frames to establish temporal correspondences. Their normalized distance-based weights are:

\quad s=\max(\min_{i\in I}d_i,10^{-4}),$$ where $d_i$ is the distance to the initial gripper center. These weights are reused for weighted Kabsch alignment. **2). Geometric reliability descriptors.** Four descriptors assess motion recovery quality. Let $\mathbf e_i^t=\mathbf R_t\boldsymbol\mu_i^0+\mathbf p_t-\widehat{\boldsymbol\mu}_i^t$ denote the registration residual, $\delta_i^t=\widehat{\boldsymbol\mu}_i^t-\boldsymbol\mu_i^0$ the Gaussian displacement, and $\sigma_1\geq\sigma_2$ the leading singular values of the weighted cross-covariance matrix $\mathbf H_t$. We compute: $$\begin{aligned} r_t&=\frac{\sum_{i\in I}w_i|\mathbf e_i^t|2} {\sum{i\in I}w_i|\delta_i^t|2+\epsilon}, &u_t&=\frac{1}{M\sum{i\in I}w_i^2}, &g_t&=\frac{\sigma_2(\mathbf H_t)} {\sigma_1(\mathbf H_t)+\epsilon}, &q_t&=|\operatorname{Log}(D_t^{-1}S_t)|_2. \end{aligned}$$ These descriptors measure registration accuracy, effective point support, geometric conditioning, and temporal consistency, respectively. Together with the three 6-D motion representations, they form the 22-D GMP token. **A5.** The residual diffusion refiner uses a learnable gate to regulate GMP's contribution. The encoded 22-D GMP feature $c_h$ is fused with the action embedding $a_h$ through: $$\alpha_h=\operatorname{sigmoid}\left(\operatorname{MLP}([a_h,c_h])\right),$$ $$a'_h=a_h+\alpha_h\cdot c_h.$$ This gate adaptively regulates geometric guidance according to the action context and encoded recovery quality. The fused feature is used to predict residual corrections to the coarse action trajectory.

Dear Reviewer zVPB,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

To clarify how ARGaM selects the gripper region and recovers the geometric motion prior, we include the pseudocode below.

Thank you for your time and effort in reviewing our paper.

 
 


Algorithm 1 Gripper region selection and geometric motion prior recovery


Input: G0\mathcal G_0, G^1:T\widehat{\mathcal G}_{1:T}, gripper pose X0X_0 in the Gaussian frame, point count MM, horizon TT.

Output: Selection mask mm and 22-D GMP token sequence M1:TGMP\mathcal M_{1:T}^{\mathrm{GMP}}.


1. μi0←Centroids⁡(G0),pgrip0←GripperCenter⁡(X0)\\{\boldsymbol\mu_i^0\\}\gets\operatorname{Centroids}(\mathcal G_0),\quad \mathbf p_{\mathrm{grip}}^0\gets\operatorname{GripperCenter}(X_0)

2. di←∥μi0−pgrip0∥2(∀i),I←SmallestIndices⁡(d,M)d_i\gets\lVert\boldsymbol\mu_i^0-\mathbf p_{\mathrm{grip}}^0\rVert_2\\ (\forall i),\quad I\gets\operatorname{SmallestIndices}(d,M)

3. mi←1i∈I,s←max⁡(min⁡i∈Idi,10−4)m_i\gets\mathbf{1}_{\\{i\in I\\}},\quad s\gets\max\bigl(\min_{i\in I}d_i,10^{-4}\bigr)

4. wi←mie−di/s/∑jmje−dj/sw_i\gets m_i e^{-d_i/s}\big/\sum_jm_je^{-d_j/s}

5. P0←μi0i∈I,ΔT0→0GMP←I4P_0\gets\\{\boldsymbol\mu_i^0\\}_{i\in I},\quad \Delta\mathbf T_{0\to0}^{\mathrm{GMP}}\gets\mathbf I_4

6. for t=1,…,T do\textbf{for }t=1,\ldots,T\textbf{ do}

7. Pt←μ^iti∈I,ΔT0→tGMP←WeightedKabsch⁡(P0,Pt,w)\quad P_t\gets\\{\hat{\boldsymbol\mu}_i^t\\}_{i\in I},\quad \Delta\mathbf T_{0\to t}^{\mathrm{GMP}}\gets\operatorname{WeightedKabsch}(P_0,P_t,w)

8. (rt,ut,gt)←FitQuality⁡(P0,Pt,w,ΔT0→tGMP)\quad (r_t,u_t,g_t)\gets\operatorname{FitQuality}(P_0,P_t,w,\Delta\mathbf T_{0\to t}^{\mathrm{GMP}})

9. end for\textbf{end for}

10. for t=1,…,T do\textbf{for }t=1,\ldots,T\textbf{ do}

11. ΔTstep,t←(ΔT0→t−1GMP)−1ΔT0→tGMP,ΔTremain,t←(ΔT0→tGMP)−1ΔT0→TGMP\quad \Delta\mathbf T_{\mathrm{step},t}\gets (\Delta\mathbf T_{0\to t-1}^{\mathrm{GMP}})^{-1}\Delta\mathbf T_{0\to t}^{\mathrm{GMP}},\quad \Delta\mathbf T_{\mathrm{remain},t}\gets (\Delta\mathbf T_{0\to t}^{\mathrm{GMP}})^{-1}\Delta\mathbf T_{0\to T}^{\mathrm{GMP}}

12. ΔTdirect,t←WeightedKabsch⁡(Pt−1,Pt,w),qt←Discrepancy⁡(ΔTdirect,t,ΔTstep,t)\quad \Delta\mathbf T_{\mathrm{direct},t}\gets\operatorname{WeightedKabsch}(P_{t-1},P_t,w),\quad q_t\gets\operatorname{Discrepancy}(\Delta\mathbf T_{\mathrm{direct},t},\Delta\mathbf T_{\mathrm{step},t})

13. (ξtcum,ξtstep,ξtremain)←PolicyFrame6D⁡(ΔT0→tGMP,ΔTstep,t,ΔTremain,t)\quad (\xi_t^{\mathrm{cum}},\xi_t^{\mathrm{step}},\xi_t^{\mathrm{remain}})\gets \operatorname{PolicyFrame6D}(\Delta\mathbf T_{0\to t}^{\mathrm{GMP}}, \Delta\mathbf T_{\mathrm{step},t},\Delta\mathbf T_{\mathrm{remain},t})

14. ftGMP←Concat⁡(ξtcum,ξtstep,ξtremain,rt,ut,gt,qt)\quad \mathbf f_t^{\mathrm{GMP}}\gets\operatorname{Concat}( \xi_t^{\mathrm{cum}},\xi_t^{\mathrm{step}},\xi_t^{\mathrm{remain}},r_t,u_t,g_t,q_t)

15. end for\textbf{end for}

16. return m,M1:TGMP=ftGMPt=1T\textbf{return }m,\\ \mathcal M_{1:T}^{\mathrm{GMP}}=\\{\mathbf f_t^{\mathrm{GMP}}\\}_{t=1}^{T}


Reviewer_zVPBOpenReview ↗

Thank authors for the rebuttal.

The author's reply addressed some of my concerns. So I decided to increase my rating.

Reviewer_kTXuOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies Gaussian world models more for robot action generation in manipulation tasks. The proposed method ARGaM predicts future 3D Gaussian states together with a coarse action trajectory. An additional motion loss AGM supervises the displacement of the gripper Gaussians using the demonstrated gripper pose. A residual diffusion model uses motion prior from the predicted gripper Gaussians, and a rendered future image to refine the coarse action. The method is validated on nine RLBench tasks, where it reaches height success rate compared with other baselines. And real world experiments also shows the effectivness of the method.

Critical strengths
  1. The paper studies a practical problem in world models. that visually accurate future predictions do not necessarily yield useful motion for control. To address this, the authors propose training the Gaussian motion with action targets and using the recovered motion to refine actions. This is a reasonable solution and the idea is clear.

  2. The experimental results are very promising, achieving higher success rates compared to other baselines. The real-world experiments are especially valuable, and this physical evidence makes the contribution much more convincing.

Critical weaknesses
  1. The results in the table and the main text are inconsistent. For example, Section 4.2 states that "Table 1 reports that ARGaM achieves the best overall performance among all baseline methods, with an average success rate of 80.0%," whereas Table 1 reports an average success rate of 72.7% for ARGaM.

  2. Minor: 2.1. The ablation in Table 4 is conducted on only one task; it would be useful to conduct it on more tasks. 2.2. In Table 5, why does performance on Phone B not show substantial improvement? It would be helpful to include a discussion on the specific conditions under which the proposed method provides the largest gains.

Questions for authors
  1. Please correct the success rate numbers in Table 1 or Section 4.2 to make them consistent. Section 4.2 states that "Table 1 reports that ARGaM achieves the best overall performance among all baseline methods, with an average success rate of 80.0%," but Table 1 reports 72.7%.

  2. Since Table 4 is conducted on only one task, could you evaluate this across more tasks?

  3. (minor:) Are the reported results averaged across multiple random seeds? If so, could you provide the standard deviations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer kTXu

We thank the reviewer for the encouraging feedback and helpful suggestions. For clarity, we have organized and renumbered the questions. We have uploaded the revised manuscript and hope our responses address the concerns.

Weaknesses:

  1. Q1. The results are inconsistent.
  2. Q2. The ablation study on more tasks. Q3. Table 5, performance on Phone B not show substantial improvement. Q4 Include a discussion on the specific conditions under which the proposed method provides the largest gains.

Questions:

  1. Q1. Correct Table 1 or Section 4.2 to make them consistent.
  2. Q2. evaluate ablation study across more tasks.
  3. Q5. 1). results across multiple random seeds? 2). provide the standard deviations?

A1. Thank you for pointing out this inconsistency and we apologize for this mistake. The first two sentences of Section 4.2 mistakenly used real-robot results from Table 3 instead of RLBench simulation results from Table 1. The abstract and Table 1 are correct. We have corrected Section 4.2 as follows:

Table 1 reports closed-loop manipulation success rates on nine RLBench tasks. ARGaM achieves the highest mean success rate of 72.7%, outperforming ACT, ManiGaussian, Act3D, and GAF by 27.9, 22.6, 19.6, and 12.3 percentage points, respectively. Compared with the strongest baseline, GAF, ARGaM achieves higher success on eight of nine tasks, with the largest improvement on Close Fridge, from 42% to 88%. On Lift Lid Up, both methods perform near the ceiling, with ARGaM and GAF achieving 98% and 100%, respectively. These results support ARGaM’s overall effectiveness across the evaluated manipulation tasks. These results highlight ARGaM’s advantage on articulated-object manipulation tasks, particularly those that require the gripper to guide an object along a constrained path.

A2. Thank you for the insightful comment.We conducted additional ablations on two tasks (as follows) and updated Section 4.5 and Table 4. The results further demonstrate the complementary benefits of D-SSIM and AGM supervision.

Close Fridge

MethodAGMD-SSIMPSNR ↑SSIM ↑LPIPS ↓DISTS ↓FSIM ↑Success (%) ↑
Baseline××18.100.550.420.320.7670
+ D-SSIM supervision×✓18.510.620.400.290.7972
+ AGM supervision✓×19.330.570.370.310.7777
Full model✓✓20.420.650.320.270.8088

Close Grill

MethodAGMD-SSIMPSNR ↑SSIM ↑LPIPS ↓DISTS ↓FSIM ↑Success (%) ↑
Baseline××17.210.450.380.350.6550
+ D-SSIM supervision×✓18.030.540.350.320.6956
+ AGM supervision✓×18.510.500.340.330.6757
Full model✓✓19.360.570.290.300.6960

A3. Thank you for raising this point. We agree that the improvement on Phone on Base is relatively modest. This task is particularly sensitive to the initial grasp position, as even a slightly off-center grasp can cause the handset to tilt during transport, which subsequent motion refinement cannot fully compensate for. Nevertheless, ARGaM improves the success rate from 35% to 43% over GAF (Table 1), while the ablation in Table 5 shows that GMP provides an additional 4 percentage points of improvement (39% to 43%). These results suggest that geometric motion recovery provides a useful benefit, although its impact remains constrained by the accuracy of the initial grasp.

A4. Thank you for this helpful suggestion. ARGaM provides the largest gains on tasks requiring sustained gripper motion along geometrically constrained trajectories, particularly when object motion involves continuous rotation or translation. In such cases, temporal correspondences among gripper Gaussians provide useful geometric motion cues for refining coarse action predictions. Close Fridge is a representative example, where the gripper must follow the door's hinge-constrained rotation. Adding GMP improves the success rate from 68% to 88% (Table 5). Moreover, ARGaM maintains higher success rates and rendering PSNR than GAF at longer prediction horizons (Fig. 6(c–d)), further demonstrating the benefit of geometry-aware motion recovery for long-horizon manipulation.

A5. 1). In fact, our results are averaged over three random seeds. 2). The table below gives the standard deviation of the success rate for each task and the low standard deviations, below 3.3 pp, indicate consistent performance across random seeds.

MetricToi.Open G.Close G.Close F.Pho.Lift L.Mic.But.Lap.
Std. (pp)2.162.723.271.412.500.941.702.623.09

Dear Reviewer kTXu,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

I thank the authors for their detailed response. The rebuttal addresses most of my concerns. I am happy to raise my score from 3 (Weak acceptance) to 4 (Accept).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.