- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ARGaM, which autoregressively predicts future Gaussian scene states and coarse action trajectories, with demonstration-derived supervision of gripper-region Gaussian displacements. Weighted rigid registration recovers a geometric motion prior from predicted Gaussian correspondences. A residual diffusion model then combines this prior, its recovery reliability, the coarse actions, and predicted visual context to refine the trajectory. Evaluations cover nine RLBench tasks and six real-robot tasks, with ablations examining motion supervision, temporal modeling, and geometric motion guidance.
Critical strengths
- A useful connection between geometric motion supervision and action generation. Demonstrated actions directly constrain the local Gaussian displacements used for motion recovery, making predicted geometry informative for action refinement.
- Reported gains and ablations provide positive evidence. Simulation and real-robot tables show substantial improvements over baselines, while ablations support local motion supervision and geometric guidance.
Critical weaknesses
- Section 4.2 contains inconsistent simulation results. The “Closed-loop manipulation” paragraph describes Table 1 using results matching the real-world Table 3, contradicting the abstract and simulation Table 1. The correct results and the cause of this mismatch require clarification.
- Geometric recovery and reliability modulation are insufficiently specified. The gripper-mask source, registration-weight computation, and reliability-based fusion mechanism remain unclear, limiting assessment of the core pipeline’s completeness and reproducibility.
Questions for authors
- Please explain the discrepancy between Section 4.2, the abstract, and Table 1, and verify the correct correspondence between simulation and real-robot results against the experimental records. Do any other results contain similar mismatches?
- How are gripper masks obtained during training and deployment, registration weights computed, and recovery reliability used to modulate GMP’s contribution to action refinement? Do these steps rely only on information available at deployment?
3 comments
Dear Reviewer dbWZ,
I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.
To clarify how ARGaM selects the gripper region and recovers the geometric motion prior, we include the pseudocode below.
Thank you for your time and effort in reviewing our paper.
Algorithm 1 Gripper region selection and geometric motion prior recovery
Input: , , gripper pose in the Gaussian frame, point count , horizon .
Output: Selection mask and 22-D GMP token sequence .
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
Thank you for correcting Section 4.2 and clarifying that the discrepancy came from reporting the real-robot results in the simulation paragraph. The added pseudocode and implementation details clarify how the gripper region is selected, how registration weights are computed, and how recovery reliability informs action refinement using only information available at deployment. These revisions address my two main concerns, so I am raising my rating from 2 to 3 (Weak acceptance).
Response to Reviewer dbWZ
We thank the reviewer for recognizing the value of our geometric motion supervision and its empirical benefits. For clarity, we have organized and numbered the concerns below. We have uploaded the revised manuscript and hope to address the concerns raised.
Weaknesses:
Questions
A1. 1). Thank you for identifying this discrepancy. The first two sentences of Section 4.2 mistakenly reported the mean success rate and percentage-point differences from the real-robot experiments in Table 3, instead of the RLBench simulation results in Table 1. The abstract and Table 1 report the correct simulation results. ARGaM achieves a mean success rate of 72.7%, compared with 60.4% for GAF, the strongest baseline, yielding a 12.3 percentage-point improvement.
We have corrected Section 4.2 as follows:
Table 1 reports closed-loop manipulation success rates on nine RLBench tasks. ARGaM achieves the highest mean success rate of 72.7%, outperforming ACT, ManiGaussian, Act3D, and GAF by 27.9, 22.6, 19.6, and 12.3 percentage points, respectively. Compared with GAF, the strongest baseline, ARGaM achieves higher success on eight of nine tasks, with the largest improvement on Close Fridge (42% to 88%). On Lift Lid Up, both methods perform near the ceiling, with ARGaM and GAF achieving 98% and 100%, respectively. These results demonstrate ARGaM's effectiveness across the evaluated manipulation tasks.
2). We also rechecked the numerical results throughout the manuscript and found no other numerical mismatches.
A2. To explain the gripper selection, registration, and reliability-guided refinement clearly, we have added Algorithm 1 and its accompanying explanation to Section 3.3 of the revised manuscript.
1).Gripper mask. Algorithm 1 (Lines 1–3) selects the M Gaussian centroids closest to the initial gripper center. During training, the current demonstrated gripper pose is transformed into the Gaussian reference frame. At deployment, the same procedure uses the current gripper pose. Future demonstrated gripper poses are used only to construct AGM displacement targets.
2). Registration weights. Algorithm 1 (Line 4) computes normalized distance-based weights:
wi=∑jmjexp(−dj/s)miexp(−di/s),s=max(mini∈Idi,10−4),
where di is the distance to the initial gripper center. Closer Gaussians receive higher weights. These weights are reused for weighted Kabsch alignment across predicted frames.
3). Reliability-guided refinement. Algorithm 1 (Lines 8 and 11–14) computes four recovery-quality descriptors: normalized registration residual (rt), effective point support (ut), covariance singular-value ratio (gt), and adjacent-frame motion discrepancy (qt). Together with cumulative, step-wise, and remaining motion, they form the 22-D GMP token.
As described in Section 3.4, the token is encoded into ch and fused with the action embedding ah through a learnable gate:
\operatorname{sigmoid} \left(\operatorname{MLP}([a_h,c_h])\right),$$ $$a'_h=a_h+\alpha_h\cdot c_h.$$ This gate adaptively regulates the geometric prior's contribution according to the action context and encoded recovery quality. The fused feature is used to predict residual corrections to the coarse action trajectory. We have also added the explicit mathematical definitions of the four reliability descriptors and the complete pipeline pseudocode to the supplementary material. We will release the full source code to facilitate reproducibility. **A3.** Yes. All operations rely exclusively on information available at deployment, including: Current stereo images and camera calibration. Current gripper pose in the Gaussian reference frame. Future Gaussian states predicted by the world dynamics model. The model-predicted coarse action trajectory. The selection mask, registration weights, motion transformations, and reliability descriptors are computed from these inputs and predicted Gaussian states. Future demonstrated gripper poses and ground-truth future actions are used only for training supervision, never during deployment.