- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GeoIDM, an inverse dynamics model that uses action-reprojected geometric supervision with VGGT features. Experiments evaluate the action prediction accuracy and the reference-video replay in simulation and the real world.
Critical strengths
The geometric objective directly links predicted end-effector motion to visual correspondences without requiring an additional geometric estimation stage at inference. Improvements on unseen tasks in LIBERO and RoboTwin provide encouraging evidence of better task generalization.
Critical weaknesses
The novelty is mainly in the geometric training objective. However, ablation studies do not establish the advantage of the geometric training objective over simpler reprojection or feature-matching losses.
Questions for authors
Under the same backbone and training setup, how does the proposed objective compare with direct 2D reprojection supervision and a simple cosine-similarity loss? Could the authors report unified endpoint and trajectory errors across methods, and test whether the improvements transfer to videos produced by a video generation model?
4 comments
Response (2/3): Q (part 2) unified evaluation protocol
Q (part 2). Unified error convention across methods.
In the original Table 1, each method was scored in its native action space. We agree that the two conventions are not directly comparable, so we now report both the 16-step mean error and the endpoint error at step 16 for all methods (Table R4). Each method only needs its missing metric; the other is the original Table 1 number. Below, and are the ground-truth and predicted end-effector positions at step of a chunk (rotation is computed analogously with the geodesic distance).
- Absolute actions (RoboDreamer, Vidar, AnyPos, DreamGen) predict the target pose at every step, so Table 1 reported the 16-step mean. We add the endpoint error, which is close to their 16-step mean.
- Endpoint:
- 16-step mean:
- Delta actions (VERA, GeoIDM) predict per-step increments that are accumulated during execution, so Table 1 reported the error of the accumulated endpoint. We add the 16-step mean, but it is not comparable with the absolute-action mean: the trajectory is accumulated from the ground-truth pose at the start of the chunk, so early steps contain only a few accumulated errors.
- Endpoint: , with
- 16-step mean:
The endpoint error is therefore the comparable metric across action representations.
Table R4. AgiBot action prediction under the unified protocol (absolute trajectories).
(a) Error averaged over 16 steps.
| Method | Native repr. | ID Pos. (mm) ↓ | ID Rot. (deg) ↓ | ID Grip ↓ | OOD Pos. (mm) ↓ | OOD Rot. (deg) ↓ | OOD Grip ↓ |
|---|---|---|---|---|---|---|---|
| RoboDreamer | absolute | 53.04 | 5.77 | 0.121 | 101.71 | 13.53 | 0.151 |
| Vidar | absolute | 62.10 | 4.58 | 0.094 | 88.97 | 11.49 | 0.124 |
| AnyPos | absolute | 49.34 | 4.42 | 0.087 | 90.20 | 10.63 | 0.119 |
| VERA | delta | 60.19 | 6.75 | 0.083 | 92.65 | 10.90 | 0.095 |
| DreamGen | absolute | 53.70 | 4.16 | 0.028 | 105.96 | 8.34 | 0.034 |
| GeoIDM | delta | 12.39 | 2.12 | 0.035 | 64.78 | 2.52 | 0.046 |
(b) Endpoint error at step 16.
| Method | Native repr. | ID Pos. (mm) ↓ | ID Rot. (deg) ↓ | ID Grip ↓ | OOD Pos. (mm) ↓ | OOD Rot. (deg) ↓ | OOD Grip ↓ |
|---|---|---|---|---|---|---|---|
| RoboDreamer | absolute | 53.02 | 5.54 | 0.109 | 101.87 | 13.52 | 0.130 |
| Vidar | absolute | 64.09 | 4.50 | 0.102 | 87.04 | 11.56 | 0.108 |
| AnyPos | absolute | 48.84 | 4.45 | 0.094 | 91.92 | 10.80 | 0.111 |
| VERA | delta | 71.11 | 7.75 | 0.102 | 94.64 | 16.61 | 0.121 |
| DreamGen | absolute | 53.52 | 4.34 | 0.047 | 104.26 | 7.63 | 0.043 |
| GeoIDM | delta | 19.39 | 3.16 | 0.115 | 74.95 | 3.58 | 0.080 |
Result. Under the endpoint error, GeoIDM has the lowest position and rotation errors on both splits. Its position-error reduction over DreamGen is 63.8% / 28.1% (ID / OOD), essentially unchanged from 63.9% / 29.3% in the original Table 1.
Response (3/3): Q (part 3) generated videos; overview of revisions
Q (part 3). Transfer to videos produced by a video generation model.
Our paper used recorded videos to isolate the IDM; here we test it on generated videos.
Setup.
- Reference videos are generated by Wan2.1-I2V-14B-720P fine-tuned on LIBERO-90.
- 50 episodes per split (10 tasks × 5 episodes). All IDMs receive the same generated videos from the same initial states and are executed open-loop as in Table 2. Recorded-video results are re-computed on the same episodes.
- Whether each generated video itself completes the task was annotated manually, before looking at policy outcomes: 48 / 50 (ID) and 49 / 50 (OOD).
Table R5. LIBERO replay: successful episodes (out of 50) with recorded vs. generated reference videos (retention = generated / recorded).
| Method | ID recorded | ID generated | ID retention | OOD recorded | OOD generated | OOD retention |
|---|---|---|---|---|---|---|
| AnyPos | 37 | 34 | 91.9% | 10 | 7 | 70.0% |
| DreamGen | 35 | 34 | 97.1% | 11 | 9 | 81.8% |
| GeoIDM | 39 | 38 | 97.4% | 25 | 21 | 84.0% |
Findings.
- Since almost all generated videos complete the task, the drop from recorded to generated videos reflects each IDM's sensitivity to generation artifacts.
- On ID tasks, all methods are comparable (34–38 successes). On OOD tasks, GeoIDM remains clearly the strongest (21 vs. 9 successes for DreamGen), and its retention (84.0%) is comparable to or higher than the baselines' (70.0–81.8%). Grounding actions in geometric correspondence therefore does not overfit to clean recorded frames.
Limitation. Full closed-loop integration with a video planner (including replanning) is beyond the scope of this IDM-focused study.
Overview of revisions. Revised text in the updated PDF is marked in blue.
- W1 / Q (part 1): gradient analysis (Sec. 3.3); comparison with direct 2D reprojection and positive-only cosine (Sec. 4.3, Appendix E).
- Q (part 2): Table 1 now reports endpoint errors for all methods with a native-representation column (Sec. 4.1, 4.2.1, abstract); 16-step mean errors are in Appendix C.
- Q (part 3): generated-video evaluation (Appendix F); closed-loop limitation (Limitations).
We hope these responses and the additional results address the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating the score. We are glad to provide any further clarification.
Dear Reviewer, thank you again for your thoughtful review. We would appreciate it if you could let us know whether our rebuttal resolves your concerns before the discussion period closes. We are glad to provide further clarification if needed. Thank you!
Response (1/3): Summary; W1 comparison with simpler losses
We thank Reviewer yQUx for the review and the helpful questions.
Summary of additional results.
W1 / Q (part 1). Advantage over simpler reprojection or feature-matching losses.
Setup. Same setting as Table 3 (balanced 50% AgiBot subset; same backbone, feature differencing, reliability gate, and training budget). We only replace the local InfoNCE in Eq. (4) with:
Table R1. Geometric objectives (AgiBot, Table-3 setting).
Local InfoNCE gives the lowest position error on both splits.
How the negatives act: an adaptive per-sample weight. The reviewer is right that, with frozen features and stop-gradient (Eq. 3), the gradient of local InfoNCE has the same direction as that of positive-only cosine: ∇u^ℓNCE=k,∇u^ℓcos, with k=(1−p+)/τ and p+ the softmax probability of the positive. The negatives therefore act through the per-sample weight k. We measured k on the LIBERO validation set (≈54k samples), with the same hyperparameters as in the paper:
Table R2. Statistics of the per-sample weight k (LIBERO validation set).
Weight sweep: positive-only cosine vs. local InfoNCE. Since E[k]≈10, our λgeo=0.1 corresponds to λcos≈1.0. We swept around this value:
Table R3. Weight sweep for positive-only cosine (AgiBot, Table-3 setting; position error in mm).
Analysis of direct 2D reprojection.