acceptodds
Under review as a conference paper at ICLR 2027

GeoIDM: Action-Reprojected Geometric Supervision for Generalizable Inverse Dynamics

Abstract

Video-based robot policies can decouple visual planning from control by translating predicted future observations into actions with an inverse dynamics model (IDM). However, IDMs supervised solely by absolute end-effector poses and gripper states can overfit appearance cues and generalize poorly to unseen trajectories and tasks. We present GeoIDM, a Geometry-supervised IDM that grounds action prediction in temporally aligned visual correspondences. A frozen VGGT encoder extracts visual tokens with geometric information from current and future observations, and a lightweight query compressor conditions an action DiT. During training, GeoIDM applies action-reprojected geometric supervision by accumulating predicted translations to estimate the future end-effector position, reprojecting it into the future view using camera calibration, and aligning the sampled feature with the corresponding feature in the current frame through a local contrastive objective. A gate suppresses static or invalid projections. The geometric branch is removed at inference, so the model still maps visual observations to actions without an additional geometry module. We evaluate GeoIDM through offline action prediction on AgiBot, replay of reference videos in LIBERO and RoboTwin 2.0, and replay of opening, clicking, and grasping tasks on a physical Franka robot. Relative to the DreamGen baseline, GeoIDM reduces position error by 63.9% and 29.3% on the cross-episode and cross-task AgiBot splits, respectively. It further improves cross-task replay success by 20.5 and 18.0 percentage points on LIBERO and RoboTwin 2.0, respectively. These results show that geometric supervision during training improves IDM generalization while preserving the standard interface at inference.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Inverse Dynamics Models, Geometric Supervision, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yQUxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GeoIDM, an inverse dynamics model that uses action-reprojected geometric supervision with VGGT features. Experiments evaluate the action prediction accuracy and the reference-video replay in simulation and the real world.

Critical strengths

The geometric objective directly links predicted end-effector motion to visual correspondences without requiring an additional geometric estimation stage at inference. Improvements on unseen tasks in LIBERO and RoboTwin provide encouraging evidence of better task generalization.

Critical weaknesses

The novelty is mainly in the geometric training objective. However, ablation studies do not establish the advantage of the geometric training objective over simpler reprojection or feature-matching losses.

Questions for authors

Under the same backbone and training setup, how does the proposed objective compare with direct 2D reprojection supervision and a simple cosine-similarity loss? Could the authors report unified endpoint and trajectory errors across methods, and test whether the improvements transfer to videos produced by a video generation model?

4 comments
AuthorsOpenReview ↗

Response (1/3): Summary; W1 comparison with simpler losses

We thank Reviewer yQUx for the review and the helpful questions.

Summary of additional results.

  • W1 / Q (part 1): local InfoNCE gives lower position error than positive-only cosine (at every weight we tried) and direct 2D reprojection on both splits (Tables R1–R3).
  • Q (part 2): all methods are now reported under both the 16-step mean and the endpoint error (Table R4); under the comparable endpoint error, GeoIDM still has the lowest position and rotation errors, and its reduction over DreamGen is essentially unchanged (63.8% / 28.1%, ID / OOD).
  • Q (part 3): with generated videos, GeoIDM succeeds on 21 / 50 OOD episodes vs. 9 / 50 for DreamGen (Table R5).

W1 / Q (part 1). Advantage over simpler reprojection or feature-matching losses.

Setup. Same setting as Table 3 (balanced 50% AgiBot subset; same backbone, feature differencing, reliability gate, and training budget). We only replace the local InfoNCE in Eq. (4) with:

  • Positive-only cosine: ℓcos⁡=1−zanc⊤z+\ell_{\cos} = 1 - \mathbf{z}_{\rm anc}^\top \mathbf{z}_{+};
  • Direct 2D reprojection: ℓ2D=∣u^t+H−ut+Hgt∣1\ell_{\rm 2D} = \\|\hat{\mathbf{u}}_{t+H} - \mathbf{u}^{\rm gt}_{t+H}\\|_1, with the same weight as ours (λ=0.1\lambda = 0.1).

Table R1. Geometric objectives (AgiBot, Table-3 setting).

ObjectiveID Pos. (mm) ↓ID Rot. (deg) ↓ID Grip ↓OOD Pos. (mm) ↓OOD Rot. (deg) ↓OOD Grip ↓
No Lgeo\mathcal{L}_{\rm geo} (= "+ Feature diff." in Table 3)21.832.150.09974.334.030.069
Direct 2D reprojection21.022.640.08387.543.660.065
Positive-only cosine (λcos⁡\lambda_{\cos} = 1.0, matched weight; sweep in Table R3)16.623.450.08868.983.410.071
Local InfoNCE (ours, λgeo\lambda_{\rm geo} = 0.1)14.602.700.10961.094.100.068

Local InfoNCE gives the lowest position error on both splits.

How the negatives act: an adaptive per-sample weight. The reviewer is right that, with frozen features and stop-gradient (Eq. 3), the gradient of local InfoNCE has the same direction as that of positive-only cosine: ∇u^ℓNCE=k,∇u^ℓcos⁡\nabla_{\hat{\mathbf u}}\ell_{\rm NCE} = k\\,\nabla_{\hat{\mathbf u}}\ell_{\cos}, with k=(1−p+)/τk = (1-p_+)/\tau and p+p_+ the softmax probability of the positive. The negatives therefore act through the per-sample weight kk. We measured kk on the LIBERO validation set (≈54k samples), with the same hyperparameters as in the paper:

Table R2. Statistics of the per-sample weight kk (LIBERO validation set).

StatisticValue
(a) Coefficient of variation of kk0.35
(b) Angle between the batch-level parameter gradients of the two losses, mean / max15° / 27°
(c) Mean kk as the 3D displacement error grows from 0 to 80 mm9.27 → 12.75
(d) Mean kk as the end-effector motion grows from < 20 mm to > 120 mm6.75 → 11.22
  • kk is far from constant (a, b). The two losses update the model in different directions, so a global weight on positive-only cosine cannot reproduce local InfoNCE.
  • kk varies systematically (c, d). Hard samples receive up to 38% more weight, so the loss fades as predictions improve. Nearly static samples, which carry little geometric signal, receive less weight; this complements the motion term of the gate.

Weight sweep: positive-only cosine vs. local InfoNCE. Since E[k]≈10\mathbb{E}[k] \approx 10, our λgeo=0.1\lambda_{\rm geo} = 0.1 corresponds to λcos⁡≈1.0\lambda_{\cos} \approx 1.0. We swept around this value:

Table R3. Weight sweep for positive-only cosine (AgiBot, Table-3 setting; position error in mm).

λcos⁡\lambda_{\cos} = 0.10.31.03.0Local InfoNCE (λgeo\lambda_{\rm geo} = 0.1)
ID Pos. ↓20.9520.0616.6215.3314.60
OOD Pos. ↓76.7870.4168.9878.1461.09
  • A larger weight trades OOD for ID: λcos⁡\lambda_{\cos} = 3.0 comes closest on ID (15.33 vs. 14.60 mm), but its OOD error (78.14 mm) is worse than using no geometric loss (74.33 mm).
  • The best OOD weight (λcos⁡\lambda_{\cos} = 1.0) still trails by 7.89 mm (68.98 vs. 61.09 mm).

Analysis of direct 2D reprojection.

  • In theory, it adds no new information. Its target ut+Hgt\mathbf u^{\rm gt}_{t+H} is the ground-truth 3D position projected through fixed, known cameras, i.e., it comes from the same 3D labels that already supervise the cumulative displacement loss Lcum\mathcal{L}_{\rm cum}. The target of Lgeo\mathcal{L}_{\rm geo} instead comes from the future observation: the predicted point must land on content that matches the anchor, which Lcum\mathcal{L}_{\rm cum} cannot provide.
  • The results agree (Table R1). Almost no gain on ID (21.83 → 21.02 mm) and a clear loss on OOD (74.33 → 87.54 mm), while local InfoNCE reaches 61.09 mm. The benefit of Lgeo\mathcal{L}_{\rm geo} comes from visual grounding, not from extra coordinate supervision.
AuthorsOpenReview ↗

Response (2/3): Q (part 2) unified evaluation protocol

Q (part 2). Unified error convention across methods.

In the original Table 1, each method was scored in its native action space. We agree that the two conventions are not directly comparable, so we now report both the 16-step mean error and the endpoint error at step 16 for all methods (Table R4). Each method only needs its missing metric; the other is the original Table 1 number. Below, pkp_k and p^k\hat p_k are the ground-truth and predicted end-effector positions at step kk of a chunk (rotation is computed analogously with the geodesic distance).

  • Absolute actions (RoboDreamer, Vidar, AnyPos, DreamGen) predict the target pose at every step, so Table 1 reported the 16-step mean. We add the endpoint error, which is close to their 16-step mean.
    • Endpoint: Eend=∣p^16−p16∣E_{\rm end} = \\| \hat p_{16} - p_{16} \\|
    • 16-step mean: Emean=116∑k=116∣p^k−pk∣E_{\rm mean} = \frac{1}{16}\sum_{k=1}^{16} \\| \hat p_k - p_k \\|
  • Delta actions (VERA, GeoIDM) predict per-step increments Δp^i\Delta \hat p_i that are accumulated during execution, so Table 1 reported the error of the accumulated endpoint. We add the 16-step mean, but it is not comparable with the absolute-action mean: the trajectory is accumulated from the ground-truth pose at the start of the chunk, so early steps contain only a few accumulated errors.
    • Endpoint: Eend=∣∑i=116ei∣E_{\rm end} = \\| \sum_{i=1}^{16} e_i \\|, with ei=Δp^i−Δpie_i = \Delta \hat p_i - \Delta p_i
    • 16-step mean: Emean=116∑k=116∣∑i=1kei∣E_{\rm mean} = \frac{1}{16}\sum_{k=1}^{16} \\| \sum_{i=1}^{k} e_i \\|

The endpoint error is therefore the comparable metric across action representations.

Table R4. AgiBot action prediction under the unified protocol (absolute trajectories).

(a) Error averaged over 16 steps.

MethodNative repr.ID Pos. (mm) ↓ID Rot. (deg) ↓ID Grip ↓OOD Pos. (mm) ↓OOD Rot. (deg) ↓OOD Grip ↓
RoboDreamerabsolute53.045.770.121101.7113.530.151
Vidarabsolute62.104.580.09488.9711.490.124
AnyPosabsolute49.344.420.08790.2010.630.119
VERAdelta60.196.750.08392.6510.900.095
DreamGenabsolute53.704.160.028105.968.340.034
GeoIDMdelta12.392.120.03564.782.520.046

(b) Endpoint error at step 16.

MethodNative repr.ID Pos. (mm) ↓ID Rot. (deg) ↓ID Grip ↓OOD Pos. (mm) ↓OOD Rot. (deg) ↓OOD Grip ↓
RoboDreamerabsolute53.025.540.109101.8713.520.130
Vidarabsolute64.094.500.10287.0411.560.108
AnyPosabsolute48.844.450.09491.9210.800.111
VERAdelta71.117.750.10294.6416.610.121
DreamGenabsolute53.524.340.047104.267.630.043
GeoIDMdelta19.393.160.11574.953.580.080

Result. Under the endpoint error, GeoIDM has the lowest position and rotation errors on both splits. Its position-error reduction over DreamGen is 63.8% / 28.1% (ID / OOD), essentially unchanged from 63.9% / 29.3% in the original Table 1.

AuthorsOpenReview ↗

Response (3/3): Q (part 3) generated videos; overview of revisions

Q (part 3). Transfer to videos produced by a video generation model.

Our paper used recorded videos to isolate the IDM; here we test it on generated videos.

Setup.

  • Reference videos are generated by Wan2.1-I2V-14B-720P fine-tuned on LIBERO-90.
  • 50 episodes per split (10 tasks × 5 episodes). All IDMs receive the same generated videos from the same initial states and are executed open-loop as in Table 2. Recorded-video results are re-computed on the same episodes.
  • Whether each generated video itself completes the task was annotated manually, before looking at policy outcomes: 48 / 50 (ID) and 49 / 50 (OOD).

Table R5. LIBERO replay: successful episodes (out of 50) with recorded vs. generated reference videos (retention = generated / recorded).

MethodID recordedID generatedID retentionOOD recordedOOD generatedOOD retention
AnyPos373491.9%10770.0%
DreamGen353497.1%11981.8%
GeoIDM393897.4%252184.0%

Findings.

  • Since almost all generated videos complete the task, the drop from recorded to generated videos reflects each IDM's sensitivity to generation artifacts.
  • On ID tasks, all methods are comparable (34–38 successes). On OOD tasks, GeoIDM remains clearly the strongest (21 vs. 9 successes for DreamGen), and its retention (84.0%) is comparable to or higher than the baselines' (70.0–81.8%). Grounding actions in geometric correspondence therefore does not overfit to clean recorded frames.

Limitation. Full closed-loop integration with a video planner (including replanning) is beyond the scope of this IDM-focused study.

Overview of revisions. Revised text in the updated PDF is marked in blue.

  • W1 / Q (part 1): gradient analysis (Sec. 3.3); comparison with direct 2D reprojection and positive-only cosine (Sec. 4.3, Appendix E).
  • Q (part 2): Table 1 now reports endpoint errors for all methods with a native-representation column (Sec. 4.1, 4.2.1, abstract); 16-step mean errors are in Appendix C.
  • Q (part 3): generated-video evaluation (Appendix F); closed-loop limitation (Limitations).

We hope these responses and the additional results address the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating the score. We are glad to provide any further clarification.

Dear Reviewer, thank you again for your thoughtful review. We would appreciate it if you could let us know whether our rebuttal resolves your concerns before the discussion period closes. We are glad to provide further clarification if needed. Thank you!

Reviewer_35nHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GeoIDM to recover actions from current and future video observations and improve performance on unseen tasks. A frozen VGGT encoder processes both observations, and compressed feature differences condition an action DiT. During training, predicted translation increments are accumulated into a future end-effector position and reprojected into the future image using camera calibration. A reliability-gated local contrastive loss is then computed on frozen dense features. The geometric supervision branch is removed at inference. Experiments cover action errors on AgiBot, reference-video-guided open-loop replay on LIBERO and RoboTwin 2.0, and a physical ablation on Franka.

Critical strengths
  1. GeoIDM connects action prediction with visual geometric correspondences through reprojection and differentiable feature sampling. Gradients from the geometric loss propagate through this path to the action model, encouraging consistency between predicted motion and observed visual changes. The geometric supervision branch is used only during training, while inference directly maps current and future images to actions.
  2. The simulation experiments provide encouraging evidence of generalization to unseen tasks. Section 4.1 defines in-domain (ID) and out-of-domain (OOD) evaluation by task identity. Table 2 compares methods with the same reference videos, paired initial states, action chunks, and controller settings. GeoIDM achieves 48.3% success on OOD LIBERO tasks, a 20.5 percentage-point gain over DreamGen's 27.8%. It also achieves 18.0% on OOD RoboTwin 2.0 tasks, exceeding the other learned methods reported.
  3. The component ablations and failure analysis help explain the method's benefits and limitations. Section 4.3 shows that ungated geometric supervision increases position error from 74.33 to 77.25 mm, whereas gating reduces it to 61.09 mm, with some loss of the improvements in rotation and gripper errors. In the physical ablation in Table 4, adding geometric supervision increases successes from 1/30 to 6/30. The discussion of contact offsets and gripper timing also provides concrete insight into execution failures.
Critical weaknesses
  1. Section 4.2.1 distinguish 16-step endpoint errors for delta actions from horizon-averaged errors for absolute actions, but Table 1 does not identify which convention each method uses. These quantities measure different aspects of trajectory error, which affects how the position-accuracy gains over DreamGen should be interpreted.
  2. Table 3 supports the position-prediction benefit of gated geometric supervision, but its effect on unseen tasks remains unclear. With action and feature differencing fixed, position error falls from 74.33 to 61.09 mm between rows 2 and 4, an improvement of approximately 17.8%. However, the table does not specify whether the results are from ID tasks, OOD tasks, or a combination of both.
  3. The added benefit of local negatives over simple feature matching lacks direct validation. Table 3 compares training with and without the geometric loss, but does not compare local InfoNCE with positive-only cosine matching. Given the frozen features and stop-gradient on negative sampling locations in Eq. (3), negatives influence coordinate optimization mainly through gradient magnitude and sample weighting. Appendix B's feature-similarity analysis illustrates the difficulty of distinguishing nearby locations, but does not establish an advantage over positive-only matching.
  4. The explanation for the RoboTwin ID performance gap needs more direct evidence. In Table 2, GeoIDM achieves 28.5% ID success versus DreamGen's 35.0%, while performing better on OOD tasks. Page 7 attributes this difference to baselines memorizing training-scene appearance and task priors. This is plausible, but the results do not rule out factors such as action-prediction accuracy, optimization trade-offs, or evaluation variability. Using task priors also does not itself imply overfitting.
Questions for authors
  1. Could the authors clarify each method's action representation in Table 1 and whether Pos. and Rot. use 16-step endpoint errors or mean per-step errors? In particular, do GeoIDM and DreamGen use the same convention? If not, could existing predictions be evaluated under a common metric to help interpret the reported 63.9%/29.3% reductions in position error?
  2. Could the authors clarify whether Table 3 reports ablation results on ID tasks, OOD tasks, or a combination of both? If OOD results are not reported separately, could rows 2 and 4 be compared on the existing OOD evaluation set to assess whether the position-prediction benefit of gated geometric supervision extends to unseen tasks?
  3. Could the authors compare the current local InfoNCE objective with positive-only cosine matching under the setting in Table 3? Reporting position, rotation, and gripper errors with the same backbone, reliability gate, training budget, and loss-weight selection rule would help assess the added benefit of local negatives over positive-only matching.
5 comments
AuthorsOpenReview ↗

Response (1/4): Summary; W1/Q1 unified error protocol

We thank Reviewer 35nH for the very careful reading and for recognizing the soundness of our method and the value of our ablations and failure analysis.

Summary of additional results.

  • W1 / Q1: all methods are now reported under both the 16-step mean and the endpoint error (Table R1); under the comparable endpoint error, GeoIDM still has the lowest position and rotation errors, and its reduction over DreamGen is essentially unchanged (63.8% / 28.1%, ID / OOD).
  • W2 / Q2: Table 3 is the OOD split; on ID, the gated geometric loss also helps (21.83 → 14.60 mm, Table R2).
  • W3 / Q3: local InfoNCE gives lower position error than positive-only cosine at every weight we tried (Tables R3–R5).
  • W4: we have removed the speculative memorization explanation (see the revision overview at the end).

W1 / Q1. Action representation and error convention in Table 1.

Thank you for raising this. No: GeoIDM and DreamGen did not use the same convention in the original Table 1. We explain why below and now report all methods under both metrics.

In the original Table 1, each method was scored in its native action space. We agree that the two conventions are not directly comparable, so we now report both the 16-step mean error and the endpoint error at step 16 for all methods (Table R1). Each method only needs its missing metric; the other is the original Table 1 number. Below, pkp_k and p^k\hat p_k are the ground-truth and predicted end-effector positions at step kk of a chunk (rotation is computed analogously with the geodesic distance).

  • Absolute actions (RoboDreamer, Vidar, AnyPos, DreamGen) predict the target pose at every step, so Table 1 reported the 16-step mean. We add the endpoint error, which is close to their 16-step mean.
    • Endpoint: Eend=∣p^16−p16∣E_{\rm end} = \\| \hat p_{16} - p_{16} \\|
    • 16-step mean: Emean=116∑k=116∣p^k−pk∣E_{\rm mean} = \frac{1}{16}\sum_{k=1}^{16} \\| \hat p_k - p_k \\|
  • Delta actions (VERA, GeoIDM) predict per-step increments Δp^i\Delta \hat p_i that are accumulated during execution, so Table 1 reported the error of the accumulated endpoint. We add the 16-step mean, but it is not comparable with the absolute-action mean: the trajectory is accumulated from the ground-truth pose at the start of the chunk, so early steps contain only a few accumulated errors.
    • Endpoint: Eend=∣∑i=116ei∣E_{\rm end} = \\| \sum_{i=1}^{16} e_i \\|, with ei=Δp^i−Δpie_i = \Delta \hat p_i - \Delta p_i
    • 16-step mean: Emean=116∑k=116∣∑i=1kei∣E_{\rm mean} = \frac{1}{16}\sum_{k=1}^{16} \\| \sum_{i=1}^{k} e_i \\|

The endpoint error is therefore the comparable metric across action representations.

Table R1. AgiBot action prediction under the unified protocol (absolute trajectories).

(a) Error averaged over 16 steps.

MethodNative repr.ID Pos. (mm) ↓ID Rot. (deg) ↓ID Grip ↓OOD Pos. (mm) ↓OOD Rot. (deg) ↓OOD Grip ↓
RoboDreamerabsolute53.045.770.121101.7113.530.151
Vidarabsolute62.104.580.09488.9711.490.124
AnyPosabsolute49.344.420.08790.2010.630.119
VERAdelta60.196.750.08392.6510.900.095
DreamGenabsolute53.704.160.028105.968.340.034
GeoIDMdelta12.392.120.03564.782.520.046

(b) Endpoint error at step 16.

MethodNative repr.ID Pos. (mm) ↓ID Rot. (deg) ↓ID Grip ↓OOD Pos. (mm) ↓OOD Rot. (deg) ↓OOD Grip ↓
RoboDreamerabsolute53.025.540.109101.8713.520.130
Vidarabsolute64.094.500.10287.0411.560.108
AnyPosabsolute48.844.450.09491.9210.800.111
VERAdelta71.117.750.10294.6416.610.121
DreamGenabsolute53.524.340.047104.267.630.043
GeoIDMdelta19.393.160.11574.953.580.080

Result. Under the endpoint error, GeoIDM has the lowest position and rotation errors on both splits. Its position-error reduction over DreamGen is 63.8% / 28.1% (ID / OOD), essentially unchanged from 63.9% / 29.3% in the original Table 1.

AuthorsOpenReview ↗

Response (2/4): W2/Q2 ablation on ID and OOD splits

W2 / Q2. Does Table 3 report ID, OOD, or both? Does the gated geo-loss benefit extend to unseen tasks?

Table 3 reports the OOD split. So the 17.8% gain of gated geometric supervision (74.33 → 61.09 mm, row 2 → row 4) is already on unseen tasks. As suggested, we also report the ID split:

Table R2. Table 3 ablation on ID and OOD splits (AgiBot, 50% subset; the OOD columns are the original Table 3).

ConfigurationΔa\Delta aΔz\Delta zLgeo\mathcal{L}_{\rm geo}GateID Pos. (mm)ID Rot. (deg)ID GripOOD Pos. (mm)OOD Rot. (deg)OOD Grip
Baseline✓×××25.263.210.08084.994.190.082
+ Feature diff.✓✓××21.832.150.09974.334.030.069
+ Geo. loss✓✓✓×19.223.360.11977.253.800.060
Full model✓✓✓✓14.602.700.10961.094.100.068
  • ID: the same comparison gives 21.83 → 14.60 mm (33.1%), so the gated loss helps on both splits.
  • Role of the gate: without it (row 3), the geometric loss still helps on ID (19.22 mm) but hurts on OOD (77.25 vs. 74.33 mm). The gate is what lets the benefit transfer to unseen tasks.
AuthorsOpenReview ↗

Response (3/4): W3/Q3 local InfoNCE vs. positive-only cosine

W3 / Q3. Local InfoNCE vs. positive-only cosine matching.

We agree with the reviewer's analysis of Eq. (3): under stop-gradient, the negatives act only through the per-sample weighting. We also include direct 2D reprojection as a further baseline.

Setup. Same setting as Table 3 (balanced 50% AgiBot subset; same backbone, feature differencing, reliability gate, and training budget). We only replace the local InfoNCE in Eq. (4) with:

  • Positive-only cosine: ℓcos⁡=1−zanc⊤z+\ell_{\cos} = 1 - \mathbf{z}_{\rm anc}^\top \mathbf{z}_{+};
  • Direct 2D reprojection: ℓ2D=∣u^t+H−ut+Hgt∣1\ell_{\rm 2D} = \\|\hat{\mathbf{u}}_{t+H} - \mathbf{u}^{\rm gt}_{t+H}\\|_1, with the same weight as ours (λ=0.1\lambda = 0.1).

Table R3. Geometric objectives (AgiBot, Table-3 setting).

ObjectiveID Pos. (mm) ↓ID Rot. (deg) ↓ID Grip ↓OOD Pos. (mm) ↓OOD Rot. (deg) ↓OOD Grip ↓
No Lgeo\mathcal{L}_{\rm geo} (= "+ Feature diff." in Table 3)21.832.150.09974.334.030.069
Direct 2D reprojection21.022.640.08387.543.660.065
Positive-only cosine (λcos⁡\lambda_{\cos} = 1.0, matched weight; sweep in Table R5)16.623.450.08868.983.410.071
Local InfoNCE (ours, λgeo\lambda_{\rm geo} = 0.1)14.602.700.10961.094.100.068

Local InfoNCE gives the lowest position error on both splits.

How the negatives act: an adaptive per-sample weight. The reviewer is right that, with frozen features and stop-gradient (Eq. 3), the gradient of local InfoNCE has the same direction as that of positive-only cosine: ∇u^ℓNCE=k,∇u^ℓcos⁡\nabla_{\hat{\mathbf u}}\ell_{\rm NCE} = k\\,\nabla_{\hat{\mathbf u}}\ell_{\cos}, with k=(1−p+)/τk = (1-p_+)/\tau and p+p_+ the softmax probability of the positive. The negatives therefore act through the per-sample weight kk. We measured kk on the LIBERO validation set (≈54k samples), with the same hyperparameters as in the paper:

Table R4. Statistics of the per-sample weight kk (LIBERO validation set).

StatisticValue
(a) Coefficient of variation of kk0.35
(b) Angle between the batch-level parameter gradients of the two losses, mean / max15° / 27°
(c) Mean kk as the 3D displacement error grows from 0 to 80 mm9.27 → 12.75
(d) Mean kk as the end-effector motion grows from < 20 mm to > 120 mm6.75 → 11.22
  • kk is far from constant (a, b). The two losses update the model in different directions, so a global weight on positive-only cosine cannot reproduce local InfoNCE.
  • kk varies systematically (c, d). Hard samples receive up to 38% more weight, so the loss fades as predictions improve. Nearly static samples, which carry little geometric signal, receive less weight; this complements the motion term of the gate.

Weight sweep: positive-only cosine vs. local InfoNCE. Since E[k]≈10\mathbb{E}[k] \approx 10, our λgeo=0.1\lambda_{\rm geo} = 0.1 corresponds to λcos⁡≈1.0\lambda_{\cos} \approx 1.0. We swept around this value:

Table R5. Weight sweep for positive-only cosine (AgiBot, Table-3 setting; position error in mm).

λcos⁡\lambda_{\cos} = 0.10.31.03.0Local InfoNCE (λgeo\lambda_{\rm geo} = 0.1)
ID Pos. ↓20.9520.0616.6215.3314.60
OOD Pos. ↓76.7870.4168.9878.1461.09
  • A larger weight trades OOD for ID: λcos⁡\lambda_{\cos} = 3.0 comes closest on ID (15.33 vs. 14.60 mm), but its OOD error (78.14 mm) is worse than using no geometric loss (74.33 mm).
  • The best OOD weight (λcos⁡\lambda_{\cos} = 1.0) still trails by 7.89 mm (68.98 vs. 61.09 mm).

Analysis of direct 2D reprojection.

  • In theory, it adds no new information. Its target ut+Hgt\mathbf u^{\rm gt}_{t+H} is the ground-truth 3D position projected through fixed, known cameras, i.e., it comes from the same 3D labels that already supervise the cumulative displacement loss Lcum\mathcal{L}_{\rm cum}. The target of Lgeo\mathcal{L}_{\rm geo} instead comes from the future observation: the predicted point must land on content that matches the anchor, which Lcum\mathcal{L}_{\rm cum} cannot provide.
  • The results agree (Table R3). Almost no gain on ID (21.83 → 21.02 mm) and a clear loss on OOD (74.33 → 87.54 mm), while local InfoNCE reaches 61.09 mm. The benefit of Lgeo\mathcal{L}_{\rm geo} comes from visual grounding, not from extra coordinate supervision.
Reviewer_eV64OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a useful auxiliary training constraint for IDM. The constraint targets the weakness of prior works where the IDM rely on visual appearance rather than robot motion. By aligning the predicted robot displacement and its visual projection in the future frame through a geometry consistency loss, the method has shown improvement in action prediction accuracy on real-world and simulation datasets, and increase task success rate in both simulation and real world given a GT reference video.

Critical strengths
  • The formulation of an auxiliary reprojection loss to associate projected actions with observed motion in IDM is straightforward and reasonable. Ablations in Table 3 is helpful to justify design decisions on the gates and feature difference.
  • Promising unseen task results: GeoIDM improves out-of-domain replay success over baselines on LIBERO and RoboTwin, and lower action error on AgiBot.
Critical weaknesses
  • The loss supervises only the projected final end-effector position. It cannot constrain intermediate motions, and does not directly supervise rotation or gripper state. Claims about resolving trajectory drift are therefore too strong.
  • The paper does not compare against directly penalizing predicted-versus-ground-truth reprojection error. This leaves the benefit of feature-based supervision unclear.
  • Evaluation does not demonstrate a complete video generation + IDM framework: Experiments use recorded reference videos and open-loop replay from matching initial states. Robustness to generated videos, execution deviations, and closed-loop operation remains untested.
  • Real-world performance improves over baseline but remains low, might be an inherent limitation with the open-loop execution itself.
Questions for authors
  • With frozen features and stop-gradient negative coordinates, the per-sample InfoNCE gradient is a scalar multiple of the positive-only cosine-similarity gradient. What benefit do local negatives provide beyond adaptive gradient weighting? Can you compare against a positive-only cosine loss with a tuned loss weight?
  • Section 4.1 uses endpoint error for delta-action predictions but horizon-averaged error for absolute-action predictions. Could you reconstruct absolute trajectories for all methods and report both metrics under an identical protocol? Do the improvements in Table 1 hold under this comparison?
  • Comparison with direct geometric supervision. Since ground-truth future end-effector projections are available, how does the proposed objective compare with directly minimizing 2D reprojection error? Does feature-based supervision offer additional generalization benefits?
  • Robustness to generated videos. Have you evaluated GeoIDM using generated future observations rather than recorded reference videos? How sensitive is it to visual artifacts or discrepancies between the reference trajectory and the robot’s actual state?
  • Reliability of feature correspondence. How does the objective handle occluded end-effectors or cases where the projected reference point falls on background?
7 comments
AuthorsOpenReview ↗

Response (1/5): Summary; W1 endpoint-only supervision

We thank Reviewer eV64 for the detailed review and the precise technical questions.

Summary of additional results.

  • W1: we agree that Lgeo\mathcal{L}_{\rm geo} only constrains the chunk endpoint and have toned down the drift claims (see the revision overview at the end).
  • W2 / Q1 / Q3: local InfoNCE gives lower position error than positive-only cosine (at every weight we tried) and direct 2D reprojection on both splits (Tables R1–R3).
  • W3 / Q4: with generated videos, GeoIDM succeeds on 21 / 50 OOD episodes vs. 9 / 50 for DreamGen (Table R4).
  • W4: we agree; the remaining failures mainly come from sub-centimeter contact offsets and gripper timing under open-loop execution.
  • Q2: all methods are now reported under both the 16-step mean and the endpoint error (Table R5); under the comparable endpoint error, GeoIDM still has the lowest position and rotation errors, and its reduction over DreamGen is essentially unchanged (63.8% / 28.1%, ID / OOD).
  • Q5: among inspected samples, we found no occlusion by other objects (Table R6).

W1. The loss only supervises the projected final end-effector position; the drift claims are too strong.

We agree that Lgeo\mathcal{L}_{\rm geo} only constrains the chunk endpoint and that our drift claims were too strong; the revised wording is listed in the revision overview at the end.

Why the endpoint constraint still matters. In our replay experiments (Table 2), consecutive chunks are executed back to back, so each chunk starts from where the previous one ended; a smaller endpoint error therefore reduces the error carried into the next chunk.

AuthorsOpenReview ↗

Response (2/5): W2/Q1/Q3 local negatives vs. cosine and 2D reprojection

W2 / Q1 / Q3. What do the local negatives add beyond positive-only cosine? Comparison with directly penalizing the reprojection error.

We thank the reviewer for this accurate observation about Eq. (3).

Setup. Same setting as Table 3 (balanced 50% AgiBot subset; same backbone, feature differencing, reliability gate, and training budget). We only replace the local InfoNCE in Eq. (4) with:

  • Positive-only cosine: ℓcos⁡=1−zanc⊤z+\ell_{\cos} = 1 - \mathbf{z}_{\rm anc}^\top \mathbf{z}_{+};
  • Direct 2D reprojection: ℓ2D=∣u^t+H−ut+Hgt∣1\ell_{\rm 2D} = \\|\hat{\mathbf{u}}_{t+H} - \mathbf{u}^{\rm gt}_{t+H}\\|_1, with the same weight as ours (λ=0.1\lambda = 0.1).

Table R1. Geometric objectives (AgiBot, Table-3 setting).

ObjectiveID Pos. (mm) ↓ID Rot. (deg) ↓ID Grip ↓OOD Pos. (mm) ↓OOD Rot. (deg) ↓OOD Grip ↓
No Lgeo\mathcal{L}_{\rm geo} (= "+ Feature diff." in Table 3)21.832.150.09974.334.030.069
Direct 2D reprojection21.022.640.08387.543.660.065
Positive-only cosine (λcos⁡\lambda_{\cos} = 1.0, matched weight; sweep in Table R3)16.623.450.08868.983.410.071
Local InfoNCE (ours, λgeo\lambda_{\rm geo} = 0.1)14.602.700.10961.094.100.068

Local InfoNCE gives the lowest position error on both splits.

How the negatives act: an adaptive per-sample weight. The reviewer is right that, with frozen features and stop-gradient (Eq. 3), the gradient of local InfoNCE has the same direction as that of positive-only cosine: ∇u^ℓNCE=k,∇u^ℓcos⁡\nabla_{\hat{\mathbf u}}\ell_{\rm NCE} = k\\,\nabla_{\hat{\mathbf u}}\ell_{\cos}, with k=(1−p+)/τk = (1-p_+)/\tau and p+p_+ the softmax probability of the positive. The negatives therefore act through the per-sample weight kk. We measured kk on the LIBERO validation set (≈54k samples), with the same hyperparameters as in the paper:

Table R2. Statistics of the per-sample weight kk (LIBERO validation set).

StatisticValue
(a) Coefficient of variation of kk0.35
(b) Angle between the batch-level parameter gradients of the two losses, mean / max15° / 27°
(c) Mean kk as the 3D displacement error grows from 0 to 80 mm9.27 → 12.75
(d) Mean kk as the end-effector motion grows from < 20 mm to > 120 mm6.75 → 11.22
  • kk is far from constant (a, b). The two losses update the model in different directions, so a global weight on positive-only cosine cannot reproduce local InfoNCE.
  • kk varies systematically (c, d). Hard samples receive up to 38% more weight, so the loss fades as predictions improve. Nearly static samples, which carry little geometric signal, receive less weight; this complements the motion term of the gate.

Weight sweep: positive-only cosine vs. local InfoNCE. Since E[k]≈10\mathbb{E}[k] \approx 10, our λgeo=0.1\lambda_{\rm geo} = 0.1 corresponds to λcos⁡≈1.0\lambda_{\cos} \approx 1.0. We swept around this value:

Table R3. Weight sweep for positive-only cosine (AgiBot, Table-3 setting; position error in mm).

λcos⁡\lambda_{\cos} = 0.10.31.03.0Local InfoNCE (λgeo\lambda_{\rm geo} = 0.1)
ID Pos. ↓20.9520.0616.6215.3314.60
OOD Pos. ↓76.7870.4168.9878.1461.09
  • A larger weight trades OOD for ID: λcos⁡\lambda_{\cos} = 3.0 comes closest on ID (15.33 vs. 14.60 mm), but its OOD error (78.14 mm) is worse than using no geometric loss (74.33 mm).
  • The best OOD weight (λcos⁡\lambda_{\cos} = 1.0) still trails by 7.89 mm (68.98 vs. 61.09 mm).

Analysis of direct 2D reprojection.

  • In theory, it adds no new information. Its target ut+Hgt\mathbf u^{\rm gt}_{t+H} is the ground-truth 3D position projected through fixed, known cameras, i.e., it comes from the same 3D labels that already supervise the cumulative displacement loss Lcum\mathcal{L}_{\rm cum}. The target of Lgeo\mathcal{L}_{\rm geo} instead comes from the future observation: the predicted point must land on content that matches the anchor, which Lcum\mathcal{L}_{\rm cum} cannot provide.
  • The results agree (Table R1). Almost no gain on ID (21.83 → 21.02 mm) and a clear loss on OOD (74.33 → 87.54 mm), while local InfoNCE reaches 61.09 mm. The benefit of Lgeo\mathcal{L}_{\rm geo} comes from visual grounding, not from extra coordinate supervision.
AuthorsOpenReview ↗

Response (3/5): W3/Q4 generated videos; W4 real-world performance

W3 / Q4. Robustness to generated videos, visual artifacts, and reference–state discrepancies.

Our paper used recorded videos to isolate the IDM; here we test it on generated videos.

Setup.

  • Reference videos are generated by Wan2.1-I2V-14B-720P fine-tuned on LIBERO-90.
  • 50 episodes per split (10 tasks × 5 episodes). All IDMs receive the same generated videos from the same initial states and are executed open-loop as in Table 2. Recorded-video results are re-computed on the same episodes.
  • Whether each generated video itself completes the task was annotated manually, before looking at policy outcomes: 48 / 50 (ID) and 49 / 50 (OOD).

Table R4. LIBERO replay: successful episodes (out of 50) with recorded vs. generated reference videos (retention = generated / recorded).

MethodID recordedID generatedID retentionOOD recordedOOD generatedOOD retention
AnyPos373491.9%10770.0%
DreamGen353497.1%11981.8%
GeoIDM393897.4%252184.0%

Findings.

  • Since almost all generated videos complete the task, the drop from recorded to generated videos reflects each IDM's sensitivity to generation artifacts.
  • On ID tasks, all methods are comparable (34–38 successes). On OOD tasks, GeoIDM remains clearly the strongest (21 vs. 9 successes for DreamGen), and its retention (84.0%) is comparable to or higher than the baselines' (70.0–81.8%). Grounding actions in geometric correspondence therefore does not overfit to clean recorded frames.

Discrepancies between the reference and the robot's actual state. This experiment contains both factors the reviewer mentions: the generated videos have visual artifacts, and the trajectory they show is not guaranteed to match what the robot actually executes. Our experiment does not separate the two. Under open-loop replay, such deviations are not corrected, which affects all IDMs equally. Correcting them requires closed-loop replanning with the video planner, which is beyond the scope of this IDM-focused study.

W4. Real-world performance remains low.

We agree with the reviewer's assessment. Most failures come from sub-centimeter contact offsets and gripper-timing errors (Sec. 4.3), which open-loop execution cannot correct. This is the same limitation discussed in W3/Q4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Thanks to Reviewers and the Area Chair

Dear Area Chair and Reviewers,

We sincerely thank you for your time and constructive feedback. We are encouraged that the reviewers found our training-only geometric objective simple and sound, linking predicted end-effector motion to visual correspondences without any extra module at inference (eV64, 35nH, yQUx); that they recognized the clear gains on unseen tasks, i.e., +20.5 / +18.0 percentage points in out-of-domain (OOD) replay success on LIBERO / RoboTwin 2.0 (eV64, 35nH, yQUx); and that they found the ablations and failure analysis informative (eV64, 35nH).

Below we summarize how we addressed the main concerns. For each item, we indicate where the full answer can be found (e.g., "eV64 2/5" refers to "Response (2/5)" under Reviewer eV64's review) and where the paper was revised. All revisions in the updated PDF are marked in blue.

  1. Advantage over simpler losses (eV64 2/5; 35nH 3/4; yQUx 1/3). In the Table-3 setting, our local InfoNCE gives the lowest position error on both splits: 14.60 / 61.09 mm (ID / OOD), vs. 16.62 / 68.98 mm for positive-only cosine at a matched weight and 21.02 / 87.54 mm for direct 2D reprojection. A weight sweep for the cosine loss does not close the gap, and a gradient analysis explains why. Revision: Sec. 3.3, Sec. 4.3, Appendix E.
  2. Error conventions in Table 1 (eV64 4/5; 35nH 1/4; yQUx 2/3). All methods are now evaluated under both the endpoint error and the 16-step mean error. Under the comparable endpoint error, GeoIDM still has the lowest position and rotation errors, and its position-error reduction over DreamGen is 63.8% / 28.1% (ID / OOD; originally 63.9% / 29.3%). Revision: Table 1, Sec. 4.1, Sec. 4.2.1, abstract, Appendix C.
  3. Generated videos (eV64 3/5; yQUx 3/3). With reference videos generated by a fine-tuned Wan2.1 model on LIBERO, GeoIDM succeeds on 21 / 50 OOD episodes vs. 9 / 50 for DreamGen. Revision: Appendix F.
  4. Split of the ablation (35nH 2/4). Table 3 is the OOD split; on the ID split, the gated geometric loss also reduces position error (21.83 → 14.60 mm). Revision: Table 3 caption, Appendix D.
  5. Overly strong claims (eV64 1/5; 35nH 4/4). We toned down the trajectory-drift claims and removed the speculative explanation of the RoboTwin ID gap. Revision: Sec. 3.3, Sec. 4.2.2, Conclusion.
  6. Occlusion (eV64 5/5). Among 88 manually inspected flagged samples, none was occluded by other objects. Revision: Appendix G, Limitations.

Reviewer eV64 kindly confirmed that the rebuttal cleared most of their concerns and raised the score to weak accept, with a remaining concern about the limited real-world improvement (eV64 3/5). We agree: in the physical ablation, geometric supervision raises successes from 1 / 30 to 6 / 30, and most remaining failures come from sub-centimeter contact offsets and gripper timing that open-loop execution cannot correct. We now state this in the Limitations.

We have not yet heard back from Reviewers 35nH and yQUx. Our responses include all the experiments they requested (35nH: items 1, 2, 4, 5; yQUx: items 1–3), and we would be glad to clarify any remaining questions.

Thank you again for your efforts in reviewing our work.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.