acceptodds
Under review as a conference paper at ICLR 2027

EVGT: Embodied Visual Geometry Transformer

Abstract

3D reconstruction is fundamental to robotic perception, providing the spatial structure needed to understand scenes and object motion. Despite recent progress in visual geometry foundation models, a unified geometry foundation model tailored to robotic scenes remains lacking. To address this gap, we propose an Embodied Visual Geometry Transformer (EVGT), a metric-scale geometry foundation model for robot video that integrates multi-view observations to recover scene geometry with the motion and 6D pose of the object of interest. From RGB observations, EVGT predicts camera poses, metric depth, point maps, and dense 3D tracks together with 6D pose, size, oriented box, mask, and visibility of the object across time in a shared reference frame. We extract visual tokens from RGB observations and augment them with object queries, with optional prompt tokens for target specification. A spatiotemporal transformer models interactions among these tokens across views and frames, and multiple prediction heads decode the resulting features into scene geometry, 3D motion, and object states. We adopt a two-stage training strategy that combines heterogeneous scene-geometry supervision with partially labeled object-state data. We report point-map, depth, and camera-pose results on six robot-operation sources, dynamic 3D tracking on four core sources, and object-center and rotation diagnostics on six object sources. In downstream closed-loop manipulation, augmented with frozen EVGT features reaches 86.31% success on LIBERO-Plus and 59.17% on RoboCasa24, compared with 84.67% for the LIBERO-Plus starting policy and 55.1% for a RoboCasa-fine-tuned RGB-only baseline.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Visual Geometry, 4D Scene Understanding, 3D Tracking, Object-Centric Perception

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sQ9AOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EVGT, a feed-forward transformer that takes multi-view RGB robot videos and jointly predicts camera poses, metric depth, point maps, and dense 3D tracks, plus 6D pose, bounding box, visibility masks for a prompted object, all in one shared metric frame. It builds on a DINOv3 encoder and a DVGT-style factorized attention backbone, adds object queries and prompt conditioning, and is trained in two stages. It beats baselines including VGGT, 4RC, and Pi3 on scene geometry and tracking, and frozen EVGT features give modest gains for VLAs such as π0.5 on LIBERO-Plus and RoboCasa24.

Critical strengths
  • Well-motivated formulation. It treats time-varying per-view camera poses and object state as first-class outputs in one shared metric frame, which fits robot setups with wrist cameras and changing states.
  • Thorough evaluation shows performance gain on pointmap reconstruction, depth prediction, camera pose estimation, and 3D point tracking over strong baselines such as VGGT. The encoded features moderatly benefit downstream robot policies. One good signal is that viewpoint-change success on LIBERO-Plus rises from 77.67% to 90.37%, the setting where multi-view geometry matters most.
Critical weaknesses
  • No ablations (biggest weakness). The paper never justifies its own design choices. There is no analysis of how each output supervision (tracks, pose, box, mask) affects the results, why two-stage training is needed, and how using two types of prompts versus one affect the results.
  • In-distribution evaluation vs. zero-shot baselines. EVGT partly trains on the same sources it is tested on, while baselines are not adapted. Humanoid probe results are never reported. It is unclear what's the train & test split and no separation of in-domain and generalization performances.
  • Text prompting is not evaluated. It is advertised in the abstract and Figure 1, but all object-state results use first-frame clicks only.
  • Lack of baselines for downstream application on VLAs. No comparison against other frozen encoders (e.g., VGGT, DINOv3) under the same adapter, and the gain is moderate (+1.65 on LIBERO-Plus).
Questions for authors
  • Can you add ablations for training stages, training objectives, prompt types (see weakness #1)?
  • What are the results on humanoid probes or any robot source absent from training, and with baselines fine-tuned on your data? Can you provide a clear breakdown of what data sources are used in training and what are in evaluation, and report in domain and out of domain results separately?
  • How do features from other encoders DINOv3/VGGT work with $\pi$0.5?
  • What are the parameter count, latency, and memory cost?

Happy to raise my score if these questions can be addressed.

6 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thanks for the constructive comments.

W1 & Q1. Ablations. Thanks for the suggestion. We ran the three ablations for training stages, training objectives, and prompt types.

(a) Training stages. Stage 1 learns metric scene geometry and dense motion on robot data. Stage 2 adds the object head and trains all tasks jointly. Two stages are needed because the object head reads the geometry and motion features, and object labels cover only a small part of the data. Trained in a single stage, the object losses act on features that do not yet encode reliable geometry. We compare with a single-stage model that trains all tasks jointly from the start for the same total number of training steps as the two stages together. The two-stage model is better on every task.

Task (average over # datasets)MetricsSingle stageTwo stages
Point map (6)Acc. ↓ / Comp. ↓ (m)0.0793 / 0.11390.0427 / 0.0361
Depth, scale-aligned (6)AbsRel ↓ / δ₁ ↑0.0988 / 0.87230.0770 / 0.9318
Moving camera (4)ATE ↓ (m) / RPE-r ↓ (°)0.0407 / 2.3810.0183 / 1.629
Fixed camera (2)Drift-t ↓ (cm) / Drift-r ↓ (°)0.0377 / 0.02570.0242 / 0.0237
Track (4)EPE ↓ (m) / APD ↑0.1182 / 0.16100.0934 / 0.2626
Object pose (6)Center ↓ (cm) / rotation ↓ (°)21.05 / 91.484.29 / 65.77

(b) Training objectives. We compare three models that differ only in which outputs are supervised: geometry only, geometry + tracks, and geometry + tracks + object (pose, size, box, and mask, where the box is computed from pose and size, Eq. 10). All start from the same initial model and use the same training data and number of updates.

MetricGeometry+ Tracks+ Tracks + Object
Depth AbsRel ↓ (scale-aligned)0.32330.31980.3173
Point Acc. ↓ (m)0.12120.10020.0961
Point Comp. ↓ (m)0.26510.23130.2371
Camera ATE ↓ (m)0.09590.09550.0872
Camera RPE-r ↓ (°)10.068.898.03
Track EPE ↓ (m) / APD ↑—0.1241 / 0.13700.1240 / 0.1376
Object center error ↓ (cm)——32.35
Object rotation error ↓ (°)——116.53
3D box IoU ↑——0.023
Mask IoU ↑——0.234

Track supervision improves point maps and camera rotation. Object supervision further improves point accuracy and camera pose and leaves tracking unchanged. The same order holds downstream when π0.5 is trained on RoboCasa24 with frozen features from EVGT variants (setup under W4 & Q3):

EVGT supervisionRoboCasa24 success ↑
Depth only59.17%
Depth + track62.94%
Depth + track + object (full)66.56%

(c) Prompt types. To test how the two prompt types affect the results, we trained two variants that see only one prompt type during training, click only or text only, and compare them with the full EVGT, which is trained with both. We evaluate each model with a first-frame click, a text prompt, and both. Cells show center error (cm) / absolute rotation error (°), lower is better.

Training promptsClick at inferenceText at inferenceText + click at inference
Click only8.49 / 73.8213.29 / 85.849.67 / 78.98
Text only10.95 / 88.0310.69 / 75.5310.53 / 77.81
Click and text (full EVGT)4.09 / 62.1812.23 / 77.917.93 / 65.83

For reference, FoundationPose (RGBD + CAD + SAM2 mask) reaches 9.54 / 87.02 and SAM2 + SAM3D reaches 111.30 cm in center error.

Training with both prompt types gives the best results with a click and with text + click by a large margin. With a text prompt alone, the three models are close.

AuthorsOpenReview ↗

Author Response - Part 2

W2 & Q2. In-distribution evaluation, unseen datasets, and fine-tuned baselines.

(a) Unseen out-of-distribution datasets. We evaluated our model on three unseen datasets: two robot datasets G1 ReplicaCAD and Humanoid Everyday, and one general indoor scene dataset ScanNet++. We compare with VGGT, 4RC, Pi3, and DA3 on the same RGB inputs, and report point-map, depth, camera, and tracking results. Depth is scale-aligned, and point maps and tracks use one Sim(3) alignment per sequence. Lower is better.

DatasetMetricEVGTVGGT4RCPi3DA3
G1 ReplicaCADPoint Acc. (m)0.10910.33770.15600.24880.2959
Depth AbsRel0.04640.08250.04770.05270.0496
Camera ATE (m)0.10020.60890.10030.23050.3319
Track EPE (m)0.09380.18840.1995——
Humanoid EverydayPoint Acc. (m)0.04860.08930.09280.06480.1073
Depth AbsRel0.08540.15790.16140.10330.1797
Camera ATE (m)0.01540.01690.01910.02060.0223
ScanNet++Point Acc. (m)0.08030.14560.07000.12170.1400
Depth AbsRel0.06040.20130.06350.08720.0966
Camera ATE (m)0.07150.32340.07600.13940.1470

EVGT has the lowest depth error on all three and the lowest point-map error on both robot datasets. 4RC achieves better performance on point maps on ScanNet++, which is included in its training datasets. This shows that our EVGT performs well on out-of-distribution data even when it is in-distribution for the competing methods.

The revised paper adds these results in Sec. 4.4 (Table 7), with full metrics in Table 10 and qualitative reconstructions on a humanoid sequence and an additional indoor scene (the TUM RGB-D long-office sequence) in Figure 5.

AuthorsOpenReview ↗

Author Response - Part 3

(b) Baselines fine-tuned on our data. We fine-tuned VGGT and 4RC on EVGT's training datasets. The table reports each benchmark dataset and the mean over datasets. Depth is scale-aligned. The four simulation datasets have moving cameras and report ATE and RPE-r, and the fixed-camera RH20T and FMB report drift.

DatasetMetricVGGT (ft)4RC (ft)EVGT
LIBEROPoint Acc. ↓ / Comp. ↓ (m)0.0957 / 0.07750.0984 / 0.09160.0326 / 0.0288
Depth AbsRel ↓ / δ₁ ↑0.2931 / 0.52550.4260 / 0.46470.0636 / 0.9381
Camera ATE ↓ (m) / RPE-r ↓ (°)0.0664 / 2.75070.1125 / 3.40740.0129 / 1.0078
Track EPE ↓ (m) / APD ↑0.1031 / 0.19250.1412 / 0.08800.0847 / 0.2872
ManiSkill3Point Acc. ↓ / Comp. ↓ (m)0.3573 / 0.33440.2573 / 0.23170.0938 / 0.0683
Depth AbsRel ↓ / δ₁ ↑0.4055 / 0.45550.3679 / 0.49980.1058 / 0.9092
Camera ATE ↓ (m) / RPE-r ↓ (°)0.0794 / 6.42080.0961 / 2.21890.0201 / 1.6436
Track EPE ↓ (m) / APD ↑0.1135 / 0.21640.1292 / 0.15050.1063 / 0.2446
MimicGenPoint Acc. ↓ / Comp. ↓ (m)0.0962 / 0.10930.1060 / 0.11120.0387 / 0.0314
Depth AbsRel ↓ / δ₁ ↑0.5042 / 0.43631.0373 / 0.29080.1191 / 0.8887
Camera ATE ↓ (m) / RPE-r ↓ (°)0.1096 / 10.33980.1025 / 3.75880.0223 / 2.4695
Track EPE ↓ (m) / APD ↑0.1134 / 0.14800.1224 / 0.12510.0789 / 0.2891
RLBenchPoint Acc. ↓ / Comp. ↓ (m)0.2269 / 0.18210.2000 / 0.13010.0465 / 0.0401
Depth AbsRel ↓ / δ₁ ↑0.4672 / 0.52690.5177 / 0.46630.0903 / 0.8973
Camera ATE ↓ (m) / RPE-r ↓ (°)0.0956 / 7.41930.1079 / 3.57840.0179 / 1.3962
Track EPE ↓ (m) / APD ↑0.1317 / 0.15630.1433 / 0.12180.1038 / 0.2295
RH20TPoint Acc. ↓ / Comp. ↓ (m)0.0528 / 0.05770.0634 / 0.05260.0334 / 0.0367
Depth AbsRel ↓ / δ₁ ↑0.1094 / 0.90040.1316 / 0.85100.0582 / 0.9629
Camera Drift-t ↓ (cm) / Drift-r ↓ (°)0.0034 / 0.12720.0580 / 0.08230.0014 / 0.0086
FMBPoint Acc. ↓ / Comp. ↓ (m)0.0293 / 0.02550.0355 / 0.03400.0110 / 0.0116
Depth AbsRel ↓ / δ₁ ↑0.1316 / 0.85160.0962 / 0.90290.0247 / 0.9943
Camera Drift-t ↓ (cm) / Drift-r ↓ (°)0.0284 / 1.41400.3407 / 0.25450.0471 / 0.0389
MeanPoint Acc. ↓ / Comp. ↓ (m)0.1430 / 0.13110.1268 / 0.10850.0427 / 0.0361
Depth AbsRel ↓ / δ₁ ↑0.3185 / 0.61600.4295 / 0.57930.0770 / 0.9318
Camera ATE ↓ (m) / RPE-r ↓ (°)0.0878 / 6.7330.1047 / 3.2410.0183 / 1.629
Camera Drift-t ↓ (cm) / Drift-r ↓ (°)0.0159 / 0.7710.1994 / 0.1680.0242 / 0.0237
Track EPE ↓ (m) / APD ↑0.1154 / 0.17830.1340 / 0.12130.0934 / 0.2626

EVGT remains best on every dataset and metric after the baselines are fine-tuned, except translation drift on FMB.

(c) Train and evaluation datasets. Training uses the datasets listed in Table 9 of the revised paper. The benchmark evaluates geometry on LIBERO, ManiSkill3, MimicGen, RLBench, RH20T, and FMB, and object pose on the four simulation datasets plus DexYCB and HO3D. The evaluation episodes are separated from the training data as follows. For LIBERO, ManiSkill3, and MimicGen, we hash each complete episode_group and assign the groups whose hash falls below a 10% threshold to a validation split, so all episodes of a group stay on the same side. RLBench uses its original validation split. From these validation splits we keep one representative episode per group and select 50 episodes per dataset in round-robin order over tasks, with base seed 20260924. RH20T and FMB are grouped by trajectory using their original collection indices, and then sampled in round-robin order over tasks. DexYCB is split by subject, and all 50 evaluation episodes come from the held-out subject 20201022-subject-10. For HO3D we first take one episode from each sequence with seed 20260926, and then sample further episodes from the remainder to reach 50. Only the data not selected for evaluation is used for training. In-domain and unseen-dataset results are now reported separately (see (a)).

W3. Text prompting evaluation. Text prompting is now evaluated.

MethodCenter ↓ (cm)Rotation ↓ (°)
EVGT, first-frame click4.0962.18
EVGT, text12.2377.91
EVGT, text + click7.9365.83
FoundationPose (RGBD + CAD + SAM2 mask)9.5487.02
SAM2 + SAM3D111.30—

Text prompting works but localizes less precisely than a click. Text + click averages the two prompt embeddings, so it inherits part of the text error.

Reviewer_R26AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors present an extension of feed-forward 3D reconstruction transformers to a more general-purpose model that outputs object tracks, bounding boxes, masks, and poses, and can be queried with a coordinate or text prompt. The model is primarily trained on robotics datasets, which translates into stronger downstream performance on robotics applications for both the target tasks as well as downstream manipulation transfer.

Critical strengths
  1. The extension of a VGGT- and DVGT-like transformer to object querying and a diverse set of outputs is valuable and interesting - in a way, it shows the value of leveraging a general 3D reconstruction architecture instead of specialized ones, and the application in robotics is particularly relevant.
  2. The metrics on the target evaluation pipelines look strong.
Critical weaknesses
  1. The evaluation on point-map reconstruction is not entirely fair - this is very in-distribution for EVGT while OOD for the others. I understand that given the focus on embodied reconstruction, general scenes may be OOD for EVGT, but it would be good to compare on unseen domains as well to understand the limitations of the method.
  2. The estimated object poses (center and absolute rotation errors) are fairly large, with some reaching nearly random errors (such as 123 degrees for DexYCB). This does not fully correspond to the claimed "accurate and temporally consistent object states."
  3. I could not find any quantitative evaluations of the object tracks, masks, and object bounding boxes (or text prompts) in the paper.
  4. The metric-scale claim is unclear, given that the evaluations are done using scale-agnostic (Sim(3) or scale-aligned) metrics.
  5. At moments, it feels like the authors tried to pack a bit too many capabilities and evaluations into the paper, and therefore it is a bit too concise in some places.
  6. The pi_0.5 improvement in the downstream manipulation transfer example is marginal.
Questions for authors
  1. On data annotation: what role does MoGe-2 (and 4RC) play in generating pseudo-labels, and how does distilling from these models affect the fairness of the baseline comparisons?
  2. Do the authors observe any capability transfer between the modalities they output - i.e., does joint prediction improve anything, or does the method simply predict a collection of separate outputs? Are there any ablations on which of these outputs actually matter for downstream manipulation transfer?
  3. Is there any rationale behind the specific choice of output modalities? Why these, and why not output something else, or take camera parameters or CAD models of objects as inputs when available?
6 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thank you for the detailed questions.

W1. Evaluation on unseen domains. As suggested, we evaluated our model on three unseen datasets: two robot datasets G1 ReplicaCAD and Humanoid Everyday, and one general indoor scene dataset ScanNet++. Cells show point-map accuracy (m) / scale-aligned depth AbsRel, lower is better.

Unseen datasetEVGTVGGT4RCPi3DA3
G1 ReplicaCAD0.109 / 0.0460.338 / 0.0830.156 / 0.0480.249 / 0.0530.296 / 0.050
Humanoid Everyday0.049 / 0.0850.089 / 0.1580.093 / 0.1610.065 / 0.1030.107 / 0.180
ScanNet++0.080 / 0.0600.146 / 0.2010.070 / 0.0640.122 / 0.0870.140 / 0.097

EVGT has the lowest depth error on all three and the lowest point-map error on both robot datasets. 4RC achieves better performance on point maps on ScanNet++, which is included in its training datasets. This shows that our EVGT performs well on out-of-distribution data even when it is in-distribution for the competing methods.

The revised paper adds these results in Sec. 4.4 (Table 7), with full metrics in Table 10 and qualitative reconstructions on a humanoid sequence and an additional indoor scene (the TUM RGB-D long-office sequence) in Figure 5.

W2. Object pose accuracy. We continued Stage 2 training for our model for more epochs. We updated the rotation-error evaluation for both EVGT and FoundationPose to account for object symmetry. For symmetric objects, multiple orientations can be visually equivalent, which was not properly handled by the previous metric and could lead to inaccurate error estimates. The updated results are shown below. Cells show center error (cm) / absolute rotation error (°), lower is better.

DatasetEVGT (RGB + one click)FoundationPose (RGBD + intrinsics + mesh + SAM2 mask)
LIBERO2.47 / 20.4114.13 / 71.99
ManiSkill31.88 / 10.764.78 / 90.62
MimicGen6.13 / 95.999.17 / 93.13
RLBench4.98 / 51.929.64 / 106.69
DexYCB4.78 / 111.5310.24 / 91.78
HO3D4.32 / 82.487.81 / 83.35
Mean4.09 / 62.189.30 / 89.59

EVGT has lower center error on all six datasets. For rotation, EVGT has lower mean error than FoundationPose (62.2° vs. 89.6°) and is accurate on LIBERO and ManiSkill3 (10–20°). The larger absolute rotation errors on MimicGen, HO3D, and DexYCB come from how the metric is defined, not from unstable tracking. Absolute orientation is measured against a canonical object frame defined by the CAD model, which our RGB-only model never sees, and severe occlusion by the hand or robot arm hides the cues that could reveal it. Given the true first-frame orientation and only its own predicted frame-to-frame rotations, EVGT's DexYCB error drops from 111.53° to 40.4°. This verifies that EVGT follows how an object moves and turns consistently over time.

W3. Supplementary quantitative evaluation.

Thanks for the suggestion. We have now included the quantitative evaluations in the revised paper.

Object tracks: same inputs and queries as Table 4, restricted to queries on the target object. EPE is the 3D end-point error in meters, and APD is the mean fraction of queries within 1, 2, 5, and 10 cm.

DatasetEVGT EPE ↓ / APD ↑VGGT4RC
LIBERO0.127 / 0.1340.140 / 0.1080.135 / 0.133
ManiSkill30.115 / 0.1530.139 / 0.1040.131 / 0.155
MimicGen0.112 / 0.1800.137 / 0.0790.138 / 0.094
RLBench0.090 / 0.2960.153 / 0.1330.130 / 0.141
Mean0.111 / 0.1910.142 / 0.1060.133 / 0.131

Masks against baselines:

MetricEVGTSAM2
Mask IoU ↑0.7520.737

Prompt types:

MethodCenter ↓ (cm)Rotation ↓ (°)Mask IoU ↑
EVGT, first-frame click4.0962.180.743
EVGT, text12.2377.910.590
EVGT, text + click7.9365.830.728
FoundationPose (RGBD + CAD + SAM2 mask)9.5487.02—
SAM2——0.737
SAM2 + SAM3D111.30——

Text prompting works but localizes less precisely than a click. Text + click averages the two prompt embeddings, and thus inherits part of the text error.

AuthorsOpenReview ↗

Author Response - Part 2

W4. Metric-scale claim. The submission used scale-aligned metrics because VGGT, 4RC, and Pi3 do not predict metric depth. We now score depth directly in meters, with no scale fitted, against the two baselines that do output metric depth. Cells show raw AbsRel ↓ / δ₁ ↑.

SplitDatasetEVGTDA3 NestedRobo3R
UnseenG1 ReplicaCAD0.064 / 0.9830.161 / 0.6450.705 / 0.003
UnseenHumanoid Everyday0.131 / 0.8980.254 / 0.6930.470 / 0.086
UnseenScanNet++0.079 / 0.9570.159 / 0.7810.454 / 0.136
In-domain, realRH20T0.234 / 0.5940.446 / 0.3380.338 / 0.395
In-domain, realFMB0.073 / 0.9910.955 / 0.0151.075 / 0.028
In-domain, simLIBERO0.507 / 0.4741.157 / 0.2091.253 / 0.066
In-domain, simManiSkill30.772 / 0.3524.119 / 0.0313.438 / 0.023
In-domain, simMimicGen0.625 / 0.3925.814 / 0.0502.777 / 0.199
In-domain, simRLBench0.383 / 0.5881.770 / 0.2341.052 / 0.076

EVGT is best on all nine datasets. Its metric scale is accurate on the unseen datasets and FMB (AbsRel 0.06–0.13) and weaker on the simulation datasets (0.38–0.77).

W5. Too many capabilities, too concise. Thanks for the suggestion. We did want to provide a more comprehensive evaluation of our model and have to move some descriptions to the supplementary due to the page limit. We have expanded the evaluation descriptions in the revision to make them easier to follow.

W6. Marginal π0.5 gain. We retrained π0.5 with Spatial Forcing on the more challenging RoboCasa24 dataset, which aligns the policy's visual features to a frozen encoder during training. Each row uses the same data, 60,000 steps, and batch size 64.

RoboCasa24 policySuccess ↑
RGB-only π0.5, fine-tuned55.12%
+ EVGT, depth supervision only59.17%
+ EVGT, depth + track62.94%
+ EVGT, depth + track + object (full)66.56%

EVGT improves 11.4 points over the RGB-only policy. On LIBERO-Plus the overall gain stays modest (84.67% → 87.22%) but on its viewpoint-change subset where geometry matters most, success rises from 77.67% to 95.18%. This demonstrates the effectiveness of our model.

Q1. MoGe-2 and 4RC pseudo-labels. They supply training targets for two real-video datasets only: MoGe-2 depth for RoboCOIN and AgiBot (calibrated with measured head-camera depth on AgiBot), and 4RC temporal geometry for AgiBot. No evaluation dataset uses pseudo-labels. Ground truth is simulator output for the four simulation datasets and sensor RGB-D for RH20T and FMB, so no method is scored against a teacher's output. EVGT also has lower error than 4RC on these datasets, which distillation alone would not produce.

Q2. Transfer between outputs. To examine whether joint prediction benefits other output modalities and which supervision signals matter for downstream manipulation, we compare three models that differ only in which outputs are supervised: geometry only, geometry + tracks, and geometry + tracks + object. All start from the same initial model and use the same training data and number of updates.

Track supervision reduces point accuracy error by 17.3%, point completeness error by 12.8%, and camera rotation error by 11.6%. Adding object supervision further improves point accuracy (0.1002 → 0.0961 m) and camera pose (ATE: 0.0955 → 0.0872 m; RPE-r: 8.89° → 8.03°), while marginally improving tracking (EPE: 0.1241 → 0.1240 m; APD: 0.1370 → 0.1376), with a slight trade-off in point completeness. These results show that motion and object supervision benefit geometric reconstruction, with object supervision also providing a small gain in tracking.

MetricGeometry+ Tracks+ Tracks + Object
Depth AbsRel ↓ (scale-aligned)0.32330.31980.3173
Point Acc. ↓ (m)0.12120.10020.0961
Point Comp. ↓ (m)0.26510.23130.2371
Camera ATE ↓ (m)0.09590.09550.0872
Camera RPE-r ↓ (°)10.068.898.03
Track EPE ↓ (m) / APD ↑—0.1241 / 0.13700.1240 / 0.1376
Object center error ↓ (cm)——32.35
Object rotation error ↓ (°)——116.53
3D box IoU ↑——0.023
Mask IoU ↑——0.234

For downstream manipulation, we evaluate the same three EVGT variants with π0.5 on RoboCasa24 (setup under W6). Adding track supervision increases success from 59.17% to 62.94% (+3.77 points), and object supervision further raises it to 66.56% (+3.62 points). These results show that our joint prediction benefits the overall perception ability of the model.

AuthorsOpenReview ↗

Author Response - Part 3

Q3. Why these outputs, and why not take cameras or CAD as input?

The outputs are what can benefit the manipulation pipelines: surfaces (depth, point maps), per-frame camera poses that register a moving wrist camera with fixed ones, scene motion (PointWorld plans on 3D point flow, and SeeR-VLA feeds point maps to a policy), and the compact state of the manipulated object (pose, box, mask). We take only RGB because we want to minimize the requirements for inputs. Casually captured robot video rarely comes with calibration, wrist extrinsics change every frame, and per-object CAD does not scale to new objects. The anchored-rotation result in W2 shows such priors help when they exist, and accepting them as optional inputs is a natural extension that we add to the discussion.

Reviewer_P7LfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EVGT, a metric-scale geometry foundation model that predicts scene geometry and object dynamics in a single forward pass. The authors use a two-stage training strategy. The first stage trains the model to predict scene geometry and dense 3D tracks. The second stage adds object-level supervision for predicting 6D object poses, sizes, bounding boxes, masks, and visibility. Experiments show improvements in geometry accuracy and downstream manipulation success rates.

Critical strengths
  1. The research problem is important. Predicting object motion together with scene geometry is useful for robotics and other real-world applications.

  2. The evaluation is comprehensive. The authors assess point-map reconstruction, depth and camera-pose accuracy, dynamic 3D tracking, and object-center and rotation errors. They also report results on downstream closed-loop manipulation.

  3. The paper is well written and easy to follow. The experimental results are organized in a clear way.

Critical weaknesses
  1. The geometry evaluation in Tables 1–3 does not include some important baselines, such as Depth Anything 3 [1]. Including these methods would provide a more complete comparison and better showcase EVGT's advantages.

  2. EVGT underperforms baselines like FoundationPose on several object-state metrics, particularly object-center error in Table 5. These results still suggest room for improvement in object-state estimation.

  3. While the experimental results are promising, the novelty of EVGT is somewhat limited. Adding object-level supervision seems like an intuitive way to improve a geometry foundation model for robotic applications, and the overall architecture builds on existing visual geometry transformer designs. However, I think this is only a minor weakness.

[1] Lin, Haotong, et al. "Depth anything 3: Recovering the visual space from any views."

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Author Response

Thank you for the supportive review.

W1. Missing geometry baselines. We have added Depth Anything 3 [1] (official model, default resolution 504, points back-projected from its own depth, intrinsics, and poses), VGGT-Ω [2], and Robo3R [3] under the Table 1 protocol. Point-map accuracy in meters after one Sim(3) alignment per sequence, lower is better:

DatasetEVGTDA3VGGT-ΩRobo3R
LIBERO0.03260.08730.05140.1152
ManiSkill30.09380.34760.33980.3972
MimicGen0.03870.10130.08240.1051
RLBench0.04650.24600.09470.2559
RH20T0.03340.04770.04700.0601
FMB0.01100.03070.02650.0325

EVGT's error is 29–72% below the best added baseline on every dataset.

Depth, with one scale correction per sequence. Cells show AbsRel ↓ / δ₁ ↑:

DatasetEVGTDA3VGGT-ΩRobo3R
LIBERO0.0636 / 0.93810.3756 / 0.55350.4199 / 0.53690.7634 / 0.4510
ManiSkill30.1058 / 0.90920.4272 / 0.44990.2633 / 0.60250.5823 / 0.3306
MimicGen0.1191 / 0.88871.2072 / 0.32920.8714 / 0.34281.1719 / 0.3601
RLBench0.0903 / 0.89730.6122 / 0.43780.3705 / 0.62500.6297 / 0.4722
RH20T0.0582 / 0.96290.1278 / 0.85780.0668 / 0.95680.1245 / 0.8469
FMB0.0247 / 0.99430.0681 / 0.95830.0532 / 0.97800.1122 / 0.8525

Camera pose, lower is better. The four moving-camera datasets show ATE (m) / RPE-r (°), and the fixed-camera RH20T and FMB show Drift-t (cm) / Drift-r (°):

DatasetEVGTDA3VGGT-ΩRobo3R
LIBERO0.0129 / 1.00780.0712 / 2.20370.0794 / 1.09020.1055 / 7.0286
ManiSkill30.0201 / 1.64360.0678 / 2.34910.0417 / 1.65690.0671 / 7.2869
MimicGen0.0223 / 2.46950.0859 / 3.89350.0885 / 3.78500.1615 / 10.1703
RLBench0.0179 / 1.39620.0610 / 3.30210.1375 / 1.72350.1153 / 8.0834
RH20T0.0014 / 0.00860.0546 / 0.03170.0564 / 0.05610.2393 / 0.1915
FMB0.0471 / 0.03890.3840 / 5.87560.3297 / 0.23960.4580 / 0.6353

EVGT also has the best depth and camera results on all six datasets.

[1] Lin et al. Depth Anything 3: Recovering the Visual Space from Any Views. arXiv:2511.10647, 2025.

[2] Wang et al. VGGT-Ω. CVPR 2026.

[3] Yang et al. Robo3R: Enhancing Robotic Manipulation with Accurate Feed-Forward 3D Reconstruction. arXiv:2602.10101, 2026.

W2. Object state trails FoundationPose.

We continued Stage 2 training for our model for more epochs. We updated the rotation-error evaluation for both EVGT and FoundationPose to account for object symmetry. For symmetric objects, multiple orientations can be visually equivalent, which was not properly handled by the previous metric and could lead to inaccurate error estimates. The updated results are shown below. Cells show center error (cm) / absolute rotation error (°), lower is better.

DatasetEVGT (RGB + one click)FoundationPose (RGBD + intrinsics + mesh + SAM2 mask)
LIBERO2.47 / 20.4114.13 / 71.99
ManiSkill31.88 / 10.764.78 / 90.62
MimicGen6.13 / 95.999.17 / 93.13
RLBench4.98 / 51.929.64 / 106.69
DexYCB4.78 / 111.5310.24 / 91.78
HO3D4.32 / 82.487.81 / 83.35
Mean4.09 / 62.189.30 / 89.59

EVGT has lower center error on all six datasets. For rotation, EVGT has lower mean error than FoundationPose (62.2° vs. 89.6°) and is accurate on LIBERO and ManiSkill3 (10–20°). The larger absolute rotation errors on MimicGen, HO3D, and DexYCB come from how the metric is defined, not from unstable tracking. Absolute orientation is measured against a canonical object frame defined by the CAD model, which our RGB-only model never sees, and severe occlusion by the hand or robot arm hides the cues that could reveal it. Given the true first-frame orientation and only its own predicted frame-to-frame rotations, EVGT's DexYCB error drops from 111.53° to 40.4°. This verifies that EVGT follows how an object moves and turns consistently over time.

W3. Novelty. We agree that the backbone follows visual geometry transformers and do not claim it as new. We think the main novelty is our new formulation for robotic perception: metric-scale object-centric geometry with motion and object tracking from multi-view unposed RGB images without calibration and object CAD inputs. It is also not trivial to instantiate such a model to achieve strong performance on comprehensive tasks.

Dear Reviewer P7Lf,

Thank you for your thoughtful review and constructive suggestions, which have significantly improved the quality and clarity of our paper. We sincerely hope that our rebuttal has fully addressed all of your concerns. As the discussion period is ending soon, if you need any further clarification or additional information, please do not hesitate to contact us. We truly appreciate your time and effort devoted to reviewing our work.

Best regards, Authors of Submission 38725

Thank you to the authors for the response. Please include the results in the revision. I will keep my initial rating at this stage.

Thank you for your feedback and consideration. We have included the results in the revised version of our paper.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.