- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces EVGT, a feed-forward transformer that takes multi-view RGB robot videos and jointly predicts camera poses, metric depth, point maps, and dense 3D tracks, plus 6D pose, bounding box, visibility masks for a prompted object, all in one shared metric frame. It builds on a DINOv3 encoder and a DVGT-style factorized attention backbone, adds object queries and prompt conditioning, and is trained in two stages. It beats baselines including VGGT, 4RC, and Pi3 on scene geometry and tracking, and frozen EVGT features give modest gains for VLAs such as π0.5 on LIBERO-Plus and RoboCasa24.
Critical strengths
- Well-motivated formulation. It treats time-varying per-view camera poses and object state as first-class outputs in one shared metric frame, which fits robot setups with wrist cameras and changing states.
- Thorough evaluation shows performance gain on pointmap reconstruction, depth prediction, camera pose estimation, and 3D point tracking over strong baselines such as VGGT. The encoded features moderatly benefit downstream robot policies. One good signal is that viewpoint-change success on LIBERO-Plus rises from 77.67% to 90.37%, the setting where multi-view geometry matters most.
Critical weaknesses
- No ablations (biggest weakness). The paper never justifies its own design choices. There is no analysis of how each output supervision (tracks, pose, box, mask) affects the results, why two-stage training is needed, and how using two types of prompts versus one affect the results.
- In-distribution evaluation vs. zero-shot baselines. EVGT partly trains on the same sources it is tested on, while baselines are not adapted. Humanoid probe results are never reported. It is unclear what's the train & test split and no separation of in-domain and generalization performances.
- Text prompting is not evaluated. It is advertised in the abstract and Figure 1, but all object-state results use first-frame clicks only.
- Lack of baselines for downstream application on VLAs. No comparison against other frozen encoders (e.g., VGGT, DINOv3) under the same adapter, and the gain is moderate (+1.65 on LIBERO-Plus).
Questions for authors
- Can you add ablations for training stages, training objectives, prompt types (see weakness #1)?
- What are the results on humanoid probes or any robot source absent from training, and with baselines fine-tuned on your data? Can you provide a clear breakdown of what data sources are used in training and what are in evaluation, and report in domain and out of domain results separately?
- How do features from other encoders DINOv3/VGGT work with $\pi$0.5?
- What are the parameter count, latency, and memory cost?
Happy to raise my score if these questions can be addressed.
6 comments
Author Response - Part 2
W2 & Q2. In-distribution evaluation, unseen datasets, and fine-tuned baselines.
(a) Unseen out-of-distribution datasets. We evaluated our model on three unseen datasets: two robot datasets G1 ReplicaCAD and Humanoid Everyday, and one general indoor scene dataset ScanNet++. We compare with VGGT, 4RC, Pi3, and DA3 on the same RGB inputs, and report point-map, depth, camera, and tracking results. Depth is scale-aligned, and point maps and tracks use one Sim(3) alignment per sequence. Lower is better.
| Dataset | Metric | EVGT | VGGT | 4RC | Pi3 | DA3 |
|---|---|---|---|---|---|---|
| G1 ReplicaCAD | Point Acc. (m) | 0.1091 | 0.3377 | 0.1560 | 0.2488 | 0.2959 |
| Depth AbsRel | 0.0464 | 0.0825 | 0.0477 | 0.0527 | 0.0496 | |
| Camera ATE (m) | 0.1002 | 0.6089 | 0.1003 | 0.2305 | 0.3319 | |
| Track EPE (m) | 0.0938 | 0.1884 | 0.1995 | — | — | |
| Humanoid Everyday | Point Acc. (m) | 0.0486 | 0.0893 | 0.0928 | 0.0648 | 0.1073 |
| Depth AbsRel | 0.0854 | 0.1579 | 0.1614 | 0.1033 | 0.1797 | |
| Camera ATE (m) | 0.0154 | 0.0169 | 0.0191 | 0.0206 | 0.0223 | |
| ScanNet++ | Point Acc. (m) | 0.0803 | 0.1456 | 0.0700 | 0.1217 | 0.1400 |
| Depth AbsRel | 0.0604 | 0.2013 | 0.0635 | 0.0872 | 0.0966 | |
| Camera ATE (m) | 0.0715 | 0.3234 | 0.0760 | 0.1394 | 0.1470 |
EVGT has the lowest depth error on all three and the lowest point-map error on both robot datasets. 4RC achieves better performance on point maps on ScanNet++, which is included in its training datasets. This shows that our EVGT performs well on out-of-distribution data even when it is in-distribution for the competing methods.
The revised paper adds these results in Sec. 4.4 (Table 7), with full metrics in Table 10 and qualitative reconstructions on a humanoid sequence and an additional indoor scene (the TUM RGB-D long-office sequence) in Figure 5.
Author Response - Part 3
(b) Baselines fine-tuned on our data. We fine-tuned VGGT and 4RC on EVGT's training datasets. The table reports each benchmark dataset and the mean over datasets. Depth is scale-aligned. The four simulation datasets have moving cameras and report ATE and RPE-r, and the fixed-camera RH20T and FMB report drift.
| Dataset | Metric | VGGT (ft) | 4RC (ft) | EVGT |
|---|---|---|---|---|
| LIBERO | Point Acc. ↓ / Comp. ↓ (m) | 0.0957 / 0.0775 | 0.0984 / 0.0916 | 0.0326 / 0.0288 |
| Depth AbsRel ↓ / δ₁ ↑ | 0.2931 / 0.5255 | 0.4260 / 0.4647 | 0.0636 / 0.9381 | |
| Camera ATE ↓ (m) / RPE-r ↓ (°) | 0.0664 / 2.7507 | 0.1125 / 3.4074 | 0.0129 / 1.0078 | |
| Track EPE ↓ (m) / APD ↑ | 0.1031 / 0.1925 | 0.1412 / 0.0880 | 0.0847 / 0.2872 | |
| ManiSkill3 | Point Acc. ↓ / Comp. ↓ (m) | 0.3573 / 0.3344 | 0.2573 / 0.2317 | 0.0938 / 0.0683 |
| Depth AbsRel ↓ / δ₁ ↑ | 0.4055 / 0.4555 | 0.3679 / 0.4998 | 0.1058 / 0.9092 | |
| Camera ATE ↓ (m) / RPE-r ↓ (°) | 0.0794 / 6.4208 | 0.0961 / 2.2189 | 0.0201 / 1.6436 | |
| Track EPE ↓ (m) / APD ↑ | 0.1135 / 0.2164 | 0.1292 / 0.1505 | 0.1063 / 0.2446 | |
| MimicGen | Point Acc. ↓ / Comp. ↓ (m) | 0.0962 / 0.1093 | 0.1060 / 0.1112 | 0.0387 / 0.0314 |
| Depth AbsRel ↓ / δ₁ ↑ | 0.5042 / 0.4363 | 1.0373 / 0.2908 | 0.1191 / 0.8887 | |
| Camera ATE ↓ (m) / RPE-r ↓ (°) | 0.1096 / 10.3398 | 0.1025 / 3.7588 | 0.0223 / 2.4695 | |
| Track EPE ↓ (m) / APD ↑ | 0.1134 / 0.1480 | 0.1224 / 0.1251 | 0.0789 / 0.2891 | |
| RLBench | Point Acc. ↓ / Comp. ↓ (m) | 0.2269 / 0.1821 | 0.2000 / 0.1301 | 0.0465 / 0.0401 |
| Depth AbsRel ↓ / δ₁ ↑ | 0.4672 / 0.5269 | 0.5177 / 0.4663 | 0.0903 / 0.8973 | |
| Camera ATE ↓ (m) / RPE-r ↓ (°) | 0.0956 / 7.4193 | 0.1079 / 3.5784 | 0.0179 / 1.3962 | |
| Track EPE ↓ (m) / APD ↑ | 0.1317 / 0.1563 | 0.1433 / 0.1218 | 0.1038 / 0.2295 | |
| RH20T | Point Acc. ↓ / Comp. ↓ (m) | 0.0528 / 0.0577 | 0.0634 / 0.0526 | 0.0334 / 0.0367 |
| Depth AbsRel ↓ / δ₁ ↑ | 0.1094 / 0.9004 | 0.1316 / 0.8510 | 0.0582 / 0.9629 | |
| Camera Drift-t ↓ (cm) / Drift-r ↓ (°) | 0.0034 / 0.1272 | 0.0580 / 0.0823 | 0.0014 / 0.0086 | |
| FMB | Point Acc. ↓ / Comp. ↓ (m) | 0.0293 / 0.0255 | 0.0355 / 0.0340 | 0.0110 / 0.0116 |
| Depth AbsRel ↓ / δ₁ ↑ | 0.1316 / 0.8516 | 0.0962 / 0.9029 | 0.0247 / 0.9943 | |
| Camera Drift-t ↓ (cm) / Drift-r ↓ (°) | 0.0284 / 1.4140 | 0.3407 / 0.2545 | 0.0471 / 0.0389 | |
| Mean | Point Acc. ↓ / Comp. ↓ (m) | 0.1430 / 0.1311 | 0.1268 / 0.1085 | 0.0427 / 0.0361 |
| Depth AbsRel ↓ / δ₁ ↑ | 0.3185 / 0.6160 | 0.4295 / 0.5793 | 0.0770 / 0.9318 | |
| Camera ATE ↓ (m) / RPE-r ↓ (°) | 0.0878 / 6.733 | 0.1047 / 3.241 | 0.0183 / 1.629 | |
| Camera Drift-t ↓ (cm) / Drift-r ↓ (°) | 0.0159 / 0.771 | 0.1994 / 0.168 | 0.0242 / 0.0237 | |
| Track EPE ↓ (m) / APD ↑ | 0.1154 / 0.1783 | 0.1340 / 0.1213 | 0.0934 / 0.2626 |
EVGT remains best on every dataset and metric after the baselines are fine-tuned, except translation drift on FMB.
(c) Train and evaluation datasets. Training uses the datasets listed in Table 9 of the revised paper. The benchmark evaluates geometry on LIBERO, ManiSkill3, MimicGen, RLBench, RH20T, and FMB, and object pose on the four simulation datasets plus DexYCB and HO3D. The evaluation episodes are separated from the training data as follows. For LIBERO, ManiSkill3, and MimicGen, we hash each complete episode_group and assign the groups whose hash falls below a 10% threshold to a validation split, so all episodes of a group stay on the same side. RLBench uses its original validation split. From these validation splits we keep one representative episode per group and select 50 episodes per dataset in round-robin order over tasks, with base seed 20260924. RH20T and FMB are grouped by trajectory using their original collection indices, and then sampled in round-robin order over tasks. DexYCB is split by subject, and all 50 evaluation episodes come from the held-out subject 20201022-subject-10. For HO3D we first take one episode from each sequence with seed 20260926, and then sample further episodes from the remainder to reach 50. Only the data not selected for evaluation is used for training. In-domain and unseen-dataset results are now reported separately (see (a)).
W3. Text prompting evaluation. Text prompting is now evaluated.
| Method | Center ↓ (cm) | Rotation ↓ (°) |
|---|---|---|
| EVGT, first-frame click | 4.09 | 62.18 |
| EVGT, text | 12.23 | 77.91 |
| EVGT, text + click | 7.93 | 65.83 |
| FoundationPose (RGBD + CAD + SAM2 mask) | 9.54 | 87.02 |
| SAM2 + SAM3D | 111.30 | — |
Text prompting works but localizes less precisely than a click. Text + click averages the two prompt embeddings, so it inherits part of the text error.
Author Response - Part 1
Thanks for the constructive comments.
W1 & Q1. Ablations. Thanks for the suggestion. We ran the three ablations for training stages, training objectives, and prompt types.
(a) Training stages. Stage 1 learns metric scene geometry and dense motion on robot data. Stage 2 adds the object head and trains all tasks jointly. Two stages are needed because the object head reads the geometry and motion features, and object labels cover only a small part of the data. Trained in a single stage, the object losses act on features that do not yet encode reliable geometry. We compare with a single-stage model that trains all tasks jointly from the start for the same total number of training steps as the two stages together. The two-stage model is better on every task.
(b) Training objectives. We compare three models that differ only in which outputs are supervised: geometry only, geometry + tracks, and geometry + tracks + object (pose, size, box, and mask, where the box is computed from pose and size, Eq. 10). All start from the same initial model and use the same training data and number of updates.
Track supervision improves point maps and camera rotation. Object supervision further improves point accuracy and camera pose and leaves tracking unchanged. The same order holds downstream when π0.5 is trained on RoboCasa24 with frozen features from EVGT variants (setup under W4 & Q3):
(c) Prompt types. To test how the two prompt types affect the results, we trained two variants that see only one prompt type during training, click only or text only, and compare them with the full EVGT, which is trained with both. We evaluate each model with a first-frame click, a text prompt, and both. Cells show center error (cm) / absolute rotation error (°), lower is better.
For reference, FoundationPose (RGBD + CAD + SAM2 mask) reaches 9.54 / 87.02 and SAM2 + SAM3D reaches 111.30 cm in center error.
Training with both prompt types gives the best results with a click and with text + click by a large margin. With a text prompt alone, the three models are close.