acceptodds
Under review as a conference paper at ICLR 2027

Deep Prior Learning for Embodied Perception

Abstract

Embodied systems need geometric perception that exploits available observations beyond images alone. Recent feed-forward 3D models incorporate geometric priors, including camera poses, intrinsics, and depth. However, handling noisy poses, preserving accurate priors, and recovering physical scale require more than simply accepting these inputs. We introduce Vision-Prior Geometry Grounded Transformer (VPGGT), a VGGT-based framework that extends OmniVGGT for prior-aware embodied perception. We formulate sensor-motivated pose corruptions from ground-truth trajectories for training and introduce a parameter-free prior residual connection (PRC) to mitigate prior dilution, where predictions are less accurate than their supplied pose priors. Our noise formulation targets camera poses; supplied intrinsics and depth receive no additional corruption. We further introduce Metric Global Attention, which conditions a global scale token on available pose and depth scales and predicts a shared metric scaling factor for the geometric outputs. Experiments across four datasets show that PRC improves translation-direction accuracy and joint pose AUC over a matched training baseline when camera priors are provided for all views, under both exact and corrupted poses. These results support explicit prior access during refinement as a useful addition to feature-level conditioning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embodied Perception, Feed-Forward 3D Reconstruction, Geometric Priors, Camera Pose Estimation, 3D Foundation Models, Metric-Scale Recovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nK8AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VPGGT, building on OmniVGGT to explore feed-forward geometry estimation with optional camera and depth priors. The core methods include: introducing synthetic pose perturbations inspired by sensor noise during training; incorporating aligned camera priors into the iterative state of the camera prediction head via parameter-free Prior Residual Connections (PRC); and predicting a unified physical scale through Metric Global Attention (MGA). Matched-training experiments across four datasets demonstrate that PRC improves translation direction accuracy and joint pose AUC when all views have camera priors. However, performance gains are less consistent under partial prior coverage, and the full model shows noticeable performance variance across different datasets.

Critical strengths

Reveals a valuable failure mode and proposes a lightweight intervention. The "prior dilution" analysis shows that even when given helpful camera estimates, the model can output poses worse than the input priors. Figure 2 further illustrates that simply feeding accurate priors into the existing iterative process is insufficient; the process itself must adapt to prior inputs. PRC addresses this without introducing additional parameters, offering a clean design with a transparent mechanism.

Core PRC results are backed by convincing matched-training experiments. The baseline and PRC variants share identical initializations, training configurations, and training budgets. When noisy camera priors are provided for all views without depth, PRC improves RTA@5° by 8.11 and 12.84 percentage points on TUM RGB-D and C3VD, respectively (Table 1). The appendix also provides results for partial coverage, directional translation metrics, and predictions on views without depth, which helps clarify the scope and limits of the approach.

Critical weaknesses

The isolated contribution of Metric Global Attention remains unverified. Table 5 reports only the full model without an MGA ablation under identical training conditions, and lacks comparisons against external physical scale reconstruction baselines. Additionally, because the full model underwent two extra epochs of fine-tuning, system-level differences cannot be cleanly attributed to MGA. While Appendix A.4 defines a baseline that directly fits scale using input camera priors, no experimental results are reported for it. This comparison is critical: on Sewerage, even with accurate camera and depth priors, the scale-aligned depth AbsRel is 0.0098, whereas the physical-scale AbsRel remains 0.1258 (Tables 4–5). As a result, it is currently unclear whether the learned scaling mechanism outperforms directly recovering scale from existing physical scale observations.

Gains over matched-trained baselines do not always translate to improvements over the raw input estimates themselves. For example, under full noisy camera priors, the input priors on TUM achieve an AUC of 81.07, whereas PRC-only achieves 76.37 (Table 1). Partial prior coverage can also degrade performance relative to matched baselines: on TartanAir-V2 at 30% accurate camera coverage, AUC drops from 72.91 to 70.92 (Table 8). In full-system comparisons, VPGGT achieves a lower AUC than the published OmniVGGT in 11 out of 12 settings on TartanAir-V2 (Table 4). While the authors explicitly acknowledge these findings, combined with the small evaluation subsets and synthetic noise setups, the practical utility of the method is not yet fully established.

Questions for authors
  1. Under the same training budget, could you compare MGA against a scale token that does not condition on prior scale, as well as the prior scale fitting baseline defined in Eq. (13)?

  2. In addition to the existing input prior reference scores, could you evaluate a baseline that retains provided poses and only predicts missing ones, reporting image pair results separately for pairs with priors on both views, one view, and neither view?

  3. Holding all other factors constant, could you contrast models trained strictly on clean priors against those trained on noisy priors, and test them across varying noise intensities or unobserved perturbation types?

4 comments

Thank you for highlighting the prior-dilution diagnosis and the matched PRC results. We also appreciate your distinction between predicting geometry and reproducing the supplied inputs. Your questions help separate three outcomes, which are beating the trained baseline, retaining the input pose, and recovering physical scale.

W1/Q1. Does direct scale conditioning outperform an unconditioned token or supplied-pose fitting?

On the 20 Sewerage windows in your example, supplied-pose fitting outperformed learned scale. We held the Full predictions fixed and fitted one scale per window to the exact camera baselines supplied as input. Median-aligned depth AbsRel remained 0.0098, whereas metric depth AbsRel fell from 0.1258 to 0.0773 and point-radius error from 12.05% to 3.96%. Thus median alignment concealed a substantial difference in physical units. This test does not establish an advantage over an equally trained unconditioned token or the same ordering when exact cameras are unavailable.

W2. Do gains over C1 imply gains over the supplied poses or across domains?

No. On TUM RGB-D with noisy cameras on every view and no depth input, AUC@30° rises from 70.89 for C1 to 76.37 for C2, still below the supplied prior's 81.07. On the same inputs, C2 exceeds the prior on RTA@5°, 67.00 versus 66.00. C3VD shows the reverse ordering across metrics: C2 exceeds the prior on AUC, 72.81 versus 71.73, but not on RTA@5°, 55.46 versus 57.73. We distinguish improvement over matched fine-tuning from improvement over the supplied pose and do not claim uniform cross-domain gains.

Q2. What happens if supplied poses are kept unchanged?

Using the same C2 checkpoint, we first predict all poses from RGB alone. We fit one similarity transform from the predicted to the supplied camera centers on the provided views, transform the full predicted trajectory, and copy the supplied poses back unchanged. The table gives joint AUC@30° as keep-provided / PRC on the same windows. AA, AU, and UU denote pairs with two, one, and zero supplied poses, respectively.

DatasetPosesCoverageWindowsAAAUUU
TartanAir-V2Exact30%660100.00 / 83.0428.64 / 67.3471.80 / 72.76
TartanAir-V2Exact70%660100.00 / 85.8651.98 / 66.6265.07 / 67.32
C3VDExact30%45100.00 / 90.574.86 / 58.9058.73 / 58.65
C3VDNoisy70%4567.76 / 69.807.18 / 54.1455.16 / 54.99

Copying preserves exact AA poses by construction. PRC gives higher AU AUC in all four examples, whereas UU AUC changes little on C3VD. On TartanAir-V2 at 70% exact coverage, these differences nearly cancel in all-pair AUC: 75.26 for keep-provided versus 75.65 for PRC. The appendix gives the four-domain breakdown.

Q3. Does training with corruptions improve robustness to test noise?

We have not isolated that training effect. C2 was trained with a mixture of clean and synthetically corrupted poses. At full camera coverage without depth input, its joint AUC changes from 80.46 with exact poses to 76.37 with the tested corruption on TUM RGB-D, and from 88.48 to 72.81 on C3VD. These comparisons measure test-time sensitivity. Without matched clean-only training, they do not establish how much robustness is caused by including corruptions during training.

Reviewer_WjSaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VPGGT which extends OmniVGGT for reconstruction with camera and depth priors. Prior Residual Connection (PRC) inserts supplied camera priors into the existing iterative pose head. Metric Global Attention (MGA) conditions a scene-scale token on available metric information. Training uses structured pose corruptions. Results show gains under full camera-prior coverage, with mixed outcomes under partial coverage and across domains.

Critical strengths
  1. The paper highlights that prior-conditioned models can produce pose estimates less accurate than the supplied priors. PRC mitigates this problem through a parameter-free modification to the camera head, with matched-training comparisons showing substantial improvements in translation-direction accuracy on TUM and C3VD under full camera-prior coverage.
  2. Input-prior references and depth evaluation on views without supplied depth help distinguish reconstruction from reproducing the inputs. Separating metric and scale-aligned results also makes the evaluation more informative.
Critical weaknesses
  1. The novelty is limited. Geometric-prior conditioning is established in Pow3R and OmniVGGT, while VGGT already uses detached, iterative residual pose updates. PRC's specific change is to replace the refinement state with aligned external priors after the initial prediction. MGA adopts MapAnything's global scale-token mechanism, adding direct conditioning on prior scale and availability while retaining VGGT's heads. These dependencies are acknowledged, but the remaining design choices represent modest extensions whose advantages need clearer experimental support.
  2. MGA's contribution is not isolated. Table 5 evaluates the full model after two additional training epochs relative to PRC-only, without matched comparisons against an unconditioned scale token, MapAnything-style scale fusion, or the supplied-prior scale fit. MapAnything is also missing as an external metric baseline.
Questions for authors
  1. Under matched training budgets, how does MGA compare with an unconditioned scale token, MapAnything-style per-view scale conditioning, and PRC-only plus Eq. (13)? How does MapAnything perform under the same metric evaluation protocol?
  2. Does inserting priors after the initial prediction outperform initialization before the first update or a simple gated blend of prior and predicted states, under the same training protocol?
3 comments

Thank you for the thoughtful review. We appreciate your distinction between improving over a trained baseline and improving over the supplied prior, and your comments on separating metric from scale-aligned evaluation. Thank you for asking us to distinguish the change to the camera state from the existing feature-conditioning and residual-update machinery.

W1. What is new beyond feature conditioning and residual updates?

The difference is the state from which refinement proceeds. Both variants use the same feature-conditioning pathway. The original head continues from its own IT0 estimate; PRC resets each supplied view's state once, after IT0, and retains the original residual-update architecture. The released-weight trace shows that subsequent updates can move away even from an exact inserted pose. Under matched fine-tuning, with noisy poses on all views and no depth input, PRC raises RTA@5° by 8.11 points on TUM RGB-D and 12.84 on C3VD. We do not claim iterative residual updates or the global scale token itself as new.

W2/Q1. Which gains are due to direct scale conditioning rather than training or fitting?

We can isolate the inference-time choice of output scale, not the training effect of conditioning the token. On 20 Sewerage windows with exact cameras and depth, the same Full predictions gave metric depth AbsRel of 0.1258 with learned scale and 0.0773 with a scale fitted to the supplied camera baselines. Median-aligned AbsRel remained 0.0098. The analytic fit was better on these windows. Full received two additional epochs after C2, and we have no equal-budget training comparison against either an unconditioned token or a per-view alternative.

Q2. Why inject after IT0 rather than earlier or through a gate?

Our alignment uses the translation scale of the initial camera prediction. IT0 supplies that reference; injection before IT0 would require a different scale rule. This explains our placement, but it does not establish that a post-IT0 reset is better than an earlier injection or a learned blend. We have no matched comparison establishing either claim.

Reviewer_4DQnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents VPGGT, which extends OmniVGGT (a VGGT-based feed-forward 3D model accepting optional camera/depth priors) with three pieces: (1) sensor-motivated pose corruption of training trajectories; (2) a parameter-free Prior Residual Connection (PRC) that initializes the camera head’s refinement state with the scale-aligned supplied pose for provided views, addressing “prior dilution”, where the released model’s refinement makes exact priors worse over iterations; (3) Metric Global Attention, a scale token conditioned on pose/depth scale statistics predicting one shared metric factor, following MapAnything. Training fine-tunes the released OmniVGGT checkpoint on TartanAir, Virtual KITTI 2, TUM RGB-D and C3VD. A matched-training ablation shows RTA@5 and AUC@30 gains at full camera-prior coverage on all four datasets (up to +33.6 RTA on C3VD), mixed results at partial coverage, and a reversal against released OmniVGGT on TartanAir-V2 subsets. Metric reconstruction is reported for the full model only.

Critical strengths
  1. The prior dilution diagnosis is real and cleanly shown: without any training, the released camera head degrades an injected exact prior at every refinement iteration (Fig. 2), on C3VD even below the prior-free output. The PRC ablation is properly controlled (same checkpoint, data, budget) and the full-coverage gains are large.

  2. The appendix has more method details and results. From the pseudo-code, it appears to be easy to implement.

Critical weaknesses
  1. The method is too thin. PRC is one equation, replace the refinement initial state with the prior, the first baseline anyone would try after observing Fig. 2. MGA is MapAnything’s scale token with minor changes. Pose corruption is data augmentation. None of the three rises to a method contribution for this venue.

  2. Almost no external comparison. The only compared methods are released OmniVGGT and, on two subsets, released VGGT. Pow3R (CVPR’25) and MapAnything, both cited as the closest prior-conditioned and metric models, never appear in any table. A paper in this line in 2027 needs at least these two.

  3. The headline system comparison is confounded by fine-tuning. VPGGT gets four extra epochs on the four evaluation-domain datasets before being compared with the released checkpoint; the matched baseline C1 alone already moves C3VD AbsRel from 0.511 to 0.055, so most of Table 3 is domain adaptation, not the proposed modules.

  4. The paper’s own results show the gains do not generalize. On TartanAir-V2 the full model loses to released OmniVGGT in 11 of 12 AUC settings and all 12 ATE settings, and PRC regresses at 30% coverage on two datasets. The method helps mainly when priors cover every view, on two datasets.

  5. The metric branch is unvalidated. No matched ablation without MGA, no MapAnything comparison, and the obvious baseline of fitting scale from the supplied priors is defined in the appendix (Eq. 13) but its numbers are never shown.

Questions for authors
  1. Add Pow3R and MapAnything under your protocol, at least on TUM and TartanAir-V2.
  2. For Table 5, report MapAnything, the Eq. 13 supplied-prior scale baseline, and a matched model without MGA. Does the learned scale branch beat fitting scale to the supplied poses?
  3. Can you characterize when PRC helps versus hurts, given the TartanAir-V2 reversal? Is the trained correction overwriting good image-only estimates?
  4. Is prior dilution specific to OmniVGGT’s camera head, or does the same injection on Pow3R / MapAnything show it too?
5 comments

Thank you for the insightful feedback and careful review. We appreciate your recognition that Fig. 2 isolates the loss of prior accuracy during refinement and that the matched-training comparison tests PRC directly. We address your concerns below.

W1. What does PRC contribute beyond an initialization change?

PRC changes the camera state once, after the initial estimate (IT0). It replaces the state of each supplied view with its scale-aligned prior and leaves missing views at their IT0 estimates. Later updates can still change every pose. Our released-weight trace shows why this distinction matters: even an exact pose inserted into the state can lose accuracy during subsequent refinement. With matched fine-tuning, camera priors on every view, and no depth input, PRC raises RTA@5° by 15.00 points on TUM RGB-D and 33.58 on C3VD with exact poses, and by 8.11 and 12.84 points with corrupted poses. We do not claim the underlying residual update as new.

W2/Q1: How do Pow3R and MapAnything compare on TUM RGB-D and TartanAir-V2?

We completed a matched comparison on the Sewerage subset of TartanAir-V2. Both models received RGB, exact camera intrinsics and poses, and reference depth on every view of the same 20 ten-view windows from four sequences. We scored the same 900 eligible pose pairs and 200 depth frames.

MetricMapAnythingFull VPGGT
RTA@5° (%) ↑100.0084.56
Joint pose AUC@30° (%) ↑99.6588.87
Median-aligned depth AbsRel ↓0.00820.0098
Metric depth AbsRel ↓0.01110.1258
Point-radius error (%) ↓0.4512.05
SE(3)-aligned ATE (m) ↓0.01400.2726

MapAnything was more accurate in this condition. The metric results use each model's native output, including Full VPGGT's learned scale. Because poses and depth were supplied on every view, this tests preservation of observed geometry, not recovery of missing geometry. We do not report a matched MapAnything result on TUM RGB-D or a verified matched Pow3R result on either dataset.

W3: Do the released-checkpoint comparisons conflate adaptation with the proposed modules?

Yes. On C3VD, RGB-only median-aligned depth AbsRel falls from 0.511 for released OmniVGGT to 0.055 after fine-tuning without PRC. We do not attribute that depth gain to PRC. Our architectural comparison is C1 versus C2: both start from the same released checkpoint and use the same training mixture, optimization settings, and budget. Full VPGGT receives two further epochs after C2.

W4/Q3: When does PRC help, and does it overwrite good visual estimates?

With camera priors on every view and no depth input, C2 improves joint AUC over C1 on all four datasets for both exact and corrupted poses. On aggregate TartanAir-V2 with full exact cameras, the released model, C1, and C2 score 93.88, 84.67, and 85.85, respectively: PRC improves over matched fine-tuning but does not recover the released model's score.

At 30% exact camera coverage on the same 660-window evaluation, all-pair AUC@30° falls from 72.91 for C1 to 70.92 for C2. Comparing a different baseline, keeping the supplied poses unchanged gives higher AUC than final C2 on pairs of two supplied views (100.00 versus 83.04), but lower AUC on mixed supplied/missing pairs (28.64 versus 67.34). This is a comparison with copying, not a reset-to-final trace. It does not show that PRC overwrote an accurate RGB-only estimate: when camera inputs are present, even IT0 uses camera-conditioned features.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.