acceptodds
Under review as a conference paper at ICLR 2027

HER: Human–Environment Reconstruction via Visual Geometry-Mediated Reasoning

13 views

Abstract

Recent monocular HMR methods have made strong progress in world-space motion recovery, yet robust global trajectory estimation remains challenging under monocular ambiguity. Feedforward 3D reconstruction offers complementary spatial cues by recovering camera motion and the geometry of the moving human. We introduce HER, which leverages the moving-human geometry recovered by feedforward 3D reconstruction as a spatial bridge between parametric body prediction and the reconstructed world. HER builds on a backbone-agnostic feedforward geometry frontend to recover spatially consistent geometry and camera trajectories over long videos. For human reconstruction, we freeze a pretrained image HMR encoder and learn a temporal human representation that jointly supports SMPL-X body and explicit structural joint prediction. We further learn a geometry-mediated spatial reasoner that couples this representation with reconstructed human geometry to recover temporally consistent global translation while preserving high-quality pose and shape. Human body structure provides an additional scale reference for establishing a common world coordinate system. A lightweight world-space optimizer further improves temporal and contact consistency. Experiments on EMDB2 and RICH show that HER outperforms state-of-the-art methods in world-space human reconstruction while maintaining strong local accuracy, with robust performance on challenging in-the-wild videos.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Human Mesh Recovery, Feed-Forward 3D Reconstruction, Human-Scene Reconstruction, Human motion Capture

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kP1hOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HER, a monocular human–environment reconstruction framework that leverages reconstructed human point clouds to assist global human localization and metric scene scale estimation. Through geometry-guided spatial and temporal reasoning, HER recovers human motion in a unified world coordinate system. Experiments demonstrate that HER improves world-space human reconstruction accuracy and scene scale estimation compared to existing methods.

Critical strengths
  1. The paper achieves competitive performance on multiple benchmarks, demonstrating the effectiveness of the proposed framework.
  2. The paper provides relatively comprehensive evaluations and ablation studies, covering multiple metrics and key components of the proposed method.
Critical weaknesses
  1. Limited evaluation of 4D reconstruction quality. The existing static qualitative results are insufficient to adequately assess 4D reconstruction quality. Additional video demonstrations would help better evaluate human–scene alignment, contact consistency, and temporal coherence.
  2. Although HER outperforms VGMR Off* (GT Scale), the comparison does not fully isolate the localization benefit. I suggest fixing camera trajectories, metric scale, human pose, and post-processing, while varying only the human localization method.
  3. The paper lacks a direct comparison of Pi3X-based scene/camera reconstruction before and after applying the predicted scale. The authors should also clarify whether Camera ATE uses Sim(3) or SE(3) alignment.
Questions for authors
  1. Could the authors report the end-to-end inference speed (FPS) and runtime of the complete pipeline?
  2. While the separately predicted metric skeleton slightly improves world-space joint accuracy, does it provide more accurate body height or bone lengths than SMPL-X-derived joints? A direct evaluation of skeletal scale accuracy would be helpful.
  3. Despite the strong results on several benchmarks, what are the limitations of the proposed method? More in-the-wild internet video examples would help illustrate its generalization capabilities and potential failure cases.
1 comment

We have also provided video demonstrations of HER via an anonymous link:

https://anonymous.4open.science/w/HER-anonymous-submitted/ for convenient viewing.

The high-resolution videos remain available in the supplementary material.

W1/Q3: Continuous 4D visualizations

Thank you for this suggestion. We have updated the supplementary material and our anonymous link with continuous reconstruction videos covering 3 RICH & 3 EMDB2 and 3 in-the-wild videos.

Each sequence is shown from 2 viewpoints to facilitate inspection of body motion, human–scene spatial relationships, foot contact behavior, and temporal consistency.

W2: Isolating VGMR's localization benefit

Thank you for this suggestion. We fixed camera-trajs, human pose and shape, and geometric inputs, and disable post-processing in both methods. Only localization changes between Points Align and VGMR. We test both a shared predicted scale and a shared GT-derived scale fitted from camera trajectories.

Results on all 25 EMDB2 sequences use common valid frames:

Shared scaleMethodW-MPJPE100 ↓ (mm)WA-MPJPE100 ↓ (mm)RTE ↓ (%)
PredictedPoints Align191.4877.881.1998
PredictedVGMR167.4066.141.1644
GT-derivedPoints Align201.0380.450.6511
GT-derivedVGMR176.3168.900.5781

VGMR consistently improves under both shared-scale settings. GT-derived scale improves RTE but does not necessarily reduce Sim(3)-aligned joint errors, since it is fitted to camera trajectories rather than human joints.

W3: Scale recovery and the camera ATE alignment protocol

Thank you for highlighting this ambiguity. The camera ATE in Table 2 uses Sim(3) alignment, which evaluates camera-traj accuracy after removing global scale differences. We have explicitly labeled the alignment protocol in the table in our updated paper.

  • To directly assess scale recovery, we additionally evaluate camera ATE using SE(3) alignment, which preserves scale errors. We evaluate all 25 EMDB2 sequences, comprising 43,205 camera frames, and report the arithmetic mean of per-sequence ATE RMSE: | Camera trajectory | SE(3) ATE ↓ | Sim(3) ATE ↓ | |---|---:|---:| | Before scale recovery | 2.086 m | 0.297 m | | After applying HER's predicted scale | 0.674 m | 0.297 m |

  • The predicted scale is applied without GT scale correction. The reduction in SE(3) ATE prove that HER improves metric camera trajectory accuracy. Sim(3) ATE remains unchanged because similarity alignment removes the global scale difference.

Q1: Computational efficiency

Thank you for asking about runtime. HER achieves 4.84 FPS on a single NVIDIA RTX 4090. We compute throughput as

FPS=NTseg+SLAM+Thuman+VGMR+Trefinement\mathrm{FPS} = \frac{N}{T_{\mathrm{seg+SLAM}} + T_{\mathrm{human+VGMR}} + T_{\mathrm{refinement}}}

where NN is the total number of processed frames. The denominator includes segmentation and SLAM, human reconstruction and VGMR, and final world space refinement. We measure computation time after warm-up with CUDA synchronization at stage boundaries.

For reference, reported speeds of related methods are summarized below:

MethodFPS ↑GPU
Human3R (ViT-L/896)5.09RTX 4090
OnlineHMR3.30RTX 6000 Ada
UniSH†2.00Not specified
UniCon3R (ViT-L/896)2.40NVIDIA A5000
HER (ours)4.84RTX 4090

† UniSH's speed is reported by UniCon3R. Baseline speeds are taken from published reports.

Q2: Skeletal scale accuracy

Thank you for suggesting this direct evaluation. We compare our directly predicted joints with SMPL-X-derived joints across all EMDB2 sequences. We preserve the original metric dimensions without scale fitting and evaluate against GT.

Skeletal representationBone-length MAE ↓ (mm)
Directly predicted joints10.21
SMPL-X-derived joints16.22

Our predicted joints achieves lower bone-length errors. This evaluation focuses on bone-length consistency rather than overall body height, which is not explicitly measured.

Q3: In-the-wild examples and Limitations

Thank you for this suggestion. We have updated the supplementary material with in-the-wild videos and failure cases.

We also updated the Limitation Section in our paper, HER would produce inaccurate results when the person is far from the camera, occupies a small image region, or is heavily occluded, due to limited visual and geometric evidence.

Reviewer_R7TjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper addresses global human and environment reconstruction from monocular videos. It proposes a VGMR module to learn the scale and translation offset of the human from the reconstructed human point cloud by a 3D geometry foundation model followed by a refinement stage to further optimize the global human trajectory. Experiments show the proposed method achieve strong performance on multiple benchmarks.

Critical strengths
  1. The proposed method achieves strong quantitative results on the open benchmarks, surpassing prior works by a clear margin.
  2. Using a data-driven method to learn the scale and translation offset of the human in the reconstructed point cloud (i.e the proposed VGMR module) seems a promising solution that could be more general and robust than optimization-based methods.
Critical weaknesses
  1. There is no video qualitative results for the human and environment reconstruction, and the all the image qualitative results focus on demonstrating the global human trajectory. It is difficult to evaluate the reconstructed human motion quality, its physical plausibility, and its alignment with the scene context without video results of the detailed motion.
  2. There is also no evaluation results of the camera-space HMR performance of VGMR and the human-scene consistency metrics.
  3. In the world-space translation refinement, the authors only optimize the SMPL-X translation without optimizing other parameters such as the joint orientations, which could degrade the local motion quality.
  4. The performance gains of VGMR is only marginal compared to direct scale regression (Tab. 4). The motivation of the VGMR model design is also not very clear. For example, why the global point feature is spitted into individual joint tokens? These joint tokens seem not relevant to each joint predictions/feature and they only interact with the entire flattened metric skeleton. Why there are separate tokens for root bias and translation instead of a single translation offset?
  5. There seems to be gaps in the training labels and inference results the human point cloud. During training, the human point cloud is from the unprojected points with ground truth depth from the BEDLAM dataset. During inference, the point cloud is estimated by a Pi3-X, which may be much noisier than the ground truth and could lead to artifacts in the human shape. This gap could affect the model performance of VGMR but is not studied or addressed in the paper.
  6. There is no section discussing the paper limitations or failure modes.
Questions for authors
  1. Can the authors provide evaluation results of camera-space HMR performance of VGMR (e.g comparing to prompt-HMR) and the human-scene consistency metrics?
  2. Can the authors explain why only optimizing the SMPL-X translation in the post-processing stage would lead to an accurate human reconstruction result?
  3. For the VGMR model architecture, why does the global point feature need to be spitted into individual joint tokens and why are there separate tokens for root bias and translation?
  4. How does the author handle the training and inference gap of the human point cloud?
4 comments

We have also provided video demonstrations of HER via an anonymous link:

https://anonymous.4open.science/w/HER-anonymous-submitted/ for convenient viewing.

The high-resolution videos remain available in the supplementary material.

W1: Continuous video demonstrations

Thank you for highlighting the continuous visualizations. We have added 3 RICH & 3 EMDB2 and 3 in-the-wild videos to the supplementary material, each shown from 2 viewpoints. These videos allow direct inspection of the evolving body motion, human–scene spatial relationships, foot contact behavior, and temporal consistency beyond the trajectory plots in the paper.

W2/Q1: Missing Evaluated on Camera-Space HMR and Human–Scene Consistency

Thank you for the suggestion. We actually have evaluated camera-space HMR and PA-MPJPE on EMDB2/RICH in Sec. 4.3.1 (Tab. 1). Additionally, App. C.1 (Tab. 7) shows HER outperforms OnlineHMR on 3DPW (PA-MPJPE: 36.0 vs 43.7; MPJPE: 55.4 vs 69.9; PVE: 67.8 vs 83.7 mm). For the requested PromptHMR comparison, we evaluated a frozen PromptHMR video head under identical shared inputs (features, keypoints, cameras):

MethodPA-MPJPE ↓ (mm)MPJPE ↓ (mm)PVE ↓ (mm)
PromptHMR vid-head38.2459.4271.67
HER human branch36.0055.4367.75

HER achieves lower errors across all 3 metrics. And we have updated the PromptHMR vid-head results on our paper.

Human–scene consistency. We conduct an additional comparison on 50 RICH test videos from static cameras 00 & 01 (32 & 18 videos, respectively). For each video, we fuse static scene geometry from 100 uniformly sampled frames.

We evaluate human-scene consistency and scene geometry across 50 videos (in metric scale): Pelvis Error (3D distance between aligned predicted and GT pelvis), Contact Distance (distance from GT-labeled contact vertices to the GT scene surface), and standard scene metrics (Accuracy, Completeness, and F-score@5cm).

MethodPelvis Position Error ↓ (mm)Contact Surface Distance ↓ (mm)Scene Accuracy ↓ (mm)Scene Completeness ↓ (mm)Scene F-score@5 cm ↑ (%)
HER264.2787.5167.2526.1477.90
Human3R366.07130.0999.6465.1755.66
UniSH496.39169.37173.3125.9748.19
OnlineHMR509.01210.5596.3541.8365.55

HER achieves the best metircs. These results support accurate human localization relative to the aligned scene and closer agreement between GT-labeled contact vertices and the GT scene surface.

W3/Q2: Effect of translation-only refinement on local motion

Thank you for raising this concern. The refinement applies the same world-space translation residual Δt\Delta_t to every joint and vertex within each frame. Thus, for any 2 joints jj and kk,

(Xt,j+Δt)−(Xt,k+Δt)=Xt,j−Xt,k.(X_{t,j} + \Delta_t) - (X_{t,k} + \Delta_t) = X_{t,j} - X_{t,k}.

Thus, the refinement preserves all root-relative joint and vertex positions, with SMPL-X pose, global orientation, and shape held fixed. MPJPE and pelvis-aligned vertex errors therefore remain unchanged. Only the global translation trajectory is modified; its temporal smoothness and foot sliding are regularized by the temporal and contact losses.

W4: Limited Gains over Direct Scale Regression and Motivation for Individual Joint Tokens

Thank you for raising these questions. Tab.4 compares scale-estimation strategies under a shared 2 epoch training pilot budget. Under this setting, our method reduces W/WA-MPJPE100 from 181.1/72.4 mm to 176.8/71.8 mm. Which demonstrate the benefit of incorporating a metric skeletal reference within this controlled comparison.

The global point features are used to initialize individual joint tokens, enabling the decoder to capture joint geometric information.

  • Each joint token attends to the point-cloud memory and interacts with other joint tokens to incorporate spatial context.
  • The tokens are explicitly supervised to predict their corresponding joints through the training objective in App.1.2 (Eq. 14): $ \mathcal{L}_{\mathrm{VGMR}} = \mathcal{L}_{\mathrm{joint}} + 0.2\mathcal{L}_{\mathrm{root}} + 0.2\mathcal{L}_{\mathrm{trans}} $, where the joint-loss term measures the MSE between predicted and GT joint positions.

Unlike direct scale regression estimates scale from point cloud features alone, our method leverages the metric skeletal reference predicted by human branch benefiting from large-scale training.

  • The flattened metric skeleton encoded as a single query to aggregate the learned joint token features, allowing scale estimation to leverage multiple joint cues while reducing reliance on unreliable ones.

Further details on global-feature initialization and the separate root-bias & translation tokens are discussed in Q3.

W5/Q4: Training–inference point clouds gap

Thank you for raising this concern. We mitigate this gap through training-time augmentation and inference-time filtering.

As described in App.1.2, training uses isotropic scaling in [0.5,2.0][0.5, 2.0], Gaussian point jitter (σ=0.01\sigma = 0.01, clipped to ±0.05\pm 0.05), left–right flipping (p=0.5p = 0.5), and rotations within [−π,π][-\pi, \pi] (p=0.5p = 0.5 per axis). Scale training also perturbs the metric skeletal reference with Gaussian noise (σ=0.01\sigma = 0.01).

At inference, we select valid human-mask points with confidence ≥0.2\geq 0.2, apply median-centered radius filtering (0.9) and 16-neighbor outlier filtering (MAD-normalized threshold 3.0), then centroid-center and voxel-sample the cloud (grid size 0.01), preserving its scale. Spatial thresholds use reconstruction units; each filter retains at least 256 available points when possible.

The updated supplementary material visualize original RGB-D clouds / Augmented clouds / Pi3X clouds. Our method uses the human's overall spatial structure and reduce sensitivity to minor local noise.

W6: Missing Discussion of Limitations and Failure Modes

Thank you for pointing out this omission. We added Limitations section in updated manuscript, discussing the following limitations and the supplementary material show the failure cases.

Limitations: HER is sensitive to subject visibility and geometric quality. Distant subjects with small image footprints provide limited informative pixels, which can lead to inaccurate pose, scale, and localization. Severe occlusion, segmentation errors, motion blur, and frontend noise can also produce incomplete or unreliable human point clouds, degrading spatial reasoning. While the temporal module and final optimization improve motion continuity and smoothness, they cannot recover entirely missing visual evidence or correct pose and shape errors from the human branch.

Q3: Motivation for Individual Joint Tokens and Separate Root-Bias and Translation Tokens

Thank you for the question. We clarify the motivations for these design choices below.

  • Individual joint tokens. The global point feature initializes 22 individual joint tokens, allowing the decoder to learn joint-specific geometric representations rather than relying on a single global descriptor. Through self-attention and cross-attention to the point-cloud memory, these tokens exchange spatial context and recover geometric information from incomplete observations. Each token is explicitly supervised to predict its corresponding joint, ensuring its relevance to the underlying body structure. Global-feature initialization also provides geometric context for our lightweight 2-layer decoder.

  • Separate root-bias and translation tokens. Although both predictions are defined relative to the point-cloud centroid, they serve distinct purposes. The translation token estimates the SMPL-X translation offset, while the root-bias token predicts the pelvis position. These quantities differ because the pelvis does not coincide with the SMPL-X template origin, as explained in Appendix A.2. The root-bias token therefore provides explicit pelvis supervision alongside SMPL-X translation prediction.

To evaluate the contribution of the root-bias branch, we conduct a controlled ablation strictly following the training setup described in the paper. The only modification is removing the root-bias token and its associated prediction head and loss, while keeping all other components and training settings unchanged. Both models are trained from scratch using the full training schedule.

ModelW-MPJPE100 ↓ (mm)WA-MPJPE100 ↓ (mm)RTE ↓ (%)
Full model167.4066.141.1577
Without root-bias token171.3968.441.1670

Removing the branch increases W-MPJPE100 and WA-MPJPE100 by 3.99 mm and 2.30 mm, respectively, with a slight increase in RTE. These results support the benefit of explicit pelvis supervision in addition to SMPL-X translation prediction.

Thanks for the comprehensive and detailed rebuttal. The additional video qualitative results help demonstrate the quality of the human-scene reconstruction and the additional experiments improve the soundness of the paper. I would raise my score to weak acceptance.

We sincerely thank the reviewer for your valuable feedback, constructive suggestions, and generous increase in the score. We greatly appreciate your time and effort in reviewing our work. We truly thank for your support and encouragement.

Reviewer_du13OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HER addresses the difficulty of separating global human position from camera motion in monocular video. Its geometric frontend uses windowed reconstruction to recover camera trajectories and point maps. Visual Geometry-Mediated Reasoning (VGMR) uses the dynamic human points in those maps to locate SMPL-X bodies and uses a root-relative metric skeleton predicted by the human branch to estimate the scale from reconstruction units to meters. The system then optimizes only the human translations in world coordinates. The paper evaluates global human motion and camera trajectories, compares against an input-matched Points Align baseline, and ablates the scale, temporal, and post-processing components.

Critical strengths
  1. The method uses dynamic human geometry to constrain global localization. It excludes human regions during static scene registration but retains their point maps for VGMR to estimate human position, while the root-relative metric skeleton provides a scale estimate. This turns geometry that would otherwise be treated as a dynamic disturbance into evidence for human localization, directly addressing the position and scale ambiguities of monocular reconstruction.
  2. The experimental comparisons are thorough. The paper evaluates global human motion and camera trajectories, and uses an input-matched Points Align comparison together with scale, temporal, and post-processing ablations to examine the main design choices. These experiments provide reasonably strong support for the conclusions about human localization.
Critical weaknesses
  1. The relative placement of humans and the scene lacks direct evaluation. W-MPJPE100 and WA-MPJPE100 use Sim(3) alignment, which removes some scale error; RTE retains scale error, so the paper does provide metric evidence for human trajectories. However, human and camera trajectories are evaluated separately, and the paper does not quantify scene geometry quality or human position and contact errors relative to the scene. This limits how directly the experiments support the claimed "shared metric world frame" for human–environment relationships.
  2. The final translation optimization is insufficiently specified for reproduction. Table 3 shows that this stage substantially affects jitter and foot skating, but Appendix A.4 does not give complete formulas and weights for each loss term or specify the source of contact predictions and the thresholds that activate the contact constraints. These details appear straightforward to add, but without them it is difficult to reproduce the post-processing or attribute its gains to particular constraints.
Questions for authors
  1. Is the camera ATE in Table 2 computed after SE(3) or Sim(3) alignment? On datasets with scene ground truth, can the authors report human localization or contact error relative to the scene to directly test the claimed "shared metric world frame"?
  2. What are the exact loss terms and weights in the final translation optimization, and where do the contact predictions and geometric thresholds come from?
  3. VGMR is trained on RGB-D human point clouds from BEDLAM1, but at inference time it uses point maps predicted by the geometric frontend. Table 2 compares camera ATE across different geometry backbones. How do the human trajectory metrics change when those backbones are substituted?
4 comments

We have also provided video demonstrations of HER via an anonymous link:

https://anonymous.4open.science/w/HER-anonymous-submitted/ for convenient viewing.

The high-resolution videos remain available in the supplementary material.

W1/Q1: Lack of Direct Evaluation of Human–Scene Relationships

Thank you for suggesting a direct evaluation of human–scene consistency and scene geometry. We conduct an additional comparison on 50 RICH test videos from static cameras 00 & 01 (32 & 18 videos, respectively). For each video, we fuse static scene geometry from 100 uniformly sampled frames.

We evaluate human-scene consistency and scene geometry across 50 videos (in metric scale): Pelvis Error (3D distance between SE(3) aligned predicted and GT pelvis), Contact Distance (distance from GT-labeled contact vertices to the GT scene surface), and standard scene metrics (Accuracy, Completeness, and F-score@5cm).

MethodPelvis Position Error ↓ (mm)Contact Surface Distance ↓ (mm)Scene Accuracy ↓ (mm)Scene Completeness ↓ (mm)Scene F-score@5 cm ↑ (%)
HER264.2787.5167.2526.1477.90
Human3R366.07130.0999.6465.1755.66
UniSH496.39169.37173.3125.9748.19
OnlineHMR509.01210.5596.3541.8365.55

HER achieves the best metircs. These results support accurate human localization relative to the aligned scene and closer agreement between GT-labeled contact vertices and the GT scene surface.

W2/Q2: Reproducibility of the final translation refinement

Thank you for requesting these details. We optimize only a per-frame world-space translation residual, keeping body pose, shape, camera parameters, and scene scale fixed:

L=Lrepr+Ltemp+1000Lcontact+Linit.\mathcal{L} = \mathcal{L}_{\mathrm{repr}} + \mathcal{L}_{\mathrm{temp}} + 1000\mathcal{L}_{\mathrm{contact}} + \mathcal{L}_{\mathrm{init}}.

The reprojection term uses bbox height-normalized Smooth L1 loss. The temporal term penalizes second-order translation-residual differences and third-order world-space joint differences, with weights of 1 and 1000.

The initialization term penalizes the squared translation residual.

Contact probabilities come from the human branch, with a threshold of 0.8. The contact term penalizes foot displacement and is enabled only when the initial weighted RMS displacement is at most 0.03 m/frame and at least 80% of the contact weight satisfies this threshold. We use Adam with a learning rate of 0.003 for up to 1000 iterations.

This refinement preserves local body geometry. Its contact term suppresses foot sliding but does not directly constrain foot-to-scene distance or penetration. And we just updated the detail of this translation refinement in Appendix. A.4 of the paper.

Q1: Camera ATE alignment & human–scene consistency

Thank you for requesting this clarification. The camera ATE in Table 2 uses Sim(3) alignment. To explicitly evaluate metric-scale recovery, we additionally report SE(3)-aligned ATE, which aligns rotation and translation without fitting scale.

On all 25 EMDB2 sequences, applying our predicted metric scale reduces the mean SE(3)-aligned ATE from 2.086 m to 0.674 m:

Camera trajectorySE(3) ATE ↓ (m)
Before metric-scale recovery2.086
After HER's predicted scale0.674

Applying HER’s predicted scale substantially reduces SE(3)-aligned ATE demonstrating that our scale estimation improves metric camera trajectory accuracy. The scale is predicted without GT-based correction, and results are averaged equally across sequences.

Human–scene consistency is addressed in our response to W1.

Q3: Replacing the geometric backbone

Thank you for suggesting this experiment. We evaluated Pi3X, VGGT, and VGGT-Ω on all 25 EMDB2 sequences, keeping the human predictions and VGMR weights fixed and disabling world- space translation refinement.

BackboneW-MPJPE100 ↓ (mm)WA-MPJPE100 ↓ (mm)Sim(3) Camera ATE ↓ (m)SE(3) Camera ATE after predicted scaling ↓ (m)
Pi3X167.4066.140.2970.674
VGGT748.05329.852.0734.945
VGGT-Ω487.83182.640.4183.067

Compare to Sim(3) ATE, SE(3) ATE preserves scale errors after applying HER's predicted scale. Camera ATE is averaged over per-sequence RMSE.

Pi3X provides an approximate metric-scale prior [1], which HER further corrects using human structure. We also augment BEDLAM1 point clouds with isotropic scaling sampled uniformly from [0.5,2.0][0.5, 2.0] during training, although accurate scale estimation is not guaranteed for inputs with scale factors far outside this augmentation range.

VGGT's higher Sim(3) ATE reveals trajectory errors beyond scale mismatch;

VGGT-Ω's closer Sim(3) but higher SE(3) ATE indicates poorer metric recovery with frozen downstream weights.

As geometry, cameras, and predicted scales change together, the degradation cannot be attributed to scale alone.

[1] Pi3X official repository: https://github.com/yyfz/Pi3 # introducing-pi3x-engineering-update

Dear Reviewer du13,

Thank you for your thoughtful review. In our rebuttal, we clarify the points you raised and report additional experimental results. With about 2 days left in the discussion period, we would appreciate your feedback on whether our response addresses your concerns. We’re happy to answer any further questions throughout the discussion.

Best regards,
The Authors

Dear Reviewer du13,

As the discussion period is coming to an end, if our responses have addressed your concerns, would you kindly consider upgrading your rating from 3 to 4, or increasing your confidence in the Weak Accept recommendation? We sincerely appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.