acceptodds
Under review as a conference paper at ICLR 2027

GroundedHuman: Implicit Interaction Grounding for Dynamic Human–Scene Reconstruction

Abstract

Capturing human motion and surroundings from monocular video requires both temporally coherent reconstruction and plausible human-scene alignment. Existing methods typically trade off the two goals: feed-forward models enable efficient dynamic reconstruction but often yield weaker human-scene consistency, whereas interaction-aware methods often target static settings or incur extra overhead. This work presents GroundedHuman, a feed-forward framework that reconstructs human motion, scene geometry, and camera motion in a shared metric world. Our key insight is to introduce implicit body-scene interaction grounding into feed-forward reconstruction: the body determines where to inspect the scene geometry, producing interaction representations that enable both metric calibration and motion positioning. Specifically, scene geometry is first calibrated with the metric human body via body-aligned scene attention to establish a shared reference for scene depth and camera motion. Then, interactions between body parts and their surrounding scene context are modeled hierarchically for better positioning of human motions. Experiments show that GroundedHuman improves world-space motion estimation and human-scene coherence while reducing floating and penetration artifacts relative to recent state-of-the-art methods on real-world scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Scene Reconstruction, Monocular 3D Reconstruction, Metric Scale Recovery, Human Mesh Recovery, World-Space Human Motion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fKjYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GroundedHuman reconstructs human motion, scene geometry, and camera motion from monocular video in a shared metric frame. Starting from frozen VGGT-Ω and NLF estimators, it calibrates scene depth and camera translation through body-aligned scene queries. It then uses hierarchical regional context to refine human root translation while preserving pose and shape. Evaluation covers global motion, metric depth, human–scene consistency, and physical-grounding diagnostics.

Critical strengths

Strength 1: The complete system reports strong global-motion results.

On EMDB-2, it achieves 58.5 mm WA-MPJPE and 150.0 mm W-MPJPE, compared with 68.9 mm and 174.7 mm for JOSH (Table 1). These results demonstrate practical value for reconstructing humans and their surroundings together.

Strength 2: The grounding module provides measurable improvements within the proposed pipeline.

With calibration retained, adding grounding reduces the collision ratio from 12.83% to 6.48% and mean penetration from 4.65 cm to 2.89 cm (Table 3). The component ablations also support recomputing context after updates and separating human and environmental evidence.

Critical weaknesses

Weakness 1: The methodological advance over existing geometric-feedback methods is not sufficiently established.

GRAFT already uses body-anchored scene queries and recurrent corrective updates without explicit contact losses (Pradyumna et al., 2026, Secs. 3.2–3.3). The shared video calibration and hierarchical translation refinement are plausible extensions, but their added value remains unclear. There is no matched-initialization comparison with GRAFT or a comparable geometric refinement. Every variant in Table 3 retains calibration. The world-motion and metric-depth evaluations also lack a baseline using the same frozen estimators with only analytic scale alignment. Consequently, the reported gains do not adequately separate the contributions of the initial estimators, learned calibration, and proposed refinement.

Weakness 2: The physical evaluation does not adequately support the broader interaction claims.

HS-CF and HS-V measure agreement between the body mesh and scene-depth points on its visible surface (App. A.3). They do not evaluate contact with surrounding objects. The physical diagnostics use the lowest body vertex relative to an estimated ground height (App. A.4). They therefore cannot establish improvements in the sofa, seat, and backrest interactions emphasized in Fig. 5. Table 3 contains only internal ablations, without a quantitative physical-grounding comparison against contact-aware methods. The qualitative examples are useful, but insufficient to establish the claimed advantage over prior methods.

Questions for authors

Question 1 (Contribution attribution): With VGGT-Ω and NLF fixed, what are the global-motion and metric-depth results for analytic calibration alone, learned calibration, and the full model?

Question 2 (Interaction evidence): Can the authors provide quantitative scene-contact or penetration comparisons against UniCon3R and a GRAFT-based refinement using matched initial geometry? This would directly address the claimed improvement in interaction quality.

Question 3 (Training clarification): Which EMDB and 3DPW sequences are used for fine-tuning in Appendix B.3? Please specify the training and evaluation splits and confirm whether the evaluation sequences are held out.

Reviewer_WKrvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

GroundedHuman takes a monocular video and outputs the scene, the camera path and the human bodies in one metric world. Two frozen networks do the heavy lifting. Specifically VGGT-Ω gives scene depth, cameras and features (up to an unknown scale), and NLF gives SMPL bodies in metres. The paper adds two small learned modules. The first, Body-Aligned Metric Calibration, uses the body as a ruler. It divides body depth by scene depth under the body pixels, takes the median as the scale, and lets a small transformer with attention from 24 body anchors to local scene features predict a correction to that scale plus a depth offset; the per-frame values are combined into one clip-level scale and depth offset. The second, Hierarchical Interaction Grounding, splits the SMPL surface into 96 regions, reads scene features around each region at four window sizes, and lets each region vote on a 3D shift of the whole body; the votes are gated, averaged, clipped and applied twice. Only scale, depth offset and root translation change; pose, shape, cameras and scene geometry stay as the backbones gave them. Training is on BEDLAM, then the grounding module is fine-tuned on EMDB and 3DPW.

Critical strengths
  1. Best on all three EMDB-2 metrics (58.5/150.0/1.1 WA-MPJPE/W-MPJPE/RTE), with only scale, depth offset and root translation learned.

  2. Clean modular design, where clip-level scale is separate from per-region positioning, context re-read after each move and body pose stays untouched. Each grounding component is ablated, and removing any one raises the RICH collision ratio above the full model’s 6.48%.

  3. Table 2 consistency beats UniSH on all eight numbers and SHOW on seven.

Critical weaknesses
  1. The model may have been trained on its own test data. App. B.3 says the grounding module is fine-tuned on EMDB and 3DPW using their ground-truth body positions (Eq. 14), but EMDB-1/2 and 3DPW are exactly the test sets of Tables 1–2, and the paper never says which sequences were used. RICH is the only benchmark not used for fine-tuning, and there the picture changes. The lead over JOSH shrinks from 10.4 mm to 1.3 mm, and the method goes from 55.2 mm ahead of UniCon3R on EMDB-2 to 6.2 mm behind it

  2. There is no proof that the two new modules cause the good numbers. The obvious baseline, frozen NLF and VGGT-Ω with only the median scale of Eq. 10, nothing learned is mssing. Table 3 ablates only the grounding module and only on contact metrics, and Table 1 ablates nothing, so the 58.5 on EMDB-2 may come from the newer backbones alone. The same holds for Fig. 4(a):

  3. The camera path in Fig. 4(a) is VGGT-Ω’s own path times one number based on (Eq. 5), and the Sim(3) alignment used for scoring re-fits that number, so Sec. 4.2 itself admits the plot “does not isolate the contribution of metric calibration”. If the plot cannot show the calibration, it shows VGGT-Ω, yet VGGT-Ω is not in it.

  4. The method loses on RICH, the benchmark with the most body–scene contact, where grounding should help most. GVHMR, which does not use the scene at all, is better on all three metrics (78.8/126.3/2.4 vs. 87.7/128.2/2.5), which goes against Sec. 2’s claim that motion alone cannot give “accurate human position without sufficient constraints from scene geometry”; UniCon3R is also better on WA-MPJPE and RTE, and the method is only fifth on RTE. Sec. 4.1 does not mention any of this, the conclusion still claims gains on “EMDB, RICH, and 3DPW”.

  5. The method is presented as online without being one. It sits in Table 1’s Online block, and Fig. 4 compares it with “online reconstruction models”, yet “online” is never defined, Eq. 4 needs the whole clip before any frame can be output, and the limitations admit it is “not yet fast enough for real-time”.

  6. Physical grounding is never compared with another method. Table 3 follows UniCon3R’s protocol but is an ablation only, so Coll./Pen./Float never stand next to UniCon3R or JOSH; the metrics cover only the ground plane, not the furniture contact Fig. 5 targets, and how often the body floats is never reported, so a variant that floats more often but less far scores better.

  7. Table 2 rewards the calibration step itself. Eq. 10 resizes the scene so that it matches the body under the body mask, and HS-V/HS-CF then score how well the scene matches the body under that same mask, so a low score cannot separate correct placement from scale fitting.

  8. Novelty is thin. The “key insight” that the body tells you where to look in the scene is the premise of MetricHMSR, GRAFT, SHOW, WHAC, HAMSt3R and Human3R as Sec. 2 describes them. Also MetricHMSR already uses the body’s depth to fix the scene depth with a learned scale and offset and GRAFT already moves the body onto the scene in rounds.

  9. Important details are missing. The paper never gives δmax\delta_{max} , the loss weights, ϵown\epsilon_{own} , the training schedule or how the 96 regions are defined, despite the reproducibility statement,. The local-pose table (PA-MPJPE/MPJPE/PVE) that Sec. 4.2 says is in App. A.2 is not there, so the paper never shows how the frozen NLF pose compares with the baselines’ own pose estimators, and part of the Table 1 lead may just be a better pose estimator.

  10. A simple method is written as a complicated one. Sec. 3 denotes three easy operations (a median of depth ratios, a clip-level combination, a weighted mean of region votes) behind the symbols ΦscaleΦ_{scale} , ΦtempΦ_{temp} and ΦaggΦ_{agg} and points to App. B, where ΦtempΦ_{temp} is still not defined, only called “robust” and “aggregated in log-scale”.

Questions for authors
  1. Which EMDB/3DPW sequences are used for fine-tuning, and are they disjoint from EMDB-1/2 and the 3DPW test split? Please report EMDB-2/RICH numbers for (a) BEDLAM-only training, (b) frozen backbones plus median scale, (c) each module removed.

  2. Fig. 2’s close-ups seem to be zooms of the output of Body-Aligned Metric Calibration, so no improvement from Hierarchical Interaction Grounding is visible. What do they show?

  3. Table 1 marks UniCon3R as unable to handle several people, yet Fig. 10 shows it reconstructing two, and its method handles any number of detected people per frame. On what basis is the × given?

  4. The video shows Human3R with duplicated, drifting scene geometry; Human3R’s own paper reports that accuracy drops as the video gets longer and fixes it with a state reset every 100 frames. Was this reset used for the video results?

  5. Also address all other weaknesses.

Reviewer_RXB3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper claims that

  • it presents a feed-forward framework for monocular dynamic human–scene reconstruction.

It actually consists of

  • Frozen base models: VGGT-Ω for the scene and cameras, and NLF for the humans.
  • Two added post-processing modules, decoupled from the base models: both base models stay frozen during training and inference, run first, and receive nothing back from the modules.
    1. Metric calibration: estimates a single global scale (plus a depth bias) for the scene depth and camera translation of the whole video, using the human body as the metric reference.
    2. Interaction grounding: refines each person's root translation frame by frame from the surrounding scene context, while pose, shape and the scene stay fixed.
  • Training: both modules are trained on synthetic data (BEDLAM) to undo perturbations of the ground truth whose distribution is not specified (scaled scene depth; shifted root translation and scene scale); interaction grounding is then fine-tuned on EMDB and 3DPW.

However, the first of the three listed contributions is the framework itself ("a feed-forward framework for monocular dynamic human-scene reconstruction"), and the main results compare this whole system with other methods. Since the scene and human reconstruction come entirely from the frozen base models, I do not consider the framework a contribution of this paper; its contribution lies in the two modules.

Such post-processing modules are a separable part: they can be attached to the outputs of any existing method, at most at the extra cost of running the scene feature extractor. Their value therefore has to be shown module by module and on top of different base methods; comparing the whole system with other methods, which could equally add these modules, cannot show it.

The paper is therefore not convincing because:

  1. The introduction logic does not support the actual contribution. It motivates and positions the work as a new framework, not as two modules on top of frozen estimators.
  2. The two modules are neither properly discussed nor evaluated on their own.
  3. The evaluation setting compares whole systems, so it does not support the contribution.

In "weaknesses", I outline the discussion and experiments that would be needed if the two modules were honestly presented as post-processing that can be attached to any scene predictor (traditional or VGGT-like) combined with a human mesh estimator.

Critical strengths

The pipeline is clearly described at the level of data flow.

Critical weaknesses

1. The experiments cannot test the actual contribution

The contribution lies in two post-processing modules, but the main results only compare the whole system with other systems.

  • What the experiments show. Table 1 (world-space human motion), Fig. 4 (camera trajectory and metric depth) and Table 2 (human–scene consistency, with baseline numbers taken from SHOW) all compare complete systems. They show that this system scores better than other systems, not that the two modules are useful. Table 2 moreover measures the agreement between two predictions, the predicted body and the predicted scene, which can agree while both are wrong.
  • What is missing for the modules themselves.
    • Metric calibration has no ablation at all, and there is no baseline of the frozen VGGT-Ω and NLF without the modules. The paper itself notes that Fig. 4(a) "does not isolate the contribution of metric calibration".
    • Interaction grounding is ablated only in Table 3, on RICH alone and with physical-plausibility metrics alone (collision, penetration, floating). Its effect on accuracy (WA-MPJPE, RTE) is not reported. These metrics also measure only the lowest body vertex relative to an estimated ground height (App. A.4), so contact and penetration with other surfaces, such as chairs or walls, are not measured, although the paper claims reduced floating and penetration in general.
    • The data split is not stated. Interaction grounding is fine-tuned on EMDB and 3DPW (App. B.3), while the main results are tested on EMDB-2 (Table 1) and on 3DPW and EMDB-1 (Table 2). The paper never states which sequences were used for fine-tuning, so an overlap with the test sequences cannot be ruled out; this concerns the strongest result of the paper, on EMDB-2.
    • Metric scale, the very quantity the first module recovers, is checked against real-world sizes only on two examples (PROX, Fig. 9), without a quantitative evaluation.
  • What the experiments should be. Since the modules only read what a scene estimator and a human estimator produce (their outputs, plus the features of the scene estimator), they can be attached to any existing method. If they work, this has to be shown by attaching them to several existing methods and reporting a consistent gain, e.g., GVHMR, which already outperforms the full system on RICH (Table 1), as well as Human3R, UniSH, SHOW and TRAM. Each module should also be evaluated on its own target against ground truth: the scale error for calibration, the root-translation error for grounding.
  • Why a whole-system comparison is not enough here. Position-refinement components of this kind are interchangeable and can be stacked, e.g., a temporal post-processing step such as GVHMR's could be added on top of this system. A whole-system comparison therefore cannot tell whether these particular modules help, or whether another or an additional component would do the same.

2. The two modules are not positioned against the straightforward solutions and the state of the art of their subproblems

Because the introduction argues for a new framework, the gap is argued only at the system level, and neither module is compared, as a module, with the methods that solve the same subproblem.

  • Metric calibration (scene scale from the human body). If the idea is to let the human estimate fix the scale of the scene, the straightforward ways have to be considered or ruled out first:
    • Optimization: joint optimization of humans, scene and cameras, such as HSfM, or optimization with contact and penetration consistency, such as JOSH and PhySIC.
    • Feed-forward human cues inside the reconstruction: predicting body correspondences, SMPL or 3D keypoints together with the scene, such as HAMSt3R, Human3R and UniSH, and then computing the scale arithmetically. SHOW, a baseline of the paper, also uses body-scale priors to guide the scene reconstruction.
    • A minimal baseline: a closed-form robust fit of the scale that aligns the visible body surface with the scene depth at the same pixels.
    • MetricHMSR, which the paper itself describes as using human surfaces as anchors to refine monocular scene depth, addresses the same subproblem, but it is compared only as a whole system.
  • Interaction grounding (human placement). Refining the global position of a person is a component of many existing methods. The module should be compared with these components on the same inputs:
    • Scene-aware, learned: GRAFT (body-anchored geometric probes with recurrent refinement, very close to this module) and UniCon3R (predicted contact fed back as a correction).
    • Scene-aware, optimization-based: contact and penetration terms, as in PROX, PhySIC and JOSH.
    • Physics-based: physics simulation in a scanned scene (Embodied Scene-aware Human Pose Estimation, NeurIPS 2022, evaluated on PROX) and plug-in physics-aware refinement of any kinematic estimate (PhysPT, CVPR 2024). This family is not discussed at all, although it directly targets the floating and penetration artifacts the paper claims to reduce.
    • Scene-free, temporal: the contact-aware trajectory refinement of WHAM and the static-joint post-processing of GVHMR.
  • Training. Both modules are trained to undo synthetic perturbations of the ground truth, and only interaction grounding is then fine-tuned on real data. The paper does not show that these perturbations resemble the actual errors of the frozen base models, e.g., errors in NLF's absolute depth, or errors in pose that a rigid translation of the body cannot correct.
Questions for authors
  1. Training perturbations. Both modules are trained to undo perturbations of the ground truth whose distribution is not specified.
    • What is this distribution, and are the inputs during training the ground-truth body and depth or the predictions of NLF and VGGT-Ω?
    • For metric calibration, the described perturbation ("multiplicative depth-scale perturbations", App. B.3) appears to be one scale factor for the whole depth map, not per-pixel noise. With ground-truth inputs, a closed-form least-squares (or robust) scale-and-shift alignment between the visible body surface and the scene depth already recovers it exactly. Why was this perturbation chosen instead of the actual errors of the frozen base models, and how would training on those errors change the results?
  2. On top of a stronger human estimator. GVHMR already outperforms the full system on RICH (Table 1). Can the two modules be applied to GVHMR's output in place of NLF, and do they still improve it?

Beyond these two, I have not listed detailed questions about the method, because my main concerns are with the introduction logic and the experimental setting. If these can be addressed, I will raise more specific questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.