acceptodds
Under review as a conference paper at ICLR 2027

MetaV3RSE: 4D Human-Object-Scene Reconstruction from Monocular Video

Abstract

We introduce , a unified feed-forward framework that reconstructs the human, the manipulated object, and the surrounding scene together in one online pass from monocular video. Existing feed-forward methods address either human-object or human-scene reconstruction, not the coupled setting, so object recovery cannot use the human's scene grounding and the scene-grounded human cannot use the constraints supplied by the object. Methods that do reconstruct all three rely on per-sequence optimization. The difficulty is that the manipulated object is the least visible part of the interaction. The reconstructed human constrains where the object lies but not its precise rigid pose, while visible object evidence is precise but incomplete. We therefore extend the recurrent human-scene state with object and human-object contact representations that produce an interaction-conditioned pose hypothesis and feed interaction back to the human, then correct that hypothesis using visual evidence gathered at canonical points of the known object template. We show that reduces combined human-object reconstruction error by 24% over the strongest feed-forward baseline on BEHAVE, and joint object reasoning also improves the human, reducing WA-MPJPE by over 23% on the unseen EMDB-2 benchmark.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
human-scene reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_g7PJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes METAV3RSE, an online, feed-forward model for jointly reconstructing a human, a single manipulated rigid object (with a known metric template), and the surrounding scene from monocular video. It builds on the frozen recurrent human-scene backbone of UniCon3R/Human3R and adds two components.

The first component is an interaction-aware object initialization. Object and human-object contact prompts, initialized from the human token, are decoded together with the recurrent state. A bidirectional patch-level Interact module operates on the predicted human mask and a supplied object mask. It produces a human-side residual ΔH^ho, which is added to the scene-contact-refined human feature before SMPL-X regression (Eq. 6), and an object-side residual, which drives human-object contact and a human-relative object pose.

The second component is a mesh-indexed pose readout. Under the initial pose, 64 canonical template points are projected into the object crop. Each point combines local and global crop features with the interaction context and predicts a 3D correction. A closed-form Kabsch step then recovers the final rigid pose.

The two stages are trained sequentially, with Stage 1 frozen while Stage 2 is trained.

The claimed contributions are: (i) formulating unified online HOSI reconstruction; (ii) extending contact-guided latent refinement to objects so that "object interaction feeds back to refine the human"; and (iii) the initialization and mesh-indexed readout.

The validation consists of four parts. Human metrics are reported on EMDB-1/2, RICH and 3DPW (e.g., EMDB-2 WA-MPJPE 113.7 to 87.0 mm vs. UniCon3R). Human-object metrics are reported on BEHAVE and InterCap against the single-image feed-forward baselines HOI-TG and CONTHO (BEHAVE CDc 24.6 to 18.8 cm). The Stage-2 readout is ablated. The appendix adds analyses of occlusion, centroid/ADD-S, first-frame alignment, backbone scaling, contact, and failure modes.

Critical strengths

Strong online human reconstruction while also recovering the scene and camera. On the held-out EMDB-2 benchmark, METAV3RSE reaches 87.0 mm WA-MPJPE. This is lower than other online / world-grounded methods as reported in the UniCon3R and Human3R papers (OnlineHMR 93.5, GVHMR 111.0, WHAM 135.6, and Human3R*/UniCon3R 118.2/113.7), and approaches offline methods such as TRAM (76.4). On EMDB-1, PVE/MPJPE (75.6/65.7 mm) are also lower than those of strong single-image methods, e.g., NLF at 82.4/69.6 as reported in Human3R. The improvement holds for the lightweight ViT-S/672 backbone as well (EMDB-2 130.7 to 110.9 mm). This matters to my assessment because a single online model that reaches this level of human accuracy while jointly producing scene geometry and camera motion is practically useful, independently of where the gain comes from (see the weakness below).

An extensible way to add a new entity type to a frozen recurrent 4D reconstruction model. The image encoder, recurrent state and scene/camera heads stay frozen, and the object enters only through additional prompts and lightweight heads. The inherited capabilities are preserved: human-scene contact on RICH remains close to UniCon3R (F1 0.69 vs. 0.71), and the RICH pose metrics do not degrade. The object pose is predicted relative to the human root and then composed with the inherited camera pose, which yields a world-frame object without a separate object-to-world estimation step. This matters because unified feed-forward 4D models are increasingly used as perception backbones, and the paper offers a concrete, modular recipe for extending such a model to manipulated objects without retraining or degrading the backbone. More broadly, the problem direction is timely: extending unified feed-forward 4D reconstruction from humans and scenes to manipulated objects is a natural next step for the community, and to my knowledge this is among the first attempts to do so within a single online model.

Critical weaknesses

The paper's central claims are not yet sufficiently supported by the experiments. The three issues below concern the paper's main contributions, which is why they dominate my assessment.

1, it is unclear whether the human gains come from object interaction. The abstract states that "joint object reasoning also improves the human", citing the EMDB-2 result. As far as I can tell, EMDB provides no object annotations, and EMDB-2 consists mostly of walking and stair sequences. The paper does not explain how ΔH^ho is computed on these sequences or what role it plays there. At the same time, METAV3RSE is trained with additional real-image data (BEHAVE, InterCap, HODome) and its human heads are retrained, yet there is no ablation that removes ΔH^ho and no same-data UniCon3R control. Without such controls, it is difficult to attribute the EMDB improvement to the proposed object-to-human feedback rather than to the change in training data and protocol.

2, the evidence for the object-side contribution is limited. The comparison is restricted to two single-image methods. Under the same BEHAVE protocol that the paper adopts (the released VisTracker evaluation), VisTracker reports 7.81/8.49 cm CDh/CDo, versus 22.5/21.6 cm for METAV3RSE. Under first-frame alignment (App. C), the feed-forward method THO reports 8.49/13.36 cm, versus 40.6/43.3 cm here. I understand that VisTracker uses test-time optimization and that THO has no public checkpoint, but these reported numbers suggest a substantial gap that the paper should acknowledge and discuss. In addition, the mesh-indexed readout closely follows the final stage of HOI-TG/CONTHO: template vertices are projected with the predicted pose, image features are sampled at the projected locations, vertex positions are regressed, and an SVD-based rigid fit recovers R and t. The object-side technical novelty therefore lies mainly in conditioning this readout on the interaction context.

3, the joint human-object-scene setting is not evaluated. Objects are evaluated only on BEHAVE and InterCap, which have static cameras, in the camera frame or an aligned frame. The scene and camera outputs are evaluated only on sequences without objects. There is no measure of world-frame object trajectories or object-scene consistency. Moreover, none of the qualitative results (Figs. 2–5) shows reconstructed scene geometry at all; the scene appears only in the schematic overview of Fig. 1, whose example input (a person carrying a box outdoors) is not accompanied by a corresponding object reconstruction. As a result, the "unified HOSI reconstruction" contribution currently remains a claim rather than a demonstrated capability.

Questions for authors
  1. On EMDB, 3DPW and RICH, which have no object annotations, how is the object mask obtained, and what does ΔH^ho evaluate to (zero, skipped, or computed from some detected region)? Could you clarify how much of the EMDB-2 improvement can be attributed to this residual?

  2. Could you report, on EMDB-2 and BEHAVE CDh, (a) UniCon3R retrained with exactly your data and schedule, (b) METAV3RSE with ΔH^ho removed, and (c) METAV3RSE with the object branch removed? This is the most important experiment for my assessment.

  3. Why is Human3R† on EMDB-2 reported in Table 1 (156.7 mm) but listed as unavailable in Table 8 for the same ViT-L/896 configuration? Why are the HOI-TG and CONTHO rows identical in Table 6?

  4. Are the supplied object masks and crops ground truth on BEHAVE and InterCap? Do the baselines receive exactly the same localization? Is the InterCap test set (MILO list) disjoint from the training data by subject and by object?

  5. Please report rotation error for the prior and the final pose in Table 6. If Stage 2 mainly corrects translation, how does it differ from a translation/depth refinement head?

  6. UniCon3R's Table 1 also includes world-grounded HMR methods (e.g., WHAM, GVHMR, TRAM, JOSH, OnlineHMR), which are not included in your Table 1. Could you explain this choice? Including them would help contextualize the results; for example, GVHMR is reported at 78.8 mm WA-MPJPE on RICH, which bears on the statement that METAV3RSE achieves the best or tied-best results there.

  7. Could you clarify the target application of the proposed setting? A natural motivation for a feed-forward model is large-scale, in-the-wild use (e.g., recovering 4D human-object data from internet video). However, the method assumes a known metric template, a supplied object mask and a single rigid object, and the scene does not constrain the object directly (e.g., through object-scene contact or support). What concrete application requires online feed-forward inference and scene reconstruction under these assumptions, and why is a joint model preferable there to a modular pipeline (e.g., a human-scene reconstruction model combined with a model-based 6D object pose estimator given the same template and mask)?

5 comments
AuthorsOpenReview ↗

Response to Reviewer g7PJ (1/3)

We thank the reviewer for the thorough assessment. In short: the controlled ablation you identified as most important shows that training with the object pathway improves the human even on object-free videos; METAV3RSE is the most accurate online feed-forward method on EMDB-2 while also reconstructing the object, scene, and camera; and it places the object more accurately than the image-based feed-forward baselines.

Weakness 1 and Questions 1–2: Are the human gains due to object-to-human feedback?

Answer. The EMDB gains come from training, not from object-to-human feedback at inference. No object input is given on EMDB, 3DPW, or RICH, so the object pathway, including ΔHho\Delta H^{ho}, is never executed: it is not computed, set to zero, or applied to a dummy region (zero executions over 43,129 EMDB-2 and 106,446 RICH frames; Q1). The control you requested (Q2) shows that, under identical data and schedule, the object pathway reduces EMDB-2 WA-MPJPE from 100.4 to 87.0 mm, which rules out more data or longer training as the cause. Removing the object pathway recovers UniCon3R, so (a) and (c) are the same model.

Table G1 (extended with human-only V2V). EMDB-2 in mm, BEHAVE in cm. Human-only metrics align the human alone; official CD$_h$ uses joint human-object alignment (Stage 1).

ConfigurationObject pathwayΔHho\Delta H^{ho}EMDB-2 WA-MPJPE ↓EMDB-2 W-MPJPE ↓BEHAVE human-only CD ↓BEHAVE human-only V2V ↓BEHAVE official CD$_h$ ↓
UniCon3R (published / initialization)NoNo113.7285.69.9311.56—
(a)/(c) Human-scene only, our trainingNoNo100.4257.88.109.05—
(b) METAV3RSE w/o ΔHho\Delta H^{ho}YesNo92.2227.48.089.0229.38
METAV3RSE (full)YesYes87.0220.67.578.4722.50

How much is due to the residual. Of the 26.7 mm gain over published UniCon3R, 13.3 mm comes from the updated training procedure as a whole, 8.2 mm from object supervision, and 5.2 mm from training with ΔHho\Delta H^{ho}; these are successive differences along one ablation path. We attribute the transfer to object supervision also training the shared human predictor: the object and contact prompts are built from the human representation, and the human is supervised through the interaction-refined representation (Eq. 6). On BEHAVE, where the residual runs, it further lowers human-only CD from 8.08 to 7.57 cm; the larger change in official CD$_h$ reflects its joint alignment, and because (b) and the full model are trained separately, this comparison reflects training and inference together.

Revision. We have corrected the abstract (L026–028), which attributed the over-23% EMDB-2 gain to joint object reasoning, and Section 5.1 (L369–375); details are in Appendix H.1.

Weakness 2: Object-side contribution and comparison with specialized methods

Answer (comparisons). We do not claim parity with VisTracker or THO. VisTracker (7.81/8.49 cm human/object CD vs. our 22.5/21.6 cm, same protocol) uses per-sequence test-time optimization, but THO (8.49/13.36 cm vs. our 40.6/43.3 cm under first-frame alignment) is also feed-forward, so the gap is not explained by optimization alone. A direct comparison with THO is further limited by protocol differences: our rescoring of HOI-TG gives 62.0 cm object CD, whereas THO reports 24.7 cm for it. Our advantage is over the image-based feed-forward baselines evaluated with identical frames and code: combined CD 18.8 vs. 24.6 cm for HOI-TG on BEHAVE, and object CD 24.9 vs. 33.4 cm under a single sequence-level alignment (Table R8). Their object V2V and acceleration errors are lower than ours, so this advantage holds for surface and centroid error only.

Answer (novelty). Our contribution is to condition template projection, feature sampling, and closed-form rigid fitting on human-object interaction inside an online human-scene model: the interaction places the initial pose relative to the scene-grounded human, decides where the template points read image evidence, conditions their corrections, and feeds a correction back to the human. With the same initialization, removing interaction conditioning from the readout raises object CD from 21.6 to 24.0 cm on BEHAVE and from 26.9 to 30.7 cm on InterCap (Table 4 of the paper).

Revision. Section 4 now opens with the principle behind the interaction-conditioned readout (L226–234).

AuthorsOpenReview ↗

Response to Reviewer g7PJ (2/3)

Weakness 3: The joint human-object-scene setting is not demonstrated clearly

Answer. We agree that the submission evaluated human-scene and human-object reconstruction separately. We add two pieces of evidence, each with a limited scope. Quantitatively, with one alignment kept for each whole BEHAVE sequence, METAV3RSE has lower object surface and centroid error than HOI-TG and CONTHO, which supports more accurate object placement over the sequence. It does not establish temporal consistency (our object acceleration error remains higher), uninterrupted tracking under a moving camera (BEHAVE has a static camera, and we process it in 16-frame windows with state resets), or object-scene consistency, which we do not model. Qualitatively, the revised Fig. 1 and the new Fig. 3 show the human, object, and scene reconstructed together in one metric frame.

Object trajectories (Table R8, all 85 BEHAVE sequences). Each protocol estimates one transformation per sequence from the human and object vertices and keeps it fixed. CD$_o$ and centroid error in cm; lower is better.

MethodStageFirst-frame Sim(3) CD$_o$ ↓Centroid ↓Sequence Sim(3) CD$_o$ ↓Centroid ↓Sequence rigid CD$_o$ ↓Centroid ↓
HOI-TG—62.047.929.124.333.429.4
CONTHO—60.847.129.024.233.429.3
METAV3RSEStage 163.147.736.427.440.432.5
METAV3RSEStage 1 + 243.334.024.118.224.919.5

Stage 1 alone is less accurate than the baselines; the gain comes from the full model, whose readout reduces object CD from 40.4 to 24.9 cm under rigid alignment.

Joint reconstruction (new Fig. 3). For the same inputs, CONTHO reconstructs the human and object without the scene, UniCon3R the human and scene without the object, and METAV3RSE all three in one metric frame. This shows the joint output, not its accuracy.

Revision. Section 6 now states these limits (L536–539); Table R8 is in Appendix H.2, and the comparison figure is Fig. 3.

Question 3: Numerical inconsistencies in Tables 1, 6, and 8

Both are presentation errors, now corrected; no results change.

Tables 1 and 8. The ViT-L/896 Human3R† EMDB-2 entries were omitted from Table 8 by mistake; we have restored them from Table 1 (L1204).

Table 6. HOI-TG and CONTHO coincide only at the displayed precision; both use template-based regression with the same ground-truth object localization:

MethodCentroid ↓ (cm)ADD-S ↓ (cm)ADD-S<10%d ↑ (%)
HOI-TG18.9459.87060.402
CONTHO18.9339.86960.337

We have added these values to the Table 6 caption (L1081–1088).

Question 4: Ground-truth localization, input parity, and the InterCap split

Answer. Yes, the object masks are ground truth, for all methods. The inputs are not identical, and they favor the baselines: HOI-TG and CONTHO also receive ground-truth human masks and boxes, which METAV3RSE predicts itself. The InterCap split is disjoint by subject (we train on subjects 01–08 and test on 09–10) but not by object: the same ten objects appear in training, as is standard for template-based evaluation. METAV3RSE additionally uses calibrated intrinsics; the baselines also load the dataset camera parameters, though not identically.

InputMETAV3RSEHOI-TGCONTHO
Object maskGround truthGround truthGround truth
Human mask and boxPredictedGround truthGround truth

Predicted masks. Replacing the ground-truth object masks with SAM 3 [R1] predictions on the same 1,396 InterCap frames degrades all three methods by similar amounts (Table R6). This is a shared difficulty, but it does not remove our dependence on ground-truth object localization, which we state as a limitation.

MethodGT-mask CD$_o$ ↓SAM 3-mask CD$_o$ ↓Increase
HOI-TG26.8766.8539.98
CONTHO27.7664.5736.81
METAV3RSE26.9965.9238.93

Revision. We have documented these inputs and the split in Section 5 (L312–314, L358–361) and the localization limitation in Section 6 (L534–536).

AuthorsOpenReview ↗

Response to Reviewer g7PJ (3/3)

Question 5: Does Stage 2 improve rotation, and how does it compare with a simpler translation/depth head?

Answer. No: Stage 2 improves placement, not orientation. Centroid error falls from 22.4 to 15.8 cm, while rotation error changes from 45.3° to 46.9° (Table R5); most of the orientation comes from the interaction-aware initialization, which is already more accurate in rotation than HOI-TG and CONTHO. We have not compared against a translation- or depth-only refinement head, so we do not claim that the mesh-indexed readout outperforms that simpler alternative. The closest evidence is the direct pose-regression variant in Table 4 of the paper, which predicts the rigid pose from the same features: the readout lowers object CD from 22.3 to 21.6 cm on BEHAVE and from 29.9 to 26.9 cm on InterCap.

MethodRotation ↓ (°)Centroid ↓ (cm)ADD-S ↓ (cm)
HOI-TG50.718.99.9
CONTHO51.018.99.9
METAV3RSE, Stage 145.322.412.4
METAV3RSE, Stage 246.915.89.4

Revision. We have added rotation error to Table 6 (L1081–1088) and stated in Appendix B.2 (L1130–1133) that Stage 2 mainly corrects placement.

Question 6: Why are stronger world-grounded methods missing?

Answer. We agree they should be included, and they change our claim. On EMDB-2 WA-MPJPE, the offline methods JOSH, PromptHMR-Vid, and TRAM outperform METAV3RSE (Table G2, global response). Among online methods, METAV3RSE has the lowest error (87.0 vs. 93.5 mm for OnlineHMR) while also reconstructing the object, scene, and camera. On RICH, GVHMR is slightly better on WA-MPJPE (78.8 vs. 79.1 mm), so our "best or tied-best" statement in Section 5.1 was too broad.

Revision. We have added this comparison to Appendix H.6 (Table 19) and restricted the accuracy claims in Section 5.1 (L373–378).

Question 7: Target application, and why a joint model over a modular pipeline?

Application. Our broader goal, as stated in the introduction, is capturing people interacting with objects in their surroundings for mixed reality, robotics, and embodied AI, for example building digital twins of everyday activity. The current assumptions fit settings where objects are known in advance: recording human demonstrations for robot learning, such as package handling or tool use with CAD-modeled objects, or capturing interactions with scanned objects in digital-twin environments. In both, the needed output is human motion, object pose, contact, camera, and scene in one metric frame, produced as the video arrives. These assumptions (a known template, an object mask, a single rigid object) limit in-the-wild use, and we state them as limitations in Section 6.

Joint vs. modular. We find a measured trade-off rather than an accuracy advantage: pairing our human-scene reconstruction with MegaPose [R2], given the same inputs, yields more accurate object poses (object CD 23.37 vs. 26.99 cm), while the joint model is 4.5× faster and also predicts human-object contact and refines the human from the interaction. The lower official CD$_h$ of the joint model is an alignment effect, not better human geometry, since both pipelines share the same human vertices.

PipelineOfficial CD$_h$ ↓Object CD ↓Object V2V ↓FPS ↑
METAV3RSE (joint)7.4026.9932.991.17
METAV3RSE human + MegaPose [R2]8.6223.3728.750.26

Revision. We report this comparison in Appendix H.3 and state in Section 6 that specialized pose estimators remain more accurate on the object and that the method assumes known templates and object masks (L534, L538–539).

References:

[R1] N. Carion et al. SAM 3: Segment Anything with Concepts. ICLR 2026.
[R2] Y. Labbé et al. MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare. CoRL 2022.

Reviewer_Y6JFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MetaV3RSE, a unified feed-forward framework for 4D HOSI reconstruction from monocular video with bi-directional human-object interaction refinement and a mesh-indexed object pose readout. Experiments demonstrate SOTA performance in 3D HMR, 3D HOI over existing methods.

Critical strengths
  1. This is pioneering paper that tackles HOSI that extends HSI and HOI reconstruction.
Critical weaknesses
  1. The paper sounds more of extension of UniCon3R paper rather than standalone research paper. While extending HSI to HOSI is promising, it is difficult to grasp the need for modeling object as objects are already part of scene in HSI.
  2. One of the main advantage of HSI methods like UniCon3R is that such methods do not require known 3D object template. However, this method refrains itself to need known 3D object template, which restricts its use when there is no known 3D object template.
  3. It is difficult to follow the core principles of the proposed methods. Most of the texts in the paper are used for explaining technical details.
Questions for authors
  1. Isn't human-object reconstruction a subset of human-scene reconstruction? Why does the paper claim "Existing feed-forward methods address either human-object or human-scene reconstruction, not the coupled setting" in L014-L015?
  2. What is human's scene grounding and scene-grounded human in L016?
  3. There is insufficient explanation in the paper regarding "online" aspect of METAV3RSE. While this is mentioned in some parts of the paper, the paper never mentions why the "online" claim is mentioned when explaining core contribution of METAV3RSE.
  4. How is METAV3RSE ran and evaluated on 3DPW, EMDB, RICH (Table 1, Table 2) that do not have known 3D object template?
  5. Could you help us understand why there are such huge improvement by METAV3RSE in EMDB-1, EMDB-2 datasets? To my knowledge, the human modules don't differ much from UniCon3R but performance differs significantly. What part of the method contribute to this improved performance?
  6. Generally, there needs more explanation on "why" METAV3RSE outperform previous methods in their corresponding tasks. What component of the method is useful for capturing 3D human motion, 3D HOI reconstruction, 3D HSI reconstruction?
  7. There should be ablation study on HSI vs HOSI setting in METAV3RSE framework.
  8. Some related works are missing: HOI-Gaussian (CVPR 2025, Wen et al.), TeHOR (CVPR 2026, Nam et al.).
  9. Please state if code will be released.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Y6JF (1/3)

Weakness 1 and Questions 1–2: Distinction from UniCon3R, explicit object reconstruction, and "scene-grounded human"

Answer (W1). The image encoder, recurrent backbone, scene/camera heads, and human-scene contact refinement are inherited from UniCon3R; template projection and closed-form rigid fitting are established in CONTHO and HOI-TG. Our contribution is to use human-object interaction in both directions within an online human-scene model: it places the object relative to the scene-grounded human, decides where and how the object template reads image evidence, and feeds a correction back to the human. Removing this interaction conditioning raises object CD from 21.6 to 24.0 cm on BEHAVE (Table 4 of the paper).

Q1: Isn't HOI a subset of HSI? Physically yes, but not in outputs: UniCon3R's scene pointmap may contain the object's visible surfaces, but not its complete rigid shape, 6-DoF pose, or contact with the body, which METAV3RSE adds.

Method familyHumanSceneObject 6-DoF poseHuman-object contact
Human-scene (UniCon3R, Human3R)✓✓✗✗
Human-object (HOI-TG, CONTHO)✓✗✓✓
METAV3RSE✓✓✓✓

Q2: "Scene-grounded human." A human reconstructed in the same metric world frame as the scene. METAV3RSE uses it as the anchor for the object: it predicts the object's pose relative to the body (Section 4.1) and maps it into the world frame with the predicted camera. We do not model object-scene contact or support.

Revision. We have revised the abstract (L014–016) and the introduction (L039–041):

Existing feed-forward methods address either human-object or human-scene reconstruction; the latter may capture the visible surfaces of a manipulated object, but not its rigid pose or contact with the body.

We posit that joint reconstruction allows the scene-grounded human, i.e., the human reconstructed in the same metric world frame as the scene, to guide object placement under occlusion, while object interaction refines the human in return.

Weakness 2: Dependence on a known 3D object template

Answer. We agree that METAV3RSE requires a rigid metric template, unlike template-free human-scene methods, and this requirement remains. However, it does not need a pre-existing CAD model: in a bounded feasibility test, templates generated at test time give comparable error without retraining.

Evidence. We replace the CAD templates with SAM 3D [Y1] templates generated from the first frame in which the object is visible, scaled using the predicted human, on the first six BEHAVE test sequences (one backpack and five boxes; 96 frames).

Template sourceObject CD ↓ (cm)
Released CAD template20.59
Generated at test time (SAM 3D [Y1])16.73

The metric reflects both shape and placement, so this does not show that the generated shapes are more accurate, and the result varies by object (e.g., boxtiny: 18.54 vs. 17.59 cm). The test is small and dominated by boxes, so it does not establish generalization to unseen objects.

Revision. We state the template requirement as a limitation (L534–535) and report this test in Appendix H.5.

Weakness 3: The core principles of the proposed method are difficult to follow

We agree, and have added the following overview at the start of Section 4 (L226–234); the method, equations, and section structure are unchanged.

The manipulated object is often the hardest part of the interaction to reconstruct, as it is small and partly occluded by the person. The reconstructed human indicates roughly where the object lies but not its exact rigid pose, while visible object pixels are precise but incomplete. METAV3RSE combines the two in two steps (Fig. 2). First, human and object image regions exchange features to predict an initial object pose relative to the human, human-object contact, and a residual update to the human (Section 4.1). Second, this pose determines where points of the known object template read image evidence; their corrected 3D positions are fitted with a single rigid pose in closed form (Section 4.2). The human thus initializes the object, image evidence refines it, and rigidity preserves its shape, without per-sequence optimization.

AuthorsOpenReview ↗

Response to Reviewer Y6JF (2/3)

Question 3: Why is online reconstruction a central contribution?

Answer. Online means causal, not real-time: predictions at frame tt use only frames up to tt, with no per-sequence optimization. It matters because existing methods that reconstruct the human, object, and scene together (e.g., HOSNeRF, RHINO) optimize over the full video, whereas METAV3RSE updates all outputs as frames arrive, as needed for streaming uses such as recording robot demonstrations. Causal human-scene reconstruction is inherited from UniCon3R; our contribution is extending it to the manipulated object without test-time pose optimization.

Evidence and scope. Changing future frames leaves earlier predictions unchanged (zero numerical difference). The model runs at about 1.1 FPS on a single RTX 8000, so it is online but not real-time. On BEHAVE, we process 16-frame windows with the recurrent state reset between windows, so our results show causal reconstruction within windows, not uninterrupted state across full sequences.

Questions 4, 5, and 7: Evaluation without object templates, the size of the EMDB gains, and the HSI vs. HOSI ablation

Answer. On 3DPW, EMDB, and RICH, no object template, mask, or pose is given, so the object pathway, including ΔHho\Delta H^{ho}, never runs (zero executions over 43,129 EMDB-2 and 106,446 RICH frames). METAV3RSE then runs the same human-scene pathway as UniCon3R with different learned weights, so the EMDB gains come from training, not from object modules at test time. EMDB and 3DPW are never used for training.

Evidence (Table G1, global response). Table G1 is also the requested HSI vs. HOSI ablation (Q7), with identical data and schedule: HSI is row (a), with the object pathway removed (the UniCon3R architecture); HOSI without human-side feedback is row (b); full HOSI is the last row. On EMDB-2, WA-MPJPE goes from 113.7 mm (published UniCon3R) to 100.4 mm with our training procedure as a whole, 92.2 mm with object supervision, and 87.0 mm with ΔHho\Delta H^{ho} (Q5). EMDB-1 shows the same trend:

ConfigurationPVE ↓MPJPE ↓PA-MPJPE ↓
UniCon3R (published)94.381.749.8
(a) HSI, our training89.977.948.7
(b) HOSI without ΔHho\Delta H^{ho}86.374.847.8
Full HOSI (METAV3RSE)75.665.745.0

On BEHAVE, where objects are present, human-only CD is 8.10, 8.08, and 7.57 cm for HSI, HOSI without ΔHho\Delta H^{ho}, and full HOSI. We attribute the transfer to object supervision also training the shared human predictor: the object and contact prompts are built from the human representation, and the human is supervised through the interaction-refined representation (Eq. 6). Beyond accuracy, HOSI adds the object's 6-DoF pose and human-object contact, while the scene and camera modules stay frozen.

Revision. We have corrected the abstract (L026–028) and Section 5.1 (L369–375) to attribute the EMDB gains to training; details are in Appendix H.1.

Question 6: Which components contribute to improvements in human motion, HOI, and HSI reconstruction?

Each task benefits from a different part of METAV3RSE, and not every component improves every task.

Human motion. The gains come from training: the updated training procedure, object supervision, and the object-to-human residual reduce EMDB-2 WA-MPJPE from 113.7 to 87.0 mm, although the object pathway does not run on EMDB (Questions 4, 5, and 7).

Human-object reconstruction. The interaction-aware initialization sets most of the object's orientation (rotation error 45.3° vs. 50.7° and 51.0° for HOI-TG and CONTHO), and the mesh-indexed readout mainly corrects its placement (centroid error 22.4 → 15.8 cm on BEHAVE). The readout's interaction conditioning matters: removing it raises object CD from 21.6 to 24.0 cm on BEHAVE (Table 4 of the paper). The readout does not improve orientation, and our object acceleration error remains higher than the baselines.

Human-scene reconstruction. The scene, camera, and human-scene contact modules are inherited and frozen, so we claim preservation, not improvement: on RICH, human-scene contact F1 stays close to UniCon3R (0.69 vs. 0.71) and RTE is unchanged at 1.7%.

Revision. Section 4 (L226–234) and Section 5.1 (L369–375) now state these separate roles.

AuthorsOpenReview ↗

Response to Reviewer Y6JF (3/3)

Question 8: Missing related work (HOI-Gaussian and TeHOR)

We have added both to the related work (L134–136). We understand HOI-Gaussian to refer to Gaussian-HOI by Wen et al. [Y2], a training-free, optimization-based method for open-vocabulary human-object reconstruction from a single image. TeHOR [Y3] reconstructs a human and object from a single image using text and appearance cues, including non-contact interactions. Both are single-image methods, whereas METAV3RSE reconstructs the human, object, camera, and scene from video in one metric frame.

Question 9: “Please state whether the code will be released.”

Yes. As stated in the revised abstract (L028–029), we will release the code, trained checkpoints, model configurations, and evaluation scripts upon acceptance.

References:

[Y1] Chen et al., SAM 3D: 3Dfy Anything in Images. CVPR 2026.
[Y2] B. Wen, D. Huang, Z. Zhang, J. Zhou, J. Deng, J. Gong, Y. Chen, L. Ma, and Y.-L. Li. Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions. CVPR 2025.
[Y3] H. Nam, D. S. Jung, and K. M. Lee. TeHOR: Text-Guided 3D Human and Object Reconstruction with Textures. CVPR 2026.

Reviewer_WYX1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The method extends UniCon3R (which builds on CUT3R and Human3R). Given an input monocular video of a person interacting with an object, along with an object segmentation mask, calibrated intrinsics, and a known metric-scale object template, this method performs reconstruction of the manipulated object pose in addition to contact-aware human and scene reconstruction from videos. Similar to UniCon3R and Human3R, it accomplishes this by adding additional prompt tokens and readout heads for the human-object contact and object pose relative to the body. Similar to the scene contact residual from UniCon3R, an object=>human residual provides feedback from the human-object interaction module back to the human latent before the human pose prediction head. After predicting the initial object pose in this manner, the object pose is then applied to transform a canonical object template into the predicted object pose, and the transformed 3D points are projected onto the image and used to select corresponding image features which are used to predict a per-point 3D offset for 64 points on the object template mesh. Kabsch alignment is then used to solve for the final object pose by aligning the object template points to the offset template points.

Critical strengths

Estimating the object in addition to human and scene all together in a single forward pass is interesting compared to prior methods, which either require slam (e.g. Droid-Slam) or perform some per-scene optimization or only estimate subsets of these three quantities in a feed-forward manner.

The method achieves improved results on human reconstruction in Table 1 and Table 2 compared to its direct baselines (UniCon3R, Human3R, etc.)

The mesh-indexed object pose readout slightly improves object chamfer distance according to ablation study, mainly by reducing translation error.

Critical weaknesses

The claim that joint object reasoning (i.e. the object => human residual) is what improves the human is not ablated. E.g. Most of the sequences in EMDB-2 do not really have manipulated objects so how can joint object reasoning improve human pose estimation on that dataset? It is unclear whether the improved human estimation results on EMDB-2 in Table 1 are from the object=>human residual, or simply from training on additional datasets (e.g. HODome, Behave-Date01, InterCap, Behave, etc.) compared to UniCon3R. An ablation study removing the object => human residual while keeping the new training recipe could help disentangle this claim.

Although the method beats baselines on methods that don't use external slam systems, existing published Droid-Slam based non-feedforward methods still outperform this method on EMDB-2 WA-MPJPE (e.g. TRAM 76.4, PromptHMR 71.0, JOSH 68.9, etc. compared to this method's 87.0). CameraHMR also gets better MPJPE on 3DPW (62.1 without fine-tuning or 56.0 with fine-tuning, compared to this method's 72.0)

Questions for authors

How is this method different from UniCon3R on videos within evaluation datasets (e.g. EMDB, 3DPW, RICH, etc.) where there is no interaction object in the video? Does the pipeline run with empty object masks / object meshes and is the object-to-human residual still active.

2 comments
AuthorsOpenReview ↗

Response to Reviewer WYX1

We thank the reviewer for the thoughtful review and valuable suggestions, and respond to each comment below.

Weakness 1: Are the human gains due to object-to-human feedback or to additional training data?

Answer. The EMDB gains come from training, not from object-to-human feedback at inference. No object input is given on EMDB, 3DPW, or RICH, so the object pathway, including ΔHho\Delta H^{ho}, never runs there: it is not computed, set to zero. Under identical data and schedule, adding the object pathway reduces EMDB-2 WA-MPJPE from 100.4 to 87.0 mm, so the gain is not explained by more data or longer training.

Evidence (Table G1, global response). From published UniCon3R (113.7 mm), the updated training procedure as a whole gives 100.4 mm, object supervision 92.2 mm, and training with ΔHho\Delta H^{ho} 87.0 mm; these are successive differences along one ablation path. We attribute the transfer to object supervision also training the shared human predictor: the object and contact prompts are built from the human representation, and the human is supervised through the interaction-refined representation (Eq. 6). On BEHAVE, where the residual runs, it further lowers human-only CD from 8.08 to 7.57 cm; as these models are trained separately, this reflects training and inference together.

Revision. We have corrected the abstract (L026–028) and Section 5.1 (L369–375), which attributed the EMDB-2 gain to joint object reasoning; details are in Appendix H.1.

Weakness 2: Stronger SLAM-based and specialized methods outperform METAV3RSE

Answer. We agree, and narrow our claim accordingly. On EMDB-2 WA-MPJPE, JOSH [R3] (68.9 mm), PromptHMR-Vid [R2] (71.0 mm), and TRAM [R1] (76.4 mm) outperform METAV3RSE (87.0 mm); all three are offline, processing the full video with SLAM or per-sequence optimization. Among online methods, METAV3RSE has the lowest error (87.0 vs. 93.5 mm for OnlineHMR [R7]) and is the only one that also reconstructs the object, scene, and camera. On 3DPW, the single-image method CameraHMR [R4] also achieves lower MPJPE (62.1 vs. 72.0 mm).

Evidence. Table G2 in the global response compares all methods, including WHAM [R5] and GVHMR [R6], with their inference requirements and outputs.

Revision. We have added this comparison to Appendix H.6 (Table 19) and restricted our accuracy claims to online methods (L373–378). Our contribution is unified online human-object-scene reconstruction, not state-of-the-art human accuracy.

Question 1: How does METAV3RSE differ from UniCon3R when no object is present? Are empty masks or meshes used, and is the residual active?

No empty masks or meshes are used, and the residual is not active: on EMDB, 3DPW, and RICH, no object input is given, so the object pathway, including ΔHho\Delta H^{ho}, is never executed. METAV3RSE then runs the same human-scene pathway as UniCon3R and differs only in its learned human weights, which were trained with object supervision and the residual. This is why the EMDB gains are a training effect (Weakness 1 and Table G1); on BEHAVE, where objects are present, the residual runs and refines the human directly.

References

[R1] Y. Wang et al. TRAM: Global Trajectory and Motion of 3D Humans from In-the-Wild Videos. ECCV 2024.
[R2] Y. Wang et al. PromptHMR: Promptable Human Mesh Recovery. CVPR 2025.
[R3] Z. Liu et al. Joint Optimization for 4D Human-Scene Reconstruction in the Wild (JOSH). ICLR 2026.
[R4] P. Patel et al. CameraHMR: Aligning People with Perspective. 3DV 2025.
[R5] S. Shin et al. WHAM: Reconstructing World-Grounded Humans with Accurate 3D Motion. CVPR 2024.
[R6] Z. Shen et al. World-Grounded Human Motion Recovery via Gravity-View Coordinates (GVHMR). SIGGRAPH Asia 2024.
[R7] Y. Zhao et al. OnlineHMR: Video-Based Online World-Grounded Human Mesh Recovery. CVPR 2026.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Revised Manuscript

Dear Reviewer WYX1,

As the discussion period is nearing its end, we wanted to kindly check whether our rebuttal, additional experiments, and revised manuscript have addressed your concerns.

We appreciate your thoughtful suggestions, which motivated the training-matched ablations, clarification of the object pathway's behavior on human-only benchmarks, and comparisons with specialized reconstruction methods. These have helped us strengthen the paper.

We would be happy to clarify any remaining questions or concerns.

Best regards,
Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response (1/2)

We sincerely thank all reviewers for their careful assessment. We are encouraged that the reviewers found reconstructing the human, object, and scene in a single forward pass "interesting compared to prior methods" (WYX1), the work "pioneering" for this setting (Y6JF), and the online reconstruction "practically useful" (g7PJ).

We have uploaded a revised PDF. Revised text is shown in blue, which is unrelated to the blue module names in the paper’s equations and figures. Line numbers refer to the revised PDF.

In summary, the paper establishes:

  1. First online model for joint human, object, and scene reconstruction. To our knowledge, METAV3RSE is the first online, feed-forward model that reconstructs the human, the manipulated object, the camera, and the scene in one metric frame, with their contacts, and without SLAM or per-video optimization.
  2. Most accurate online global human motion. On EMDB-2, METAV3RSE has the lowest global human-motion error among all online methods compared, 23% below the next best (Table G2).
  3. Learning about objects improves the human, even without objects. With identical data and training, adding the object pathway reduces global human-motion error on EMDB-2 by 14%, on videos that contain no objects (Table G1). We attribute this to the object losses also training the shared human predictor.
  4. More accurate object placement. METAV3RSE places the manipulated object more accurately than HOI-TG and CONTHO, with 25% lower error when a single alignment is kept for the whole video (Table R8).

Together, these results support a training-time transfer from object supervision to the human, interaction-conditioned object placement, and a faster integrated pipeline.


Issue 1: Separate the contributions of training data, object supervision, and object-to-human feedback to human reconstruction.

Addressed in: WYX1 — Weakness 1, Question 1; Y6JF — Questions 4, 5, 7; g7PJ — Weakness 1, Questions 1–2.

Conclusion. The EMDB gains are a training effect, not object-to-human feedback at inference: the object modules, including ΔHho\Delta H^{ho}, never run on EMDB, 3DPW, or RICH. Of the 26.7 mm EMDB-2 WA-MPJPE improvement over published UniCon3R, 13.3 mm comes from the updated training procedure (as a whole, not the additional data alone), 8.2 mm from object supervision, and 5.2 mm from training with the object-to-human residual; these are successive differences along one ablation path, not a unique decomposition. On BEHAVE, where the residual runs, it further lowers human-only CD (8.08 → 7.57 cm). We have revised the abstract (L026–028) and Section 5.1 (L369–375) accordingly.

Table G1: Training-matched ablation. EMDB in mm, BEHAVE in cm. Labels (a)–(c) follow g7PJ's Question 2. Human-only CD aligns the human alone; official CD$_h$ uses the benchmark's joint human-object alignment.

ConfigurationObject branchΔHho\Delta H^{ho}EMDB-2 WA-MPJPE ↓EMDB-2 W-MPJPE ↓EMDB-1 PVE ↓BEHAVE human-only CD ↓BEHAVE official CD$_h$ ↓
UniCon3R (published/initialization)NoNo113.7285.694.39.93—
(a)/(c) UniCon3R, our data and scheduleNoNo100.4257.889.98.10—
(b) METAV3RSE without ΔHho\Delta H^{ho}YesNo92.2227.486.38.0829.38
METAV3RSE (full)YesYes87.0220.675.67.5722.50

Rows (a) to full share initialization, data, and schedule. Configuration (b) and the full model are trained separately, so the BEHAVE comparison reflects the residual's training and inference effects together.

AuthorsOpenReview ↗

Global Response (2/2)

Issue 2: Compare with stronger specialized human reconstruction methods.

Addressed in: WYX1 — Weakness 2; g7PJ — Question 6; Y6JF — Weakness 1, Question 1.

Conclusion. On EMDB-2 WA-MPJPE, JOSH (68.9 mm), PromptHMR-Vid (71.0 mm), and TRAM (76.4 mm) outperform METAV3RSE (87.0 mm); all three are offline, processing the full video with SLAM or per-sequence optimization. Among online methods, METAV3RSE has the lowest error (87.0 vs. 93.5 mm for OnlineHMR) and is the only one that also reconstructs the object. We have added this comparison to Appendix H.6 (Table 19) and restricted our accuracy claims accordingly (L373–378).

Table G2: EMDB-2 accuracy, requirements, and outputs. WA-/W-MPJPE in mm, RTE in %; lower is better. Online: uses only current and past frames. Bold marks our method. HS/HO: human-scene/human-object contact.

MethodOnlineNo SLAM/VO or test-time opt.OutputsWA-MPJPE ↓W-MPJPE ↓RTE ↓
JOSHNoNoHuman, scene, camera68.9174.71.3
PromptHMR-VidNoNoHuman, camera71.0216.51.3
TRAMNoNoHuman, camera76.4222.41.4
GVHMRNoNoHuman, camera111.0276.52.0
WHAMConditional*NoHuman, camera135.6354.86.0
OnlineHMRYesNoHuman, camera93.5310.42.2
UniCon3RYesYesHuman, scene, camera, HS113.7285.62.3
METAV3RSEYesYesHuman, object, scene, camera, HS/HO87.0220.62.2

WHAM's motion network is online, but its world trajectory relies on an external SLAM pipeline.


Issue 3: Clarify the novelty relative to UniCon3R and HOI-TG/CONTHO, and the role of each object component.

Addressed in: Y6JF — Weaknesses 1, 3, Questions 1, 2, 6; g7PJ — Weakness 2, Question 5.

Novelty. The image encoder, recurrent backbone, scene/camera heads, and human-scene contact refinement are inherited from UniCon3R; template projection and closed-form rigid fitting are established in CONTHO and HOI-TG. Our contribution is to use human-object interaction in both directions within an online human-scene model: it places the object relative to the scene-grounded human, decides where and how the object template reads image evidence, and feeds a correction back to the human. Removing this interaction conditioning raises object CD from 21.6 to 24.0 cm on BEHAVE (Table 4 of the paper).

Components. The initialization sets most of the object's orientation, and the readout mainly corrects its placement (Table R5, g7PJ Q5).

Revision. Section 4 now opens with this principle (L226–234).


Issue 4: Strengthen the evidence for object trajectories and the joint human-object-scene setting.

Addressed in: g7PJ — Weaknesses 2–3, Question 5; Y6JF — Question 6.

Conclusion. With one alignment kept for each whole BEHAVE sequence, METAV3RSE has lower object surface error (24.9 vs. 33.4 cm) and object-centroid error (19.5 vs. 29.4 and 29.3 cm) than HOI-TG and CONTHO (Table R8), which supports more accurate object placement over the sequence. It does not establish temporal consistency (object acceleration error remains higher than the baselines'), uninterrupted tracking under a moving camera (BEHAVE has a static camera, and we process it in 16-frame windows with the state resets), or object-scene consistency, which we do not model. The revised Fig. 1 and the new Fig. 3 show a qualitative joint reconstruction of a moving-camera sequence. We state these limits in Section 6 (L536–539); details are in Appendix H.2.


Issue 5: Clarify assumptions, evaluation fairness, and joint versus modular reconstruction.

Addressed in: Y6JF — Weakness 2, Questions 3, 4, 8, 9; g7PJ — Questions 3, 4, 7; WYX1 — Question 1.

Conclusion. The evaluation gives the baselines more ground-truth localization than our method: all use ground-truth object masks, but HOI-TG and CONTHO also receive ground-truth human masks and boxes, which METAV3RSE predicts itself. With predicted SAM 3 masks, all three methods degrade by similar amounts (Table R6); this is a shared difficulty, but our dependence on ground-truth object localization remains a limitation. A modular pipeline with MegaPose is more accurate on the object, while the joint model is 4.5× faster and also predicts contact (Table R7). Templates generated at test time with SAM 3D give comparable error in a small feasibility test, but the method still requires a template. We state these assumptions in Section 5 (L312–314, L358–361) and Section 6 (L534–536); details are in Appendix H.3–H.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.