- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes METAV3RSE, an online, feed-forward model for jointly reconstructing a human, a single manipulated rigid object (with a known metric template), and the surrounding scene from monocular video. It builds on the frozen recurrent human-scene backbone of UniCon3R/Human3R and adds two components.
The first component is an interaction-aware object initialization. Object and human-object contact prompts, initialized from the human token, are decoded together with the recurrent state. A bidirectional patch-level Interact module operates on the predicted human mask and a supplied object mask. It produces a human-side residual ΔH^ho, which is added to the scene-contact-refined human feature before SMPL-X regression (Eq. 6), and an object-side residual, which drives human-object contact and a human-relative object pose.
The second component is a mesh-indexed pose readout. Under the initial pose, 64 canonical template points are projected into the object crop. Each point combines local and global crop features with the interaction context and predicts a 3D correction. A closed-form Kabsch step then recovers the final rigid pose.
The two stages are trained sequentially, with Stage 1 frozen while Stage 2 is trained.
The claimed contributions are: (i) formulating unified online HOSI reconstruction; (ii) extending contact-guided latent refinement to objects so that "object interaction feeds back to refine the human"; and (iii) the initialization and mesh-indexed readout.
The validation consists of four parts. Human metrics are reported on EMDB-1/2, RICH and 3DPW (e.g., EMDB-2 WA-MPJPE 113.7 to 87.0 mm vs. UniCon3R). Human-object metrics are reported on BEHAVE and InterCap against the single-image feed-forward baselines HOI-TG and CONTHO (BEHAVE CDc 24.6 to 18.8 cm). The Stage-2 readout is ablated. The appendix adds analyses of occlusion, centroid/ADD-S, first-frame alignment, backbone scaling, contact, and failure modes.
Critical strengths
Strong online human reconstruction while also recovering the scene and camera. On the held-out EMDB-2 benchmark, METAV3RSE reaches 87.0 mm WA-MPJPE. This is lower than other online / world-grounded methods as reported in the UniCon3R and Human3R papers (OnlineHMR 93.5, GVHMR 111.0, WHAM 135.6, and Human3R*/UniCon3R 118.2/113.7), and approaches offline methods such as TRAM (76.4). On EMDB-1, PVE/MPJPE (75.6/65.7 mm) are also lower than those of strong single-image methods, e.g., NLF at 82.4/69.6 as reported in Human3R. The improvement holds for the lightweight ViT-S/672 backbone as well (EMDB-2 130.7 to 110.9 mm). This matters to my assessment because a single online model that reaches this level of human accuracy while jointly producing scene geometry and camera motion is practically useful, independently of where the gain comes from (see the weakness below).
An extensible way to add a new entity type to a frozen recurrent 4D reconstruction model. The image encoder, recurrent state and scene/camera heads stay frozen, and the object enters only through additional prompts and lightweight heads. The inherited capabilities are preserved: human-scene contact on RICH remains close to UniCon3R (F1 0.69 vs. 0.71), and the RICH pose metrics do not degrade. The object pose is predicted relative to the human root and then composed with the inherited camera pose, which yields a world-frame object without a separate object-to-world estimation step. This matters because unified feed-forward 4D models are increasingly used as perception backbones, and the paper offers a concrete, modular recipe for extending such a model to manipulated objects without retraining or degrading the backbone. More broadly, the problem direction is timely: extending unified feed-forward 4D reconstruction from humans and scenes to manipulated objects is a natural next step for the community, and to my knowledge this is among the first attempts to do so within a single online model.
Critical weaknesses
The paper's central claims are not yet sufficiently supported by the experiments. The three issues below concern the paper's main contributions, which is why they dominate my assessment.
1, it is unclear whether the human gains come from object interaction. The abstract states that "joint object reasoning also improves the human", citing the EMDB-2 result. As far as I can tell, EMDB provides no object annotations, and EMDB-2 consists mostly of walking and stair sequences. The paper does not explain how ΔH^ho is computed on these sequences or what role it plays there. At the same time, METAV3RSE is trained with additional real-image data (BEHAVE, InterCap, HODome) and its human heads are retrained, yet there is no ablation that removes ΔH^ho and no same-data UniCon3R control. Without such controls, it is difficult to attribute the EMDB improvement to the proposed object-to-human feedback rather than to the change in training data and protocol.
2, the evidence for the object-side contribution is limited. The comparison is restricted to two single-image methods. Under the same BEHAVE protocol that the paper adopts (the released VisTracker evaluation), VisTracker reports 7.81/8.49 cm CDh/CDo, versus 22.5/21.6 cm for METAV3RSE. Under first-frame alignment (App. C), the feed-forward method THO reports 8.49/13.36 cm, versus 40.6/43.3 cm here. I understand that VisTracker uses test-time optimization and that THO has no public checkpoint, but these reported numbers suggest a substantial gap that the paper should acknowledge and discuss. In addition, the mesh-indexed readout closely follows the final stage of HOI-TG/CONTHO: template vertices are projected with the predicted pose, image features are sampled at the projected locations, vertex positions are regressed, and an SVD-based rigid fit recovers R and t. The object-side technical novelty therefore lies mainly in conditioning this readout on the interaction context.
3, the joint human-object-scene setting is not evaluated. Objects are evaluated only on BEHAVE and InterCap, which have static cameras, in the camera frame or an aligned frame. The scene and camera outputs are evaluated only on sequences without objects. There is no measure of world-frame object trajectories or object-scene consistency. Moreover, none of the qualitative results (Figs. 2–5) shows reconstructed scene geometry at all; the scene appears only in the schematic overview of Fig. 1, whose example input (a person carrying a box outdoors) is not accompanied by a corresponding object reconstruction. As a result, the "unified HOSI reconstruction" contribution currently remains a claim rather than a demonstrated capability.
Questions for authors
-
On EMDB, 3DPW and RICH, which have no object annotations, how is the object mask obtained, and what does ΔH^ho evaluate to (zero, skipped, or computed from some detected region)? Could you clarify how much of the EMDB-2 improvement can be attributed to this residual?
-
Could you report, on EMDB-2 and BEHAVE CDh, (a) UniCon3R retrained with exactly your data and schedule, (b) METAV3RSE with ΔH^ho removed, and (c) METAV3RSE with the object branch removed? This is the most important experiment for my assessment.
-
Why is Human3R† on EMDB-2 reported in Table 1 (156.7 mm) but listed as unavailable in Table 8 for the same ViT-L/896 configuration? Why are the HOI-TG and CONTHO rows identical in Table 6?
-
Are the supplied object masks and crops ground truth on BEHAVE and InterCap? Do the baselines receive exactly the same localization? Is the InterCap test set (MILO list) disjoint from the training data by subject and by object?
-
Please report rotation error for the prior and the final pose in Table 6. If Stage 2 mainly corrects translation, how does it differ from a translation/depth refinement head?
-
UniCon3R's Table 1 also includes world-grounded HMR methods (e.g., WHAM, GVHMR, TRAM, JOSH, OnlineHMR), which are not included in your Table 1. Could you explain this choice? Including them would help contextualize the results; for example, GVHMR is reported at 78.8 mm WA-MPJPE on RICH, which bears on the statement that METAV3RSE achieves the best or tied-best results there.
-
Could you clarify the target application of the proposed setting? A natural motivation for a feed-forward model is large-scale, in-the-wild use (e.g., recovering 4D human-object data from internet video). However, the method assumes a known metric template, a supplied object mask and a single rigid object, and the scene does not constrain the object directly (e.g., through object-scene contact or support). What concrete application requires online feed-forward inference and scene reconstruction under these assumptions, and why is a joint model preferable there to a modular pipeline (e.g., a human-scene reconstruction model combined with a model-based 6D object pose estimator given the same template and mask)?
5 comments
Response to Reviewer g7PJ (2/3)
Weakness 3: The joint human-object-scene setting is not demonstrated clearly
Answer. We agree that the submission evaluated human-scene and human-object reconstruction separately. We add two pieces of evidence, each with a limited scope. Quantitatively, with one alignment kept for each whole BEHAVE sequence, METAV3RSE has lower object surface and centroid error than HOI-TG and CONTHO, which supports more accurate object placement over the sequence. It does not establish temporal consistency (our object acceleration error remains higher), uninterrupted tracking under a moving camera (BEHAVE has a static camera, and we process it in 16-frame windows with state resets), or object-scene consistency, which we do not model. Qualitatively, the revised Fig. 1 and the new Fig. 3 show the human, object, and scene reconstructed together in one metric frame.
Object trajectories (Table R8, all 85 BEHAVE sequences). Each protocol estimates one transformation per sequence from the human and object vertices and keeps it fixed. CD$_o$ and centroid error in cm; lower is better.
| Method | Stage | First-frame Sim(3) CD$_o$ ↓ | Centroid ↓ | Sequence Sim(3) CD$_o$ ↓ | Centroid ↓ | Sequence rigid CD$_o$ ↓ | Centroid ↓ |
|---|---|---|---|---|---|---|---|
| HOI-TG | — | 62.0 | 47.9 | 29.1 | 24.3 | 33.4 | 29.4 |
| CONTHO | — | 60.8 | 47.1 | 29.0 | 24.2 | 33.4 | 29.3 |
| METAV3RSE | Stage 1 | 63.1 | 47.7 | 36.4 | 27.4 | 40.4 | 32.5 |
| METAV3RSE | Stage 1 + 2 | 43.3 | 34.0 | 24.1 | 18.2 | 24.9 | 19.5 |
Stage 1 alone is less accurate than the baselines; the gain comes from the full model, whose readout reduces object CD from 40.4 to 24.9 cm under rigid alignment.
Joint reconstruction (new Fig. 3). For the same inputs, CONTHO reconstructs the human and object without the scene, UniCon3R the human and scene without the object, and METAV3RSE all three in one metric frame. This shows the joint output, not its accuracy.
Revision. Section 6 now states these limits (L536–539); Table R8 is in Appendix H.2, and the comparison figure is Fig. 3.
Question 3: Numerical inconsistencies in Tables 1, 6, and 8
Both are presentation errors, now corrected; no results change.
Tables 1 and 8. The ViT-L/896 Human3R† EMDB-2 entries were omitted from Table 8 by mistake; we have restored them from Table 1 (L1204).
Table 6. HOI-TG and CONTHO coincide only at the displayed precision; both use template-based regression with the same ground-truth object localization:
| Method | Centroid ↓ (cm) | ADD-S ↓ (cm) | ADD-S<10%d ↑ (%) |
|---|---|---|---|
| HOI-TG | 18.945 | 9.870 | 60.402 |
| CONTHO | 18.933 | 9.869 | 60.337 |
We have added these values to the Table 6 caption (L1081–1088).
Question 4: Ground-truth localization, input parity, and the InterCap split
Answer. Yes, the object masks are ground truth, for all methods. The inputs are not identical, and they favor the baselines: HOI-TG and CONTHO also receive ground-truth human masks and boxes, which METAV3RSE predicts itself. The InterCap split is disjoint by subject (we train on subjects 01–08 and test on 09–10) but not by object: the same ten objects appear in training, as is standard for template-based evaluation. METAV3RSE additionally uses calibrated intrinsics; the baselines also load the dataset camera parameters, though not identically.
| Input | METAV3RSE | HOI-TG | CONTHO |
|---|---|---|---|
| Object mask | Ground truth | Ground truth | Ground truth |
| Human mask and box | Predicted | Ground truth | Ground truth |
Predicted masks. Replacing the ground-truth object masks with SAM 3 [R1] predictions on the same 1,396 InterCap frames degrades all three methods by similar amounts (Table R6). This is a shared difficulty, but it does not remove our dependence on ground-truth object localization, which we state as a limitation.
| Method | GT-mask CD$_o$ ↓ | SAM 3-mask CD$_o$ ↓ | Increase |
|---|---|---|---|
| HOI-TG | 26.87 | 66.85 | 39.98 |
| CONTHO | 27.76 | 64.57 | 36.81 |
| METAV3RSE | 26.99 | 65.92 | 38.93 |
Revision. We have documented these inputs and the split in Section 5 (L312–314, L358–361) and the localization limitation in Section 6 (L534–536).
Response to Reviewer g7PJ (3/3)
Question 5: Does Stage 2 improve rotation, and how does it compare with a simpler translation/depth head?
Answer. No: Stage 2 improves placement, not orientation. Centroid error falls from 22.4 to 15.8 cm, while rotation error changes from 45.3° to 46.9° (Table R5); most of the orientation comes from the interaction-aware initialization, which is already more accurate in rotation than HOI-TG and CONTHO. We have not compared against a translation- or depth-only refinement head, so we do not claim that the mesh-indexed readout outperforms that simpler alternative. The closest evidence is the direct pose-regression variant in Table 4 of the paper, which predicts the rigid pose from the same features: the readout lowers object CD from 22.3 to 21.6 cm on BEHAVE and from 29.9 to 26.9 cm on InterCap.
| Method | Rotation ↓ (°) | Centroid ↓ (cm) | ADD-S ↓ (cm) |
|---|---|---|---|
| HOI-TG | 50.7 | 18.9 | 9.9 |
| CONTHO | 51.0 | 18.9 | 9.9 |
| METAV3RSE, Stage 1 | 45.3 | 22.4 | 12.4 |
| METAV3RSE, Stage 2 | 46.9 | 15.8 | 9.4 |
Revision. We have added rotation error to Table 6 (L1081–1088) and stated in Appendix B.2 (L1130–1133) that Stage 2 mainly corrects placement.
Question 6: Why are stronger world-grounded methods missing?
Answer. We agree they should be included, and they change our claim. On EMDB-2 WA-MPJPE, the offline methods JOSH, PromptHMR-Vid, and TRAM outperform METAV3RSE (Table G2, global response). Among online methods, METAV3RSE has the lowest error (87.0 vs. 93.5 mm for OnlineHMR) while also reconstructing the object, scene, and camera. On RICH, GVHMR is slightly better on WA-MPJPE (78.8 vs. 79.1 mm), so our "best or tied-best" statement in Section 5.1 was too broad.
Revision. We have added this comparison to Appendix H.6 (Table 19) and restricted the accuracy claims in Section 5.1 (L373–378).
Question 7: Target application, and why a joint model over a modular pipeline?
Application. Our broader goal, as stated in the introduction, is capturing people interacting with objects in their surroundings for mixed reality, robotics, and embodied AI, for example building digital twins of everyday activity. The current assumptions fit settings where objects are known in advance: recording human demonstrations for robot learning, such as package handling or tool use with CAD-modeled objects, or capturing interactions with scanned objects in digital-twin environments. In both, the needed output is human motion, object pose, contact, camera, and scene in one metric frame, produced as the video arrives. These assumptions (a known template, an object mask, a single rigid object) limit in-the-wild use, and we state them as limitations in Section 6.
Joint vs. modular. We find a measured trade-off rather than an accuracy advantage: pairing our human-scene reconstruction with MegaPose [R2], given the same inputs, yields more accurate object poses (object CD 23.37 vs. 26.99 cm), while the joint model is 4.5× faster and also predicts human-object contact and refines the human from the interaction. The lower official CD$_h$ of the joint model is an alignment effect, not better human geometry, since both pipelines share the same human vertices.
| Pipeline | Official CD$_h$ ↓ | Object CD ↓ | Object V2V ↓ | FPS ↑ |
|---|---|---|---|---|
| METAV3RSE (joint) | 7.40 | 26.99 | 32.99 | 1.17 |
| METAV3RSE human + MegaPose [R2] | 8.62 | 23.37 | 28.75 | 0.26 |
Revision. We report this comparison in Appendix H.3 and state in Section 6 that specialized pose estimators remain more accurate on the object and that the method assumes known templates and object masks (L534, L538–539).
References:
[R1] N. Carion et al. SAM 3: Segment Anything with Concepts. ICLR 2026.
[R2] Y. Labbé et al. MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare. CoRL 2022.
Response to Reviewer g7PJ (1/3)
We thank the reviewer for the thorough assessment. In short: the controlled ablation you identified as most important shows that training with the object pathway improves the human even on object-free videos; METAV3RSE is the most accurate online feed-forward method on EMDB-2 while also reconstructing the object, scene, and camera; and it places the object more accurately than the image-based feed-forward baselines.
Answer. The EMDB gains come from training, not from object-to-human feedback at inference. No object input is given on EMDB, 3DPW, or RICH, so the object pathway, including ΔHho, is never executed: it is not computed, set to zero, or applied to a dummy region (zero executions over 43,129 EMDB-2 and 106,446 RICH frames; Q1). The control you requested (Q2) shows that, under identical data and schedule, the object pathway reduces EMDB-2 WA-MPJPE from 100.4 to 87.0 mm, which rules out more data or longer training as the cause. Removing the object pathway recovers UniCon3R, so (a) and (c) are the same model.
Table G1 (extended with human-only V2V). EMDB-2 in mm, BEHAVE in cm. Human-only metrics align the human alone; official CD$_h$ uses joint human-object alignment (Stage 1).
How much is due to the residual. Of the 26.7 mm gain over published UniCon3R, 13.3 mm comes from the updated training procedure as a whole, 8.2 mm from object supervision, and 5.2 mm from training with ΔHho; these are successive differences along one ablation path. We attribute the transfer to object supervision also training the shared human predictor: the object and contact prompts are built from the human representation, and the human is supervised through the interaction-refined representation (Eq. 6). On BEHAVE, where the residual runs, it further lowers human-only CD from 8.08 to 7.57 cm; the larger change in official CD$_h$ reflects its joint alignment, and because (b) and the full model are trained separately, this comparison reflects training and inference together.
Revision. We have corrected the abstract (L026–028), which attributed the over-23% EMDB-2 gain to joint object reasoning, and Section 5.1 (L369–375); details are in Appendix H.1.
Answer (comparisons). We do not claim parity with VisTracker or THO. VisTracker (7.81/8.49 cm human/object CD vs. our 22.5/21.6 cm, same protocol) uses per-sequence test-time optimization, but THO (8.49/13.36 cm vs. our 40.6/43.3 cm under first-frame alignment) is also feed-forward, so the gap is not explained by optimization alone. A direct comparison with THO is further limited by protocol differences: our rescoring of HOI-TG gives 62.0 cm object CD, whereas THO reports 24.7 cm for it. Our advantage is over the image-based feed-forward baselines evaluated with identical frames and code: combined CD 18.8 vs. 24.6 cm for HOI-TG on BEHAVE, and object CD 24.9 vs. 33.4 cm under a single sequence-level alignment (Table R8). Their object V2V and acceleration errors are lower than ours, so this advantage holds for surface and centroid error only.
Answer (novelty). Our contribution is to condition template projection, feature sampling, and closed-form rigid fitting on human-object interaction inside an online human-scene model: the interaction places the initial pose relative to the scene-grounded human, decides where the template points read image evidence, conditions their corrections, and feeds a correction back to the human. With the same initialization, removing interaction conditioning from the readout raises object CD from 21.6 to 24.0 cm on BEHAVE and from 26.9 to 30.7 cm on InterCap (Table 4 of the paper).
Revision. Section 4 now opens with the principle behind the interaction-conditioned readout (L226–234).