acceptodds
Under review as a conference paper at ICLR 2027

HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction

Abstract

Recent advances in 3D foundation models have enabled joint reconstruction of humans and their surrounding environments. However, combining independently trained human and scene priors often produces misalignment in scale and depth. We observe that the two priors have complementary strengths. The scene prior provides consistent depth but approximate scale, while the human prior provides fixed body scale but less reliable depth. Motivated by this observation, we present HARMONI, a feed-forward framework that reconstructs cameras, scene, and humans with identities from monocular or multi-view video without test-time optimization. We introduce bidirectional anchoring, in which scene depth guides human placement, while human keypoints calibrate the scene scale to match the human body. While most existing approaches target monocular inputs and multi-view methods rely on optimization or re-identification, our approach naturally extends to multiple views. Building on bidirectional anchoring, we introduce a multi-view fusion module that merges per-view estimates, while reducing the influence of unreliable views and visually similar individuals. Experiments show that our framework outperforms previous human-scene methods in global motion and multi-view pose estimation by up to 28% and 64%, while running 28 faster than optimization-based approaches.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Scene Reconstruction, 4D Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_46QnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HARMONI reconstructs cameras, scene geometry, and multiple SMPL-X humans from monocular or multi-view video without test-time optimization. It combines a frozen Pi3X scene model with human features from a frozen Multi-HMR encoder and trainable human reconstruction modules. Its central contribution is bidirectional anchoring: scene depth anchors human placement through a predicted head-depth residual, while human body proportions calibrate the scene’s metric scale. For multi-view inputs, the method combines confidence-weighted parameter averaging, head triangulation, and a refinement transformer. A separate association procedure matches people across time and views using learned features and geometric similarities. Results show improvements in global human reconstruction on RICH and EMDB-2 and in multi-view localization on EgoHumans and EgoExo4D. The reported reductions of approximately 28% and 64% are supported by the corresponding table entries. The principal contribution is the specific geometric coupling and its integration into an efficient system: human-informed scene scaling and feed-forward human–scene alignment already appear in HSfM and UniSH, respectively.

Critical strengths
  1. The alignment mechanism is well motivated and supported by meaningful ablations.

  2. The gains in global localization are substantial, while local pose quality remains competitive.

  3. The multi-view design has interpretable component-level evidence.

  4. The supplementary evaluation adds useful practical evidence.

Critical weaknesses
  1. The main reconstruction results do not establish performance of the fully automatic identity pipeline. Table 2 explicitly uses ground-truth identities for every method. This is appropriate for comparing reconstruction conditional on correct correspondence, but it bypasses a central advertised capability of HARMONI. Table 7 evaluates association separately, reporting 88.5% precision and 78.7% recall; the consequences of these errors for triangulation, fusion, and final reconstruction are not reported. Moreover, the comparison between 90.9% association accuracy and the previously reported 51.2% appearance-based result needs clearer protocol matching: the manuscript does not establish identical temporal context, detection inputs, or accuracy definitions. The earlier result comes from HSfM’s evaluation of an UnCaliPose-based association procedure. HSfM, supplementary discussion.

  2. The central scale-calibration operation is insufficiently specified and its metric effect is not fully established. Equations 1–3 first construct human translation from scene depth and a residual, then update the scene scale using s∗=rss^*=rs. The manuscript does not explicitly state the resulting transformation of human translations or the depth residual. For example, scaling the entire depth as r(danchor+Δd)r(d_{\mathrm{anchor}}+\Delta d) differs from scaling only scene depth as rdanchor+Δdr d_{\mathrm{anchor}}+\Delta d. This distinction matters because the residual represents an anatomical surface-to-joint distance. The projected-length ratio also depends on pose, shape, perspective, and keypoint errors; it is not automatically an exact scene-scale estimator. The paper should state its approximation assumptions and fallback behavior when reliable joints are scarce. Table 4 provides useful scale-sensitive evidence, but Table 5 reports identical Pi3X and HARMONI camera/depth scores to the displayed precision. The evaluation description does not clarify whether these are calibrated outputs or whether scale alignment removes the calibration’s effect. Appendix B.3 explicitly uses Sim(3) alignment for the Table 1 joint metrics, so those metrics alone cannot establish absolute metric-scale accuracy.

  3. The association algorithm does not guarantee the claimed global consistency. Algorithm 1 independently performs Hungarian matching for each connected view pair and then merges accepted matches using union-find. Pairwise one-to-one matching does not ensure consistency across cycles. For example, accepted matches A1↔B1A_1\leftrightarrow B_1, B1↔C1B_1\leftrightarrow C_1, and C1↔A2C_1\leftrightarrow A_2 merge two different people from view A into one identity, even though each pairwise assignment is individually one-to-one. No conflict-resolution rule or restriction against merging simultaneous same-view detections is described. Such a merge could directly corrupt the subsequent fusion.

  4. The long-video inference protocol is too incomplete to assess temporal comparisons and practical scalability. Section 3.1 jointly processes T×VT\times V frames using the scene model, but the evaluation details do not specify inference window length, temporal sampling, overlap, or how separately processed windows share a coordinate system and scale. These choices can materially affect trajectory accuracy and resource usage. This matters when comparing with Human3R, which explicitly processes video online using a recurrent state. Different access to future frames is a legitimate design trade-off, but it should be disclosed when interpreting the accuracy comparison. Human3R, method and efficiency discussion. The four-view runtime measurement supports efficiency for that configuration; it does not characterize long-video processing. The qualitative long trajectories in Appendix A.1 do not resolve the missing inference details.

Questions for authors
  1. What happens to the main reconstruction results when ground-truth identities are replaced by your predicted associations? Please report the resulting localization errors and how missed detections, duplicate identities, and incorrect merges enter the evaluation.
  2. What are the exact equations after applying s∗=rss^*=rs? Which quantities are rescaled, is the anatomical depth residual rescaled, and is the calibration ratio recomputed? What happens when no sufficiently reliable calibration joints are available?
  3. Does Table 5 evaluate the final calibrated scene? Please specify the camera and depth alignment protocols and explain why all four reported metrics match the frozen Pi3X baseline to the displayed precision.
  4. How does Algorithm 1 resolve inconsistent association cycles? Can a union-find component contain two people observed simultaneously in the same view, and how often does this occur before any corrective handling?
  5. What temporal context and windowing are used for each benchmark? Please specify frame counts, sampling, window alignment, and resource usage, and clarify whether Table 7’s association comparison uses the same observations and metric definition as the cited appearance-based baseline.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 46Qn

We thank the reviewer for the constructive feedback. We address most questions in the General Response.

11. Difference with HSfM (Müller et al., CVPR 2025) and UniSH (Li et al., CVPR 2026)

HSfM (Müller et al., CVPR 2025) uses the human body as a scale reference, but this addresses only the scale misalignment. The remaining depth misalignment is resolved through test-time optimization with external 2D keypoints and known identities. UniSH (Li et al., CVPR 2026) aligns humans and the scene in a feed-forward manner with AlignNet, but UniSH (Li et al., CVPR 2026) directly estimates human depth in the scene. As a result, UniSH (Li et al., CVPR 2026) may show depth misalignment (Fig. 1b). In contrast, our bidirectional anchoring explicitly addresses both scale and depth alignment in a single feed-forward pass.

12. Clarification on appearance-based protocol

The 51.2% in Sec. 4.3 is quoted from the HSfM (Müller et al., CVPR 2025) supplementary, which evaluates an UnCaliPose (Xu and Kitani, BMVC 2022)-based appearance association. We included it to illustrate that appearance cues are unreliable in multi-person scenes with similarly dressed people, but we agree that it was not measured under our protocol, which we revised in Sec. 4.3, L468–471. We further compared the ReID method in UnCaliPose (Xu and Kitani, BMVC 2022) under our protocol in Table R7.

Table R7. Comparison with appearance-based matching.

Acc (↑)F1 (↑)Prec (↑)Recall (↑)
UnCaliPose (Xu and Kitani, BMVC 2022)80.266.465.367.5
Ours-multi90.983.388.578.7

Under the identical protocol, ours outperforms the appearance baseline. Using appearance cues tends to merge similarly dressed people, which results in lower precision. We believe the appearance baseline achieves a higher accuracy than the quoted 51.2% because our protocol matches 25-frame tracklets rather than single-frame boxes.

Question summary

Q1. What happens to the main reconstruction results when ground-truth identities are replaced by your predicted associations? Report the resulting localization errors and how missed detections, duplicate identities, and incorrect merges enter the evaluation.

A1. We kindly refer the reviewer to Item 1 (predicted identities) and Item 2 (identity consistency).

Q2. What are the exact equations after applying r? What happens when no sufficiently reliable calibration joints are available?

A2. Please see Item 5 (update equations with r).

Q3. Does Table 5 evaluate the final calibrated scene? Please specify the camera and depth alignment protocols.

A3. Please see Item 6 (clarification on Table 5).

Q4. How does Algorithm 1 resolve inconsistent association cycles? How often does this occur before any corrective handling?

A4. Please see Item 4 (inconsistent cycles).

Q5. What temporal context and windowing are used for each benchmark? Please specify frame counts, sampling, window alignment, and resource usage, and clarify whether Table 7’s association comparison uses the same observations and metric definition as the cited appearance-based baseline.

A5. Please see Item 7 (long-sequence protocol) and Item 12 (appearance-based protocol comparison).

AuthorsOpenReview ↗

Gentle reminder for discussion

Dear Reviewer 46Qn,

Thanks again for your thoughtful review. We addressed your questions as follows:

  • Table 2 with predicted identities (Item 1)
  • The appearance-based comparison under our protocol (Item 12; revised in Sec. 4.3, L468–471)
  • The update equations after scale calibration and the Table 5 protocol (Items 5–6; revised in Sec. 3.2, L281–283 and Sec. 4.2, L421–423)
  • Inconsistent association cycles (Item 4)
  • The long-sequence inference protocol (Item 7; revised in Sec. B.3, L1205–1210)

We would appreciate it if you could check these responses and let us know if any point needs further clarification.

Reviewer_VnKnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HARMONI, a feed-forward framework for joint human and scene reconstruction from monocular and multi-view videos. The key contribution is bidirectional anchoring, which uses scene depth to guide human placement and human body proportions to calibrate scene scale. The paper further introduces a multi-view fusion module combining confidence-weighted aggregation, triangulation, transformer refinement, and geometric identity association. Experiments demonstrate improved reconstruction accuracy and substantially faster inference compared to existing approaches.

Critical strengths
  • The paper identifies the scale and depth inconsistencies between human and scene priors and proposes an effective solution through bidirectional anchoring.
  • The method achieves significant improvements over existing human-scene reconstruction methods in both monocular and multi-view settings.
  • The optimization-free design supports multi-person, multi-view reconstruction with significantly reduced inference time.
  • The ablation studies demonstrate the effectiveness of the proposed alignment and multi-view fusion components.
Critical weaknesses
  • The method relies on head-based detection and depth anchoring, making it vulnerable to complete head occlusion and challenging human interactions.
  • The sensitivity of bidirectional anchoring to scene depth, human body shape, and keypoint predictions is not sufficiently investigated.
  • Temporal consistency and motion smoothness are not explicitly evaluated.
Questions for authors
  • How does the visibility of human's head impact this method?
  • Can the authors provide quantitative evaluations of temporal consistency, such as motion jitter or acceleration error?
2 comments
AuthorsOpenReview ↗

Response to Reviewer VnKn

We thank the reviewer for the constructive feedback. We address the questions in the General Response.

Question summary

Q1. How does the visibility of human's head impact this method?

A1. We kindly refer the reviewer to Item 3 (head occlusion robustness).

Q2. Can the authors provide quantitative evaluations of temporal consistency, such as motion jitter or acceleration error?

A2. Please see Item 2 (jitter and acceleration error).

AuthorsOpenReview ↗

Gentle reminder for discussion

Dear Reviewer VnKn,

Thank you again for your thoughtful review and your positive assessment of our work. We have addressed your questions in our rebuttal as follows,

  • Robustness to head occlusion (Item 3, Table R5; added in Sec. A.2, L889–903 and L998–1012)
  • Sensitivity to body shape, 2D keypoints, and scene depth (Item 3, Table R6; added in Sec. A.2, L906–913 and L957–963)
  • Temporal consistency with jitter and acceleration error (Item 2, Table R3)

We would appreciate it if you could check these responses and let us know if any point needs further clarification.

Reviewer_VmaVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HARMONI, which reconstructs cameras, scene point maps, and multiple SMPL-X humans with identities from monocular or multi-view videos without test-time optimization. Its core idea, bidirectional anchoring, places each person using the depth of a frozen scene model (Pi3X) and calibrates the scene scale by the ratio of projected SMPL-X body lengths to image-space body lengths. For multi-view inputs, per-view estimates are fused, and identities are associated across views based on 3D positions and poses. The paper reports comparisons and ablations for monocular global motion on EMDB-2 and RICH and for multi-view pose estimation on EgoHumans and EgoExo4D.

Critical strengths
  1. A single feed-forward model handles both monocular and multi-view videos, and in the multi-view setting it is more accurate and much faster than optimization-based methods. On EgoHumans, it reduces W-MPJPE^\dagger from 1.04 m (HSfM) to 0.37 m and processes a scene with three people and four views in about 1.1 s, versus about 118 s for HSfM and 32 s for HAMSt3R (Tables 2 and 9). This is the main strength, as it directly serves large-scale joint human-scene capture.
  2. The anchoring design is simple and interpretable, and controlled ablations on the same backbone support it. Removing scale calibration increases EMDB-2 WA-MPJPE from 84.7 mm to 157.8 mm (Table 6), and calibrating the scene outperforms rescaling SMPL-X (Table 4). These results show that the gains do not stem only from the choice of backbone.
Critical weaknesses
  1. Both technical pillars have precedents, so the contribution is mainly their integration.
    • Scale calibration: Since the projected length \ell^{smpl}_k is approximately inversely proportional to the depth d at which SMPL-X is placed, the calibration ratio r is essentially the ratio between the depth implied by the body size and the scene depth. The same principle sets the scene scale in the initialization of HSfM (compared in Table 2 as the optimization-free HSfM (init.)), in HAC (Yang et al., ICCV 2025), and in SynCHMR (Zhao et al., CVPR 2024). HAC and SynCHMR are not cited, and HSfM is described only as optimization-based (L.149-156).
    • Association: Appearance-free cross-view association by geometry and pose also appears in Huang et al. (Simultaneously Recovering Multi-Person Meshes and Multi-View Cameras with Human Semantics, TCSVT 2024; not cited) and in Hwang et al. (SIGGRAPH 2026), which the paper cites. In Table 7, position alone already reaches 89.5% accuracy, and adding pose improves it by only 1.4 points.
    • The methodological novelty is therefore moderate, and the paper's value depends on its accuracy-speed trade-off.
  2. The evidence for this trade-off is insufficient.
    • Monocular accuracy: On EMDB-2, the optimization-based JOSH (its JOSH_3 variant initialized with VIMO and MASt3R; Liu et al., 2026b) and HAC (as reported in its arXiv version) outperform Ours-mono on all three metrics, and PromptHMR-Vid (Wang et al., CVPR 2025) and TRAM (Table 10) achieve lower WA-MPJPE. The JOSH3R numbers in Table 1 come from the same table that reports JOSH_3, yet Table 1 includes only feed-forward methods and omits JOSH_3.
    • Runtime: No monocular runtime is reported; runtime is measured only on one multi-view scene at a single timestep. The claim of running 28 times faster than optimization-based approaches (abstract, introduction) is the ratio to HAMSt3R, which Table 2 itself marks as only partially optimization-free. It thus remains unclear how large the speed advantage is over the more accurate methods.
    • Multi-view evaluation: The evaluation is per timestep with ground-truth identities, so neither the proposed association nor temporal consistency is evaluated.
    • These gaps concern the title and the main claims, so they weigh heavily in my assessment.
Questions for authors

Clarifications that could change the assessment:

  1. Please report the runtime of the proposed method on monocular video, including all modules (e.g., FPS), together with the GPU and the window length used at inference, and discuss the accuracy-speed trade-off against the runtimes reported for more accurate methods such as JOSH_3 and TRAM (e.g., Table 3 of Liu et al., 2026b).
  2. Please report Table 2 with identities obtained by the proposed association instead of ground truth, together with temporal metrics on multi-view sequences (e.g., acceleration error and identity switches).
  3. After calibration, are the human translations \tau also scaled by r? How are long sequences split into windows, and how are scale and identities kept consistent across windows? How are the intrinsics K_n obtained from the outputs of Pi3X at test time?
  4. Hwang et al. (SIGGRAPH 2026), cited in the paper, also use Pi3 depth for 3D person positions, Hungarian matching on position and pose, triangulation, and a bone-length-based scale. Please clarify how the proposed multi-view fusion and association technically differ from this work.
  5. How much of the multi-view improvement over HSfM is attributable to the scene backbone (Pi3X instead of DUSt3R) rather than to the calibration design? A brief discussion would suffice.

Suggestions for future work (not affecting the rating):

  1. An analysis of sensitivity to the body-size prior (e.g., for children) and to head occlusion.
3 comments
AuthorsOpenReview ↗

Response to Reviewer VmaV (1/2)

We thank the reviewer for the constructive feedback. We address most questions in the General Response.

8. Our contribution relative to prior work

  • Contribution. HSfM (Müller et al., CVPR 2025), HAC (Yang et al., ICCV 2025), and SynCHMR (Zhao et al., CVPR 2024) also use the human body as a scale reference. However, this addresses only the scale misalignment between humans and the scene. Humans can still be placed too close to or too far from the camera relative to the scene (Fig. 1b). Our bidirectional anchoring explicitly addresses both scale and depth alignment. Removing either one degrades our model (Table 6), and applying scale calibration alone to another baseline, Human3R (Chen et al., ICLR 2026), yields inconsistent gains (Table 13, revised version). With bidirectional anchoring, our method runs in a single feed-forward pass without test-time optimization, offering a favorable accuracy-runtime trade-off compared to TRAM (Wang et al., ECCV 2024) and JOSH₃ (Liu et al., ICLR 2026) (Item 9).
  • Hwang et al. We agree that our association shares its ingredients with Hwang et al. (SIGGRAPH 2026). However, the difference lies in the fusion process. Hwang et al. triangulate 3D keypoints from external 2D detections, fit SMPL parameters to them through 3D-to-3D optimization, and further refine them during Gaussian splatting reconstruction. In contrast, our multi-view fusion module obtains the global SMPL parameters in a single feed-forward pass, without test-time optimization.
  • HSfM (Müller et al., CVPR 2025) Backbone. HSfM (Müller et al., CVPR 2025) uses DUSt3R (Wang et al., CVPR 2024) for initialization and optimization with bundle adjustment. In Table 4, when we keep the Pi3X scale ss and instead rescale the SMPL meshes by rr (Ours-s), W-MPJPE† on EgoHumans (Khirodkar et al., ICCV 2023) is 1.56 m, worse than HSfM (Müller et al., CVPR 2025) (1.04 m). With calibration, it reduces to 0.37 m, showing that the gain does not come from the backbone alone.
  • We revised the HAC (Yang et al., ICCV 2025), PromptHMR (Wang et al., CVPR 2025), SynCHMR (Zhao et al., CVPR 2024), and Huang et al. (TCSVT 2024) citations in the related work (Sec. 2, L135–136, L140, L153–154).

9. Runtime

  • Runtime-accuracy trade-off. Following the reviewer’s request, we compare the accuracy–speed trade-off of our method with those of the optimization-based methods JOSH₃ (Liu et al., ICLR 2026) and TRAM (Wang et al., ECCV 2024). All runtimes are measured on one NVIDIA RTX 4090, and the JOSH₃ (Liu et al., ICLR 2026) and TRAM (Wang et al., ECCV 2024) numbers are taken from Liu et al. (2026b). Ours runs at 6.9 FPS, JOSH₃ (Liu et al., ICLR 2026) at 0.8 FPS, and TRAM (Wang et al., ECCV 2024) at 1.1 FPS. Since a 100-frame window exceeds the 24 GB memory of the RTX 4090, we process 75-frame windows for this measurement.
  • WA-MPJPE is 84.9 / 102.6 / 127.8 mm on RICH (Huang et al., CVPR 2022) and 83.6 / 68.9 / 76.4 mm on EMDB-2 (Kaufmann et al., ICCV 2023) (ours / JOSH₃ (Liu et al., ICLR 2026) / TRAM (Wang et al., ECCV 2024)). Overall, ours is 8.6× and 6.2× faster than JOSH₃ (Liu et al., ICLR 2026) and TRAM (Wang et al., ECCV 2024) without test-time optimization or an external SLAM. It is more accurate on RICH (Huang et al., CVPR 2022), while JOSH₃ (Liu et al., ICLR 2026) and TRAM (Wang et al., ECCV 2024) remain more accurate on EMDB-2 (Kaufmann et al., ICCV 2023).
  • Multi-view, multi-frame runtime. On one RTX A6000, a window of 100 frames runs at 4.0 FPS (25 s, 27.9 GB), for both a monocular window and a multi-view window of 4 cameras × 25 timesteps.
  • Claim of running 28× faster. Our method is at least 28× faster than optimization-based approaches, including HAMSt3R (Rojas et al., ICCV 2025), which uses partial optimization (28×), and HSfM (Müller et al., CVPR 2025), which uses full optimization (over 100×). We count HAMSt3R as optimization-based, since it requires optimization-based SMPL fitting to obtain human meshes.
AuthorsOpenReview ↗

Response to Reviewer VmaV (2/2)

10. Impact of pose term in association and how intrinsic K is obtained

  • Impact of pose term. As shown in Table 7, adding the pose term has an effect on the positive pairs. Recall rises from 67.9 to 78.7 and F1 from 78.9 to 83.3, at a 5.6-point cost in precision. The weighting therefore sets the precision–recall trade-off of the association, rather than its accuracy.
  • Deriving K at inference. Following MoGe (Wang et al., CVPR 2025), we recover the focal length of each view from the predicted local point map by solving for the focal length and depth shift over confident pixels. We set the principal point to the image center. We revised it in Sec. 3.1, L205–206.

Question summary

Q1. Please report the runtime of the proposed method on monocular video, including all modules (e.g., FPS), together with the GPU and the window length used at inference, and discuss the accuracy-speed trade-off against the runtimes reported for more accurate methods such as JOSH₃ (Liu et al., ICLR 2026) and TRAM (Wang et al., ECCV 2024) (e.g., Table 3 of Liu et al., 2026b).

A1. We kindly refer the reviewer to Item 9 (runtime-accuracy trade-off).

Q2. Please report Table 2 with identities obtained by the proposed association instead of ground truth, together with temporal metrics on multi-view sequences (e.g., acceleration error and identity switches).

A2. Please see Item 1 (predicted identities) and Item 2 (identity consistency).

Q3. After calibration, are the human translations τ\tau also scaled by rr? How are long sequences split into windows, and how are scale and identities kept consistent across windows? How are the intrinsics KnK_n obtained from the outputs of Pi3X at test time?

A3. Please see Item 5 (Update equations with r), Item 7 (long-sequence protocol), and Item 10 (obtaining intrinsic K).

Q4. Hwang et al. (SIGGRAPH 2026), cited in the paper, also use Pi3 depth for 3D person positions, Hungarian matching on position and pose, triangulation, and a bone-length-based scale. Please clarify how the proposed multi-view fusion and association technically differ from this work.

A4. Please see Item 8 (Hwang et al.).

Q5. How much of the multi-view improvement over HSfM (Müller et al., CVPR 2025) is attributable to the scene backbone (Pi3X instead of DUSt3R) rather than to the calibration design? A brief discussion would suffice.

A5. Please see Item 8 (HSfM (Müller et al., CVPR 2025) backbone).

AuthorsOpenReview ↗

Gentle reminder for discussion

Dear Reviewer VmaV,

Thanks again for your thoughtful review. We addressed your questions as follows:

  • Our contribution relative to HSfM, HAC, SynCHMR, and Hwang et al. (Item 8; revised in Sec. 2, L135–136, L140, L153–154)
  • The monocular runtime and the accuracy–speed trade-off against JOSH₃ and TRAM (Item 9)
  • Table 2 with predicted identities and temporal metrics (Items 1–2)
  • The calibration, windowing, and intrinsics details (Items 5, 7, 10; revised in Sec. 3.2, L281–283, Sec. B.3, L1205–1210, and Sec. 3.1, L205–206)

We would appreciate it if you could check these responses and let us know if any point needs further clarification.

Reviewer_qtEcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HARMONI is a feed-forward framework for jointly reconstructing cameras, scenes, and identified humans from monocular or multi-view video without test-time optimization of reconstruction parameters. It combines frozen scene and human encoding priors through bidirectional anchoring: scene depth anchors human placement, while image keypoints and projected body proportions calibrate scene scale. For multiple views, it combines confidence-weighted parameter aggregation, head triangulation, and a refinement transformer, together with geometric person association. Experiments on EMDB-2, RICH, EgoExo4D, and EgoHumans report improved global reconstruction accuracy and favorable runtime. Component ablations and physical plausibility measurements support the alignment design. The main multi-view accuracy comparison uses ground-truth identities.

Critical strengths
  1. The alignment mechanism is well motivated and supported by complementary evidence. Table 6 compares residual anchoring against direct translation/depth prediction and evaluates the effect of scale calibration. Table 4 reports improvements under scale-sensitive evaluation, and Appendix A.1/Table 8 adds penetration and floating measurements. Together, these results provide substantive evidence that the proposed coupling addresses human–scene misalignment.
  2. The framework offers an attractive accuracy–runtime tradeoff for joint reconstruction. Tables 1–2 show substantial reductions in global reconstruction errors across monocular and multi-view settings while maintaining competitive local pose accuracy. Table 3 supports the fusion components, and the runtime breakdown in Table 9 makes the computational benefit concrete. The approximately 1.13-second runtime for the reported three-person, four-view example compares favorably with the optimization-based baselines, although its scope is limited to that measurement setup.
Critical weaknesses

The reconstruction accuracy of the fully automatic multi-person pipeline is not sufficiently established. Table 2 explicitly uses ground-truth identities for every method. This is a reasonable controlled comparison, but it does not show how errors from the proposed geometric association affect the headline reconstruction gains. Table 7 reports an association F1 of 83.3% and recall of 78.7%, so the downstream impact could be material. The qualitative temporal-occlusion examples are useful, but they do not quantify identity consistency across sequences. Since automatic association is an important part of the claimed system capability, a matched comparison using predicted identities, with an explicit treatment of missed and incorrectly associated people, would materially strengthen the paper.

Questions for authors
  1. End-to-end evidence: On the same inputs and test samples as Table 2, how do W-MPJPE†, GA-/PA-MPJPE, and person coverage change when predicted identities replace ground-truth identities? How are missed detections, unmatched people, and incorrect merges counted? If temporal tracklets are used, please also quantify identity consistency or clarify the evaluated temporal scope.
  2. Scale-update and evaluation clarification: After computing the ratio in Eq. (3), how are the point maps, camera translations, human head translations, and metric depth residuals updated? Please specify the operation order. Does Table 5 use scale alignment or uncalibrated scene outputs, explaining the identical values for HARMONI and Pi3X? Table 4 already provides scale-sensitive evidence; this question concerns the implementation and interpretation of the other protocol.
  3. Association consistency and reproducibility: In Algorithm 1, how are conflicting matches handled when union-find merges pairwise associations—for example, if a cycle merges overlapping tracklets from the same view? Please also specify the outlier and association thresholds and how they were selected. These details affect the reliability of the automatic association component.
2 comments
AuthorsOpenReview ↗

Response to Reviewer qtEc

We thank the reviewer for the constructive feedback. We address the questions in the General Response.

Question summary

Q1. How does human pose accuracy and person coverage change when predicted identities replace ground-truth identities in Table 2? How are missed detections, unmatched people, and incorrect merges counted? Please also quantify identity consistency or clarify the evaluated temporal scope.

A1. We kindly refer the reviewer to Item 1 (predicted identities) and Item 2 (identity consistency).

Q2. How are the point maps, camera translations, human head translations, and metric depth residuals updated after Eq. (3)? Does Table 5 use scale alignment or uncalibrated scene outputs?

A2. Please see Item 5 (update equations with r) and Item 6 (Clarification on Table 5).

Q3. In Algorithm 1, how are conflicting matches handled when union-find merges pairwise associations? Please also specify the outlier and association thresholds and how they were selected.

A3. Please see Item 4 (inconsistent cycles and thresholds).

AuthorsOpenReview ↗

Gentle reminder for discussion

Dear Reviewer qtEc,

Thank you again for your thoughtful review and your positive assessment of our work. We have addressed your questions in our rebuttal as follows,

  • Table 2 with predicted identities and identity consistency (Items 1–2)
  • The update equations after scale calibration and the Table 5 protocol (Items 5–6; revised in Sec. 3.2, L281–283 and Sec. 4.2, L421–423)
  • Inconsistent association cycles and thresholds (Item 4; revised in Sec. B.1, L1079)

We would appreciate it if you could check these responses and let us know if any point needs further clarification.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (4/4)

4. Association cycles and thresholds (R1, R4)

In Alg. 1, we use γreid\gamma_{\text{reid}} to filter unreliable pairwise matches, but we do not explicitly handle inconsistent cycles. We set both γoutlier\gamma_{\text{outlier}} and γreid\gamma_{\text{reid}} empirically to 1.0 m after scale calibration. On EgoHumans (Khirodkar et al., ICCV 2023), inconsistent cycles affect 4.0% of global identities, and their effects are included in Table R1. This can be prevented by skipping a union when the two components contain tracklets of the same view with overlapping timesteps. With this constrained union, the association results in Table 7 change as follows: Acc 90.9 → 91.6 (+0.7), F1 83.3 → 84.1 (+0.8), Precision 88.5 → 92.9 (+4.4), and Recall 78.7 → 76.8 (−1.9). We added the threshold values in Sec. B.1, L1079.

5. Details on update equations with calibration ratio r (R1, R2, R4)

We apply the scale calibration in the following order: (1) Pi3X (Wang et al., ICLR 2026) predicts point maps PP, cameras (Rcam,tcam,K)(R^{\text{cam}}, t^{\text{cam}}, K), and scale ss, and the scene is reconstructed as sPsP and stcams t^{\text{cam}}. (2) For each person, HARMONI predicts the 2D head location pheadp^{\text{head}} and the residual Δd\Delta d, samples the anchor depth danchord^{\text{anchor}} from PP, and computes the depth sdanchor+Δds d^{\text{anchor}} + \Delta d and translation τ\tau by Eq. 1. (3) The 2D keypoints and the projected SMPL-X joints are used to calculate rr (Eq. 3). (4) We replace ss with s∗=rss^\ast = r s and update each element as follows:

P∗=s∗P,tcam∗=s∗tcam,τ∗=(s∗danchor+Δd)K−1[phead;1].P^\ast = s^\ast P,\qquad t^{\text{cam}\ast} = s^\ast t^{\text{cam}},\qquad \tau^\ast = \big(s^\ast d^{\text{anchor}} + \Delta d\big) K^{-1} [p^{\text{head}};1].

We keep Δd\Delta d unchanged since it represents the anatomical surface-to-joint distance. The SMPL-X mesh, rotations and intrinsics are also unchanged. rr is computed once and shared by all views of a window. In the extreme case where no body joint is available in any frame, we keep ss. We added this detail in Sec. 3.2, L281–283.

6. Why is Table 5 identical to Pi3X? (R1, R4)

We compare camera and depth accuracy in Table 5 to evaluate the effect of fine-tuning the 3D foundation backbone, showing that our frozen design preserves the 3D foundation model's capabilities. The reported numbers are our final calibrated outputs (s∗s^\ast), but camera trajectories are evaluated after Sim(3) alignment and depth after per-frame scale–shift alignment, so the uncalibrated outputs give the same values. We also note that the main purpose of the scale calibration is to align the humans and the scene priors. We revised the evaluation detail in Sec. 4.2, L421–423.

7. Inference protocol for long sequences (R2, R4)

We explain the long-sequence evaluation protocol (Table 1) below. We revised details in Sec. B.3, L1205–1210.

  • Monocular videos. Our model processes a sequence as non-overlapping windows of 100 frames without subsampling. Windows are aligned by an SE(3) fit on the camera positions and 3D person positions of the two boundary frames.
  • Multi-view videos. We use all available cameras with 25 frames per view per window, without subsampling or overlap. Since the cameras are static, each window is aligned directly to the first window by a Sim(3) fit on the camera poses averaged within the window.
  • Per window. Scale calibration ratio rr is computed independently for each window, and identities are not carried across windows.
  • Resources. Windows are processed one at a time, so memory is bounded by one window (27.9 GB for 100 frames on an RTX A6000).
  • Online vs. offline. Human3R (Chen et al., ICLR 2026) runs online with a recurrent state, whereas our model is offline within a window. We are open to extending our framework to the online setting as future work.
AuthorsOpenReview ↗

General Response (3/4)

2. Identity consistency and temporal evaluation (R1, R2, R3)

Following WHAM (Shin et al., CVPR 2024), we evaluate temporal motion using jitter and acceleration error, and compare with the baseline Human3R (Chen et al., ICLR 2026). As shown in Table R3, our monocular model improves both metrics on EMDB-2 (Kaufmann et al., ICCV 2023) and the acceleration error on RICH (Huang et al., CVPR 2022), while jitter on RICH (Huang et al., CVPR 2022) is higher than Human3R (Chen et al., ICLR 2026)’s. We presume that, unlike our method, Human3R (Chen et al., ICLR 2026) is a recurrent model that updates its state over frames, and on RICH (Huang et al., CVPR 2022) the cameras are static, so jitter mainly reflects per-frame pose noise. Using all views further improves both metrics by a large margin.

Table R3. Temporal motion evaluation on EMDB-2 (Kaufmann et al., ICCV 2023) and RICH (Huang et al., CVPR 2022).

EMDB-2 Jitter (10 m/s³) (↓)EMDB-2 Accel (mm/frame²) (↓)RICH Jitter (10 m/s³) (↓)RICH Accel (mm/frame²) (↓)
Human3R (Chen et al., ICLR 2026)231.948.9174.636.4
Ours-mono188.640.3191.235.6
Ours-multi--87.219.2

We also evaluate identity consistency on EgoHumans (Khirodkar et al., ICCV 2023) in Table R4. We split each sequence into 25-frame clips and report IDF1 (Ristani et al., ECCVW 2016) and IDSW (Bernardin & Stiefelhagen, EURASIP JIVP 2008) per 1K for Ours-mono and Ours-multi. Using all views increases IDF1 and reduces IDSW compared to tracking each view independently.

Table R4. Tracking results on EgoHumans (Khirodkar et al., ICCV 2023).

IDF1 (↑)ID switches per 1K (↓)
Ours-mono84.4%3.09
Ours-multi94.8%1.65

3. Sensitivity analysis on head occlusions, scene depth, body shape, keypoint (R2, R3, R4)

To further analyze the robustness to head occlusion, we occluded each person on EMDB-2 (Kaufmann et al., ICCV 2023) with a gray box extending downward from the top of the head, covering 30%, 60%, and 90% of the height between the top of the head and the shoulders. As shown in Table R5, HARMONI detects 96% of the persons with almost unchanged accuracy at 30%, and still 75% at 60%, where most of the head is covered. When the entire head is covered (90%), the detection rate drops to 37%. The errors for detected persons increase gradually. We added occluded samples and Table R5 in the revision (Sec. A.2, L889–903 for Table 9 and L998–1012 for Fig. 14).

Table R5. Robustness to top-down occlusion on EMDB-2 (Kaufmann et al., ICCV 2023).

WA-MPJPE (↓)W-MPJPE (↓)RTE (↓)Detection rate (↑)
0%83.6214.41.40.98
30%86.8218.91.50.96
60%105.0258.71.70.75
90%129.0279.72.40.37

We also directly perturbed each element in our method on EMDB-2 (Kaufmann et al., ICCV 2023). We applied Gaussian noise with σ = 0.5 to the body shape parameters (≈ ±5 cm in body height), Gaussian noise with σ = 5% of the image length to the 2D keypoints, or multiplicative Gaussian noise with σ = 5% to the scene depth danchord^{\text{anchor}}. As shown in Table R6, perturbing body shape and 2D keypoints changes WA-MPJPE by at most 1.5 mm and RTE by at most 0.2. The ratio rr pools all joints, people, and frames of a window with Huber reweighting. Perturbing danchord^{\text{anchor}} has a larger effect, as it shifts the depth of each person in every frame. Since frame-wise noise does not accumulate into a trajectory drift, RTE is unchanged under danchord^{\text{anchor}} noise. We added Table R6 in the revision (Sec. A.2, L906–913 and L957–963 for Table 10).

Table R6. Perturbation on body shape, 2D kpt, danchord^{\text{anchor}}.

WA-MPJPE (↓)W-MPJPE (↓)RTE (↓)
clean83.6214.41.4
body shape84.9218.81.6
2D kpt85.1219.21.4
danchord^{\text{anchor}}120.3256.21.4
AuthorsOpenReview ↗

General Response (2/4)

1. Effect of predicted identity on Table 2 (R1, R2, R4)

We followed the HSfM (Müller et al., CVPR 2025) protocol in Table 2, which uses GT identities to compare reconstruction under correct correspondence. To quantify how association errors affect the final reconstruction, we additionally report results on the same test samples using identities predicted by our association module. The results are shown in Tables R1 and R2.

Protocol. We use the same per-view outputs as the GT-identity run of Table 2, and only replace the cross-view association. Since predicted identities do not correspond to the GT annotations, we first match them to GT. For this matching only, we align the reconstruction to GT by a Sim(3) fit on the camera centers and assign each annotated person to a prediction by Hungarian matching on pelvis distance. No distance threshold is applied, so failed reconstructions remain in the evaluation. The pose metrics are then computed exactly as in Table 2.

Metrics. In addition to the pose metrics, we report the following counts, each divided by the number of annotated people. These metrics are reported in Table R2. The 1 m pelvis-distance threshold is used only for these counts, not for the pose metrics.

  • Coverage: annotated people matched to a prediction within 1 m.
  • Missed: 1 − Coverage.
  • Duplicate: unmatched predictions within 1 m of an annotated person.
  • Incorrect merge: matched predictions that fuse a per-view detection of another person. A detection is counted as another person’s if its mean 2D joint error to this person’s GT, normalized by the image long side, exceeds 0.1.

As shown in Table R1, even with predicted identities, our W-MPJPE† remains lower than HSfM (Müller et al., CVPR 2025) and HAMSt3R (Rojas et al., ICCV 2025) on both datasets. For GA-MPJPE on EgoHumans (Khirodkar et al., ICCV 2023), ours is higher than HSfM (Müller et al., CVPR 2025), which uses GT identities. Compared to Ours-GT, PA-MPJPE is unchanged, while W-MPJPE† and GA-MPJPE increase. This suggests that the increased error mainly comes from global placement rather than local pose.

Table R1. Multi-view pose accuracy with predicted identities (m) on EgoHumans (Khirodkar et al., ICCV 2023) and EgoExo4D (Grauman et al., CVPR 2024).

EgoHumans W-MPJPE† (↓)EgoHumans GA-MPJPE (↓)EgoHumans PA-MPJPE (↓)EgoExo4D W-MPJPE† (↓)EgoExo4D PA-MPJPE (↓)
HSfM (Müller et al., CVPR 2025)1.040.210.050.560.06
HAMSt3R (Rojas et al., ICCV 2025)3.800.420.140.510.09
Ours-GT0.370.200.050.230.06
Ours-Pred0.530.320.050.470.06

Table R2 reports the coverage and association failure rates of Ours-Pred. Coverage exceeds 90% on both datasets, while the duplicate rate is higher on EgoExo4D (Grauman et al., CVPR 2024). This is likely because EgoExo4D (Grauman et al., CVPR 2024) contains more challenging scenes with partial observations and unannotated people standing next to the annotated person. Increasing the number of neighboring views in Alg. 1 from 2 to 3 lowers the duplicate rate from 14.6% to 2.9%, at the cost of slightly lower coverage (88.8%) and more incorrect merges (9.8%).

Table R2. Failure rates of Ours-Pred.

Coverage (↑)Missed (↓)Duplicate (↓)Incorrect Merge (↓)
EgoHumans (Khirodkar et al., ICCV 2023)93.2%6.8%3.1%7.5%
EgoExo4D (Grauman et al., CVPR 2024) (k=2)90.2%9.8%14.6%8.3%
EgoExo4D (Grauman et al., CVPR 2024) (k=3)88.8%11.2%2.9%9.8%
AuthorsOpenReview ↗

General Response (1/4)

We sincerely thank all reviewers qtEc (R1), VmaV (R2), VnKn (R3), and 46Qn (R4), for their constructive feedback. We are encouraged by their consensus on the following contributions of our work.

1. Effective and well-motivated design Reviewers recognized our bidirectional anchoring as an “effective solution” (R3) that is “well motivated” (R1, R4), “simple and interpretable” (R2), and “supported by meaningful ablations” (R4), with gains that “do not stem only from the choice of backbone” (R2). Our multi-view design was also noted to have “interpretable component-level evidence” (R4).

2. Substantial accuracy gains Reviewers highlighted “substantial reductions in global reconstruction errors” (R1), “significant improvements over existing methods” (R3), and that “gains in global localization are substantial” (R4), with our method being “more accurate and much faster than optimization-based methods” (R2).

3. Efficient optimization-free inference Reviewers acknowledged that our “single feed-forward model handles both monocular and multi-view videos” (R2) with “significantly reduced inference time” (R3), offering an “attractive accuracy–runtime tradeoff” (R1).

We address the common concerns raised by multiple reviewers with additional experiments and clarifications in the general response. Reviewer-specific questions are answered in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.