- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HARMONI reconstructs cameras, scene geometry, and multiple SMPL-X humans from monocular or multi-view video without test-time optimization. It combines a frozen Pi3X scene model with human features from a frozen Multi-HMR encoder and trainable human reconstruction modules. Its central contribution is bidirectional anchoring: scene depth anchors human placement through a predicted head-depth residual, while human body proportions calibrate the scene’s metric scale. For multi-view inputs, the method combines confidence-weighted parameter averaging, head triangulation, and a refinement transformer. A separate association procedure matches people across time and views using learned features and geometric similarities. Results show improvements in global human reconstruction on RICH and EMDB-2 and in multi-view localization on EgoHumans and EgoExo4D. The reported reductions of approximately 28% and 64% are supported by the corresponding table entries. The principal contribution is the specific geometric coupling and its integration into an efficient system: human-informed scene scaling and feed-forward human–scene alignment already appear in HSfM and UniSH, respectively.
Critical strengths
-
The alignment mechanism is well motivated and supported by meaningful ablations.
-
The gains in global localization are substantial, while local pose quality remains competitive.
-
The multi-view design has interpretable component-level evidence.
-
The supplementary evaluation adds useful practical evidence.
Critical weaknesses
-
The main reconstruction results do not establish performance of the fully automatic identity pipeline. Table 2 explicitly uses ground-truth identities for every method. This is appropriate for comparing reconstruction conditional on correct correspondence, but it bypasses a central advertised capability of HARMONI. Table 7 evaluates association separately, reporting 88.5% precision and 78.7% recall; the consequences of these errors for triangulation, fusion, and final reconstruction are not reported. Moreover, the comparison between 90.9% association accuracy and the previously reported 51.2% appearance-based result needs clearer protocol matching: the manuscript does not establish identical temporal context, detection inputs, or accuracy definitions. The earlier result comes from HSfM’s evaluation of an UnCaliPose-based association procedure. HSfM, supplementary discussion.
-
The central scale-calibration operation is insufficiently specified and its metric effect is not fully established. Equations 1–3 first construct human translation from scene depth and a residual, then update the scene scale using . The manuscript does not explicitly state the resulting transformation of human translations or the depth residual. For example, scaling the entire depth as differs from scaling only scene depth as . This distinction matters because the residual represents an anatomical surface-to-joint distance. The projected-length ratio also depends on pose, shape, perspective, and keypoint errors; it is not automatically an exact scene-scale estimator. The paper should state its approximation assumptions and fallback behavior when reliable joints are scarce. Table 4 provides useful scale-sensitive evidence, but Table 5 reports identical Pi3X and HARMONI camera/depth scores to the displayed precision. The evaluation description does not clarify whether these are calibrated outputs or whether scale alignment removes the calibration’s effect. Appendix B.3 explicitly uses Sim(3) alignment for the Table 1 joint metrics, so those metrics alone cannot establish absolute metric-scale accuracy.
-
The association algorithm does not guarantee the claimed global consistency. Algorithm 1 independently performs Hungarian matching for each connected view pair and then merges accepted matches using union-find. Pairwise one-to-one matching does not ensure consistency across cycles. For example, accepted matches , , and merge two different people from view A into one identity, even though each pairwise assignment is individually one-to-one. No conflict-resolution rule or restriction against merging simultaneous same-view detections is described. Such a merge could directly corrupt the subsequent fusion.
-
The long-video inference protocol is too incomplete to assess temporal comparisons and practical scalability. Section 3.1 jointly processes frames using the scene model, but the evaluation details do not specify inference window length, temporal sampling, overlap, or how separately processed windows share a coordinate system and scale. These choices can materially affect trajectory accuracy and resource usage. This matters when comparing with Human3R, which explicitly processes video online using a recurrent state. Different access to future frames is a legitimate design trade-off, but it should be disclosed when interpreting the accuracy comparison. Human3R, method and efficiency discussion. The four-view runtime measurement supports efficiency for that configuration; it does not characterize long-video processing. The qualitative long trajectories in Appendix A.1 do not resolve the missing inference details.
Questions for authors
- What happens to the main reconstruction results when ground-truth identities are replaced by your predicted associations? Please report the resulting localization errors and how missed detections, duplicate identities, and incorrect merges enter the evaluation.
- What are the exact equations after applying ? Which quantities are rescaled, is the anatomical depth residual rescaled, and is the calibration ratio recomputed? What happens when no sufficiently reliable calibration joints are available?
- Does Table 5 evaluate the final calibrated scene? Please specify the camera and depth alignment protocols and explain why all four reported metrics match the frozen Pi3X baseline to the displayed precision.
- How does Algorithm 1 resolve inconsistent association cycles? Can a union-find component contain two people observed simultaneously in the same view, and how often does this occur before any corrective handling?
- What temporal context and windowing are used for each benchmark? Please specify frame counts, sampling, window alignment, and resource usage, and clarify whether Table 7’s association comparison uses the same observations and metric definition as the cited appearance-based baseline.
2 comments
Gentle reminder for discussion
Dear Reviewer 46Qn,
Thanks again for your thoughtful review. We addressed your questions as follows:
- Table 2 with predicted identities (Item 1)
- The appearance-based comparison under our protocol (Item 12; revised in Sec. 4.3, L468–471)
- The update equations after scale calibration and the Table 5 protocol (Items 5–6; revised in Sec. 3.2, L281–283 and Sec. 4.2, L421–423)
- Inconsistent association cycles (Item 4)
- The long-sequence inference protocol (Item 7; revised in Sec. B.3, L1205–1210)
We would appreciate it if you could check these responses and let us know if any point needs further clarification.
Response to Reviewer 46Qn
We thank the reviewer for the constructive feedback. We address most questions in the General Response.
11. Difference with HSfM (Müller et al., CVPR 2025) and UniSH (Li et al., CVPR 2026)
HSfM (Müller et al., CVPR 2025) uses the human body as a scale reference, but this addresses only the scale misalignment. The remaining depth misalignment is resolved through test-time optimization with external 2D keypoints and known identities. UniSH (Li et al., CVPR 2026) aligns humans and the scene in a feed-forward manner with AlignNet, but UniSH (Li et al., CVPR 2026) directly estimates human depth in the scene. As a result, UniSH (Li et al., CVPR 2026) may show depth misalignment (Fig. 1b). In contrast, our bidirectional anchoring explicitly addresses both scale and depth alignment in a single feed-forward pass.
12. Clarification on appearance-based protocol
The 51.2% in Sec. 4.3 is quoted from the HSfM (Müller et al., CVPR 2025) supplementary, which evaluates an UnCaliPose (Xu and Kitani, BMVC 2022)-based appearance association. We included it to illustrate that appearance cues are unreliable in multi-person scenes with similarly dressed people, but we agree that it was not measured under our protocol, which we revised in Sec. 4.3, L468–471. We further compared the ReID method in UnCaliPose (Xu and Kitani, BMVC 2022) under our protocol in Table R7.
Table R7. Comparison with appearance-based matching.
Under the identical protocol, ours outperforms the appearance baseline. Using appearance cues tends to merge similarly dressed people, which results in lower precision. We believe the appearance baseline achieves a higher accuracy than the quoted 51.2% because our protocol matches 25-frame tracklets rather than single-frame boxes.
Question summary
A1. We kindly refer the reviewer to Item 1 (predicted identities) and Item 2 (identity consistency).
A2. Please see Item 5 (update equations with r).
A3. Please see Item 6 (clarification on Table 5).
A4. Please see Item 4 (inconsistent cycles).
A5. Please see Item 7 (long-sequence protocol) and Item 12 (appearance-based protocol comparison).