acceptodds
Withdrawn from ICLR 2027

COHERE4D: Training-Free Metric Alignment for Coherent 4D Human–Scene Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present COHERE4D, a training-free framework that geometrically aligns the outputs of independently pretrained human and scene foundation models to reconstruct a coherent, metric 4D world from monocular videos. Recent unified human–scene reconstruction methods largely learn metric scene scale and human alignment from training data, but generalize poorly to out-of-distribution scenes. Instead, COHERE4D performs test-time geometric alignment of human and scene predictions using dense pixel-aligned correspondences and RANSAC-based robust estimation. This alignment provides mutual geometric grounding: near-metric human meshes anchor the scale of scene geometry and camera translation, while scene geometry in turn refines human translation to improve global human trajectories. This test-time alignment requires no additional training and uses RANSAC-based robust estimation to suppress outliers caused by occlusion, background contamination, and erroneous depth. Across benchmarks, COHERE4D significantly outperforms prior methods in human–scene alignment, metric-scale scene and camera pose reconstruction, and global human motion estimation. Our training-free alignment method adds only about 4 ms/frame, while the overall inference reaches up to 10 FPS.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Scene Reconstruction, 4D Reconstruction, Metric Scale Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZVEoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper propose a training-free optimization framework for 4D human-scene reconstruction. It first estimates the correspondence between human mesh and scene points and use it to calibrate the global scene scale, and then optimize the global human translation using the scaled scene points. The propose method achieves very strong performance on multiple datasets with different initialization methods.

Critical strengths
  1. The proposed method is a flexible, training-free framework and is compatible with different human and scene initialization methods.
  2. The quantitative and qualitative results are strong with a very comprehensive ablation study.
Critical weaknesses
  1. Lack of novelty. Jointly optimizing the scene scale and the human global translation is not a new idea and is already proven effective is existing training-free optimization frameworks like JOSH.
  2. There are no regularization terms to ensure that the optimized global human motion is plausible and valid in areas like motion smoothness, foot sliding, and physical correctness.
  3. It seems that the initial correspondences between the human mesh vertices and scene points are determined by 2D projection of their initial positions and are fixed afterwards (Sec. 3.2 is very difficult to read and understand). However, the initial human mesh projection can be inaccurate and some vertex may not be even projected within the 2D human mask. In addition, shouldn't the correspondence be updated after the translation refinement with some iterative process?
Questions for authors
  1. Why are the correspondences between the human mesh vertices and scene points fixed in the optimization procedure?
  2. Can the authors also evaluate the human motion quality to show that the proposed method can still generate plausible human motion?
Reviewer_x89hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors present COHERE4D, a training-free framework designed for metric 4D human-scene reconstruction using monocular video. This method integrates separately pretrained models for human mesh recovery and scene reconstruction. It establishes pixel-aligned correspondences between the reconstructed human meshes and the scene point maps. By employing RANSAC, the framework estimates a common metric scale for the scene and calculates per-frame human translations. This estimated scale is essential for calibrating the scene geometry and camera trajectories, while refining scene-grounded translations enhances the positioning of humans in the scene when the reconstructed geometry is reliable. Additionally, overlapping reconstruction segments are aligned to enable the processing of longer video sequences.

Experiments conducted on datasets such as Bonn, BEDLAM2, RICH, BEHAVE, EMDB-2, and 3DPW demonstrate significant improvements in metric scene reconstruction, global human motion analysis, and the geometric alignment between humans and scenes. The framework is compatible with various pretrained human and scene models without the need for further training.

Critical strengths
  1. The paper addresses a significant issue in monocular reconstruction: scene foundation models often produce accurate relative geometry but fail to predict the correct metric scale. The proposed solution of using reconstructed human meshes to calibrate scene geometry is practical, especially since both the human and scene models are typically pretrained.

  2. The framework is modular and does not require training. The authors evaluate Multi-HMR 2, Fast SAM 3D Body, and HMR2.0 using various scene backbones, including DA3, Pi3X, and VGGT-Ω. This modularity is beneficial because it allows the alignment procedure to leverage stronger foundation models without the need for costly joint training.

  3. The metric reconstruction results are significant. Unlike other evaluations that eliminate scale errors by using ground-truth alignment, COHERE4D retains the predicted metric scale. For instance, on the Bonn dataset, calibrating Pi3X with Multi-HMR 2 reduces point error from 30.48 cm to 13.08 cm, which directly supports the paper's main contribution to metric calibration.

  4. The authors present valuable ablation studies of the core components. Appendix Table 10 indicates that RANSAC significantly outperforms ordinary least-squares fitting in situations where the correspondences include outliers. Furthermore, Appendix Table 9 demonstrates that separating scale calibration from translation refinement shows that refinement is not always beneficial. The authors also acknowledge instances where inaccurate foreground geometry can lead to alignment failures.

Critical weaknesses

Major Weaknesses

1. The paper overlooks HAC, a directly relevant prior work that introduced optimization-free human-guided metric scale calibration.

The authors present reconstructed human meshes as natural metric references to calibrate monocular scene geometry. However, Humans as Checkerboards (HAC, ICCV 2025) has already proposed an optimization-free framework that utilizes metric human reconstruction to calibrate monocular SLAM scale. HAC combines independently estimated human and scene information, uses human–scene contact joints as metric depth references, and recovers scaled camera trajectories and global human motion. Despite this close relationship, COHERE4D neither cites HAC nor compares its results against it.

The overlap is not limited to motivation; both methods use pretrained human reconstructions to resolve scene-scale ambiguity without training a joint reconstruction model. COHERE4D differs in that it uses dense pixel-aligned body-surface correspondences instead of contact-joint references and jointly estimates scene scale and human translation through robust geometric fitting. These differences may be significant, but the paper fails to explain them in relation to HAC.

HAC is also absent from Table 4, despite reporting the same global human motion metrics on EMDB-2. HAC achieves 71.0 mm WA-MPJPE, 197.2 mm W-MPJPE, and 1.2% RTE, while COHERE4D reports its best results as 68.1 mm, 152.9 mm, and 1.15%. Although I recognize that the two systems utilize different human and scene backbones, making cross-comparison of their published metrics challenging, HAC remains a directly relevant baseline. Its complete omission makes it difficult to assess the novelty and empirical progress of COHERE4D.

2. The choice of dense pixel-aligned correspondences is not adequately justified, especially considering prior evidence that the specific human reference points used for scale calibration significantly impact results.

COHERE4D generates correspondences by rendering the predicted human mesh at its initial camera-space translation and retrieving scene points at the same image pixels. This method avoids 3D nearest-neighbor matching and provides many candidate correspondences. However, it does not ensure that paired points represent the same physical surface.

If the initial human translation is incorrect, the rendered body can overlap background geometry, nearby objects, or even another person. Consequently, the resulting correspondences may originate from unrelated surfaces. While RANSAC can aid in identifying correct correspondences when enough valid ones are present, it cannot recover the correct locations if the initial projection is significantly displaced. Moreover, Appendix A.2 constructs translation-refinement correspondences using the original human placement without an explicit correspondence-update step after refinement.

The choice of dense matching is particularly crucial given HAC's findings regarding reference-point ablations. HAC reports a WA-MPJPE of 372.86 mm when using the head for scale calibration, 289.76 mm with the pelvis, 71.94 mm with the feet, and 70.95 mm using contact joints. These results demonstrate that having more human geometry does not necessarily equate to more reliable geometry for scale estimation.

COHERE4D does not compare all visible body-surface points with contact or support-region correspondences, nor does it evaluate how alignment behaves when the initial human translation is perturbed. I would expect these controls to be included since dense surface matching is a primary methodological difference between COHERE4D and earlier human-based scale calibration approaches. The current experiments show that dense fitting works well with robust scene backbones, but they do not establish why it is preferable to selecting fewer, more physically meaningful correspondences.

3. The reliability of the metric scale estimator depends on the reconstructed human's dimensions and the first scene chunk; however, neither of these dependencies has been sufficiently examined.

Equation 8 estimates the scene scale based on the relative spatial distribution of human mesh points and scene points. This estimation relies on the assumption that the reconstructed human provides an accurate metric reference. If the human backbone consistently predicts a body size that is too large or too small, the estimated scene scale will inherit the same error. RANSAC cannot eliminate this bias, as correspondences may agree with one another while still sharing the same incorrect metric reference.

Moreover, the method selects a shared scale from the first reconstructed chunk and keeps it fixed throughout the sequence, making initialization particularly crucial. If the video starts with distant humans, significant occlusions, inaccuracies in body dimensions, or poor foreground depth, the initial metric scale may be unreliable. Although later frames might contain more accurate human observations, the method does not utilize them to reassess the initial scale.

There is also an unexplained scenario in which the first chunk contains no valid human reconstruction. Appendix A.1 discusses how unreliable scale hypotheses are filtered and what occurs when none meets the quality thresholds, but it does not clearly explain how initialization proceeds when no valid correspondence or scale candidate is available.

HAC explores the temporal aggregation of human-based scale estimates, although its full-sequence approach is not directly suitable for causal inference. Instead, COHERE4D could assess a running robust estimate or use delayed initialization based solely on past observations. Since the paper claims reliable long-video reconstruction, the stability of its metric initialization should be evaluated rather than simply assumed.

4. The RANSAC ablation demonstrates the importance of robust fitting but fails to establish why the proposed estimator is preferable to other robust alternatives.

The authors employ two-point RANSAC hypotheses, body-size-dependent inlier thresholds, and cross-frame consensus to estimate scene scale. Appendix Table 10 compares the proposed estimator against ordinary least squares and reports significant improvements. These results are beneficial; however, ordinary least squares is known to be sensitive to outliers, particularly when correspondences involve occluders or incorrect depth.

Thus, the comparison shows that robust estimation is necessary rather than demonstrating that the specific RANSAC and scale-consensus procedure is the best option available.

This distinction is even more evident because COHERE4D already employs weighted medians, median absolute deviation, confidence-weighted Procrustes, and IRLS for inter-chunk alignment. These methods present plausible alternatives for parts of the main scale-calibration procedure. Additionally, the selected scene backbones can provide geometric confidence estimates, yet the paper does not clarify whether confidence-weighted robust fitting offers any advantage over treating all candidate correspondences equally before inlier selection.

The authors do not need to compare against every robust estimator. However, given that robust geometric alignment is the central methodological contribution, a controlled comparison against confidence-weighted IRLS, trimmed fitting, or another robust scale estimator would be more enlightening than comparing RANSAC with non-robust least squares. This comparison would help clarify whether the observed gains stem from the specific consensus formulation or simply from utilizing standard outlier rejection methods.

5. Scene-grounded translation refinement is not consistently beneficial, and the method does not clarify when to trust the scene over human predictions.

The paper introduces human-anchored scale calibration and scene-grounded translation refinement as complementary components of bidirectional alignment. However, Appendix Table 9 indicates that translation refinement can actually worsen global human motion.

In the EMDB-2 dataset, adding translation refinement after scale calibration increases the G-MPJPE in seven out of the nine human/scene-backbone combinations. For example, with the 3DB + VGGT-Ω configuration, the G-MPJPE rises from 1203.9 mm to 1208.6 mm. On the RICH dataset, the same configuration causes an increase from 138.9 mm to 184.9 mm.

The authors acknowledge this limitation and retain the original 3DB translation in their main results. While this is a reasonable engineering decision, it raises questions about the generality of the refinement procedure.

The effectiveness of the current choice depends heavily on the specific human backbone used. However, the relative reliability of the human and scene predictions can fluctuate across frames, even within the same sequence. A strong human backbone may occasionally make translation errors, while a reliable scene backbone might fail near occlusion boundaries.

The method lacks a mechanism to determine when to apply translation refinement based on the current predictions' reliability. Although the RANSAC inlier ratio, geometric residuals, and scene-model confidence could potentially provide useful signals, this has not been explored. Therefore, the experiments demonstrate that scene-grounded translation refinement is beneficial under certain conditions, but not that it is a reliably corrective method in all cases.

6. The paper claims to present more plausible human–scene interactions, but its quantitative evaluation focuses on geometric agreement rather than physical grounding.

The introduction and qualitative comparisons repeatedly link improved alignment to more plausible contact and interaction scenarios. Figures 7, 8, and 15 provide examples of humans interacting with benches, supporting surfaces, and other individuals.

However, the primary alignment evaluation employs HS-CF, which measures the agreement between the visible human mesh surface and scene points within the human region. These scene points can actually represent the human's surface rather than the static surfaces that support the body.

As a result, a reconstruction could yield a low HS-CF value while the person still appears to float above the ground, penetrate nearby objects, or slide during contact. Aligning two reconstructed representations does not inherently establish physically correct interactions with the surrounding environment.

This is a significant oversight, as JOSH and UniCon3R—both of which are included in COHERE4D's global-motion comparison—already evaluate relevant physical grounding errors. JOSH reports issues such as foot sliding and floating, while UniCon3R assesses collision ratios, penetration depths, floating distances, and maximum penetration on the RICH dataset.

I believe that a geometric alignment method does not necessarily need to predict contact forces or perform physical optimization. However, improved physical interaction is presented as an outcome of COHERE4D, yet the reported alignment metric fails to substantiate this claim. At a minimum, one quantitative comparison of physical grounding is necessary to support this assertion. Otherwise, conclusions should be limited to improved geometric alignment between the predicted human and scene surfaces.

7. Baseline Results Require Further Explanation

Several baseline results are not adequately explained, particularly the substantial difference between COHERE4D's reproduced UniSH depth results and those reported in the original paper.

COHERE4D reevaluates Human3R and UniSH on Bonn, focusing on scene depth and camera reconstruction under a protocol that maintains the predicted metric scale. This approach is appropriate for evaluating metric calibration.

However, the reproduced UniSH results show a significant discrepancy when compared to the original publication. UniSH reports an AbsRel of 0.035 and a threshold accuracy of 0.980 on Bonn, while COHERE4D reports an AbsRel of 0.318 and a threshold accuracy of 0.5541. This represents approximately a ninefold increase in AbsRel and a drop of over 42 percentage points in threshold accuracy. In contrast, the reproduced Human3R results are much closer to the original publication.

I acknowledge that this discrepancy may stem from differences in scale-alignment conventions, checkpoints, preprocessing, or evaluation subsets. I am not asserting that COHERE4D's results are necessarily incorrect; however, such a significant difference warrants a clearer explanation, especially as these results are used to illustrate the limitations of learned metric-scale reconstruction.

The authors should specify precisely which evaluation settings differ from UniSH's original benchmark to ensure reproducibility. Without this information, it is challenging to ascertain whether COHERE4D is highlighting a genuine metric-scale generalization failure or comparing methods under significantly different evaluation conventions.

Minor Weaknesses

  1. Discussion of HumanBA: The relationship to HumanBA should be addressed. HumanBA (CVPR 2026) utilizes human motion to enhance camera pose estimation through motion decoupling and reliability-weighted bundle adjustment. Its optimization-based framework differs from COHERE4D's approach, so a direct comparison in the main feed-forward reconstruction table is not necessary. However, a brief discussion would help delineate COHERE4D's human-guided scale calibration from human-aware camera-motion refinement.

  2. Implementation Details: Implementation details should be documented in greater detail. The method introduces chunk length LL and overlap OO, but does not provide their actual values. These choices significantly impact latency, correspondence availability, and cumulative alignment error. The authors should also specify the exact DA3, Pi3X, and VGGT-Ω checkpoints used, along with preprocessing and confidence settings. This information is particularly crucial as some backbone repositories contain multiple model variants and updated checkpoints.

Questions for authors

Questions

  1. Comparison with HAC and the Role of Dense Correspondences. HAC (ICCV 2025) introduced optimization-free human-guided metric scale calibration and provided code along with EMDB-2 evaluation scripts. To highlight the novelty and empirical advancements of COHERE4D, the authors are asked to: (i) explicitly discuss HAC and differentiate its contact-based calibration from the proposed dense-surface alignment, (ii) include HAC's published EMDB-2 results as a contextual baseline, clearly identifying differences in pretrained backbones, and (iii) compare dense visible-body correspondences, contact/support-region correspondences, and confidence-weighted correspondences using the same human and scene predictions. The last experiment is particularly important, as HAC demonstrates that the choice of human reference points significantly impacts scale estimation. (Refer to major weaknesses 1 and 2.)

  2. Sensitivity to Initial Human Placement and Correspondence Errors. COHERE4D generates correspondences based on the original human projection but does not demonstrate how effectively the estimator can recover from inaccuracies in this projection. The authors are requested to: (i) perturb initial human translations by controlled amounts and measure the errors in scale, human translation, and scene reconstruction, (ii) compare the current fixed-correspondence refinement with an alternative approach that re-renders the human model and updates correspondences after each translation estimate, and (iii) report the valid-inlier ratio and failure rate as initialization error increases. These experiments would clarify whether COHERE4D can recover from poor initial localization or primarily enhances predictions that are already somewhat aligned. (Refer to major weakness 2.)

  3. Reliability of Human-Anchored Scale and First-Chunk Initialization. Equation 8 renders metric scale dependent on predicted human dimensions, while Appendix A.1 establishes the scale using the first reconstruction chunk. The authors are requested to: (i) perturb the predicted human body scale by ±5%, ±10%, and ±20%, and measure the resulting errors in scene and camera scale, (ii) estimate scale from different chunks of the same video to quantify sensitivity to initialization, (iii) compare fixed first-chunk calibration with a causal running robust scale estimate, and (iv) explain how the system initializes when the first chunk lacks valid human detection or sufficient correspondences. These tests would directly evaluate the assumptions behind the metric anchor and the claimed reliability over long sequences. (Refer to major weakness 3.)

  4. Justification for the Robust Scale Estimator. Appendix Table 10 compares RANSAC against ordinary least squares. To highlight the benefits of the proposed scale-consensus procedure, the authors are requested to conduct a comparison with at least one other robust estimator, such as confidence-weighted IRLS, trimmed least squares, or median-based scale fitting. These comparisons should utilize identical human and scene predictions, covering the same evaluation sequences and including a reported runtime cost. Additionally, it would be beneficial to analyze the inlier threshold, correspondence sampling strategy, and the use of predicted scene confidence. If simpler robust fitting yields similar results, the paper should clarify which specific aspects of the proposed estimator contribute to the observed improvements. (Refer to major weakness 4.)

  5. When Should Scene-Grounded Translation Refinement Be Applied? Appendix Table 9 indicates that translation refinement worsens G-MPJPE for seven of the nine backbone combinations on EMDB-2. The authors are requested to: (i) report how often refinement helps or hinders performance at the frame or sequence level, (ii) investigate whether RANSAC inlier support, geometric residuals, or scene confidence can predict these outcomes, and (iii) compare the current backbone-dependent approach to a straightforward confidence-gated refinement policy. The objective is to determine if bidirectional grounding can ascertain when the scene provides a more reliable estimate, rather than relying on a fixed decision for each pretrained backbone. (Refer to major weakness 5.)

  6. Quantitative Evidence for Physically Plausible Interaction. The paper claims to improve contact and achieve more plausible interactions, but the primary alignment metric does not assess physical support. The authors are requested to present physical grounding results on RICH using established metrics from UniCon3R, including penetration depth, maximum penetration, floating distance, and collision ratio, with assessments of Human3R, UniSH, and UniCon3R conducted under the same protocol. Foot sliding or contact stability metrics would provide additional evidence where temporal support annotations are available. These results are crucial for determining whether lower HS-CF corresponds to enhanced interaction with the actual environment. If not, the claims should be limited to discussing geometric alignment. (Refer to major weakness 6.)

  7. Clarification of the Reproduced UniSH Results. UniSH originally reported 0.035 AbsRel and 0.980 threshold accuracy on Bonn, while COHERE4D reports 0.318 and 0.5541. The authors are requested to specify: (i) the exact UniSH checkpoint and inference configuration, (ii) the Bonn sequences and preprocessing used, (iii) whether either evaluation employs scale alignment, and (iv) whether the original published result can be reproduced before applying COHERE4D's metric-scale evaluation protocol. This clarification is essential before interpreting the difference as a failure of learned metric-scale reconstruction, rather than a discrepancy in evaluation settings. (Refer to major weakness 7.)

Reviewer_bjg3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents COHERE4D, a training-free framework for metric 4D human-scene reconstruction from monocular videos. It combines independently pretrained human and scene models through geometric alignment: metric human reconstruction is used to calibrate scene/camera scale, while scene geometry can further refine human translation.

Critical strengths
  • The framework is simple and training-free, and can directly combine different pretrained human and scene models without joint retraining.
  • Human-anchored scale calibration gives particularly consistent improvements across scene backbones and datasets. Even for scene models that already predict metric scale, such as DA3 and Pi3X, the human-derived calibration generally improves scene geometry and camera translation.
  • The experiments cover multiple datasets and backbone combinations, with strong results across metric scene reconstruction, human-scene reconstruction, and global human motion, while adding relatively little computational overhead.
Critical weaknesses
  1. Novelty relative to closely related human-based metric calibration work

The main idea of using metric human reconstruction to resolve scene/camera scale ambiguity is closely related to prior work.

In particular, Humans as Checkerboards (HAC, ICCV 2025) also uses metric humans to calibrate monocular camera-motion scale for world-coordinate human reconstruction. SynCHMR (CVPR 2024) similarly explores the broader idea that humans can help recover metric scene/camera information and that scene information can in turn improve human reconstruction.

COHERE4D has meaningful differences, especially its dense human-scene alignment and its ability to combine recent foundation models without retraining. However, I think the paper currently overstates the novelty of the broader human-anchored metric grounding idea. The relation to HAC and SynCHMR should be discussed more clearly, and a direct comparison with HAC would be valuable.

  1. The “bidirectional grounding” claim appears stronger than the results support

Human-to-scene scale calibration is consistently effective and appears to be the strongest part of the method.

In contrast, scene-grounded human translation refinement is less consistent. On EMDB-2, adding translation refinement sometimes provides little improvement or slightly hurts performance, and for 3DB the main results retain the original human translation.

This suggests that the two directions are not equally reliable. I think the paper should either moderate the bidirectional-grounding claim or explain more clearly when scene-to-human refinement is expected to help.

  1. It is difficult to fully separate the alignment contribution from the use of stronger recent backbones

Many of the strongest results combine recent human models with DA3, Pi3X, or VGGT-Ω, while several unified baselines use fixed or older backbones.

The HMR2.0 experiments help, but a more controlled comparison using the same human and scene backbones with different scale/alignment strategies would better isolate the contribution of COHERE4D itself.

Questions for authors
  1. Why is Sim(3) stitching used for scene backbones that already predict metric scale?

Consecutive scene chunks are aligned using a similarity transform with an independent scale factor. This is natural for scale-ambiguous VGGT-Ω, but DA3 and Pi3X already predict metric-scale geometry.

For such models, an SE(3) alignment seems like a natural alternative because it preserves the backbone's predicted scale. Allowing per-chunk rescaling may also complicate the interpretation of how much the subsequent human-based calibration improves the original metric prediction.

Could the authors explain this choice and, ideally, compare SE(3) versus Sim(3) stitching for metric scene backbones?

  1. How should scene-grounded translation refinement be enabled in practice?

The ablations suggest that translation refinement helps when scene geometry is reliable but can hurt under challenging moving-camera conditions.

Is there a confidence criterion that could automatically decide whether to use the refined translation or preserve the original human prediction? This seems especially relevant for making the method robust across different scene backbones.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.