acceptodds
Under review as a conference paper at ICLR 2027

StableCrowd: Spatio-Temporally Consistent Feed-Forward Crowd Reconstruction with Multi-Joint Height

Abstract

Reconstructing crowd motion from monocular large-scene video requires efficiency and spatio-temporal consistency. Among feed-forward methods, per-person video methods recover each person's motion but not the relative positions of the people, while large-scene crowd methods anchor each person in the shared scene through a point regressed as a pixel distance in a crop, a target that varies with the crop and so jitters over time. On the other hand, existing methods that achieve spatio-temporal consistency rely on expensive per-sequence optimization. We present StableCrowd, a feed-forward method that reconstructs crowd motion from such videos. We introduce a new target, Multi-Joint Height, the metric height of every joint above the scene surface. It is crop-invariant, stays stable over time, and can be lifted in closed form to a metric 3D estimate of the joint. We propose Global Trajectory Fusion, which regresses the trajectory in the same space as these estimates, committing directly to absolute positions in the shared scene and covering a sequence of any length. We also propose Canonical Camera Normalization, which absorbs the position and size of each crop into a camera rotation and a zoom, so that close-range single-person data trains the model for large-scene video. StableCrowd is the first feed-forward method to achieve spatio-temporal consistency from large-scene video, zero-shot on VirtualCrowd and WorldPose, showing that per-sequence optimization is no longer required to reach it. Code and models will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
monocular video, multi-person human pose and shape estimation, feed-forward, spatio-temporal consistency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_t2M5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents StableCrowd, a feed-forward method for reconstructing 3D human motion from monocular large-scene videos. The method predicts the height of multiple body joints above the ground and uses camera geometry to recover their 3D positions. A temporal fusion network then combines these estimates to produce stable global trajectories. The authors also introduce camera normalization to improve the generalization across different person crops. Experiments on VirtualCrowd and WorldPose show strong results in both spatial accuracy and temporal consistency without per-sequence optimization.

Critical strengths
  • The paper addresses an important and challenging problem. Using multi-joint heights for global localization is a simple but effective idea with a clear geometric motivation.

  • The overall framework is well designed. The proposed components work together to improve both spatial and temporal consistency.

  • The experimental results are strong. The authors compare with several recent methods on both synthetic and real-world datasets. The ablation studies also support the effectiveness of the main components.

Critical weaknesses
  • Incomplete ablation of temporal fusion. The idea of MJH is good, but I wonder how much of the improvement actually comes from GTF. In Table 5, HVIP is tested without temporal fusion, while MJH uses GTF. Could the authors also test HVIP with GTF? This would help show the benefit of MJH itself.

  • Limited analysis of Phase-shifted Coverage. The current ablation only compares whole-sequence prediction with the full phase-shifted method. It would be useful to compare with fixed-length windows with 0% or 50% overlap and simple blending. The effect of different numbers of refinement rounds is also unclear.

  • Sensitivity to camera and ground errors. The method relies on accurate camera and ground geometry, but the experiments do not fully evaluate its robustness to errors in these inputs. This is important for real-world videos where camera and ground parameters need to be estimated.

Questions for authors
  • Could the authors add an experiment using HVIP with the same temporal GTF to better separate the contributions of the representation and temporal fusion?

  • Could the authors compare Phase-shifted Coverage with fixed-length windows using 0% or 50% overlap and report results with one, two, and three refinement rounds?

  • How sensitive is the method to errors in camera and ground estimation? It would be helpful to test different error levels (e.g., 1%–10%) in focal length and camera height, angular errors in the ground normal. Camera height errors are especially worth studying.

  • CCN appears related to existing perspective crop methods, such as PersPose[1]. Could the authors briefly clarify the differences?

  • Could the authors provide a detailed runtime breakdown of the full pipeline, including detection, tracking, camera and ground estimation, CCN, MJH, MHL, and GTF? This would help readers better understand the actual inference cost.

[1] Hao, Xiaoyang, and Han Li. "Perspose: 3d human pose estimation with perspective encoding and perspective rotation." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

5 comments
AuthorsOpenReview ↗

Response to Reviewer t2M5 (1/3)

(1/3)

We thank the reviewer for finding MJH "a simple but effective idea with a clear geometric motivation", the framework "well designed" and that "the experimental results are strong". We would be grateful to hear whether the revision addresses the concerns raised, and will answer any further question promptly within the discussion period.

HVIP with temporal fusion (Weakness 1; Question 1). The revision adds GTF retrained on a single point, HVIP or the anchor height of the body root, in place of the per-joint estimates of MJH, with nothing else changed (Table Rc-1, also added to Table 5). Under the same GTF, MJH is the most accurate, on WorldPose the root height with GTF being slightly smoother at a 30% higher MPJPE, and even with per-frame softmax fusion and no GTF MJH has a lower MPJPE than either single point with GTF (Table 5), so the accuracy does not come from temporal fusion alone.

Table Rc-1. HVIP, root height and MJH with matched temporal decoders (same as Table Ra-1). Cells read VirtualCrowd / WorldPose; GTF† is GTF trained with a single point as its input.

MPJPE (m) ↓SEQ-RT-MPJPE (mm) ↓ACC error (mm/f²) ↓
HVIP, direct (Table 5)1.80 / 0.57450 / 277201.36 / 75.63
HVIP + GTF†1.76 / 0.61280 / 2595.97 / 2.68
Root height, direct (Table 5)1.67 / 0.41209 / 22961.94 / 36.65
Root height + GTF†1.69 / 0.43182 / 2313.69 / 1.96
MJH + GTF (Table 5)1.51 / 0.33156 / 1992.65 / 2.26

Phase-shifted coverage (Weakness 2; Question 2). The whole-sequence row of Table 5 showed that bounded windows give the accuracy on long sequences. Within windows of one length, a single round with 50% overlap and linear blending already bridges the window boundaries, and the three shifted rounds of the submitted model, whose windows within a round lie end to end, smooth them further at unchanged accuracy (Table Rc-2). The revised appendix reports this table (Appendix I, Table 19).

Table Rc-2. Phase-shifted coverage against windows of one length, by overlap and rounds (new). Cells read VirtualCrowd / WorldPose; best in bold.

Windows of one lengthMPJPE (m) ↓ACC error (mm/f²) ↓ACC norm ratio (→ 1)
One round, windows end to end1.520 / 0.3333.50 / 3.721.27 / 4.31
One round, 50% overlap, linear blending1.515 / 0.3263.01 / 2.461.19 / 3.53
Two rounds, the second shifted by half a window1.513 / 0.3292.96 / 2.601.06 / 3.50
Three rounds, each shifted by half a window (submitted)1.507 / 0.3272.65 / 2.260.85 / 3.14
AuthorsOpenReview ↗

Response to Reviewer t2M5 (2/3)

(2/3)

Sensitivity to the scene (Weakness 3; Question 3).

  • Input. Like Crowd3D, Crowd4D and DyCrowd, StableCrowd takes the camera and scene surface as its input, calibrated or estimated with existing tools; this input is not among our contributions, and every method we can re-run is given the same one (Sec. 4.2).
  • Real-world videos. Large-scene videos from deployed cameras, such as surveillance, are the main target of StableCrowd; even with unknown intrinsics and lens distortion, these cameras can be calibrated accurately and automatically against a 3D reconstruction of the site from an automatically flown drone survey, as shown in Appendix L, Fig. 13, with median reprojection residuals from 0.60 to 2.37 pixels. The revision states this dependence in the limitations, together with the automatic calibration of deployed cameras (Appendix M.4).
  • Error levels. The revision perturbs the focal length, camera pitch, camera height and ground orientation one at a time (Table Rc-3): an error of 1 to 10% in the camera height moves each person's ground point by the same share of its distance.
  • Estimated scene. Table 2 and Table Rc-4 evaluate StableCrowd with an estimated ground, on VirtualCrowd and, under the protocol of the FIFA Skeletal Light challenge, on WorldPose, where the per-frame intrinsics and lens distortion are given and the camera rotation of every frame after the first, and with it the ground in each camera's frame, is estimated from the pitch lines (Appendix I, Table 23). Against the given camera, the estimated one changes the MPJPE of StableCrowd little and leaves its spatial consistency (PPDS, PA-PPDS) essentially unchanged, StableCrowd remaining far more accurate than TRAM; apart from the one sequence in which the court lines are lost, its rotation error lies far below the angles of Table Rc-3, its pitch component the smallest of the three (Appendix I, Fig. 9). In the wild, on three in-the-wild clips, two of whose scenes are shown in Figs. 12 and 15, estimating the scene surface, or the camera intrinsics as well, moves each person by 4.5 to 6.2% of the distance to the camera on average, while the layout of the people is largely kept (Appendix L, Table 24; Fig. 12 shows the camera intrinsics and scene surface estimated from the video).

Table Rc-3. Error propagation of the focal length, camera pitch, camera height and ground orientation (new; same as Table Ra-3). Ground error as a share (%) of the distance, one factor at a time, in the worse direction. VirtualCrowd: means over the person-frames of the six evaluated videos, whose rays meet the ground at a median of 9°, that the lifting still takes (under the 1, 2 and 5° tilts toward or away, 9.8, 15.5 and 29.4% fall below the 2° threshold and are left out); other columns: a single ray meeting the ground at that angle. Full table: Appendix F, Table 10.

ErrorVirtualCrowd45°30°10°
Camera height, 1 / 5 / 10%1 / 5 / 101 / 5 / 101 / 5 / 101 / 5 / 10
Focal length, 1 / 5 / 10%0.56 / 2.8 / 5.60.71 / 3.5 / 7.11.0 / 5.1 / 103.0 / 16 / 38
Camera pitch or ground tilt toward or away, 1 / 2 / 5°13 / 27 / 811.8 / 3.7 / 103.1 / 6.5 / 1811 / 25 / 99
Ground tilt sideways, 1 / 2 / 5°2.0 / 3.9 / 9.5–––

Table Rc-4. Absolute localization on WorldPose with an estimated ground (new; same as Table Ra-4). The per-frame intrinsics and lens distortion, which change with the zoom, are given in both. Given: the standard protocol (Table 3). Estimated: the protocol of the FIFA Skeletal Light challenge; ours is re-run with it, and TRAM is the row of Table 3, placed in the world by the same camera. Best in bold.

GroundMethodMPJPE (m) ↓SEQ-RT-MPJPE (mm) ↓ACC error (mm/f²) ↓PPDS ↑PA-PPDS ↑
GivenTRAM2.711928628.770.8410.845
GivenOurs0.331992.260.9840.985
EstimatedTRAM2.721936634.910.8410.845
EstimatedOurs0.362442.350.9830.984
AuthorsOpenReview ↗

Response to Reviewer t2M5 (3/3)

(3/3)

CCN and PersPose (Question 4).

Differences. CCN differs from PersPose in two respects.

  • Source of the warp rotation. PersPose turns the camera along the shortest arc onto the box center, with no ground plane; CCN fixes the rotation about the optical axis against the ground normal.
  • Network input. PersPose encodes the cropped camera as a per-pixel map (its Sec. 4.1 and 4.2); CCN passes the focal length and the residual pitch to the network, which regresses the pose with the pitch set to zero and adds it back analytically.

CCN as a design choice. Both apply the perspective warp of a camera rotating about its optical center and zooming (Hartley & Zisserman), each where its task needs it. On reflection, we regard the StableCrowd pipeline and its core component, MJH, as the substantive technical contributions; CCN is a design choice that serves them. Sec. 4.4 cites PersPose alongside PCL (Yu et al., CVPR 2021), and Appendix M.3 discusses CCN together with PersPose, PCL and Crowd3D++.

Runtime (Question 5). The revision reports the runtime of each stage (Appendix M.2):

  • Detection, tracking and keypoints precede any method in a large scene, where cropping is necessary; on 1920$×$1080 frames with about 207 people, they run at 14.9 frames per second (29.0 batched), 70.6 frames per second and 166 person-frames per second (255 batched).
  • The camera and scene surface are estimated offline, once per video or once per site, within minutes.
  • The stages of StableCrowd run at 4262 (CCN), 174 (image features), 1227 (MJH, 4741 batched), 7825 (MHL) and 1623 (GTF) person-frames per second.
  • After the detection, tracking and keypoints, StableCrowd reconstructs 103 person-frames per second, so that a scene of 100 people over 200 frames takes about 195 seconds, against about four hours reported by Crowd4D and DyCrowd (Appendix M.1).
Reviewer_gTFMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The authors introduce StableCrowd, a feed-forward framework designed to reconstruct crowd motion within a shared world coordinate system using large-scene monocular videos. The method comprises three main components:

  1. Multi-Joint Height (MJH): This component predicts the height of ten body joints above the scene surface and analytically converts them into metric 3D positions by utilizing known camera and ground geometry.

  2. Global Trajectory Fusion (GTF): GTF integrates these joint-specific estimates into a singular root trajectory by employing overlapping temporal windows and multiple refinement rounds.

  3. Canonical Camera Normalization (CCN): This component normalizes individual person crops through virtual camera rotation and zoom, minimizing discrepancies between close-range training images and large-scene videos.

The authors conduct evaluations on the VirtualCrowd and WorldPose datasets, comparing their results with those of existing crowd reconstruction methods, including Crowd3D, Crowd4D, DyCrowd, TRAM, GVHMR, and GEM. The findings indicate improvements in absolute localization, local pose accuracy, and temporal stability according to the reported evaluation protocols.

Critical strengths
  1. The paper addresses an important problem in large-scene human reconstruction. Existing whole-image methods often lose detail when hundreds of people occupy a large image, while independently reconstructing each person makes it difficult to recover a shared spatial layout. The paper provides a clear distinction between spatial consistency, temporal consistency, and absolute localization, and evaluates these properties separately.

  2. Multi-Joint Height is a useful representation for metric human localization. A joint's height, image observation, and calibrated camera/ground geometry determine its 3D location when the viewing ray is well conditioned. Using several joints provides multiple localization estimates and avoids relying entirely on one root or ground-contact point. The representation ablation in Table 5 supports the benefit of replacing HVIP with metric height and combining multiple joint estimates.

  3. GTF addresses a real limitation of absolute trajectory regression over long videos. Regressing an entire sequence in one window works reasonably on shorter sequences but generalizes poorly to the longer WorldPose videos. The phase-shifted method reduces WorldPose MPJPE from 3.27 m for whole-sequence GTF to 0.33 m. Figure 7 provides a useful qualitative example covering 1,948 frames.

  4. The reported results are strong under the stated protocols. StableCrowd achieves 0.33 m absolute MPJPE on WorldPose, compared with 2.71 m for TRAM, and 133.5 mm SEQ-RT-MPJPE on VirtualCrowd. The supplementary material also describes the dataset populations, baseline reruns, fine-tuning procedures, and implementation settings in considerable detail.

Critical weaknesses

Major Weaknesses

1. The novelty of Canonical Camera Normalization (CCN) is not adequately distinguished from earlier geometric crop normalization methods.

In Section 3.1, CCN is introduced as one of the three main contributions of the work. This method creates a virtual camera whose principal axis points toward the person, whose roll is aligned with the ground, and whose focal length is adjusted according to the crop.

However, Perspective Crop Layers (PCL, CVPR 2021) have already introduced perspective-correct crops using a virtual camera with the same optical center, along with a rotation-and-zoom transformation. The mathematical formulation of its warp is essentially the same as that in StableCrowd's Equation 1. PCL also explores the choice of virtual focal length and demonstrates that correcting for perspective effects improves human pose estimation.

A closely related work is RCR/Crowd3D++ (arXiv:2411.06232), which specifically addresses large-scene crowd reconstruction. RCR introduces canonical upright 2D/3D spaces and ground-aware normalization to mitigate the effects of camera orientation, field of view, and crop location. While StableCrowd cites Crowd3D, it does not discuss RCR or PCL.

It is important to note that CCN is not identical to either method. Its ground-normal-aligned virtual camera and its integration with temporal MJH regression are significant differences. However, the fundamental principle of camera normalization is derived from prior work.

Table 4 only compares CCN to ordinary crops and CLIFF-style box encoding without contrasting it with existing geometric crop normalization methods. Since CCN is presented as an independent contribution, the authors should clarify what is novel about its formulation and demonstrate how this specific construction improves upon previous geometry-aware alternatives. While the integrated StableCrowd system may still be novel, the current presentation exaggerates the novelty of the normalization component.

2. The assertion that Crowd3D's HVIP representation cannot yield temporally stable reconstruction—even with a perfect regressor—is mathematically incorrect as stated.

The introduction claims that Crowd3D represents HVIP as a crop-relative pixel distance, which changes with crop adjustments. It then states that even a perfect regressor cannot recover temporally consistent 3D positions from this representation.

This conclusion does not logically follow from the dependence on the crop.

Let AtA_t denote the known transformation between full-image coordinates and crop coordinates at frame tt, and let uu be the full-image projection of a fixed HVIP. A perfect crop-relative regressor predicts At(u)A_t(u). By applying the inverse crop transformation, we recover At−1(At(u))=uA_t^{-1}(A_t(u)) = u. Thus, changes in crop size or position do not inherently alter the recovered full-image point.

Crowd3D already transforms the predicted HVIP back into the original image coordinates before integrating it into the shared scene. With perfect prediction and unchanged camera/ground geometry, the recovered world point can remain consistent.

While the authors may be correct that crop-dependent targets are more challenging to learn and that imperfect HVIP predictions lead to jitter, these factors are different from claiming that perfect HVIP regression cannot achieve temporal consistency.

This distinction is crucial because the purported fundamental limitation of HVIP serves as a primary motivation for MJH. The authors should either provide a derivation supporting the impossibility claim or revise it to reflect a learning and generalization challenge. Conducting an experiment that converts ground-truth HVIP predictions into world coordinates would help differentiate between representation limitations and regression errors.

3. The main evaluation relies on ground-truth camera and tracking information, so the results do not fully establish reconstruction accuracy from monocular video without additional scene knowledge.

Section 4.2 states that the standard protocol provides every method with ground-truth person tracks, camera intrinsics, ground geometry, and, on WorldPose, per-frame camera extrinsics.

These inputs directly affect StableCrowd's central geometric reconstruction step.

Metric Height Lifting uses the joint's image ray, the ground normal, and the camera height above the ground. Errors in the camera or ground estimates can therefore produce systematic errors in every reconstructed person's location.

The paper describes a camera/ground estimation procedure in Appendix F and shows additional qualitative reconstructions on real videos. However, the main WorldPose results are obtained using supplied camera parameters and ground geometry, rather than estimating these quantities from the monocular input.

The distinction is substantial because WorldPose contains a moving broadcasting camera. Its original benchmark demonstrates that camera-estimation errors can dominate global human reconstruction accuracy. Supplying ground-truth camera extrinsics removes a major source of difficulty.

I appreciate that giving all methods identical inputs is useful for controlled comparison. My concern is that the paper's headline reconstruction claims are broader than what this evaluation establishes.

The authors should evaluate the effect of replacing ground-truth calibration with estimated calibration, or at least perturb camera pitch, focal length, camera height, and ground orientation to quantify sensitivity. The paper should also distinguish reconstruction from calibrated monocular video with provided tracks from a complete system that starts with RGB video alone.

4. The comparison with Crowd4D, DyCrowd, and the world-space outputs of per-person methods needs stronger treatment of coordinate-frame ambiguity.

StableCrowd's Table 2 reports absolute MPJPE values of 14.46 m for Crowd4D and 22.08 m for DyCrowd. However, their sequence-rigid-aligned errors are 135.2 mm and 308.5 mm, respectively. In the original publications, both methods report local pose errors measured in tens of millimeters rather than tens of meters.

The authors explain that Crowd4D and DyCrowd estimate their own scenes, and they appropriately caution against interpreting the comparison as a direct victory over optimization-based approaches. Nevertheless, the enormous gap between absolute and sequence-aligned errors raises an important evaluation question: how were the reconstructed coordinate systems registered to the reference world frame before computing absolute MPJPE?

If much of the error comes from an unresolved global rotation, translation, or scene-coordinate gauge difference, absolute MPJPE may not isolate the quality of the recovered human trajectories.

A related issue arises with GVHMR and GEM. Appendix G.3 explains that these methods output both camera-frame and world-frame reconstructions. StableCrowd evaluates their world-space trajectories after a first-frame alignment, because these preserve the continuous trajectory predicted by each model.

This is a defensible choice for evaluating accumulated drift. However, the standard protocol already provides ground-truth camera extrinsics. Under those conditions, their camera-frame human predictions could also be transformed into the shared world coordinate system frame by frame. This alternative would test spatial localization using the same camera information provided to StableCrowd, rather than relying exclusively on each baseline's relative-motion rollout.

The authors should explain the registration of Crowd4D and DyCrowd world frames, and report the camera-frame-plus-given-extrinsics option for GVHMR and GEM where supported. The existing world-trajectory comparison is useful, but it should not be the only evaluation of these baselines under a protocol that supplies accurate camera trajectories.

5. The GTF ablations do not separate the proposed phase-shifted fusion from ordinary temporal smoothing and other architectural choices.

Global Trajectory Fusion combines per-joint 3D estimates using temporal convolutions, multiple refinement rounds, phase-shifted windows, acceleration regularization, and a shift-consistency loss.

Table 5 includes useful comparisons with softmax fusion and whole-sequence GTF. However, the softmax baseline performs fusion independently for each frame, whereas GTF uses temporal context and losses that explicitly encourage smoothness.

The large improvement in acceleration error therefore does not establish that the learned fusion or phase-shifted coverage is specifically responsible. A much simpler temporal filter applied to robustly fused per-joint positions might remove a substantial portion of the jitter.

The whole-sequence GTF baseline is more controlled, and it provides evidence that bounded windows generalize better to long videos. Nevertheless, the paper does not separately ablate the half-window shift, number of inference rounds, acceleration loss, or shift-consistency loss.

The quantitative results also show an accuracy–smoothness trade-off. On VirtualCrowd, MJH with softmax fusion achieves 1.49 m absolute MPJPE, slightly better than the 1.51 m of phase-shifted GTF, although its acceleration error is much larger.

Furthermore, the WorldPose acceleration-norm ratio of StableCrowd is 3.142, whereas the ideal ratio is 1. This is better than the reported baselines but still indicates substantially different acceleration statistics from the reference trajectories.

I would expect a temporally matched non-learned fusion baseline, followed by controlled ablations of the phase shift, refinement rounds, and temporal losses. These experiments would show whether GTF's gains come from its specific architecture rather than temporal smoothing alone.

6. The numerical conditioning of Metric Height Lifting is analyzed, but its practical reliability is not demonstrated for difficult viewing geometries and imperfect keypoints.

Equation 3 recovers a joint's 3D position from its height and viewing ray. Appendix B derives that the amplification of height error is inversely proportional to the sine of the angle between the viewing ray and ground.

This means that the problem becomes poorly conditioned when the ray is nearly parallel to the ground.

The implementation uses a minimum angle of 2 degrees. At that angle, a 5 cm error in joint height can produce approximately 1.43 m error along the viewing ray. This is substantial compared with the localization errors reported in the main tables.

The authors argue that using multiple joints provides redundancy because lower joints are usually better conditioned while upper joints may remain visible under occlusion. This is reasonable, but no experiment quantifies how often individual joints are invalid, how errors vary with the ray-ground angle, or which joints contribute most to the final trajectory.

The choice of ten joints is also not ablated. The authors exclude the hips because the detector and SMPL skeleton define them differently, but do not compare lower-body-only, upper-body-only, or alternative joint subsets.

In addition, the lifting formula uses the detected 2D keypoints, making the reconstructed positions sensitive to keypoint localization errors. The main evaluation uses ground-truth tracks, but detected keypoints and camera geometry remain important sources of uncertainty.

A conditioning analysis stratified by subject distance, ray angle, joint visibility, and keypoint noise would directly test the claimed benefit of MJH redundancy. It would also help identify the conditions under which the fixed 2-degree threshold is sufficient.

Minor Weaknesses

  1. Clarification Needed on Baseline Checkpoint Selection Protocol. Section 4.2 states that the reported baseline checkpoint is the version—either released or JTA-fine-tuned—that achieves the lower MPJPE on each benchmark and protocol. Appendix H mentions that fine-tuning epochs are selected based on JTA validation without utilizing test metrics. These statements appear to describe two separate selection steps. If the final decision regarding the released versus fine-tuned models incorporates VirtualCrowd or WorldPose test results, this should be explicitly stated. While selecting the best baseline checkpoint benefits the competing methods, the procedure must remain unambiguous.

  2. Scope of the Reported Zero-Shot Claim Should Be Defined More Clearly. The learned reconstruction models are not trained on the VirtualCrowd and WorldPose sequences that are evaluated. However, the camera and ground estimation parameters are chosen using two held-out views from VirtualCrowd. This does not invalidate the zero-shot reconstruction claim, but it does differentiate between model generalization and an entirely untuned evaluation pipeline.

  3. Runtime Figure Excludes Important Upstream Stages. StableCrowd reports a processing speed of 103 person-frames per second, excluding the detection and 2D keypoint estimation stages. The authors clarify this in the supplementary materials, which is appreciated. A comprehensive runtime breakdown would facilitate a clearer interpretation of the computational comparisons for dense crowds, where tracking and keypoint estimation can be substantial.

  4. The temporal metrics use different acceleration stencils. Appendix E computes ACC error with adjacent-frame second differences but computes the acceleration-norm ratio using a wider two-frame difference. Both are applied consistently to all methods, so I do not consider this a baseline fairness error. However, the choice should be justified because the two stencils respond differently to high-frequency oscillations, which are relevant to the paper's temporal stability claim.

Summary

StableCrowd addresses a significant issue and offers a promising solution for large-scale crowd reconstruction. The Multi-Joint Height representation effectively helps recover accurate human locations, while the phase-shifted trajectory network demonstrates strong performance in long-sequence evaluations under the specified conditions.

However, the paper has several issues that impact its novelty and the validity of its experimental conclusions. The geometric camera normalization is not adequately compared to Parallel Camera Layout (PCL) and Real-World Camera Rig (RCR), and the assertion about the inherent temporal instability of perfect Human Video Inpainting (HVIP) regression lacks mathematical justification.

Additionally, the quantitative results heavily rely on known camera and ground geometry, and the lifting formulation is sensitive to errors in these parameters. The evaluation of competing methods requires a clearer explanation regarding coordinate-frame ambiguities, and the Group Temporal Filtering (GTF) ablations do not sufficiently isolate the proposed temporal mechanisms from simpler smoothing alternatives.

While I find the reported results promising, the existing evidence does not fully support the extent of the novelty and generalization claims. The authors should address the following questions to clarify which contributions are genuinely new and how reliably StableCrowd performs outside of the ground-truth-assisted evaluation setting.

Questions for authors
  1. Camera-normalization novelty. PCL already provides a virtual-camera crop based on rotation and focal-length adjustment, while RCR/Crowd3D++ uses canonical upright spaces and ground-aware normalization for large-scene crowd reconstruction. The authors are requested to: (i) acknowledge and discuss these two methods, (ii) distinguish CCN's mathematical formulation from their transformations, and (iii) compare CCN with at least one previous geometric crop normalization using the same training data and architecture. This would establish whether the gain comes from StableCrowd's specific normalization or from a known perspective-correction mechanism. (Refer to major weakness 1.)

  2. The perfect-HVIP argument. The introduction states that even perfect HVIP regression cannot yield temporally consistent world positions. The authors are requested to: (i) provide a formal derivation or counterexample supporting this claim, (ii) reconstruct 3D positions from ground-truth HVIP after reversing the crop transformation, and (iii) compare its temporal consistency with predicted HVIP and predicted MJH. If perfect HVIP is consistent, the motivation should be revised to focus on regression difficulty rather than an inherent representational impossibility. (Refer to major weakness 2.)

  3. Evaluation using estimated scene calibration. The authors are requested to: (i) report absolute localization and spatial consistency when camera and ground parameters are estimated rather than supplied, especially on WorldPose, (ii) perturb focal length, camera pitch, camera height, and ground orientation to measure error propagation, and (iii) distinguish calibrated-video reconstruction from a complete RGB-only pipeline. Please also clarify which calibration parameters are tuned using the held-out VirtualCrowd scene and whether those values are reused unchanged across datasets. (Refer to major weakness 3 and minor weakness 2.)

  4. Baseline coordinate systems and alternative outputs. The authors are requested to: (i) explain the coordinate transformation used before computing absolute MPJPE for Crowd4D and DyCrowd, (ii) clarify why their absolute errors are tens of meters while their sequence-rigid-aligned errors are in the hundreds of millimeters, and (iii) evaluate the camera-frame outputs of GVHMR and GEM transformed using the provided ground-truth camera extrinsics, alongside their world-trajectory outputs. These comparisons would separate global-frame registration errors from human reconstruction and trajectory errors. (Refer to major weakness 4.)

  5. Controlled GTF ablations. The authors are requested to: (i) compare GTF against robust per-frame joint fusion followed by a simple temporal filter, using identical MJH predictions, (ii) ablate one, two, and three inference rounds, (iii) compare shifted and non-shifted windows with the same window length, and (iv) separately remove the acceleration and shift-consistency losses. Please report both absolute position accuracy and temporal error, including the acceleration-norm ratio. These experiments would isolate the benefit of the phase-shifted architecture from generic temporal smoothing. (Refer to major weakness 5.)

  6. Metric Height Lifting conditioning. Equation 3 becomes poorly conditioned when the viewing ray is close to parallel with the ground. The authors are requested to: (i) report 3D position error as a function of ray-ground angle and subject distance, (ii) quantify the proportion of invalid lifted joints, (iii) compare lower-body-only, upper-body-only, and complete ten-joint configurations, and (iv) perturb 2D keypoints and predicted joint heights to measure sensitivity. This would show whether the proposed multi-joint redundancy actually handles the difficult viewing conditions highlighted by the paper. (Refer to major weakness 6.)

  7. Reproducibility and evaluation protocol. Please clarify whether released-versus-fine-tuned checkpoint selection uses the test benchmark, as suggested by Section 4.2, or a separate validation set, as described for fine-tuning epoch selection in Appendix H. Please also provide an end-to-end runtime breakdown including detection, tracking, keypoint estimation, CCN, MJH, and GTF, and explain why different temporal stencils are used for ACC error and acceleration-norm ratio. (Refer to minor weaknesses 1, 3, and 4.)

References

  1. Yu et al., PCLs: Geometry-Aware Neural Reconstruction of 3D Pose With Perspective Crop Layers, CVPR 2021. https://arxiv.org/abs/2011.13607
  2. Huang et al., RCR: Robust Crowd Reconstruction with Upright Space from a Single Large-scene Image, arXiv:2411.06232.
  3. Wen et al., Crowd3D: Towards Hundreds of People Reconstruction From a Single Image, CVPR 2023.
  4. Kang et al., Crowd4D: Scene-Aware Monocular 4D Crowd Reconstruction, ICML 2026.
  5. Wen et al., DyCrowd: Towards Dynamic Crowd Reconstruction from a Large-scene Video, TPAMI 2025.
  6. Shen et al., World-Grounded Human Motion Recovery via Gravity-View Coordinates, SIGGRAPH Asia 2024.
  7. Wang et al., TRAM: Global Trajectory and Motion of 3D Humans from In-the-Wild Videos, ECCV 2024.
  8. Li et al., GENMO: A Generalist Model for Human Motion, ICCV 2025.
  9. Jiang et al., WorldPose: A World Cup Dataset for Global 3D Human Pose Estimation, ECCV 2024.
  10. Li et al., Coordinate Transformer: Achieving Single-stage Multi-person Mesh Recovery from Videos, ICCV 2023.
10 comments
AuthorsOpenReview ↗

Response to Reviewer gTFM (1/8)

(1/8)

We thank the reviewer for the detailed review, and for finding Multi-Joint Height "a useful representation for metric human localization", that GTF "addresses a real limitation of absolute trajectory regression over long videos", and that "the reported results are strong under the stated protocols". We would be grateful to hear whether the revision addresses the concerns raised, and will answer any further question promptly within the discussion period.

Camera normalization, PCL and RCR (Major 1; Question 1).

Differences [(ii) of Question 1]. Table Rb-1 sets CCN beside PCL and RCR.

Table Rb-1. CCN, PCL and RCR. PCL: its Secs. 3.2 and 3.3; RCR: its Sec. 3.4; CCN: Secs. 3.1 and 3.2.

PCLRCRCCN
Where it is applieda differentiable layer inside the network, for 3D posebefore a pre-trained single-person regressoron the input crops of a network trained on mostly close-range data, for large-scene video
Source of the warp rotationthe position in the imagethe position in the image and the ground normalthe position in the image and the ground normal
Description of the warped imagea perspective camera, its virtual cameraa fixed orthographic projectionan equivalent perspective camera, without approximation, the canonical camera, through which the lifting casts the ray of each joint
Network inputthe warped imagethe warped imagethe warped image, the focal length and the residual pitch
Regression spacethe warped camera, the pose containing the pitchthe warped camera, the pose containing the pitchthe pitch set to zero and added back analytically

CCN as a design choice. On reflection, we regard the StableCrowd pipeline and its core component, MJH, as the substantive technical contributions; CCN is a design choice that serves them. The warp underneath, that of a camera rotating about its optical center and zooming (Hartley & Zisserman), is the one PCL, RCR and CCN all use.

Comparison [(iii) of Question 1]. Table Rb-2 (Appendix I, Table 21) compares CCN with RCR's normalization under the same training data and network, with the warp kept and the network and GTF retrained. With RCR's normalization, the network has the same image input and regression space as that of RCR: it sees the warped image alone and regresses the pose directly in the warped camera. CCN has the lower T-MPJPE on both datasets.

Discussion [(i) of Question 1]. The revision cites PCL in the ablation of CCN (Sec. 4.4), and Appendix M.3 discusses CCN together with PCL, PersPose and RCR.

Table Rb-2. RCR's normalization and CCN under our training data and network (same as Table Ra-2). Cells read VirtualCrowd / WorldPose.

Crop handlingT-MPJPE (mm) ↓
No extra input (Table 4)74.7 / 71.3
RCR's normalization74.5 / 63.9
CCN (Table 4)61.8 / 59.9
AuthorsOpenReview ↗

Response to Reviewer gTFM (2/8)

(2/8)

The perfect-HVIP argument (Major 2; Question 2).

Derivation [(i) of Question 2]. We thank the reviewer for the careful analysis of this sentence, which left out a condition: it holds for a regressor with errors, not for a perfect one. To make the point clearer, the revision replaces it, "Even a perfect regressor therefore cannot recover temporally consistent 3D positions from it", with the dilemma that motivated MJH (Sec. 1):

  • Crop that follows the person. The person stays at a size suited to regression, but the HVIP target, a pixel distance in the crop, changes with the crop size, and its regression errors reach the 3D position as jitter.
  • Crop of fixed size along the track. The target stays stable, but a person walking away shrinks within the crop and becomes hard to regress.

MJH avoids this dilemma: a metric height does not depend on the crop, which can therefore follow the person. The caption of Fig. 2 now states the same condition: the HVIP jitters with the size of a crop that follows the person.

Ground-truth HVIP [(ii) of Question 2]. We agree that ground-truth HVIP, with the ground-truth 2D detection, can be mapped back through the known crop transform to the ground-truth position exactly; the problem therefore lies in the regression dilemma above.

Temporal consistency [(iii) of Question 2]. As ground-truth HVIP is exact, the comparison leaves it out and sets predicted HVIP against predicted MJH: under the same GTF, predicted HVIP is both less accurate and less smooth on both datasets (Table Rb-3); on WorldPose the root height with GTF is slightly smoother than MJH, at a 30% higher MPJPE.

Table Rb-3. HVIP, root height and MJH with matched temporal decoders (same as Table Ra-1). Cells read VirtualCrowd / WorldPose; GTF† is GTF trained with a single point as its input.

MPJPE (m) ↓SEQ-RT-MPJPE (mm) ↓ACC error (mm/f²) ↓
HVIP, direct (Table 5)1.80 / 0.57450 / 277201.36 / 75.63
HVIP + GTF†1.76 / 0.61280 / 2595.97 / 2.68
Root height, direct (Table 5)1.67 / 0.41209 / 22961.94 / 36.65
Root height + GTF†1.69 / 0.43182 / 2313.69 / 1.96
MJH + GTF (Table 5)1.51 / 0.33156 / 1992.65 / 2.26
AuthorsOpenReview ↗

Response to Reviewer gTFM (3/8)

(3/8)

Evaluation with an estimated scene (Major 3; Question 3).

The main results do not all use a ground-truth camera: on VirtualCrowd, the camera and scene surface are estimated from the video (Table 2), and on WorldPose, filmed by a moving broadcast camera, (i) below estimates the camera as well.

  • Automatic calibration in our applications. Large-scene videos from deployed cameras are the main target of StableCrowd; even with unknown intrinsics and lens distortion, these cameras can be calibrated automatically against a 3D reconstruction of the site from an automatically flown drone survey, as shown in Appendix L, Fig. 13, with median reprojection residuals from 0.60 to 2.37 pixels. Beyond the benchmarks, the supplementary video and Appendix L show StableCrowd on deployed cameras calibrated this way, with the people seen by different cameras of a site in one coordinate system, and, with the camera intrinsics and scene surface estimated from the video, on planar ground and on steps (Fig. 12; with the ground alone estimated, Fig. 15). The revision states this dependence in the limitations, together with the automatic calibration of deployed cameras (Appendix M.4).
  • Estimated against calibrated camera and scene surface [(i) of Question 3]. On WorldPose, following the protocol of the FIFA Skeletal Light challenge, which gives the per-frame intrinsics and lens distortion and the camera of the first frame only, the camera rotation of every later frame, and with it the ground in each camera's frame, is estimated from the pitch lines. The revised appendix shows the distribution of this estimation error over all frames, a real perturbation of the camera (Appendix I, Fig. 9). Against the given camera, the estimated one changes the MPJPE of StableCrowd little and leaves its spatial consistency (PPDS, PA-PPDS) essentially unchanged, still far more accurate than TRAM (Table Rb-4). The same comparison in the wild, with the calibrated camera and scene surface as the reference: on three in-the-wild clips, two of whose scenes are shown in Figs. 12 and 15, estimating the scene surface, or the camera intrinsics as well, moves each person by 4.5 to 6.2% of the distance to the camera on average, while the layout of the people, their positions relative to one another, is largely kept (Table Rb-5; Appendix L, Table 24); for deployed cameras, our main application, the automatic calibration above gives the calibrated input instead.
  • Perturbations [(ii) of Question 3]. Beyond this real perturbation, the revision perturbs the focal length, camera pitch, camera height and ground orientation one at a time (Table Rb-6). Apart from the one sequence in which the court lines are lost, the real perturbation above lies far below these angles, its pitch component the smallest of the three (Appendix I, Fig. 9).
  • Calibrated video [(iii) of Question 3]. The revised introduction states the input of StableCrowd: a video, and the camera and scene surface, calibrated or estimated (Sec. 1); as for Crowd3D, Crowd4D and DyCrowd, this input is not among our contributions, and every method we can re-run is given the same one (Sec. 4.2).
Reviewer_kbcGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents StableCrowd, a feed-forward pipeline for world-space HMR from monocular videos of crowded scenes. StableCrowd uses Canonical Camera Normalization (CCN) to warp per-person crops into canonical camera views and predicts Multi-Joint Height (MJH), a crop-invariant representation that can be lifted to per-joint 3D positions using scene geometry. Global Trajectory Fusion (GTF), a temporal convolutional decoder, then fuses these estimates into a coherent root trajectory for each person. Experiments on VirtualCrowd and WorldPose show strong overall performance in global localization and trajectory reconstruction compared with the evaluated baselines.

Critical strengths

I find MJH a natural and intuitive crop-invariant prediction target, although its geometric lifting is sensitive to estimation errors and its effectiveness is not sufficiently established (see "weaknesses"). Also, the feed-forward pipeline avoids expensive per-sequence optimization and achieved 100x performance against Crowd4D in supplementary video, making reconstruction of long crowd videos more practical.

Critical weaknesses
  1. The ablation does not sufficiently isolate the contribution of MJH from that of GTF. Table 5 states that a single-point representation produces one estimate per frame and therefore has “nothing to fuse.” However, a temporal convolutional decoder could still aggregate HVIP or root-height estimates across frames. Comparisons using comparable temporal decoders would better establish the benefit of MJH itself. On VirtualCrowd, replacing softmax fusion with GTF substantially reduces acceleration error while slightly increasing MPJPE (1.49 → 1.51 m), further motivating this controlled comparison.

  2. Robustness to errors in scene geometry and to non-planar terrain is insufficiently established. The pipeline depends on reliable ground orientation and geometry, but the continuity assumption underlying the existence argument in Appendix B does not generally hold for stairs, where the ground-height function can be discontinuous at step boundaries. Quantitative evaluation on stair-climbing sequences, such as those available in EMDB and 3DPW, would better support the claim of applicability to arbitrary terrain.

  3. The novelty of CCN is insufficiently established relative to Crowd3D++’s Ground-aware Normalization Transform. Both methods use camera and ground geometry to normalize per-person observations before reconstruction. The paper should clarify their geometric differences and demonstrate the benefit of CCN over this closely related approach, beyond comparisons with ordinary cropping and box encoding.

Questions for authors
  1. Could you compare HVIP, root-height, and MJH with matched temporal decoders to isolate MJH’s contribution in table 5?

  2. Could you ablate CCN against Crowd3D++'s normalization in table 4?

  3. How sensitive is the method to ground and camera estimation errors and how it works on arbitrary terrains (not the easy ones in Appendix L)? Ideally some stair / non-flat scene from standard benchmarks like EMDB or 3DPW should justify here.

8 comments
AuthorsOpenReview ↗

Response to Reviewer kbcG (1/3)

(1/3)

We thank the reviewer for finding Multi-Joint Height "a natural and intuitive crop-invariant prediction target", the feed-forward pipeline "making reconstruction of long crowd videos more practical", and the experiments to "show strong overall performance". We would be grateful to hear whether the revision addresses the concerns raised, and will answer any further question promptly within the discussion period.

Isolating MJH from GTF (Weakness 1; Question 1). Our representation went from HVIP to a single anchor height, that of the body root, and then to Multi-Joint Height; the step to multiple joints is the one that introduced the fusion, so Table 5 compares each representation in the form it was designed with. This is what "nothing to fuse" meant: a single point has no joints to fuse within a frame. The revision makes this explicit and adds the comparison across frames (Table Ra-1): under the same GTF, MJH is the most accurate on both datasets, and on WorldPose the root height with GTF is slightly smoother, at a 30% higher MPJPE. Even with per-frame softmax fusion and no GTF (Table 5), MJH has a lower MPJPE than either single point with GTF (Table Ra-1).

Table Ra-1. HVIP, root height and MJH with matched temporal decoders. Cells read VirtualCrowd / WorldPose; GTF† is GTF trained with a single point as its input.

MPJPE (m) ↓SEQ-RT-MPJPE (mm) ↓ACC error (mm/f²) ↓
HVIP, direct (Table 5)1.80 / 0.57450 / 277201.36 / 75.63
HVIP + GTF†1.76 / 0.61280 / 2595.97 / 2.68
Root height, direct (Table 5)1.67 / 0.41209 / 22961.94 / 36.65
Root height + GTF†1.69 / 0.43182 / 2313.69 / 1.96
MJH + GTF (Table 5)1.51 / 0.33156 / 1992.65 / 2.26

CCN and Crowd3D++ (Weakness 3; Question 2).

Differences. CCN differs from Crowd3D++ in two respects.

  • Projection. Crowd3D++ treats the warped image as a fixed orthographic projection (its Sec. 3.4); CCN describes it, without approximation, as taken by an equivalent perspective camera, the canonical camera (Sec. 3.1), through which the lifting casts the ray of each joint (Sec. 3.2).
  • Regression space. Crowd3D++ regresses the pose directly in the orthographic space of the warped image, which still looks down on the person, so the regressed pose contains the pitch; CCN gives the network the focal length and the residual pitch, has it regress the pose with the pitch set to zero, and adds the pitch back analytically.

CCN as a design choice. On reflection, we regard the StableCrowd pipeline and its core component, MJH, as the substantive technical contributions; CCN, which applies the perspective warp of a camera rotating about its optical center and zooming (Hartley & Zisserman) so that a network trained on mostly close-range data can take crops from large-scene video, is a design choice that serves them.

Comparison. Table Ra-2 (Appendix I, Table 21) compares CCN with Crowd3D++'s normalization under our training data and network, with the warp kept and the network and GTF retrained. With Crowd3D++'s normalization, the network has the same image input and regression space as that of Crowd3D++: it sees the warped image alone and regresses the pose directly in the warped camera. CCN has the lower T-MPJPE on both datasets. The revised appendix discusses CCN together with PCL, PersPose and Crowd3D++.

Table Ra-2. Crowd3D++'s normalization and CCN under our training data and network. Cells read VirtualCrowd / WorldPose.

Crop handlingT-MPJPE (mm) ↓
No extra input (Table 4)74.7 / 71.3
Crowd3D++'s normalization74.5 / 63.9
CCN (Table 4)61.8 / 59.9
AuthorsOpenReview ↗

Response to Reviewer kbcG (2/3)

(2/3)

Robustness to scene geometry and to non-planar terrain (Weakness 2; Question 3).

  • Automatic calibration is feasible in our applications. Large-scene videos from deployed cameras are the main target of StableCrowd; even with unknown intrinsics and lens distortion, these cameras can be calibrated automatically against a 3D reconstruction of the site from an automatically flown drone survey (Appendix L, Fig. 13), with median reprojection residuals from 0.60 to 2.37 pixels. Beyond the benchmarks, the supplementary video and Appendix L show StableCrowd on deployed cameras calibrated this way, with the people seen by different cameras of a site in one coordinate system, and, with the camera intrinsics and scene surface estimated from the video, on planar ground and on steps (Fig. 12; with the ground alone estimated, Fig. 15).
  • On both benchmarks, we evaluate StableCrowd with an estimated camera and scene surface. The revision measures the effect of errors in the camera and scene surface, one factor at a time (Table Ra-3), and of estimating them: Table 2 on VirtualCrowd, and Table Ra-4 on WorldPose, where, following the protocol of the FIFA Skeletal Light challenge, which gives the per-frame intrinsics and lens distortion and the camera of the first frame only, the camera rotation of every later frame, and with it the ground in each camera's frame, is estimated from the pitch lines. Against the given camera, the estimated one changes the MPJPE of StableCrowd little and leaves its spatial consistency (PPDS, PA-PPDS) essentially unchanged, still far more accurate than TRAM.
  • In the wild, the absolute positions may shift, but the layout is largely kept. In addition to the benchmarks above, we show StableCrowd on in-the-wild clips with an estimated scene: the absolute positions may shift against the calibrated result, but the layout is largely kept on these clips, qualitatively and quantitatively (Appendix L, Fig. 15 and Table 24).
  • The above is the trade-off of taking the scene as the 3D cue. Lifting every person onto the same scene keeps the people consistent with one another, but they also shift with the scene when it is off. The revised limitations (Appendix M.4) now discuss this trade-off and the automatic calibration of deployed cameras.
  • The description of the lifting now covers discontinuities, such as stairs. We thank the reviewer for pointing this out. The revised description (Appendix B) holds on continuous and discontinuous ground alike: the lifting takes the first point on the ray, sampled on the depth map and counted from the camera, at which the height above the ground reaches the regressed height or less, which every ray meeting the ground has, as this height runs from the camera's own to zero; on continuous ground this point attains it exactly, and where the height jumps past it at a stair edge, the point is where the ray passes over the edge.
  • Applicability to steps and non-planar terrain; EMDB and 3DPW. Besides the calibrated site with steps (supplementary video, Appendix L), the revision adds one whose camera intrinsics and scene surface are estimated from the video (Appendix L, Fig. 12). On this scene, against the calibrated result, estimating the scene surface, or the camera intrinsics as well, shifts each person's absolute position by 4.7 to 6.2% of their distance from the camera on average, while the layout is largely kept (PA-PPDS 0.97 to 0.98; Appendix L, Table 24). For terrain harder than the scenes of Appendix L, StableCrowd works with the scene surface it is given: in our applications, which use deployed cameras, the accuracy of this surface is ensured by the automatic calibration above, and for internet videos, it depends on the monocular geometry model that estimates it. StableCrowd is designed for large scenes seen from afar. EMDB and 3DPW record one or a few people close to hand-held cameras and come without their scene surface, so a quantitative comparison there would test this close-range setting rather than large scenes, and would add to the error of StableCrowd that of a per-frame estimate of the scene surface from the moving camera. We thank the reviewer for pointing this out: where Appendix B stated that the lifting remains solvable over "arbitrary terrain", the revision states that it remains solvable over non-planar terrain, whose ground height varies continuously or jumps at stair edges.
  • Clarification of the input. The revised introduction states the input of StableCrowd: a video, and the camera and scene surface, calibrated or estimated (Sec. 1). As for Crowd3D, Crowd4D and DyCrowd, this input is not among our contributions, and every method we can re-run is given the same one (Sec. 4.2).
AuthorsOpenReview ↗

Response to Reviewer kbcG (3/3)

(3/3)

Table Ra-3. Error propagation of the focal length, camera pitch, camera height and ground orientation (new). Ground error as a share (%) of the distance, one factor at a time, in the worse direction. VirtualCrowd: means over the person-frames of the six evaluated videos, whose rays meet the ground at a median of 9°, that the lifting still takes (under the 1, 2 and 5° tilts toward or away, 9.8, 15.5 and 29.4% fall below the 2° threshold and are left out); other columns: a single ray meeting the ground at that angle. Full table: Appendix F, Table 10.

ErrorVirtualCrowd45°30°10°
Camera height, 1 / 5 / 10%1 / 5 / 101 / 5 / 101 / 5 / 101 / 5 / 10
Focal length, 1 / 5 / 10%0.56 / 2.8 / 5.60.71 / 3.5 / 7.11.0 / 5.1 / 103.0 / 16 / 38
Camera pitch or ground tilt toward or away, 1 / 2 / 5°13 / 27 / 811.8 / 3.7 / 103.1 / 6.5 / 1811 / 25 / 99
Ground tilt sideways, 1 / 2 / 5°2.0 / 3.9 / 9.5–––

Table Ra-4. Absolute localization on WorldPose with an estimated ground (new). The per-frame intrinsics and lens distortion, which change with the zoom, are given in both. Given: the standard protocol (Table 3). Estimated: the protocol of the FIFA Skeletal Light challenge; ours is re-run with it, and TRAM is the row of Table 3, placed in the world by the same camera. Best in bold.

GroundMethodMPJPE (m) ↓SEQ-RT-MPJPE (mm) ↓ACC error (mm/f²) ↓PPDS ↑PA-PPDS ↑
GivenTRAM2.711928628.770.8410.845
GivenOurs0.331992.260.9840.985
EstimatedTRAM2.721936634.910.8410.845
EstimatedOurs0.362442.350.9830.984

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response

We thank the three reviewers for their recognition of Multi-Joint Height, "a natural and intuitive crop-invariant prediction target" (kbcG), "a useful representation for metric human localization" (gTFM) and "a simple but effective idea with a clear geometric motivation" (t2M5), and of the results: "strong overall performance" (kbcG), "the reported results are strong under the stated protocols" (gTFM) and "the experimental results are strong" (t2M5).

The revision addresses the reviews as follows; each reply reports in full the results its reviewer asked for.

  • Revised contributions; CCN (kbcG, gTFM, t2M5). The contributions are now the StableCrowd pipeline and its core component MJH; GTF and CCN are presented as design choices (Sec. 1). Under our training data and network, CCN has a lower T-MPJPE than Crowd3D++'s normalization on both datasets (Appendix I).
  • Perfect-HVIP argument (gTFM). We agree that ground-truth HVIP maps back to the ground-truth position exactly; Sec. 1 now states instead the regression dilemma that motivated MJH, and under the same GTF predicted HVIP is less accurate and less smooth than MJH (Table 5).
  • MJH isolated from GTF (kbcG, t2M5). We retrained GTF on HVIP and on the root height, with nothing else changed; under the same GTF, MJH remains the most accurate on both datasets (Table 5).
  • Camera and scene surface (all three reviewers). On VirtualCrowd the main results already use an estimated camera and scene surface (Table 2). We added an analysis of how errors in this input propagate (Appendix F), re-ran StableCrowd on WorldPose with the camera rotation estimated from the pitch lines, where it remains far more accurate than TRAM (Appendix I), and compared estimated against calibrated input in the wild, where the absolute positions may shift but the layout of the people is largely kept (Appendix L). The limitations now state this as the trade-off of taking the scene as the 3D cue, together with the automatic calibration of deployed cameras (Appendix M.4).
  • Stairs (kbcG). The description of the lifting in Appendix B now covers discontinuities such as stairs, and Appendix L adds a scene with steps whose camera intrinsics and terrain are estimated from the video (Fig. 12, Table 24).
  • GTF (gTFM, t2M5). We compared GTF with a temporal filter on identical predictions, where GTF is more accurate on JTA validation and WorldPose, and ablated its windows, rounds and loss terms (Appendix I).
  • Lifting under hard conditions (gTFM). We report the error by ray angle and distance, the person-frames left without a position, joint subsets and keypoint noise; even the largest keypoint noise changes the MPJPE of the whole pipeline by at most 4 mm (Appendices B and I).
  • Compared methods and runtime (gTFM, t2M5). We clarified the registration of Crowd4D and DyCrowd: after one rigid registration per sequence, ours remains the most accurate; we added the camera-frame outputs of GVHMR and GEM, and report the runtime of each stage (Appendices G, I and M).
  • Clarifications (gTFM). We narrowed the zero-shot claim (Sec. 4.1) and made the checkpoint selection clearer (Sec. 4.2).
AuthorsOpenReview ↗

Update: new results

We have added the results of two experiments completed since our first responses:

  1. StableCrowd in the wild with an estimated camera and scene surface, against calibrated ones (Appendix L, Table 24).
  2. The distribution of the estimated camera rotation error on WorldPose (Appendix I, Fig. 9).

We have also revised the paper and our earlier responses above, so that each response is self-contained, with its own tables, and the general response summarizes them. We would be glad to discuss any remaining concern.

AuthorsOpenReview ↗

Thanks to the reviewers and the Area Chair

We sincerely thank the three reviewers and the Area Chair for their time, the careful reviews and the discussion, which made the paper clearer and more complete. Following their suggestions, the revision adds matched comparisons isolating MJH, GTF and CCN, the sensitivity to calibration errors, results on in-the-wild videos, and the runtime analysis. We have also made sure that the limitations discuss the dependence on the camera and scene surface, as the trade-off of taking the scene as the 3D cue, and that deployed cameras, our main target, can be calibrated automatically.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.