- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents MuPeT, a VGGT-based framework for multi-person 3D reconstruction from uncalibrated sports broadcast footage. To address severe occlusions, repetitive visual patterns, and the lack of real-world 3D supervision, the authors propose training exclusively on synthetic scenes using player segmentation masks, dense landmarks, and pitch lines instead of RGB images. The framework jointly predicts camera geometry, 3D player landmarks, and cross-view associations, followed by an optional EM-based refinement. Experiments on RugbyBench demonstrate substantial improvements over existing general-purpose 3D reconstruction and human pose estimation methods.
Critical strengths
- The paper addresses a challenging and practically relevant setting involving uncalibrated cameras, large viewpoint variations, and dense multi-person interactions. Motivation is clear, and the model design is closely aligned with the problem being addressed.
- Using structured geometric representations to bridge synthetic training and real-world reconstruction is a reasonable and interesting design choice.
- The experiments demonstrate substantial improvements over existing baselines, with ablations supporting the proposed landmark-based representation and geometric refinement.
Critical weaknesses
- There is a potential overfitting to RugbyBench-specific spatial priors. The synthetic training data is generated using camera configurations and player spatial distributions estimated directly from RugbyBench, which is also used for evaluation. Although the ground-truth data is not used in training, it remains unclear whether the model learns transferable geometric reasoning or primarily exploits benchmark-specific priors. Evaluation on unseen rugby matches, ideally with a match-level split that excludes test matches from synthetic data distribution estimation, would help establish its generalization within the same sport.
- The scale and representativeness of real-world evaluation seem limited. The evaluation relies on only 35 short sequences from seven rugby matches, raising concerns about whether the reported results adequately represent the diversity of player interactions, camera conditions, and scene configurations encountered in real broadcasts. A more comprehensive analysis of performance variability across matches and interaction scenarios would strengthen the experimental conclusions.
- A central claim of MuPeT is that structured inputs, including segmentation masks, dense landmarks, and pitch lines, effectively bridge the synthetic-to-real gap. However, the current ablations mainly focus on architectural choices rather than validating this key hypothesis. A more focused analysis of the contributions of different input representations would better substantiate the proposed design and its advantages over conventional VGGT-based approaches.
- The reported W-MPJPE jointly reflects global player localization and body pose errors, making it difficult to assess the accuracy of individual 3D poses, particularly during dense physical interactions. The final SMPL-X reconstructions also lack direct 3D quantitative evaluation.
Questions for authors
- How would MuPeT perform on unseen rugby matches if their camera configurations and player spatial statistics were excluded from synthetic data generation?
- Could the authors provide an input-representation ablation (e.g., removing pitch lines or dense landmarks) under the same synthetic training setup, and report performance on both synthetic and real RugbyBench data to verify whether these representations effectively reduce the sim-to-real gap?
- RugbyBench retains only 321 of 525 frames with valid BA-derived 3D annotations. Are the excluded frames also included in MuPeT's evaluation, particularly when computing reconstruction coverage and reprojection errors? If not, how might their exclusion affect the reported results?
- Could the authors additionally report root-aligned MPJPE to distinguish individual 3D pose accuracy from global player localization errors?
6 comments
-
Missing frames. At the time of submission, RugbyBench already provided 3D annotations of the same quality for all 525 frames, and all MuPeT results in the paper are already reported on this full set, so no frames are excluded from coverage, reprojection or W-MPJPE. The mention of fewer frames was from a slightly-out-of-date version of the RugbyBench paper, which contained this single difference.
-
Root-aligned MPJPE. To isolate per-player pose from global placement, we report root-aligned MPJPE. We evaluate on all 525 frames in the three close-up views (Tight, High-Behind Right, Reverse), where players are large enough for pose to be meaningful, on the 8,689 players reconstructed by both MuPeT and Multi-HMR 2. For MuPeT, we use the joints of SMPL-X fitted to its predicted 3D landmarks, rather than the landmark-to-joint mapping used for W-MPJPE in the paper. Raw landmarks have no body-shape or limb-length constraints, which matters little at the scale of W-MPJPE but dominates an error measured per joint after root alignment; the fit enforces a plausible body. On individual pose accuracy, the monocular regressor is ahead, as expected given the design of each method. The comparison favours Multi-HMR 2: it predicts each player’s pose from the high-resolution close-up alone, with a strong body prior, whereas MuPeT’s landmarks are estimated jointly from all six views, most of which film players from 50–100 m. Small errors in detection and camera estimation in these distant views become decimetre-level landmark errors, which the fitted body inherits. However, Multi-HMR 2 sees one view at a time: it cannot associate players across cameras or place them in a shared metric scene. The two metrics are therefore complementary: W-MPJPE shows that MuPeT recovers and places players consistently across views, while root-aligned MPJPE shows that the remaining gap lies in fine per-player pose, which more precise landmarks or a pose prior during reconstruction could close.
| Method | MPJPE (mm) ↓ |
|---|---|
| Multi-HMR 2 | 187 |
| MuPeT + SMPL-X fit | 221 |
For Q2, seems like the dense landmarks hinder the model's performance on real scenes? Could you identify in which situations the quality of MAMMA landmarks are reduced? Perhaps adding a short discussion would be better. The current explaination remain not convincing without actual evidences.
We thank the reviewer for the comments and address each point below.
Synthetic validation (525 scenes)
Real RugbyBench (525 frames)