acceptodds
Under review as a conference paper at ICLR 2027

MuPeT: Multi-Person Transformer for 3D Reconstruction from Broadcast Sports

Abstract

High-level sporting events are often recorded from multiple viewpoints, providing abundant visual data to train models for 3D reconstruction and human pose estimation. However, these settings remain a significant blind spot for current models: they involve highly dynamic scenes with limited texture and repetitive backgrounds, captured by uncalibrated, long-range cameras. In this paper, we present MuPeT (Multi-Person Transformer), a model to reconstruct player poses and impacts in 3D under severely challenging conditions. Since training a Visual Geometry Guided Transformer (VGGT) requires precise 3D supervision, absent in real sports data, we train it exclusively with synthetic renderings. The key enabler is the input modality: instead of pure RGB, which would present a large sim-to-real gap, our model consumes human segmentations and lines, which are easily rendered or computed from real data. We thus leverage the strengths of pretrained models while enabling purely-synthetic training, to learn the geometric biases inherent to these environments. We show that MuPeT successfully reconstructs difficult impact scenarios where previous approaches fail.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Pose Estimation, Broadcast Footage, Sports, Multi-view Videos

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AffrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MuPeT, a VGGT-based framework for multi-person 3D reconstruction from uncalibrated sports broadcast footage. To address severe occlusions, repetitive visual patterns, and the lack of real-world 3D supervision, the authors propose training exclusively on synthetic scenes using player segmentation masks, dense landmarks, and pitch lines instead of RGB images. The framework jointly predicts camera geometry, 3D player landmarks, and cross-view associations, followed by an optional EM-based refinement. Experiments on RugbyBench demonstrate substantial improvements over existing general-purpose 3D reconstruction and human pose estimation methods.

Critical strengths
  1. The paper addresses a challenging and practically relevant setting involving uncalibrated cameras, large viewpoint variations, and dense multi-person interactions. Motivation is clear, and the model design is closely aligned with the problem being addressed.
  2. Using structured geometric representations to bridge synthetic training and real-world reconstruction is a reasonable and interesting design choice.
  3. The experiments demonstrate substantial improvements over existing baselines, with ablations supporting the proposed landmark-based representation and geometric refinement.
Critical weaknesses
  1. There is a potential overfitting to RugbyBench-specific spatial priors. The synthetic training data is generated using camera configurations and player spatial distributions estimated directly from RugbyBench, which is also used for evaluation. Although the ground-truth data is not used in training, it remains unclear whether the model learns transferable geometric reasoning or primarily exploits benchmark-specific priors. Evaluation on unseen rugby matches, ideally with a match-level split that excludes test matches from synthetic data distribution estimation, would help establish its generalization within the same sport.
  2. The scale and representativeness of real-world evaluation seem limited. The evaluation relies on only 35 short sequences from seven rugby matches, raising concerns about whether the reported results adequately represent the diversity of player interactions, camera conditions, and scene configurations encountered in real broadcasts. A more comprehensive analysis of performance variability across matches and interaction scenarios would strengthen the experimental conclusions.
  3. A central claim of MuPeT is that structured inputs, including segmentation masks, dense landmarks, and pitch lines, effectively bridge the synthetic-to-real gap. However, the current ablations mainly focus on architectural choices rather than validating this key hypothesis. A more focused analysis of the contributions of different input representations would better substantiate the proposed design and its advantages over conventional VGGT-based approaches.
  4. The reported W-MPJPE jointly reflects global player localization and body pose errors, making it difficult to assess the accuracy of individual 3D poses, particularly during dense physical interactions. The final SMPL-X reconstructions also lack direct 3D quantitative evaluation.
Questions for authors
  1. How would MuPeT perform on unseen rugby matches if their camera configurations and player spatial statistics were excluded from synthetic data generation?
  2. Could the authors provide an input-representation ablation (e.g., removing pitch lines or dense landmarks) under the same synthetic training setup, and report performance on both synthetic and real RugbyBench data to verify whether these representations effectively reduce the sim-to-real gap?
  3. RugbyBench retains only 321 of 525 frames with valid BA-derived 3D annotations. Are the excluded frames also included in MuPeT's evaluation, particularly when computing reconstruction coverage and reprojection errors? If not, how might their exclusion affect the reported results?
  4. Could the authors additionally report root-aligned MPJPE to distinguish individual 3D pose accuracy from global player localization errors?
6 comments

We thank the reviewer for the comments and address each point below.

  1. Generalisation to unseen matches. We retrained MuPeT with all camera and player statistics from one match removed from the synthetic generator, and evaluated it on that match. We held out game 21, whose stadium appears in no other match. Both models use the same reduced training recipe as the input-representation ablation (Q2), evaluated at 50k iterations without EM on game 21 only. The model that never saw game 21's statistics performs on par with the model trained on all matches; the small differences between them are likely within run-to-run variation. This is consistent with the statistics themselves: rebuilding them without any single match shifts camera positions by at most 0.25 standard deviations and median focal lengths by at most 8%, and player-layout statistics by at most 0.2 interquartile ranges. MuPeT therefore does not rely on having seen a match's specific camera or player statistics.
Synthetic data fromCoverage ↑Reproj. (px) ↓Pair F1 ↑W-MPJPE (m) ↓
All 7 matches94.7%198.572.73.17
6 matches (game 21 excluded)100%188.673.43.33
  1. Input-representation ablation. As suggested, we trained three variants with identical synthetic data, architecture and reduced schedule (50k iterations), changing only the input: the full input (instance masks, pitch lines, dense landmarks), the input without pitch lines, and the input without landmarks. We evaluated each on 525 held-out synthetic scenes and on the 525 real RugbyBench frames without EM. Pitch lines give a clear benefit that transfers to real data, reducing W-MPJPE from 3.78 to 2.79 m. Pair F1 is saturated on synthetic data and identical across variants; the inputs differ mainly in their effect on geometry. The benefit of landmarks is clear on synthetic data but not on real data, where the version without landmarks is slightly ahead (within run-to-run variations). We attribute this mainly to the reduced quality of MAMMA landmarks in certain situations, such as distant players and dense contact, which the synthetic degradations only partly capture. More generally, the synthetic-to-real gap also reflects that real rugby poses differ substantially from the AMASS and simulated collision poses seen in training. Absolute numbers are lower than our final model due to the shorter schedule.

Synthetic validation (525 scenes)

InputCoverage ↑Reproj. (px) ↓Pair F1 ↑W-MPJPE (m) ↓
Full99.8%38.994.10.84
No landmarks99.8%46.594.10.93
No pitch lines99.8%80.394.11.32

Real RugbyBench (525 frames)

InputCoverage ↑Reproj. (px) ↓Pair F1 ↑W-MPJPE (m) ↓
Full99.0%241.875.12.79
No landmarks99.6%206.676.82.76
No pitch lines98.5%273.874.83.78
  1. Missing frames. At the time of submission, RugbyBench already provided 3D annotations of the same quality for all 525 frames, and all MuPeT results in the paper are already reported on this full set, so no frames are excluded from coverage, reprojection or W-MPJPE. The mention of fewer frames was from a slightly-out-of-date version of the RugbyBench paper, which contained this single difference.

  2. Root-aligned MPJPE. To isolate per-player pose from global placement, we report root-aligned MPJPE. We evaluate on all 525 frames in the three close-up views (Tight, High-Behind Right, Reverse), where players are large enough for pose to be meaningful, on the 8,689 players reconstructed by both MuPeT and Multi-HMR 2. For MuPeT, we use the joints of SMPL-X fitted to its predicted 3D landmarks, rather than the landmark-to-joint mapping used for W-MPJPE in the paper. Raw landmarks have no body-shape or limb-length constraints, which matters little at the scale of W-MPJPE but dominates an error measured per joint after root alignment; the fit enforces a plausible body. On individual pose accuracy, the monocular regressor is ahead, as expected given the design of each method. The comparison favours Multi-HMR 2: it predicts each player’s pose from the high-resolution close-up alone, with a strong body prior, whereas MuPeT’s landmarks are estimated jointly from all six views, most of which film players from 50–100 m. Small errors in detection and camera estimation in these distant views become decimetre-level landmark errors, which the fitted body inherits. However, Multi-HMR 2 sees one view at a time: it cannot associate players across cameras or place them in a shared metric scene. The two metrics are therefore complementary: W-MPJPE shows that MuPeT recovers and places players consistently across views, while root-aligned MPJPE shows that the remaining gap lies in fine per-player pose, which more precise landmarks or a pose prior during reconstruction could close.

MethodMPJPE (mm) ↓
Multi-HMR 2187
MuPeT + SMPL-X fit221

For Q2, seems like the dense landmarks hinder the model's performance on real scenes? Could you identify in which situations the quality of MAMMA landmarks are reduced? Perhaps adding a short discussion would be better. The current explaination remain not convincing without actual evidences.

Reviewer_uUnhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MuPeT, a method for multi-person 3D reconstruction from six uncalibrated sports views. It generates synthetic scenes and trains a VGGT-style model using instance masks, dense anatomical landmarks, and pitch lines. The model predicts camera parameters, 3D landmarks, and cross-view player associations. An optional EM procedure refines the associations and geometry, after which SMPL-X bodies can be fitted to the reconstructed players.

Critical strengths
  1. I like the simulation data pipeline and the choice of input modalities. Using masks, landmarks, and pitch lines simplifies synthetic data generation and provides a reasonable way to reduce the appearance gap between simulation and real footage. The simulated detection errors also make the training setup more realistic.
  2. The method itself seems reasonable, and the reported improvements are substantial. The network already achieves 80.3% player-association F1 before EM, showing that it learns useful cross-view information.
Critical weaknesses
  1. My main concern is the motivation for the VGGT-style reconstruction model. Once masks, dense landmarks, and pitch lines are available, there are already strong geometric constraints. Together with the camera-role priors, I wonder whether conventional calibration and multi-view reconstruction followed by EM could already work well. The comparisons with COLMAP + LightGlue and UncaliPose do not fully answer this, since they rely on feature matching and appearance-based player matching respectively. Although UncaliPose receives ground-truth intrinsics, it does not test the same landmark-based geometric pipeline. I would like to understand which limitations of such a pipeline motivate the learned model.
  2. As a minor point, the exact six-channel input layout and the conversion from predictions to evaluation joints could be stated more explicitly. It is unclear whether MuPeT’s W-MPJPE in Table 2 uses joints extracted from the optional SMPL-X fits or another mapping from the predicted landmarks.
Questions for authors
  1. Could you compare against a geometry-based pipeline using the same inputs and priors? For example, the fitted camera-position and focal-length distributions could initialize camera hypotheses, pitch-line alignment could refine them, and epipolar consistency between MAMMA landmarks could initialize player matching before the same EM/BA refinement without learned affinities. This would help clarify what the learned reconstruction model adds.
  2. Could you clarify why these six camera feeds were used? Were they the only feeds available or selected from a larger set? Were any additional fixed-view feeds or calibration metadata available, and what does the cable-camera view contribute?
  3. Could you show an actual six-channel input and specify what each channel stores, including how landmark labels are represented?
  4. For Table 2, are MuPeT’s joints extracted from the fitted SMPL-X bodies using the joint regressor described in RugbyBench? If not, how are the predicted landmarks mapped to the benchmark’s skeletal joints?
3 comments

We thank the reviewer for the comments and address each point below.

  1. Non-learned geometric baseline. As suggested, we implemented a geometric baseline using exactly MuPeT's inputs and priors, described in detail below. Since this approach cannot constrain the close-up cameras, it estimates only the three distant views (Wide, High-Behind Left, Cable), and all methods in the table are scored on these views.
FramesMethodReproj. (px) ↓Rot. error (°) ↓Pair F1 ↑
Test (10)Geometric baseline204.935.243.0
Geometric baseline + EM211.755.145.4
MuPeT + EM26.37.281.2
Calibration (5)Geometric baseline202.213.246.5
Geometric baseline + EM61.934.552.4
MuPeT + EM19.45.784.0

Distant views only (Wide, High-Behind Left, Cable). Means over all frames, failures included.

The baseline's output is all-or-nothing: on about half the test frames it finds a usable rig (3–9°, 7–25 px, F1 56–78); on the other half it locks onto a wrong rig that scores better than the true cameras (24–174°, 260–980 px, F1 0–20). EM refines the usable rigs (e.g. 22 → 7 px reprojection) but cannot recover the wrong ones, where it drifts further from the true cameras, which is why the mean rotation error increases after EM. Even on its own calibration frames, the baseline fails on several frames, and further re-tuning of the weights against the wrong rigs it found never improved this.

This is because each term admits wrong geometries: landmarks can match the wrong players, pitch lines align under many shifts, and zoom varies too much for the priors to constrain it. MuPeT's learned backbone resolves these ambiguities jointly in a single feed-forward pass, providing the initialisation that EM relies on.

Overall, we agree that an optimisation-based pipeline could in principle work and would be interesting to develop. However, our experiments suggest that a successful approach would require substantial hand-tuning and testing to incorporate all the necessary priors robustly; in our view, MuPeT is a more reliable way to achieve the same result without hand tuning.

Baseline details. The baseline uses SAM person and pitch-line masks, MAMMA landmarks, and the generator's camera distributions. The close-up views (Tight, High-Behind Right, Reverse) are out of reach for this approach: at high zoom they show few or no pitch lines and only a handful of players, leaving too little geometry to constrain the cameras. A candidate camera setup is scored by a weighted sum of cross-view pitch-line distance and overlap on the ground plane, player ground-position agreement, landmark epipolar error, and focal-length and camera-position priors. The rig is found by iterated sample-and-refine search: each round draws 1,000 cameras per role, from the prior or around the current best rigs, and scores all 10⁹ combinations exactly. The weights are calibrated on 5 frames using their ground truth, by repeatedly solving for weights that rank the true rig above every wrong rig found so far and searching again. We evaluate on 10 held-out frames from other clips. From the estimated cameras, we match players across views by projecting each player's feet onto the pitch and pairing players whose pitch positions agree (within 3 m). We then run the same EM as MuPeT (alternating player re-assignment and bundle adjustment of cameras and 3D landmarks), initialised with these cameras and matches instead of MuPeT's predictions.

For completeness, we also tested recovering the close-up views, fixing the three distant cameras at their ground truth and reusing the weights optimised on the calibration frames, evaluated on those same five frames. The objective still fails on all of them: the close-up cameras' leave-one-out reprojection error averages about 1,000 px, and in 11 of 15 cases the wrong camera scores better than the true one. Player matching fails accordingly, with an F1 of only 33 on links involving a close-up view, against 80 for MuPeT.

  1. Camera feeds. The six feeds are the complete set available in RugbyBench; additional fixed views or calibration metadata are not available. The cable camera, suspended above the pitch, provides a high, steep viewpoint close to the play that adds useful spatial information, particularly in dense interactions where other views are heavily occluded. It is also the view where pretrained models struggle most, as its unusual top-down pose is rare in their training data.

  2. Input channels. Each view is a 1024×576 image with six channels:

  • Channels 1–3 show the player masks and pitch lines. Each player mask is filled with a random colour. Colours are re-drawn for every view and frame, so they only tell players apart within one image, not across views. Pitch lines are drawn in grey.
  • Channels 4–6 show where each body landmark is. At the pixel of each detected MAMMA landmark, we store the 3D position of that landmark on a standard SMPL-X body in T-pose; all other pixels are zero. So the three values tell the network which body point it is (e.g. the left knee always has the same value, in every view), and nearby body points have similar values. Please see Figure 8 of the updated manuscript for a visualisation.
  1. Evaluation joints. We do not use SMPL-X fits for Table 2. For each of the 13 RugbyBench joints, we pick the one MAMMA landmark closest to that joint on a standard SMPL-X body, and use its predicted 3D position as the joint.

Thanks for implementing the geometric baseline and clarifying the other points. I find the additional evidence helpful and will keep my weak accept rating.

Reviewer_mS8QOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary
  • The paper focuses on multi-view 3D reconstruction of a sport scene (cameras + players). Input is uncalibrated videos of a rugby game.
  • The proposed method, MuPeT is a VGGT style feedforward model: its inputs are the segmentation masks of the players (using SAM3/ViTPose), field lines as edge map; and the output is camera params, pitch plane, depth, pitch points, image matches and people matches.
  • The key idea is to leverage synthetic data to supervise the model. No even attempt to use photorealistic renders, but instead of segmentation masks as a domain agnostic unifying representation.
  • The method appears to be for a benchmark work RugbyBench (also simultaneuously in submission at ICLR?), which forms the main data for training and evaluations.
  • The baselines are VGGT, VGGT-omega, Pi for reconstruction. PromptHMR and Multi-HMR2 for human recovery.
  • The evaluations show that the proposed method outperforms the baselines consistently across tasks.
Critical strengths
  • Uncalibrated multi-view 3D scene reconstruction and correspondence matching - this is the core focus problem. It is age old, however, I would say still unsolved to the level of accuracy it would take to deploy in real-life for a live sports game like rugby. This paper attempts to solve this problem with a specific focus on this scenario.

  • The key idea is simple and approach straightforward.

Critical weaknesses
  • Limited technical contributions: The core of the model is multi-view feedforward design applied to the domain specific sports data. There is limited technical learnings from this work for the field of 3D reconstruction. The paper does proposed EM like approach to minimize the domain gap between real-syn, however, this is not central to the paper and although novel, doesn't appear to be substantially general lesson which can be applied to other problem settings.

  • The scope of the experiments: The train data and eval data are both coming from an unpublished dataset RugbyBench. This limits the reproducibility of the method and its evaluations. Given the proposed method is also heavily designed for the sports setup, evaluating it on public datasets is infeasible.

  • Missing qualitative results: The core problem is temporal and 3D. No visual results as video are part of the submission. Supplementary material is an excellent opportunity to truly showcase the effectiveness of the method compared to the baselines. It is currently difficult to understand if the proposed method is simply incrementally better or a step gain compared to other methods from the limited images shown in the paper.

  • Ablations and design choices: Some design choices seem abrupt, although they are justified in text, very few are supported with quantitative evidence. The most central one, the choice of using masks as the input representation - this is unusual. Synthetic data with photorealistic renders and color augmentations, if (and this is important) executed correctly, should be able to have similar effects as the proposed method achieves. Many existing works like BEDLAM2, Syn4D and other synthetic datasets have shown this to be the case. The choice of EM as way to minimize the domain gap, it is presented as a the only obvious choice, however I would argue it is "a" solution and not the only solution. The paper's writing style currently over indexes on these uniquely specific design choices, while this takes away focus from the more important aspects of the core problem of multi-view 3D reconstruction and correspondence estimation.

Questions for authors
  • Given the limited novelty of the proposed method, please consider sharing your opinion on the technical strengths of the work.
  • The scope of the evaluations currently is limited to a single dataset (RugbyBench). Is it possible to widen the scope and enable a fairer comparison to baselines?
  • The lack of qualitative video results.
  • Thoughts on the uniquely specific design choices and the writing style.
2 comments

We thank the reviewer for the comments and address each point below.

  1. Technical contributions. We respectfully disagree with the assessment of limited novelty, and we believe the reviewer's points support it:

    1. The reviewer states that the EM is "novel" - whether central or not, it is a technical contribution (with verified impact).
    2. The reviewer highlights "the choice of using masks as the input representation - this is unusual". We interpret "unusual" as novel, unless there's prior art.

    We understand that novelty thresholds can vary, but we hope that this means the reviewer agrees that, ultimately, there is some amount of novelty. More than technically, our work contributes two broader lessons:

    1. People can serve as the calibration signal when the scene lacks texture. Existing human–scene methods recover geometry from static scene structure and then place people in it, which fails on textureless, repetitive scenes. MuPeT inverts this paradigm and learns cameras and scene geometry primarily from the players themselves.
    2. Abstract inputs make training on synthetic data alone feasible. Masks, dense landmarks and lines can be rendered synthetically and computed from real footage alike, so the model transfers to real data without photorealistic rendering.

    Together, these turn a setting where all prior methods fail into one that works: player-matching F1 rises from at most 28.8 to 81.0 (almost 3x), with 97.5% coverage (Table 1).

  2. Evaluation scope. Full reproducibility will be enabled by the release of RugbyBench and our code. To our knowledge, no public benchmark provides uncalibrated multi-view broadcast footage with 3D annotations, so this is the scope of evaluation. MuPeT does not need RugbyBench training data - it is all synthetic (other than very broad statistics about camera placements). We believe that providing a method for an under-served, practically relevant setting that the reviewer considers "still unsolved" is a strength.

  3. Qualitative results. We have uploaded a video of MuPeT’s predictions, compared with the baselines, to the supplementary material. It shows (1) fly-throughs of single-frame scene reconstructions against VGGT, MASt3R and π³, (2) SMPL-X fits on close-up views against Multi-HMR 2, and (3) reconstruction over a video sequence against π³, with the wide broadcast view shown for reference.

  1. Design choices and writing. We agree that EM and our other design choices are not the only possible solutions, and we do not intend to claim otherwise. We have revised the manuscript to remove language implying they are:

    • Abstract. Before: "The key enabler is the input modality: instead of pure RGB, which would present a large sim-to-real gap, our model consumes human segmentations and lines, which are easily rendered or computed from real data." After: "The key enabler is the input modality: rather than RGB, which would require photorealistic rendering at scale to bridge the sim-to-real gap, our model consumes human segmentations, dense landmarks and lines, which are cheap to rasterise synthetically and easy to compute from real data."
    • Introduction. Before: "...a residual sim-to-real gap remains. We therefore introduce an optional EM-based optimisation procedure that jointly refines cross-view player associations and scene geometry on real data." After: "Since MuPeT is trained entirely on synthetic scenes without sports-specific poses, a residual sim-to-real gap remains. We reduce it with an EM-based optimisation that jointly refines cross-view player associations and scene geometry on real data."
    • Sec. 3.2. Before: "Even with these measures to reduce the sim-to-real gap, some residual difference always remains. To refine the model predictions and enforce geometric consistency on real sequences, we therefore introduce an post-prediction optimisation stage." After: "Even with these measures to reduce the sim-to-real gap, a residual difference remains in practice. To refine the model predictions and enforce geometric consistency on real sequences, we introduce a post-prediction optimisation stage."

    We welcome feedback on these changes to improve further, and in particular if the reviewer had any specific literature in mind when mentioning "I would argue it is "a" solution and not the only solution".

On RGB rendering. Regarding the choice of masks and landmarks over photorealistic RGB, we do not claim that RGB-based training could not work in principle; rather, our representation makes training at this scale feasible. MuPeT trains in 45 h on 8 GH200 GPUs, whereas VGGT required 9 days on 64 A100 GPUs, in addition to the photorealistic rendering an RGB model would need. To quantify that rendering cost, we rendered photorealistic versions of our training scenes following the CISyn protocol (XRFeitoria, Blender). We simplified the character meshes (~173k to 30k faces) and rendered only the players (Cycles path tracing, 16 samples with denoising, 1024×576, six views per scene, ~24 players each), compositing the pitch and crowd separately on the GPU. Even so, a scene takes ~20 s per CPU worker or ~11 s on a GPU. Rendering 64,000 scenes took 26 h on 14 CPU workers (~5,800 CPU-hours), yet covers only the first 1% of MuPeT's training schedule (1,000 of 100,000 steps at 64 scenes per step). The full schedule would require ~580,000 CPU-hours, or ~800 GPU-days, against ~15 GPU-days for MuPeT's entire training. A faster engine does not change this. Blender's rasteriser (Eevee) renders the same scenes only ~20% faster on the same GPU and introduces material artefacts.

We also used these renders to train an RGB model directly. Since rendering the full schedule is infeasible, we reused the 64,000 rendered player layers, re-compositing them with varied pitch textures and crowds. To favour RGB, we used a pretrained DINOv2 encoder, whereas MuPeT's ResNet-34 is trained from scratch. Both models are trained using the ablation schedule (reviewer Affr R2), and evaluated at 30k iterations since the RGB model is still currently training. The masks model is trained with freshly generated scenes at every iteration.

Input (30k iterations)DataCoverage ↑Reproj. (px) ↓Pair F1 ↑W-MPJPE (m) ↓
Masks + landmarks + lines (ResNet-34)Synthetic (64 val. scenes)100%126.993.81.65
RGB (DINOv2)Synthetic (64 val. scenes)100%309.993.62.79
Masks + landmarks + lines (ResNet-34)Real (525 frames)99.4%282.973.73.21
RGB (DINOv2)Real (525 frames)91.0%1128.221.65.44

The results at 30k iterations (for all methods) already show clear patterns. On synthetic data, the RGB model associates players as well as ours (pair F1 93.6 vs 93.8), although its geometry is worse. On real footage, however, it collapses (pair F1 21.6, 1,128 px reprojection), while the mask-based model degrades far less (73.7, 283 px). With the rendering budget available, photorealistic RGB therefore does not close the sim-to-real gap even with a pretrained encoder, whereas our representation transfers. We provide an example of the rendered RGB panels in Figure 11 of the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.