acceptodds
Withdrawn from ICLR 2027

SportCraft: Agentic 4D Reconstruction of Sports Scenes from Monocular Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Monocular sports reconstruction must align venue geometry, human motion and equipment interactions despite ambiguous scale, rapid motion and changing contacts. Corrections tailored to one action or view can also fail when reused across videos. We propose SportCraft, an agentic framework for editable 4D sports reconstruction organized around two coupled feedback loops, with Codex serving as the supervisory agent for scene diagnosis, code proposals and update review. Agent-directed model fusion registers complementary pretrained predictions in a shared scene with consistent coordinates and timing. Knowledge-guided scene refinement forms the inner loop, using rendered feedback, venue references and phase-specific sports constraints to refine cameras, human motion, object trajectories and contacts. Visual, geometric and contact checks govern the acceptance of scene updates. Agent-driven tool evolution forms the outer loop: the agent records repair experience, generates and revises parameterized geometric functions with explicit applicability conditions, and uses replay and regression checks to review candidate revisions before reuse. Pretrained model weights remain fixed.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Reconstruction, Sports Scene Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mQAwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SportCraft reconstructs editable 4D sports scenes from monocular videos. It combines pretrained estimates of human motion, scene geometry, and object locations in a shared coordinate frame. Codex supervises a refinement loop that inspects rendered feedback, selects editable variables and sports constraints, and calls numerical tools to update cameras, human motion, and equipment trajectories. Candidate updates are checked before being retained. The framework also records repair experience and proposes reusable geometric tools for subsequent videos. Experiments on 13 videos across 12 sports evaluate source-view reconstruction consistency against OVOW and RHINO, with additional qualitative results and an ice-hockey ablation.

Critical strengths

1. Combining sports knowledge with feedback-guided refinement is a useful approach to scene reconstruction. Camera, human, and equipment estimates can disagree even when each looks plausible on its own. The framework addresses this by selecting which variables to refine and applying constraints such as venue dimensions, support, grasping, and flight only when the observations and interaction phase support them. Checking candidate updates before retaining them also helps avoid fixing one component at the expense of another. I find this a promising system design for bringing complementary predictions into a coherent sports scene.

2. The results show improved source-view consistency across a range of sports. In Table 1, human reprojection error decreases from 180.5 px for OVOW to 28.2 px for SportCraft, while human and object IoU also improve. The evaluation covers 12 sports, and the qualitative results show reconstructions of seven of them, including racket sports, ball games, and close human interactions. These results support the usefulness of the overall system beyond a single sports setting.

Critical weaknesses

1. The benefit of agent-guided refinement needs a quantitative ablation. Table 1 compares complete systems, while Figure 4 only provides a qualitative ablation on one ice-hockey example. I would like to see a quantitative comparison with a fixed refinement pipeline using the same predictors, constraints, and numerical tools, to understand how much the agent’s diagnosis and refinement decisions improve reconstruction quality.

2. The evaluation leaves 3D geometry and interaction accuracy unclear. The reported metrics measure agreement with source-view keypoints and masks. They do not directly establish whether depth, contact, and motion trajectories are recovered accurately. Evaluation on a subset with 3D ground truth or additional views would help assess these aspects.

Questions for authors
  1. With the same predictors, constraints, and numerical tools, how much does agent-guided refinement improve reconstruction quality over a fixed pipeline? Please provide a quantitative ablation.
  2. Some supplementary videos contain more than two people, but the reconstructions appear to include only two. Does the method support reconstructing all visible people, and how are participants selected?
  3. How much manual intervention is needed, including reviewing venue features and masks? What are the typical agent-call count and end-to-end runtime per video?
Reviewer_3s94OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SportCraft, an agent coding framework for reconstructing 4D sports scenes from monocular videos. It combines off-the-shelf pretrained models for human motion, scene reconstruction, and object localization, and uses a coding agent to diagnose reconstruction errors, refine scenes, and develop reusable repair tools. This frameworks is evaluated on 13 videos across 12 sports.

Critical strengths

The paper intends to solve a challenging problem of reconstructing dynamic sports scenes involving both human motion and objects motion. With recent advances such as Astra, using coding agents to iteratively refine reconstructions is a particularly promising direction.

Critical weaknesses

The main contribution of agent-driven tool evolution is not comprehensively validated. First of all, the evaluation is very limited, with only 13 videos across 12 sports. This is insufficient to demonstrate the claimed generalization and transferability of the accumulated repair experience.

The ablation is limited to a single ice-hockey example, without convincing evidence of autonomous tool generation or cross-video reuse. And the reconstruction quality is unsure. Despite improvements over the baselines, Human IoU and Object IoU are only 0.209 and 0.195. In addition, all quantitative evaluations rely on 2D metrics, without validating 3D reconstruction accuracy or physical consistency.

The contribution of the agent itself is unclear. The paper lacks comparisons with a conventional optimization pipeline using the same pretrained models and geometric constraints but without Codex.

The paper does not report the computational cost of the proposed agentic approach, including the number of Codex calls, token usage, execution time, or required human intervention, etc. These are important for evaluating its practicality and scalability.

The supplementary videos mainly present final reconstructions and baseline comparisons, with little direct evidence of the proposed agentic refinement and tool evolution processes.

Questions for authors

Can the authors provide concrete examples of repair tools autonomously generated or modified by Codex, and demonstrate how these tools improve reconstruction on novel videos? How often are these tools successfully reused across different videos or sports?

How much of the reported improvement comes from Codex rather than the pretrained models and conventional geometric optimization?

How much human intervention, computation, and agent execution time are required to reconstruct each video?

Reviewer_isR1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sportcraft is an AI agent that takes in sports videos, infers results from existing vision models, and refines the scene. Evaluation is done on 13 videos across 12 sports with improved 2d reprojected metrics.

Critical strengths
  1. Challenging problem- specifically due to the lack of datasets for complete systems like this.
  2. Sensible design of the system and useful experiments across different sports.
Critical weaknesses
  1. No 3d validation is done
  2. The evaluation set is too small.
  3. Multiperson sports scalability is not demonstrated
  4. Limited method novelty- it's primarily an integration of existing pretrained estimators
Questions for authors
  1. Can we evaluate subsets of this problem on existing datasets?
  2. Is the method limited to 2 players? The videos show a max of 2 players, and the rest are skipped in the reconstruction. why?
  3. It's a bit confusing which parts of the pipeline are manual and automatic. For example, how are contact constraints checked quantitatively?
  4. How does the system change under varying camera motion, occlusion, and blur? Is the ceiling limited by all the pretrained models used in this work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.