- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents VideoArtGS, a framework for reconstructing articulated objects from monocular video. It fits motion patterns to 3D tracks, filters unreliable trajectories, and clusters the remaining tracks to initialize part centers and joint parameters. A hybrid assignment module combines center-based scores for movable parts with a hash-grid score for the static base. The initialized deformation field and canonical Gaussians are then jointly optimized using rendering and tracking supervision. The paper reports joint-estimation and mesh-reconstruction results on iTACO-S, iTACO-L, and the newly curated VideoArtGS-20 benchmark.
Critical strengths
- VideoArtGS reports substantial improvements in joint estimation and part-level reconstruction on the evaluated benchmarks, including objects with multiple movable parts. The paper also provides component ablations, threshold sensitivity analysis, tracking evaluation, efficiency comparisons, and failure cases.
- Converting noisy 3D tracks into explicit part and articulation initialization is a useful design. The ablations demonstrate the importance of coherent part-center initialization. They also support the effectiveness of the hybrid assignment module: replacing it with center-based assignment substantially degrades movable-part reconstruction.
Critical weaknesses
- The Abstract and Introduction frame the task as jointly recovering geometry and articulation from entangled camera and object motion. However, iTACO-S uses ground-truth depth and camera poses, while VideoArtGS-20 begins with 150 static frames from multiple viewpoints before articulation. Thus, the strongest quantitative results are obtained under substantially simplified conditions, whereas the fully RGB-only real-world setting is evaluated only qualitatively.
- It remains unclear whether existing reconstruction methods could achieve comparable performance when provided with the proposed track-derived initialization. Evaluating a prior reconstruction framework with the same initialized part centers and articulation parameters would clarify how much of the reported performance gap can be closed by transferring this initialization.
Questions for authors
- The paper presents articulated reconstruction from a single monocular video. But the current evidence is insufficient to establish applicability to uncontrolled monocular videos. How does VideoArtGS perform when depth and camera poses are estimated from RGB, and when the initial static segment is shortened or provides less viewpoint coverage? Quantitative results under these conditions would help clarify the method’s practical scope.
- Could the authors evaluate a representative reconstruction baseline with the proposed track-derived part-center and articulation initialization, under matched geometry inputs and a comparable optimization budget?
- The hybrid center-grid design is motivated by improved representation of complex static geometry. Could the authors report part-assignment accuracy, or analyze the w/o hybrid results across objects with different static-base complexity to directly test this explanation for the observed gains?
4 comments
Thank you for the response and clarifications. However, my two main concerns remain unresolved.
-
Shared GT inputs ensure fair comparisons but do not establish effectiveness to predicted depth/poses or shorter static observations. These can be quantitatively evaluated on existing synthetic benchmarks.
-
Internal ablations do not replace the requested cross-method comparison with shared initialization, which is necessary to isolate its contribution to the reported gains.
-
The explanation is reasonable, but direct assignment accuracy would better support the claimed mechanism.
Overall, I appreciate the clarifications, but the response does not provide sufficient evidence to resolve my primary concerns. I maintain my current rating for now.
Thank you for the follow-up. We now provide direct assignment measurements in addition to the reconstruction ablations, and respectfully clarify the remaining points.
1. Predicted geometry and static observations
Our additional DA3 camera audit finds a median positional RMSE of 0.764 m on the canonical segments even after optimal Sim(3) alignment.DA3 produces meter-scale camera-pose errors on these background-free synthetic sequences, despite performing reasonably on our real-world captures. Using these unreliable estimates would confound upstream pose-estimation failures with downstream reconstruction performance. This limitation of DA3 should not be attributed to VideoArtGS or negate the reconstruction gains established by our fair comparisons.
The static segment is a shared acquisition condition, rather than extra input reserved for VideoArtGS. We agree that this establishes comparison fairness, not robustness to shorter observations. Additional first-frame-initialization and no-static-image evaluations are running; complete quantitative results are not yet available. This limits the demonstrated operating range, but does not invalidate the gains under the reported protocol.
2. Initialization attribution
We disagree that cross-method initialization transfer is necessary to validate our contributions. Transfer into another adapted algorithm addresses a different question and should not be a prerequisite for recognizing those contributions.
3. Direct assignment accuracy
We evaluate the models on identical GT canonical-surface points: 20,000 samples per part with three seeds, surface-area weighting, and Hungarian matching of moving-part labels while fixing the static label. Means and medians are computed across all 20 objects; assignment metrics below are percentages.
| Metric (%) | Mean: w/o hybrid | Mean: Full | Median: w/o hybrid | Median: Full |
|---|---|---|---|---|
| Static-part IoU ↑ | 86.00 | 91.62 | 92.08 | 95.19 |
| Moving-part mIoU ↑ | 71.50 | 83.00 | 82.05 | 90.44 |
| All-part mIoU ↑ | 75.28 | 84.68 | 84.43 | 90.70 |
| Assignment accuracy ↑ | 90.16 | 94.80 | 93.68 | 96.31 |
| Static-to-moving misassignment ↓ | 5.78 | 2.07 | 1.02 | 1.22 |
| Moving-to-static misassignment ↓ | 19.39 | 12.77 | 8.90 | 7.66 |
All-part mIoU improves on 20/20 objects, with mean and median gains of 9.39 and 6.26 percentage points. Mean static-to-moving misassignment decreases from 5.78% to 2.07%. On the nine-movable-part table (30666), it decreases from 68.73% to 2.62%, directly supporting improved separation of this complex static structure.
These results provide the direct assignment evidence requested and complement the reconstruction gains. We respectfully ask that this additional evidence receive due weight and that you reconsider your assessment and raise your score.
Addtional experiments for canonical Gaussian init
We have completed additional experiments on all 20 objects. Directly initializing Gaussians from first-frame points, achieves comparable performance to the original model (axis error 0.35°; CD-w 0.08 cm). In the variant excluding all initial static images, articulation remains accurate (0.38° axis error and 0.13 cm position error), while mesh error increases (CD-w 0.49 cm). These results demonstrate robustness to the alternative initialization and suggest that the initial observations primarily benefit mesh completeness. Adequate surface coverage remains important for high-quality geometry.
Both additional variants share the same motion initialization and use 20,000 joint-optimization steps. Results are mean ± standard deviation over all 20 objects; lower is better.
| Setting | Axis (°) | Position (cm) | CD-w (cm) | CD-m (cm) | CD-s (cm) |
|---|---|---|---|---|---|
| Original model | 0.34 ± 0.80 | 0.10 ± 0.10 | 0.09 ± 0.09 | 0.26 ± 0.61 | 0.24 ± 0.58 |
| Direct first-frame initialization; static views retained | 0.35 ± 0.81 | 0.10 ± 0.12 | 0.08 ± 0.09 | 0.24 ± 0.45 | 0.25 ± 0.69 |
| Direct first-frame initialization; static views removed | 0.38 ± 0.80 | 0.13 ± 0.12 | 0.49 ± 0.58 | 1.08 ± 3.28 | 0.57 ± 0.69 |
Response to Reviewer dnAD
We thank Reviewer dnAD for the questions on scope, component attribution, and hybrid assignment.
1. Predicted geometry and static initialization
Synthetic experiments use GT depth and poses under the benchmark protocol. Real captures use RGB-estimated geometry (Sec. 4.3, Appendix C.3) and are evaluated qualitatively because GT part meshes and joint parameters are unavailable.
The 150-frame static segment is a shared acquisition condition across the comparisons, not extra input reserved for VideoArtGS. It initializes canonical geometry while the camera moves around the stationary object. Alternatives include first-frame points or later-frame points warped back through the initialized dynamic field. In our experience, these are feasible but give somewhat lower geometric fidelity.
2. Initialization and hybrid assignment
Track-derived initialization is itself a contribution, with center, axis, and deformation-field initialization separately ablated in Table 3. Cross-method transfer is therefore unnecessary for validating these component claims.
Table 3 directly tests whether initialization alone explains the gains: retaining the same initialization but removing hybrid assignment increases CD-m from 0.26 to 10.35 cm, axis error from 0.34° to 1.21°, and position error from 0.10 to 2.51 cm. This establishes the additional contribution of hybrid assignment.
3. Why a spatial grid for the static base?
Movable parts form coherent motion clusters, whereas the static base may be extended, non-convex, or enclose movable parts. Our dedicated static hash-grid branch models spatial membership without relying on a static-center distance prior, while retaining center-based scores for movable parts. This differs from ArtGS's combination of center distances and residual assignment scores.
For enclosing structures, a center-distance prior can favor a nearby movable part for static points, coupling assignment and articulation errors. The spatial branch addresses this limitation. Table 3 supports its reconstruction benefit, although it does not directly measure assignment IoU or performance stratified by base complexity.
These requested extensions do not negate the contributions demonstrated by the controlled ablations and substantial quantitative gains. We respectfully ask that this evidence receive due weight and that you reconsider your assessment and consider raising your score.