- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper develops a fast, feedforward method based on VGGT to acquire 3D Gaussians from sparse views for an articulated object, along with joint information. The model outputs the joint information and geometry in seconds, faster than prior similar methods. The model is trained on the partnet-mobility dataset, and evaluated on a held out set from the same dataset, as well as the AKB-48 dataset. The model results are compared against relevant reconstruction and feedforward methods.
Critical strengths
- The lightweight input required (sparse view inputs) and inference time that are much faster than comparable methods are a great advantage.
- The framework is clear and easy to follow, elegant in simplicity.
Critical weaknesses
- There is no real-world tests or validation that show that the method is applicable to more noisy and challenging real world inputs. Datasets worth considering includes the two real-world objects of PARIS.
- Segmentation usually is a challenging topic in articulated objects. The paper doesn't discuss the accuracy of the clustering in detail.
Questions for authors
- How well would ArtSplat work on real world data that may have more noisy and complex input?
- What are the categories of the objects in the evaluation set; how diverse are they?
- Often times in a partnet-mobility object with many parts, some parts have very similar motion such as two drawers or two doors that move similarly. How reliable is the segmentation on those type of objects?
- At inference, how is the background segmented from the foreground? And how reliable is the clustering, how accurate are the clustering results against GT part masks?
- The reconstruction baselines were tested on sparse inputs as well, not their usual setting of dense view input. How badly did this affect their results?
2 comments
[Q5] The baselines were tested on sparse inputs, not their usual dense-view setting; how badly did this affect their results?
Yes, considerably. To measure it, we re-ran ArtGS and ScrewSplat with the dense (100 and 50 per state, respectively) views they were originally designed for. We report both settings in Table II in App. D.1, copied below.
Table. Optimization baselines with sparse (4) and their intended dense views vs. ArtSplat with 4 views per state.
| Method | # Views / State | CD-w↓ | CD-s↓ | CD-m↓ | Ang$_m$↓ | Pos$_m$↓ | PSNR↑ | SSIM↑ | Time |
|---|---|---|---|---|---|---|---|---|---|
| ArtGS | 4 | 0.096 | 0.102 | 0.782 | 49.31 | 1.825 | 25.21 | 0.893 | 15m 50s |
| ArtGS | 100 | 0.021 | 0.045 | 0.168 | 21.25 | 0.40 | 36.66 | 0.989 | 18m 59s |
| ScrewSplat | 4 | 0.035 | 0.044 | 0.405 | 46.78 | 3.368 | 23.49 | 0.912 | 23m 05s |
| ScrewSplat | 50 | 0.018 | 0.026 | 0.055 | 24.46 | 0.327 | 34.03 | 0.980 | 26m 59s |
| ArtSplat (Ours) | 4 | 0.034 | 0.024 | 0.030 | 5.79 | 0.864 | 26.30 | 0.921 | <2 s |
With dense views, they render better than ArtSplat and their axis error is roughly halved, so the sparse input does account for a large part of their drop. On multi-joint objects, however, their axis error still remains well above that of ArtSplat with 4 views. The sparse setting therefore explains the gap in appearance, but only part of the gap in motion.
We thank the reviewer again for the valuable suggestions, which made the paper more complete. We are happy to answer any further questions and will respond promptly.
We thank the reviewer for the encouraging review and the concrete suggestions. We followed them with new experiments, discussed point by point below. Edits to the paper are marked in blue in the revised manuscript.
[W1] No real-world tests or validation on noisy and challenging real-world inputs (e.g., the two real-world objects of PARIS)
We agree that real-world results were missing, so we added two sets of real captures: the two real PARIS objects, which have joint ground truth, and seven RealScan objects (Table III and Fig. I in App. D.3; the table is reproduced below).
Table. Real-world evaluation.
On PARIS, LARM is given GT camera poses and intrinsics, the GT joint type for axis estimation, and the articulation value of each input state.
With only uncalibrated RGB images and no real training data, ArtSplat recovers the joints of both PARIS objects, second to ScrewSplat and ahead of ArtGS and LARM, which each miss one joint, with the highest SSIM and a PSNR on par with the best method, and outperforms the feed-forward LARM on RealScan. We acknowledge that the overall performance on real data is still well below that on synthetic data, and that this evaluation is small. We attribute the gap mainly to training on synthetic data only. Closing it is what we need to improve next, and we state this in the limitations.
[W2, Q4] The paper doesn't discuss the accuracy of the clustering in detail; how is the background segmented at inference?
(1) Background. ArtSplat uses an object mask to separate the object from the background. In all experiments, we use the object masks provided with the dataset; i.e., the alpha channel of the synthetic renderings and the released masks of the real captures.
(2) Clustering accuracy. As suggested, we compared the predicted parts with the GT part masks, using ARI and the mean IoU of matched parts (Table V in App. E.1, reproduced below).
Table. Clustering accuracy against GT part masks.
The predicted parts closely match the GT parts on both single-joint and multi-joint objects. Qualitative examples of the clustering and of the predicted parts are provided in Figs. II and III (App. E.1). Adjacent parts that share an axis and move similarly are the hardest case, which we examine in Q3 below. To see how much this affects the articulation, we also replaced the clustering with the GT part labels at inference, and the axis error changes only slightly (Table IV in App. E.1, copied below).
Table. HDBSCAN settings and GT part labels at inference.
min_cluster_sizemin_samples[Q2] What are the categories of the objects in the evaluation set; how diverse are they?
The 68 test objects (18 single-joint, 50 multi-joint) cover all 15 categories of our PartNet-Mobility subset: StorageFurniture 24, Table 17, Oven 3, and 2 for each of the other 12 (App. A). The most complex object has 14 joints.
[Q3] How reliable is the segmentation on parts with very similar motion, such as two drawers or two doors?
This is an important case, so we looked at it directly. 37 of the test objects have parts that share an axis, such as two drawers or two doors, and we re-rendered them so that these parts move identically (table below).
Table. Same-axis parts moving identically (37 objects).
The segmentation is unchanged from the original states, because the clustering also uses the 3D position of each Gaussian. By combining the predicted joint axes with the 3D position of each Gaussian, ArtSplat thus separates same-axis parts even when they move identically, a case in which motion cues alone would merge them.