- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ViewStruct predicts part connectivity (which regions move together as one rigid body and how bodies are attached) and articulation parameters (joint type, axis, pivot, limits) from a reference RGB image with given part segmentation and depth, optionally with extra views or a video. Qwen3-VL-4B is fine-tuned with LoRA to output this structure as JSON. Joints are expressed in a gravity-aligned, view-relative frame, parts are grouped into rigid bodies, and revolute pivots are predicted as an image location plus a relative depth. Training uses only synthetic data, with split/merge augmentation of the segmentation. The method is evaluated on PhysX-Mobility, SimReady and held-out GRScenes categories, on multi-object GRScenes room views, on real Arti4D videos, and in simulated manipulation scenarios
Critical strengths
- Single model and output format for objects and scenes, and for single image, multi-view or video input. This is a practical formulation.
- Fast inference (around 6.5 s per object), compared to minutes for the VLM stages of PhysX-Anything/Omni.
- Wide evaluation: unseen asset source, held-out categories, scenes, real videos with Mobile-SAM regions, and simulated manipulation. Gains in joint recall and in success over all joints are large, especially in the generalisation settings.
- Evaluation is done carefully (matched vs. all-joint success, geometric matching for baselines, gravity error and visibility diagnostics). The paper does not hide where baselines are better.
Critical weaknesses
W1: Novelty is limited: it is mostly a LoRA fine-tuning of an off-the-shelf VLM to output a JSON. The representation choices (gravity-aligned frame, rigid-body grouping, image-grounded pivots) make sense but are rather engineering, and the frame only brings 3.9 points.
W2: There are only two ablations (frame and split/merge augmentation). The image-grounded pivots and the joint-centred grouping, which are claimed as contributions, are not ablated.
W3: Strong input assumptions: GT part segmentation and GT depth in the main results, known gravity, an the number of joints is also supplied. This last point should be clearly stated in the main paper, since it directly helps joint recall/F1 and success over all joints, which is where most of the gain over baselines comes from.
W4: Comparison is not really fair: PhysX-Anything/Omni only get the RGB image. On matched joints PhysX-Omni is actually better (85.3 vs 81.5 success, 95.8 vs 89.9 axis accuracy), and Particulate has a much lower pivot error (0.036 vs 0.084, with some training overlap).
W5: Robustness to predicted segmentation is only tested on Arti4D. Results on PhysX-Mobility or SimReady with a real segmenter (e.g. SAM) instead of GT masks would be needed, as this is what is available in practice.
W6: “Inference in seconds” is only compared with methods that also generate the geometry, so not exactly the same task.
W7: Manipulation experiments are weak: contact points are given, physics is simplified (no gravity in PyBullet, no robot-scene collisions in MuJoCo), and there are only 10 Panda trials. Hinge opening is only 56.7% before retry, as the axis direction is often wrong.
W8: Required AI disclosure is not complete: missing all required items.
W9: References: venue is missing for most of them (arXiv only, instead of actual conference).
Questions for authors
See weaknesses
5 comments
Dear Authors,
Thank you for your detailed response and for the additional experiments and clarifications. I believe your rebuttal adequately addresses my concerns, and I have no further questions at this stage. I will take your responses and the new results into account when making my final recommendation.
However, while you refer to a revised manuscript throughout your rebuttal, I have not been able to find it. Could you please upload the revised version so that I can verify that the reported changes have been incorporated into the paper?
Best regards,
Reviewer ip9p
Dear Reviewer ip9p,
Thank you for your prompt response and for considering our additional results and clarifications.
We have already uploaded the revised manuscript, which is available at https://openreview.net/pdf?id=1fpq9euXmj. The revised version contains 25 pages, with the additional experiments mainly in Section C.2 of the supplementary material.
Please let us know if you encounter any issues accessing it, and we will contact the AC to help resolve the issue.
Best regards,
The Authors
We thank the reviewer for recognising our unified formulation, efficiency and generalisation. The revised paper adds controlled representation ablations, predicted-input and no-count experiments, and a larger manipulation evaluation, together with clearer protocols and AI disclosure.
W1 — Novelty. Our contribution is a unified, view-grounded articulation representation supporting objects and scenes, and single-image, multi-view and video inputs within one model. Joint-centred grouping avoids arbitrary internal rigid connections, while view-relative coordinates and 2D-pivot-plus-depth decoding ground geometry in the observation. These designs support generalisation to unseen asset sources and held-out categories, and real-world human-interaction videos without real-data fine-tuning.
W2 — Controlled ablations (Tables 9–10). We independently replace each component while retaining the same backbone, training data and schedule. PhysX-Mobility matched-joint success decreases from 82.9% to 76.6% with per-part connectivity, 69.5% with direct pivot regression, and 77.6% without gravity alignment. Object-canonical coordinates reduce success on held-out GRScenes categories from 48.7% to 30.0%, demonstrating the representation’s contribution to generalisation.
W3 — Input assumptions and joint counts (Tables 1, 8 and 11). We now explicitly disclose joint-count hints in the main paper. They primarily provide a stopping cue and are supplied for only a subset of training samples. Retraining and evaluating without hints yields 95.5% joint F1 on PhysX-Mobility and improves held-out GRScenes all-joint success from 48.5% to 55.9%. Estimated depth retains 70.5% all-joint success on PhysX-Mobility. Gravity perturbations of 2° leave matched-joint success unchanged, while 5° reduces it by 1.6 points.
W4 — Comparison protocols and parameter accuracy (Section B.2; Tables 1 and 7). In our comparisons, PhysX-Anything and PhysX-Omni also receive the reference part decomposition through reference part meshes that replace their generated geometry, and their reported scores depend on this supplied geometry. ViewStruct achieves 75.0% all-joint success versus PhysX-Omni’s 66.6%, reflecting more complete recovery of correct mechanisms.
W4 — Particulate’s pivot error. Particulate achieves lower conditional pivot error on PhysX-Mobility, but receives complete reference meshes, and 81 of the 387 test objects overlap its training data. These differences contextualise the comparison without establishing the cause of its lower error. ViewStruct achieves higher joint F1 (93.9% versus 61.0%) and matched-joint success (82.9% versus 68.2%). On GRScenes seen-category objects, ViewStruct also has lower pivot error (0.096 versus 0.150).
W5 — Predicted segmentation (Table 12). We add SAM-predicted regions with MoGe depth on PhysX-Mobility. Matched-joint success remains 77.5% versus 80.1% with reference regions and the same estimated depth, suggesting robustness when relevant parts are recovered, although the matched subsets differ. Predicted regions cover 71% of reference joints, limiting all-joint success to 30.7%. Segmentation coverage remains a limitation, but improving the segmentation models is beyond the scope of this work.
W6 — Efficiency (Section B.4). PhysX-Anything/Omni timings cover only their VLM joint-prediction stages and exclude geometry generation. On the same three objects and A100 GPU, ViewStruct takes 6.5 seconds, compared with 267 and 241 seconds, respectively. We clarify that these are joint-prediction timings, not complete reconstruction costs.
W7 — Manipulation evaluation (Tables 5 and 16; Section D.1). We add a Panda evaluation across the PhysX-Mobility test set with real-scale geometry, gravity, contact, automatic handle selection and collision-free approach planning. Over 898 robot-feasible joints, ViewStruct achieves 79.0% opening success, rising to 82.1% with one direction reversal; missing predictions count as failures. The original ten GRScenes trials remain separately identified as assisted-contact demonstrations.
W8 — AI disclosure. We expanded the AI Use Statement on page 10 to provide further details of AI assistance and clarify the authors’ responsibility for the final content.
W9 — References. We have updated the conference information for published references in the revised paper.