acceptodds
Under review as a conference paper at ICLR 2027

ViewStruct: View-Grounded Articulation and Connectivity Inference

Abstract

Interacting with unfamiliar objects requires understanding how their parts connect and move. We introduce ViewStruct, a framework for jointly predicting connectivity and articulation in objects and scenes. Given an RGB image with aligned part segmentation and depth, optionally supplemented by images or video, ViewStruct predicts rigid-body groupings, attachments, and joint parameters. Its view-grounded representation links predictions to observed regions and expresses joint geometry in a gravity-aligned, view-relative frame, without requiring category-specific canonical orientations. A pretrained vision–language model is adapted to predict structure in seconds, without requiring observed articulation motion or per-instance optimisation. Experiments demonstrate improved structural prediction over evaluated baselines and transfer to unseen categories, asset sources, and real-world human-interaction videos, despite task-specific training exclusively on synthetic data. Simulated manipulation experiments further demonstrate the utility of predicted joints for generating robot motion trajectories.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
kinematic structure, articulated objects, scene understanding, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ip9pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ViewStruct predicts part connectivity (which regions move together as one rigid body and how bodies are attached) and articulation parameters (joint type, axis, pivot, limits) from a reference RGB image with given part segmentation and depth, optionally with extra views or a video. Qwen3-VL-4B is fine-tuned with LoRA to output this structure as JSON. Joints are expressed in a gravity-aligned, view-relative frame, parts are grouped into rigid bodies, and revolute pivots are predicted as an image location plus a relative depth. Training uses only synthetic data, with split/merge augmentation of the segmentation. The method is evaluated on PhysX-Mobility, SimReady and held-out GRScenes categories, on multi-object GRScenes room views, on real Arti4D videos, and in simulated manipulation scenarios

Critical strengths
  • Single model and output format for objects and scenes, and for single image, multi-view or video input. This is a practical formulation.
  • Fast inference (around 6.5 s per object), compared to minutes for the VLM stages of PhysX-Anything/Omni.
  • Wide evaluation: unseen asset source, held-out categories, scenes, real videos with Mobile-SAM regions, and simulated manipulation. Gains in joint recall and in success over all joints are large, especially in the generalisation settings.
  • Evaluation is done carefully (matched vs. all-joint success, geometric matching for baselines, gravity error and visibility diagnostics). The paper does not hide where baselines are better.
Critical weaknesses

W1: Novelty is limited: it is mostly a LoRA fine-tuning of an off-the-shelf VLM to output a JSON. The representation choices (gravity-aligned frame, rigid-body grouping, image-grounded pivots) make sense but are rather engineering, and the frame only brings 3.9 points.

W2: There are only two ablations (frame and split/merge augmentation). The image-grounded pivots and the joint-centred grouping, which are claimed as contributions, are not ablated.

W3: Strong input assumptions: GT part segmentation and GT depth in the main results, known gravity, an the number of joints is also supplied. This last point should be clearly stated in the main paper, since it directly helps joint recall/F1 and success over all joints, which is where most of the gain over baselines comes from.

W4: Comparison is not really fair: PhysX-Anything/Omni only get the RGB image. On matched joints PhysX-Omni is actually better (85.3 vs 81.5 success, 95.8 vs 89.9 axis accuracy), and Particulate has a much lower pivot error (0.036 vs 0.084, with some training overlap).

W5: Robustness to predicted segmentation is only tested on Arti4D. Results on PhysX-Mobility or SimReady with a real segmenter (e.g. SAM) instead of GT masks would be needed, as this is what is available in practice.

W6: “Inference in seconds” is only compared with methods that also generate the geometry, so not exactly the same task.

W7: Manipulation experiments are weak: contact points are given, physics is simplified (no gravity in PyBullet, no robot-scene collisions in MuJoCo), and there are only 10 Panda trials. Hinge opening is only 56.7% before retry, as the axis direction is often wrong.

W8: Required AI disclosure is not complete: missing all required items.

W9: References: venue is missing for most of them (arXiv only, instead of actual conference).

Questions for authors

See weaknesses

5 comments

We thank the reviewer for recognising our unified formulation, efficiency and generalisation. The revised paper adds controlled representation ablations, predicted-input and no-count experiments, and a larger manipulation evaluation, together with clearer protocols and AI disclosure.

W1 — Novelty. Our contribution is a unified, view-grounded articulation representation supporting objects and scenes, and single-image, multi-view and video inputs within one model. Joint-centred grouping avoids arbitrary internal rigid connections, while view-relative coordinates and 2D-pivot-plus-depth decoding ground geometry in the observation. These designs support generalisation to unseen asset sources and held-out categories, and real-world human-interaction videos without real-data fine-tuning.

W2 — Controlled ablations (Tables 9–10). We independently replace each component while retaining the same backbone, training data and schedule. PhysX-Mobility matched-joint success decreases from 82.9% to 76.6% with per-part connectivity, 69.5% with direct pivot regression, and 77.6% without gravity alignment. Object-canonical coordinates reduce success on held-out GRScenes categories from 48.7% to 30.0%, demonstrating the representation’s contribution to generalisation.

W3 — Input assumptions and joint counts (Tables 1, 8 and 11). We now explicitly disclose joint-count hints in the main paper. They primarily provide a stopping cue and are supplied for only a subset of training samples. Retraining and evaluating without hints yields 95.5% joint F1 on PhysX-Mobility and improves held-out GRScenes all-joint success from 48.5% to 55.9%. Estimated depth retains 70.5% all-joint success on PhysX-Mobility. Gravity perturbations of 2° leave matched-joint success unchanged, while 5° reduces it by 1.6 points.

W4 — Comparison protocols and parameter accuracy (Section B.2; Tables 1 and 7). In our comparisons, PhysX-Anything and PhysX-Omni also receive the reference part decomposition through reference part meshes that replace their generated geometry, and their reported scores depend on this supplied geometry. ViewStruct achieves 75.0% all-joint success versus PhysX-Omni’s 66.6%, reflecting more complete recovery of correct mechanisms.

W4 — Particulate’s pivot error. Particulate achieves lower conditional pivot error on PhysX-Mobility, but receives complete reference meshes, and 81 of the 387 test objects overlap its training data. These differences contextualise the comparison without establishing the cause of its lower error. ViewStruct achieves higher joint F1 (93.9% versus 61.0%) and matched-joint success (82.9% versus 68.2%). On GRScenes seen-category objects, ViewStruct also has lower pivot error (0.096 versus 0.150).

W5 — Predicted segmentation (Table 12). We add SAM-predicted regions with MoGe depth on PhysX-Mobility. Matched-joint success remains 77.5% versus 80.1% with reference regions and the same estimated depth, suggesting robustness when relevant parts are recovered, although the matched subsets differ. Predicted regions cover 71% of reference joints, limiting all-joint success to 30.7%. Segmentation coverage remains a limitation, but improving the segmentation models is beyond the scope of this work.

W6 — Efficiency (Section B.4). PhysX-Anything/Omni timings cover only their VLM joint-prediction stages and exclude geometry generation. On the same three objects and A100 GPU, ViewStruct takes 6.5 seconds, compared with 267 and 241 seconds, respectively. We clarify that these are joint-prediction timings, not complete reconstruction costs.

W7 — Manipulation evaluation (Tables 5 and 16; Section D.1). We add a Panda evaluation across the PhysX-Mobility test set with real-scale geometry, gravity, contact, automatic handle selection and collision-free approach planning. Over 898 robot-feasible joints, ViewStruct achieves 79.0% opening success, rising to 82.1% with one direction reversal; missing predictions count as failures. The original ten GRScenes trials remain separately identified as assisted-contact demonstrations.

W8 — AI disclosure. We expanded the AI Use Statement on page 10 to provide further details of AI assistance and clarify the authors’ responsibility for the final content.

W9 — References. We have updated the conference information for published references in the revised paper.

Dear Authors,

Thank you for your detailed response and for the additional experiments and clarifications. I believe your rebuttal adequately addresses my concerns, and I have no further questions at this stage. I will take your responses and the new results into account when making my final recommendation.

However, while you refer to a revised manuscript throughout your rebuttal, I have not been able to find it. Could you please upload the revised version so that I can verify that the reported changes have been incorporated into the paper?

Best regards,
Reviewer ip9p

Dear Reviewer ip9p,

Thank you for your prompt response and for considering our additional results and clarifications.

We have already uploaded the revised manuscript, which is available at https://openreview.net/pdf?id=1fpq9euXmj. The revised version contains 25 pages, with the additional experiments mainly in Section C.2 of the supplementary material.

Please let us know if you encounter any issues accessing it, and we will contact the AC to help resolve the issue.

Best regards,
The Authors

Reviewer_wdWTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript proposes ViewStruct for predicting part connectivity and articulation of objects from images and videos. It uses a gravity-aligned, view-relative representation to predict rigid-body groups, attachments, and joint parameters. A VLM is fine-tuned on synthetic articulated objects using RGB images, part segmentation, and depth maps, with optional additional images or videos. The method is evaluated on PhysX-Mobility, GRScenes, SimReady, and real-world Arti4D videos. It achieves 78.2% articulation success over all reference joints on PhysX-Mobility and 24.8% on GRScenes.

Critical strengths

S1. Given that embodied agents need to interact with unseen objects and scenes, the manuscript addresses an important problem of inferring object connectivity and articulation from visual observations.

S2. The evaluation goes beyond articulation prediction to include downstream robotic manipulation tasks. This is valuable as it demonstrates the practical utility of the predicted articulation parameters for embodied agents.

Critical weaknesses

W1. Evaluation relies on additional ground-truth information: The method uses ground-truth part segmentation and depth, while other methods like PhysX-Anything and PhysX-Omni use only RGB images, as shown in Tab. 1. The proposed method achieves 78.2% success compared to 66.6% for PhysX-Omni. With predicted depth using MoGe, it reaches 74.8%, but this baseline still uses GT part segmentation. Additionally, Appendix (L1177) mentions that the GT joint count is provided for image-based experiments. This might simplify the problem of identifying articulated connections. Therefore, it is difficult to know how much of the reported improvement comes from the proposed method or the additional ground-truth information. This directly questions the evaluation protocol.

W2. Missing ablations for the proposed representation: The manuscript proposes image-grounded pivot prediction as a key component of its representation (Sec. 3.2). However, Tab. 9 only ablates gravity alignment and part-decomposition augmentation. It is not clear how much the proposed image-grounded pivot prediction contributes to the overall performance. Since the view-grounded representation is one of the main contributions, the lack of this ablation makes it less convincing.

Questions for authors

Q1. How sensitive is the method to part segmentation quality? The MoGe experiment in Tab. 1 evaluates predicted depth, but ground-truth segmentation is still used. It would be interesting to see the results with automatically predicted part segmentation and depth. Could the authors also provide a comparison where the proposed method and baselines use comparable input information? (see W1)

Q2. How does the method perform when the ground-truth joint count is not provided? It would be informative to report F1 score for the presence of joints and articulation success under this setting on PhysX-Mobility and GRScenes (see W1).

Q3. Comparing the proposed image-grounded pivot representation with direct 3D pivot regression, while keeping the training setup unchanged would be insighful. (see W2)

Q4. How does the initial articulation state affect prediction accuracy? Appendix Sec. 2 describes static training examples rendered with objects at rest. Does the method perform differently when an object is fully closed, partially open, or fully open? An evaluation across different articulation states would help understand the limitations of single-image prediction.

3 comments

We thank the reviewer for recognising the practical relevance of articulation inference and its value for robotic manipulation. The revised paper adds predicted-input experiments, training without joint-count hints, controlled representation ablations, and evaluation across articulation states.

W1, Q1 — Comparable inputs and predicted segmentation (Section B.2; Tables 1 and 12). In Table 1, PhysX-Anything and PhysX-Omni also receive the reference part decomposition through reference part meshes that replace their generated geometry, and their reported scores depend on this supplied geometry. ViewStruct achieves 70.5% all-joint success with estimated depth, compared with PhysX-Omni’s 66.6%. With SAM-predicted regions and MoGe depth, matched-joint success remains 77.5% versus 80.1% with reference regions and the same estimated depth. This suggests robustness when the relevant parts are recovered, although the matched subsets differ. Predicted regions cover 71% of reference joints, limiting all-joint success to 30.7% and identifying segmentation coverage as a remaining limitation. Improving the segmentation models is beyond the scope of this work.

W1, Q2 — Unknown joint counts (Tables 1 and 11). Count hints primarily serve as a stopping cue and are supplied for only a subset of training samples. We now explicitly disclose their use in standard image-based ViewStruct and zero-shot Qwen3-VL evaluations in the main paper. We additionally retrain and evaluate ViewStruct without count hints. On PhysX-Mobility, it achieves 95.5% joint F1 and 67.3% all-joint success, remaining above PhysX-Omni’s 66.6% all-joint success. On held-out GRScenes categories, it achieves 96.9% joint F1, while all-joint success improves from 48.5% to 55.9%. Thus, strong joint recovery and generalisation persist without known joint counts, although the effect varies across datasets.

W2, Q3 — Controlled pivot ablation (Tables 9–10). We replace 2D-pivot-plus-depth decoding with direct pivot-coordinate regression, retaining the same backbone, training data and training schedule. Matched-joint success on PhysX-Mobility decreases from 82.9% to 69.5%, with performance also decreasing on the other evaluation sets. This isolates the benefit of explicitly grounding pivot geometry in the input image. We additionally provide independent ablations of joint-centred grouping and the coordinate frame.

Q4 — Initial articulation state (Table 13 and discussion). We evaluate the same PhysX-Mobility objects and cameras with randomly articulated leaf joints and additionally test PhysX-Omni on the same posed images. ViewStruct retains higher all-joint success under articulation: 63.3% versus 51.9%, with a smaller decrease from rest-state performance (11.7 versus 14.7 percentage points). Its joint-type and axis accuracy remain largely stable, while pivot localisation and range estimation deteriorate. These results demonstrate robustness beyond rest-state observations and show that sensitivity to articulation state also affects the baseline, while leaving room for improvement through training on articulated-state images.

Thank you for the detailed rebuttal and additional experiments. The additional experiments substantially address my original concerns. While the performance degradation with predicted segmentation highlights an important practical limitation, I consider this adequately investigated and documented.

I have no further questions and will take the additional results into account in my final assessment.

Thank you for your careful review and for considering our additional experiments. We are glad that these results substantially address your concerns. We have incorporated the new experiments and a discussion of the practical limitations into the revised manuscript.

Reviewer_VV29OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ViewStruct fine-tunes Qwen3-VL-4B to predict rigid-body connectivity and articulation parameters from RGB images, part segmentation, and depth. The method uses a gravity-aligned coordinate frame, rigid-body grouping, and image-grounded pivot parameterization. Experiments report improved joint recovery and inference efficiency across synthetic benchmarks, with additional evaluation on real-world interaction videos and simulated manipulation.

Critical strengths
  1. The problem is practically relevant, and the reported improvements in joint coverage and inference efficiency are substantial. The evaluation also extends beyond synthetic objects to real-world interaction videos.
  2. The paper provides useful evidence for some design choices. In particular, removing part-decomposition augmentation reduces articulation success by 10 percentage points, indicating that this component contributes meaningfully to performance.
Critical weaknesses
  1. Limited methodological novelty. The framework essentially fine-tunes an existing VLM to produce structured articulation parameters. The gravity-aligned frame, rigid-body grouping, and 2D-plus-depth pivot decoding are reasonable but largely conventional representation choices. I struggle to identify a substantial methodological contribution beyond adapting the output format. More importantly, two of these three designs are not independently ablated, and there is no fine-tuned Qwen3-VL baseline using the same training data with a conventional articulation representation. The zero-shot comparison does not establish the value of the proposed representation.
  2. The main empirical advantages are confounded by additional structural information. Appendix E reveals that ground-truth joint counts are provided in image-based experiments, while ViewStruct also receives reference part segmentation and depth that the main RGB-based baselines do not. This is particularly consequential because ViewStruct's strongest advantage lies in joint recovery rather than parameter accuracy. On PhysX-Mobility, its matched-joint success is actually lower than PhysX-Omni (81.5% vs. 85.3%). Without evaluating unknown joint counts and more comparable input conditions, it is difficult to determine how much of the reported improvement comes from the proposed method itself.
Questions for authors
  1. How does ViewStruct perform when the ground-truth joint count is not provided? Please clarify which experiments and baselines receive this information.
  2. Can the authors compare against Qwen3-VL fine-tuned on the same data and training budget, but using conventional part-level connectivity and direct 3D joint parameter prediction? This would help establish the independent contribution of the proposed representation.
4 comments

We thank the reviewer for recognising the practical relevance, joint coverage and efficiency of our approach. The revised paper adds controlled representation ablations, experiments without joint-count hints, and evaluation with predicted segmentation and depth.

W1 — Novelty and representation. Our contribution is a unified, view-grounded articulation representation. It supports objects and scenes, associates mechanisms with observed regions, and accommodates single-image, multi-view and video inputs within one model. Joint-centred grouping avoids arbitrary internal connections between rigidly attached regions, while view-relative coordinates and 2D-pivot-plus-depth decoding ground joint geometry in the observation. These designs enable generalisation across asset sources and held-out categories, and transfer to real-world human-interaction videos without real-data fine-tuning.

W1, Q2 — Controlled fine-tuned baselines (Tables 9–10). We independently replace the representation components while retaining the same Qwen3-VL backbone, training data and training schedule. On PhysX-Mobility, matched-joint success decreases from 82.9% to 76.6% with conventional per-part connectivity and to 69.5% with direct pivot-coordinate regression. Replacing view-relative coordinates with an object-canonical frame reduces success on held-out GRScenes categories from 48.7% to 30.0%. These controlled comparisons demonstrate the contributions of the representation beyond backbone fine-tuning.

W2, Q1 — Joint-count hints (Tables 1 and 11). Count hints primarily serve as a stopping cue and are supplied for only a subset of training samples. The standard image-based ViewStruct evaluations and zero-shot Qwen3-VL baseline use these hints. We additionally retrain and evaluate ViewStruct without them: joint F1 reaches 95.5% on PhysX-Mobility, and all-joint success on held-out GRScenes categories improves from 48.5% to 55.9%. Their effect is dataset-dependent: PhysX-Mobility all-joint success decreases to 67.3%, while remaining above PhysX-Omni’s 66.6%. Thus, strong joint recovery and generalisation persist without known joint counts.

W2 — Predicted inputs and practical generalisation (Tables 4 and 12). With SAM-predicted regions and MoGe depth, matched-joint success remains 77.5%, compared with 80.1% using reference regions and the same estimated depth. This suggests robustness when the relevant parts are recovered, although the matched subsets differ. Predicted regions cover 71% of reference joints, limiting all-joint success to 30.7%. Together with transfer to real human-interaction videos using Mobile-SAM regions, these results demonstrate applicability with predicted segmentation while identifying coverage as a remaining limitation. Improving the underlying segmentation models is beyond the scope of this work.

W2 — Input conditions and joint recovery (Section B.2; Table 1). In our comparisons, PhysX-Anything and PhysX-Omni also receive the reference part decomposition through reference part meshes that replace their generated geometry, and their reported scores depend on this supplied geometry. ViewStruct achieves 75.0% all-joint success, compared with PhysX-Omni’s 66.6%, and retains 70.5% with estimated depth. These results highlight successful recovery of the complete mechanism alongside conditional parameter accuracy.

Dear Reviewer VV29,

As the rebuttal deadline is approaching, please review the authors’ response and participate in the discussion with them at your earliest convenience.

Best regards,

The AC

Dear Reviewer VV29,

As the discussion period draws to a close, we would appreciate your feedback on whether our additional experiments and clarifications address your concerns. The revised manuscript includes controlled representation ablations and experiments without joint-count hints, which directly address your main questions.

Please let us know if any concerns remain, and thank you again for your time and consideration.

Best regards,
The Authors

I appreciate the careful revisions and the new experimental evidence provided in your response. At this point, I consider my main concerns sufficiently addressed and do not have any additional questions. I will take the additional evidence and clarifications into consideration when making my final recommendation.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We thank the Area Chair and all reviewers for their careful evaluation and constructive discussion. We appreciate that all three reviewers have indicated that our additional experiments and clarifications address their main concerns. We briefly summarise the key additions below.

  • Evidence for the representation’s contribution (Tables 9–10). Controlled ablations retain the same backbone, training data and schedule. Replacing image-grounded pivot decoding with direct regression reduces matched-joint success from 82.9% to 69.5%. Object-canonical coordinates reduce success on held-out GRScenes categories from 48.7% to 30.0%, supporting the role of our view-grounded representation in generalisation.

  • Strong performance without known joint counts (Tables 1 and 11). A model retrained and evaluated without count hints achieves 95.5% joint F1 on PhysX-Mobility and 55.9% all-joint success on held-out GRScenes categories. These results show that joint recovery and generalisation do not depend solely on the supplied count.

  • Practical input conditions and limitations (Tables 12–13). We add evaluations with predicted segmentation and depth, and with changed articulation states. With predicted regions, matched-joint success remains 77.5%, although incomplete segmentation substantially reduces overall coverage and all-joint success. We explicitly document this limitation and clarify the reference geometry supplied to the baselines.

  • Expanded downstream validation (Tables 5 and 16). The new Panda evaluation includes gravity, contact, automatic handle selection and collision-free approach planning. ViewStruct achieves 79.0% opening success across 898 robot-feasible joints, counting missing predictions as failures.

These additions strengthen the evidence for a unified articulation representation supporting objects and scenes, multiple observation modalities, and transfer to unseen categories and real-world human-interaction videos. We have also clarified timing protocols, expanded the AI disclosure, and corrected bibliographic information.

The experiments and clarifications have been incorporated into the revised manuscript. We hope that the strengthened evidence and the discussion will be reflected in the final assessment. Thank you again for your time and consideration!

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.