acceptodds
Under review as a conference paper at ICLR 2027

FACT: Fidelity-Aware Construction of Articulated Twins

Abstract

Visually plausible articulated assets may still fail during contact interactions or exhibit inaccurate motion. We present FACT (Fidelity-Aware Construction of Articulated Twins), an agentic framework that progressively constructs articulated twins to improve geometry, contact, and dynamic fidelity. The agent drives an evidence–diagnosis–revision loop on a shared editable representation, selecting measurements and model edits using quantitative feedback, while numerical tools execute and validate the updates. It reconstructs editable articulated geometry from images through feature planning, targeted measurements, and diagnostic refinement. On this reference, it repairs collision proxies through task-aware local repartitioning before fidelity-constrained compression. Finally, it constructs response models from passive-response videos, using simulation residuals to guide model revision and constrained physical parameter fitting. Experiments show that FACT improves geometric reconstruction over baselines, enables more reliable interaction with simpler collision proxies, and better reproduces held-out physical responses than direct parameter inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Articulated Objects, Digital Twins, Physical Simulation, Agent System

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4tQqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses articulated object reconstruction with agentic self-correction loops. The framework consists of geometric feature generation, collision refinement, and dynamics refinement. The first loop geometric feature planning and measurement selection. The collisions are then addressed by task-aware local edits. Lastly, response models are refined with the observed joint trajectories. The proposed framework is evaluated on 21 objects, showing comparative benefits over the baselines in Chamfer Distance and Surface Distance.

Critical strengths
  • The motivation of fuctional plausibility is presented well and connected to the corresponding technical contributions in Introduction section.
  • It seems logical and technically interesting to disentangle self-corrections loops into geometry, contact, and dynamics. Table 2 provides empirical benefits of each block comprehensively.
  • Comparative benefits of the proposed method is supported in Table 1 and Table 3, especially CD, SD, and ISR.
  • Figure 6 comprehensively provides qualitative comparisons against existing flagship AI agents, effectively emphasizing the technical benefits of the proposed method.
Critical weaknesses
  • The comparative benefit might be attributed from the input mesh. While Codex and Articraft do not support mesh input, FACT rigorously utilizes the mesh features. Table 1 shows that FACT is inferior to the Rodin in SD-P95, F@1, and F@2. As FACT reconstructs from Rodin results under part mesh setup, the results in Table 1 provides limited benefits over the Rodin baseline.
  • In Table 4, the comparative baselines are all direct inference methods, and no fitting baseline is comapred. It would be helpful to compare the proposed method with existing works in the related fields (e.g., [1, 2]. [1]: VLM-grounded critic. [2]: geometric optimization. I could find these works within 1 mintue via Google search. More relevant work could be found.).
  • As reported in the Appendix A, Table 3 excludes two objects as those objects would degrade the performance of the proposed method. It would be better to have the same evaluation set across the tables to ensure research integrity and Code of Ethics.
  • The object and interaction diversity are limited. Although the original ArtVIP provides 992 articulated objects and 394 interactions, the paper only uses 13 objects from ArtVIP. It would be better to incorporate more diverse object categories and interactions.

[1] Yipeng Gao et al., LAM: Language Articulated Object Modelers, CVPR 2026. [2] Xuelu Li et al., ArtPro: Self-Supervised Articulated Object Reconstruction with Adaptive Integration of Mobility Proposals, CVPR 2026.

Questions for authors
  • What are the comparative benefits against the baselines that allows mesh inputs (e.g., Rodin), other than image-based direct inference baselines?
  • In Table 4, would there be comparative benefits against naive refinement-based baselines under the same evaluation protocol?
  • How does the results in Table 3 change if all 21 object classes are used in evaluation?
  • Would the comparative benefits remain consistent if more object categories or interactions are added?
Reviewer_MEjvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FACT is an agent-based framework for building articulated objects with accurate geometry, reliable contact, and realistic motion responses. It first reconstructs editable geometry through feature planning, measurements, and repeated checks. It then repairs collision proxies and reduces their complexity while preserving interaction space and valid joint motion. Finally, it uses observed motion trajectories to select response models and fit physical parameters. Experiments evaluate geometry on 21 objects, contact on 19 objects, and passive motion prediction on six real objects.

Critical strengths

The collision-proxy stage addresses a practical problem that visual reconstruction alone does not solve. A model can look correct while its collision geometry blocks a handle opening or prevents a joint from moving. FACT directly checks these problems and reports substantially higher interaction success with fewer convex hulls than the diagnostic-search baseline.

Critical weaknesses
  1. Results showing articulated reconstruction from a single real image are limited. More examples with real input images, clear views of the reconstructed geometry and joints, and comparisons with baselines would help assess how well the method works in real scenes.

  2. What are the typical runtime and VLM cost per object for each stage? The reported physics-step time measures simulation efficiency after construction, rather than the cost of constructing the asset.

  3. In the dynamics experiments, most baselines predict physical parameters directly, while FACT also adjusts them to match the observed motion. It is unclear whether the improvement mainly comes from this parameter fitting or from the agent’s changes to the motion model. A comparison using the same parameter fitting but keeping the motion model fixed would help show why the agent’s changes are needed.

Questions for authors
  1. In Figure 1, the “Contact” label is too broad and not very accurate for the open-box example.

  2. Some citations are missing:

[a] SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge [b] PAct: Part-Decomposed Single-View Articulated Object Generation [c] FreeArt3D: Training-Free Articulated Object Generation using 3D Diffusion [d] REACTO: Reconstructing Articulated Objects from a Single Video [e] VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video [f] Self-supervised Neural Articulated Shape and Appearance Models

  1. Are there results for Codex given the same Rodin meshes as FACT, and for contact evaluation using FACT’s reconstructed geometry?
Reviewer_xnKROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FACT, an agentic framework that builds simulation-ready articulated assets in three sequential stages, each running an evidence-diagnosis-revision loop in which an LLM agent picks measurements and model edits while numerical tools execute and score them. The geometry stage takes an image-generated mesh, has the agent plan parametric CAD-style features and select a measurement procedure per parameter, and emits an executable program that regenerates part meshes from features rather than copied triangles, with surface-distance and silhouette diagnostics routing errors back to replanning, remeasurement, or code correction. The contact stage freezes geometry and joints, repairs CoACD convex proxies by local repartitioning driven by task free-space and sampled-motion diagnostics, then compresses the proxy lexicographically on hull and vertex count under bounded error increases. The dynamics stage freezes everything upstream, extracts joint trajectories from passive-response videos, has the agent construct a response-model form and fit its parameters by bounded nonlinear least squares, and uses residuals to revise the model. Evaluation covers 21 assets for geometry, 19 for contact, and six real objects for held-out motion prediction, with FACT ahead of the baselines on all three.

Critical strengths

The contact stage shows that volumetric fidelity and hull count do not predict whether an interaction succeeds. In Fig. 4 the finest Component configuration uses close to ten times as many hulls as FACT and still reaches barely more than half its interaction success rate, and in Tab. 3 FACT is also the cheapest to simulate. Most work in this area treats CoACD thresholds and hull budgets as the knob to turn, and this is evidence that the knob is the wrong one.

The evaluation protocol is careful. Samples, reference labels, controllers, solver settings, and trial perturbations are shared across methods, success criteria are explicit, and the dynamics evaluation windows come from the reference trajectory alone. This is unusual for an agentic systems paper and it makes the contact and dynamics numbers mean something specific.

Critical weaknesses

The dynamics experiment compares fitting against not-fitting. FACT gets an agent-chosen response-model form plus least-squares fitting on per-object calibration clips, while all six baselines do no per-object fitting at all (App. D.4). That fitting beats guessing is not in doubt, and it does not show that the agentic part is what helps. Nothing in Sec. 4.3 tests whether the geometry and contact stages were needed for the fit either, which is what the progressive-grounding story most needs.

The geometry evidence is weaker than the headline. The large CD reduction over Codex comes from the oracle-mesh setting, i.e. fitting a program to the reference geometry it is then scored against. On image input FACT is actually worse than the Rodin mesh it was built from on SD-P95 and Avg. SIoU (Tab. 1), but Sec. 4.1 reports only that it improves "CD and surface F-scores over their upstream meshes." Tab. 2 is the same: refinement improves SD-P95 while degrading both SIoU numbers, and only the improvement is mentioned.

Sec. 1 motivates the work by the need for diverse assets at scale, but nothing in the paper shows the pipeline can get there. The study is 21 assets for geometry, 19 for contact and six real objects for dynamics, and the set is narrow as well as small: many of the assets are cabinets, drawers and appliance doors, and every fitted response model in Tab. 10 is a one-dimensional ODE in a single joint coordinate. The scaffold is also run only on GPT-6 Astra at High reasoning, and even that is stated only for the geometry stage, so the scaffold cannot be separated from the backbone. The dynamics baselines already span five models, so swapping them in would have cost little. There is no token count, wall-clock time or iteration count anywhere either, even though each revision round re-runs the numerical backend as well as the agent.

Questions for authors

How are mass and inertia obtained for each reconstructed body? "Inertial estimates" are held fixed during fitting (App. D.2, D.3) but never defined, and three of the six models in Tab. 10 are mass-normalized accelerations while the other three are torques that do depend on inertia.

How much of the result depends on the backbone, and what does the pipeline cost? The scaffold is only run on GPT-6 Astra at High reasoning, and even that is stated only for the geometry stage. Results on at least one weaker or open model, together with per-stage token counts and wall-clock time per object, would show how much comes from the scaffold rather than the backbone and whether the method can reach the asset scale Sec. 1 motivates.

Running the same dynamics calibration on a deliberately degraded twin, say the raw Rodin mesh with default CoACD proxies, would directly test whether the upstream stages matter.

App. B.2 says Codex and Articraft receive only the image while FACT folds Rodin generation into its pipeline. What does Articraft produce given the same Rodin mesh?

Every agentic result appears to be a single run. What is the variance across repeated runs at fixed inputs, and how many revision iterations does a typical object take?

The dynamics study is six objects and under three minutes of video in total, and the chest and trash can have two clips each, so at most one is available for calibration. What keeps the response-model form from overfitting, given that the form is selected by inspecting residuals on that same clip?

All contact baselines are CoACD at different granularities and settings. Andrews (2024) is cited for navigable-space preservation but not compared, and Empart (arXiv 2509.22847) assigns region-specific simplification tolerances for the same reason, so a comparison against a method that already targets task-relevant regions would be useful.

Every observed motion in Tab. 6 is a lid, door or drawer. Does the response-model construction extend to coupled or multi-DOF mechanisms, and is there a failure case worth showing?

4 comments

Dear Reviewer xnKR,

As the rebuttal discussion period is approaching its deadline, we would greatly appreciate it if you could review our responses and share your thoughts on whether they address your concerns. If our responses have satisfactorily addressed your concerns, we would also be grateful if you would consider updating your rating accordingly.

Thank you for your time and consideration.

Dear Reviewer xnKR,

As today is the final day of the rebuttal discussion, we would sincerely appreciate it if you could review our detailed responses and revised manuscript. If our additional experiments and clarifications have satisfactorily addressed your concerns, we would be grateful if you would consider updating your rating accordingly.

Thank you very much for your time and consideration.

I thank the authors for the detailed response and the new experiments. Most of my concerns have been addressed, and the additional results with DeepSeek show that the method is not limited to GPT-6 Astra. I will therefore raise my rating.

We sincerely thank you for your thoughtful feedback and greatly appreciate your time and support.

Comments on the paper

3 comments
Area_Chair_kXtcOpenReview ↗

Reminder

Dear Reviewers,

Thank you for reviewing this paper. If you haven't done so, please respond to the rebuttal as discussion deadline is approaching.

Best, AC

AuthorsOpenReview ↗

General Response to Reviewers (1/2)

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their constructive feedback and thoughtful discussions, which have helped us substantially strengthen our paper. We have carefully addressed the concerns through additional experiments, quantitative analyses, and methodological clarifications. Below, we summarize our responses to each reviewer. The corresponding manuscript revisions are detailed in our follow-up comment (2/2).

Response to Reviewer xnKR

We appreciate the reviewer's detailed questions regarding the contribution of our agentic framework beyond numerical fitting, the importance of upstream geometry and contact refinement, and the generalization and reproducibility of FACT.

To address these concerns, we introduced fitting-enabled vanilla Codex and SINDy as additional dynamics baselines, demonstrating FACT's advantages beyond direct parameter inference and conventional dynamics identification. We further investigated dynamics calibration on degraded articulated reconstructions, showing that unresolved collision and clearance errors cannot be corrected through parameter fitting alone. To evaluate robustness and generalization, we conducted three independent runs, expanded the ArtVIP evaluation, and performed cross-backbone ablations using DeepSeek. We also reported per-stage computational costs, clarified mass/inertia estimation and held-out evaluation protocols, and added coupled-linkage visualizations.

Together, these results provide stronger evidence for the effectiveness, reliability, and practical applicability of FACT's progressive framework.

Response to Reviewer MEjv

We thank the reviewer for the constructive suggestions concerning real-world reconstruction examples, computational costs, the contribution beyond parameter fitting, and experimental fairness.

We added real-world visualizations showing reconstructed geometry, articulated structures, and motion sequences. We reported per-stage runtime, token consumption, and refinement iterations to clarify construction costs. To better isolate the contribution of FACT's agentic workflow, we introduced vanilla Codex comparisons under matched contact-refinement and dynamics-fitting settings, demonstrating improved interaction success and held-out motion prediction. We also clarified the matched-input geometry evaluation, revised Figure 1 terminology, and incorporated the suggested related work.

Following the reviewer's request, we highlighted all manuscript revisions in blue. We also appreciate the subsequent suggestion regarding comparative construction costs across baselines.

Response to Reviewer 4tQq

We appreciate the reviewer's detailed feedback and constructive follow-up discussions regarding mesh-input fairness, dynamics baselines, reference-data validity, and evaluation diversity.

We clarified that Rodin serves as an upstream static mesh generator rather than an independent articulated reconstruction baseline, and explained the matched-input comparisons. To strengthen the dynamics evaluation, we added fitting-enabled Codex and SINDy baselines. We also clarified that the two objects excluded from contact evaluation have non-manifold reference geometry, independently of FACT's performance, and added visual evidence of these limitations.

Most importantly, we expanded our evaluation with 30 additional ArtVIP objects and conducted a quantitative diversity analysis across all 476 publicly released assets. Our combined selection of 43 ArtVIP objects covers all 10 identified clusters, providing stronger evidence of geometric and articulation diversity. We are grateful for the reviewer's follow-up acknowledgement that these additional results and clarifications addressed their concerns.

We sincerely thank all reviewers for their valuable suggestions. Please see our accompanying comment (2/2) for a detailed summary of the manuscript revisions.

AuthorsOpenReview ↗

Summary of Manuscript Revisions (2/2)

Dear Reviewers and Area Chair,

Following our general response to individual reviewers (1/2), we summarize the major changes incorporated into the revised manuscript. All revisions are highlighted in blue for easy reference.

Summary of Manuscript Revisions

  1. Additional Baseline Comparisons (Secs. 4.2–4.3, Tables 3–4).

    We added vanilla Codex baselines for contact refinement and fitting-enabled dynamics calibration, together with SINDy for dynamics identification. FACT outperforms these additional baselines on the key interaction and held-out dynamics metrics, strengthening the evidence for our agentic framework.

  2. Expanded Generalization and Diversity Evaluation (Appendix B.5, Table 13, Figs. 7 and 10).

    We evaluated 30 additional ArtVIP objects, demonstrating improvements over Codex across all six geometry metrics. We further analyzed our 43 selected ArtVIP objects against the 476 publicly released assets, showing coverage of all 10 identified clusters and broad geometric and articulation diversity.

  3. Reproducibility, Efficiency, and Backbone Ablations (Appendices A.3–A.4 and E, Tables 7–10 and 17–19).

    We conducted three independent runs across all three stages, demonstrating stable performance with low variability. We additionally reported per-stage runtime, token consumption, and revision iterations, and performed DeepSeek backbone ablations to demonstrate FACT's effectiveness beyond a specific foundation model.

  4. Methodological Clarifications and Visualizations (Sec. 2, Appendices A and D, Figs. 1, 8, and 9).

    We clarified physical parameter estimation, calibration and held-out trajectory separation, and non-manifold reference geometry. We also added real-world articulated structure and coupled-linkage visualizations, refined the teaser terminology, and expanded the related work.

We believe these additional experiments and clarifications substantially strengthen the empirical evidence, reproducibility, and presentation of FACT.

We sincerely appreciate all reviewers and the Area Chair for their valuable feedback and time throughout the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.