acceptodds
Under review as a conference paper at ICLR 2027

ArticulateArena: A Metric for Articulated Kinematics

Abstract

Modern methods reconstruct or generate simulation-ready articulated objects, predicting not only their geometry but also how their parts are connected and allowed to move. Evaluating the geometry is straightforward, but evaluating the predicted articulation is not, because articulation specifies a motion rather than a shape, and there is no agreed distance between two motions. More specifically, existing protocols score joint type, axis direction, origin, and motion limits separately, although these parameters jointly describe a single physical motion, and the same motion can be written as different parameter values. As a result, a joint can score maximally wrong against an equivalent encoding of itself, and several component errors are ill-conditioned or undefined exactly where predictions become accurate. We propose ArticulateArena, a representation-invariant counterpart of Chamfer distance for articulation that compares the motions one-DOF joints induce rather than the parameters that encode them. It represents each joint by the unordered pair of its Lie-algebra endpoint twists, and we prove that the resulting quotient distance is a metric. It unifies fixed, revolute, prismatic, and helical joints, brings continuous joints into the same score through a compactification, and reads as the RMS motion of the moving part in meters when weighted by its mass distribution. A motion-aware tree edit distance lifts the metric to full kinematic trees, pricing structural errors such as spurious or missing joints in the same motion units as joint errors, and for a fixed inner product it remains a metric on trees up to relabeling. Alongside the metric we release ArticulateArena-20K, a new suite of 19,977 articulated objects with verified kinematics, and we re-evaluate published reconstruction methods on it under the new metric.

Primary area:
datasets and benchmarks
Keywords:
articulated object, articulated kinematics, evaluation metrics, 3D vision, robotics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_5LqNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ArticulateArena, a framework for evaluating articulated-object kinematics. It represents joints using endpoint twists and defines a quotient metric that accounts for equivalent parameterizations, extends to continuous joints, and compares kinematic trees through motion-dependent edit costs. The paper also assembles ArticulateArena-20K and evaluates eight methods on a 200-object subset, alongside controlled perturbation studies and metric ablations.

Critical strengths
  1. Principled treatment of kinematic equivalence. The formulation addresses meaningful limitations of conventional parameter-wise errors, including equivalent joint encodings and discontinuities near fixed joints, with theoretical analysis and controlled demonstrations.

  2. Useful physical grounding and evaluation resources. The kinetic formulation connects joint errors to material displacement, supported by calibration against finite rigid-body motions. The standardized asset library and reevaluation of existing methods provide a useful foundation for more consistent evaluation.

Critical weaknesses
  1. Limited validation of the final object-level metric. Figure 4 supports the single-joint kinetic metric under controlled perturbations, but does not establish that the compactified tree metric reliably measures whole-object motion quality. Its behavior under coupled parameter errors, large rotations, and incorrect kinematic structures remains insufficiently validated.

  2. Unclear sensitivity to joint–part binding and error propagation. Matching trees with unlabeled nodes may overlook incorrect associations between joints and physical parts. Moreover, the metric’s ability to capture upstream errors affecting multiple downstream parts is unclear. Results stratified by movable-joint count and tree depth are missing.

  3. Unclear added value of the 20K asset library. The library aggregates and standardizes substantial existing resources, but its additional evaluation value is not convincingly demonstrated. The 200-object evaluation subset emphasizes semantic coverage without sufficiently establishing coverage of kinematic difficulty, rare joint types, or structural complexity.

  4. Limited diagnostic insight. The experiments demonstrate changes in method rankings, but provide limited evidence that these changes better reflect actual motion quality. The benchmark offers few systematic conclusions about failure modes, scaling with structural complexity, or the remaining bottlenecks in articulated-object generation.

  5. Incomplete coverage of methods and backbone effects. Relevant methods such as PhysX-Omni [1], SIMART [2], SINGAPO [3], PWM-ArtGen [4], Sketch2Arti [5], and LAM [6] are absent. Controlled comparisons across foundation models and against direct generation are also missing, limiting insight into framework contributions versus backbone capabilities.

References:

[1] PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects.
[2] SIMART: Decomposing Monolithic Meshes into Sim-ready Articulated Assets via MLLM.
[3] SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects.
[4] PWM-ArtGen: Part World Model for Articulated Object Generation.
[5] Sketch2Arti: Sketch-based Articulation Modeling of CAD Objects.
[6] LAM: Language Articulated Object Modelers.

Questions for authors
  1. Can you validate the final tree metric against whole-object displacement errors computed through forward kinematics across multiple joint configurations, using actual method outputs? Where does agreement deteriorate, particularly under large rotations and simultaneous structural and parameter errors?

  2. Does tree matching constrain joint–part correspondence using geometry? Can it distinguish correct joint parameters attached to the wrong parts, and capture upstream errors propagated to descendants? Please report results by movable-joint count and tree depth, and clarify the effect of aggregation and normalization.

  3. What evaluation capabilities does the full 20K library provide beyond its constituent datasets? Please clarify the reproducible selection protocol for the 200 objects and their coverage of joint types, structural complexity, and source datasets, including potential overlap with baseline training data.

  4. Beyond changed rankings, what actionable conclusions does the benchmark reveal about existing methods? Can you analyze disagreement cases and demonstrate that the proposed ranking better reflects whole-object motion quality than existing metrics?

  5. Could you include the additional methods above under clearly specified input protocols? For Articraft, could you compare alternative backbones, such as GPT-6 Astra and Opus 5.5 where available, against direct generation by the same models under matched conditions to isolate framework and backbone contributions?

Reviewer_85yeOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a novel parameterization and evaluation method that provides a unified way to represent the motion characteristics of movable parts and assess their reconstruction quality in articulated object generation tasks. Previous work on articulated object generation typically evaluates geometric reconstruction quality and motion characteristics separately. The latter are usually assessed using multiple independent metrics, such as joint-type classification error, angular error of the joint axis, joint-origin error, and motion-range error.

To evaluate these motion characteristics, the authors use a parameterization based on Lie algebra to provide a unified representation of rotational and translational motion characteristics and joint limits. They then use a quotient distance to assess the reconstruction quality of the motion characteristics for 1-DoF joints. This metric expresses results on a common scale, is invariant to equivalent encodings of generated axes, and can be extended to evaluate articulation-tree matching for complete generated objects. As a unified evalutaion benchmark, the authors provide a dataset containing 19,977 articulated objects. Comparative experiments on this dataset demonstrate that the proposed metric exhibits good continuity and discriminative power.

Critical strengths
  1. The authors use a distance in Lie algebra space to provide a unified measure of joint motion reconstruction error in articulated objects. This provides an alternative to separate angular and distance error metrics for joint parameters and supports consistent evaluation of joint motion under a common protocol. It also better handles edge cases that are challenging for traditional evaluation approaches, such as generating symmetry-equivalent axes or motion ranges, or predicting a joint with a very small motion range for a fixed part.
  2. The authors provide ArticulateArena-20K, a diverse dataset for benchmarking articulation reconstruction and supporting reliable comparisons across configurations. The dataset statistics show high object diversity, broad category coverage, and a diverse range of joint types. This provides a reliable and practical evaluation benchmark for future work.
Critical weaknesses

The core metric used in this method, namely the quotient distance, evaluates reconstruction quality only at the level of individual joint pairs and is based on a simplified linkage representation of joints and movable parts. This may limit the contribution of this work in two ways:

  1. This metric cannot replace geometric reconstruction metrics or collision-based metrics. Therefore, in the ArticulateArena evaluation, the proposed metric can replace only some joint-level evaluation metrics, while additional geometric reconstruction and collision detection metrics are still required. In other words, its contribution may be relatively limited in scope. Nevertheless, it still makes a solid contribution by addressing the problem of standardized evaluation of joint reconstruction.
  2. As discussed above, this metric also cannot directly evaluate the articulation tree of the final generated object. The authors address this issue using a metric based on edit distance to compare two different articulation trees. However, the proposed algorithm for computing this edit distance does not appear to guarantee a deterministic solution in polynomial time, and its approach is similar to traditional methods. In other words, the paper’s contribution to evaluating articulation trees as a whole appears relatively limited.
Questions for authors
  1. In future ArticulateArena evaluations, will it still be necessary to include metrics based on geometry and collision detection in addition to the proposed metrics?
  2. ArticulateArena appears to include data from multiple open-source datasets, as well as data generated by the authors. Could the authors provide statistics for these different data sources?
  3. ArticulateArena includes data that the authors generated using models. Has the reliability and realism of this portion of the data been checked?
  4. The top-k lists obtained using the new and traditional metrics in Table 4 appear to differ. For example, when selecting the top-1 method for each metric, the traditional metrics yield {Particulate, Articulate AnyMesh, Articraft}, whereas the new metrics yield {Particulate, Articulate-Anything}. Could the authors provide some qualitative results on the test set to support these difference?
  5. (minor) Could the authors provide a computational complexity analysis of Algorithm 2 in Appendix A.3 to clarify whether the metric in Section 4.3 can be consistently computed within practical runtimes for typical data sizes?
Reviewer_ACtkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper makes two primary contributions:

  1. It proposes a new evaluation metric for the kinematic reconstruction of articulated objects, designed to measure the actual discrepancy in motion between predicted joints and the ground truth (GT).
  2. It introduces the ArticulateArena-20K dataset, which contains approximately 20,000 articulated objects with kinematic annotations, providing a unified benchmark for evaluating different methods.
Critical strengths
  1. It addresses a real-world problem that has long been overlooked. Current evaluation practices for articulated reconstruction typically calculate errors for joint axes, origins, types, and limits separately. These metrics fail to accurately reflect whether a reconstructed object can actually move in the same way as the ground truth. The proposed metric, by contrast, directly measures the reconstruction error of the joint motion itself.
  2. The dataset is large-scale, and the experimental evaluation is comprehensive. ArticulateArena-20K comprises approximately 20,000 objects, spanning 628 categories and five types of joints, all standardized into the URDF format. Furthermore, the authors re-evaluated eight existing methods, analyzing the differences between the new and traditional metrics based on actual algorithmic outputs, rather than relying solely on constructed mathematical counterexamples.
Critical weaknesses
  1. The new metric is not equivalent to the true physical motion error. Although the paper demonstrates that endpoint distance can constrain the entire Lie algebra motion segment, it is not identical to the true motion distance on the SE(3) manifold—a point the authors acknowledge in Appendix B.4. Linearization errors may become more pronounced in cases involving large rotation angles, complex geometry, or extensive ranges of motion.
  2. The new metric still incorporates design choices. While the authors emphasize that their approach reduces the reliance on arbitrary thresholds found in traditional evaluation methods, several choices remain: the α\alpha parameter in the split norm, the compactification scale, and the decision of whether to apply geometry-aware weighting.
  3. There is a lack of downstream validation, making it difficult to definitively claim that the new ranking is "more correct." While the authors show that the new metric produces different rankings for existing algorithms compared to traditional metrics, it remains unclear why the new ranking is inherently more accurate. For instance, the paper notes that "Articulate-Anything" performs well under the object-level tree metric, but this does not automatically prove that the articulated assets it generates would perform better in a physical simulation.
Questions for authors
  1. Is there empirical evidence demonstrating that the proposed metric correlates better with downstream performance—such as the simulation accuracy of articulated objects or the success rate of robotic manipulation tasks? Providing such evidence would help establish the metric's practical advantages, beyond its mathematical consistency.
  2. While ArticulateArena-20K contains 19,977 objects, the primary evaluation is based on only 200 objects, with just 67 objects common to all eight methods. Is this subset sufficiently representative, and would the observed rankings remain stable if evaluated on a larger or differently sampled subset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.