acceptodds
Under review as a conference paper at ICLR 2027

Structure-Guided Alternating Inference for Monocular Robot State Refinement

Abstract

Estimating the pose and joint angles of an articulated robot from a single RGB image supports spatial reasoning without access to its internal measurements. Existing approaches use learned visual representations to predict the robot state, sometimes with iterative refinement. However, articulation and global pose can produce similar image changes, making it difficult to determine which variable should correct a remaining discrepancy. To address this, we introduce Structure-Guided Alternating Inference (SGAI), a training-free method that formulates state refinement as bounded geometric optimization using robot geometry and perceptual cues extracted by visual foundation models. Specifically, SGAI solves separate articulation and pose subproblems while holding the other block fixed, preventing compensating changes between the two blocks within each solve. Point-to-link, silhouette, and projection anchor constraints assign geometric evidence to the appropriate updates and limit corrections driven by imperfect depth and foreground predictions. Re-rendering and renewed geometric associations allow each block to use the shape or placement corrected by the other. Experiments on DREAM show improved 3D localization in most tested settings across two initializers and three robot models, using a shared configuration without additional training or initializer- or robot-specific retuning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Joint Angle Estimation, Camera-to-Robot Pose Estimation, Alternating Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZTpEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SGAI, a training-free method for refining the output of monocular robot state estimators (joint angles and robot-to-camera pose). Given an initial state from RoboPose or RoboPEPP, it alternates between two bounded optimization steps:

  • An articulation step with pose fixed.
  • A pose step with articulation fixed.

The losses use three kinds of evidence: point-to-link consistency from Depth Anything 3 points, silhouette consistency from RobotSeg masks, and a projection anchor. The motivation is a Jacobian conditioning analysis, which suggests that joint and pose updates can compensate for each other. The method is evaluated on the nine DREAM settings with one shared hyperparameter set. ADD-AUC improves in 6/9 settings for RoboPose and 7/9 for RoboPEPP, with the largest gains on Panda AK/XK (up to +2.8 pp). Ablations cover the constraints, joint versus decoupled updates, the number of rounds, and initialization perturbations.

Critical strengths
  1. Practical and modular. SGAI is a plug-in, training-free refiner that works on the state output of different estimators. It uses one fixed configuration and does not touch the initializer's features. This is useful and easy to adopt, and the paper reports its limitations honestly.

  2. Informative ablations. The paper includes ablations on constraints, update strategy, rounds, and initialization perturbation, together with a Jacobian diagnostic motivating the decoupling. The mean-ADD and rendered-depth results go beyond ADD-AUC alone.

Critical weaknesses
  1. Gains are small and inconsistent, and the evidence does not isolate the claimed mechanism.

    • Most ΔADD-AUC values are below 0.4 pp, and several are negative. KUKA and Baxter are essentially unchanged. The "6/9 and 7/9" counting overstates the benefit.
    • No seeds, confidence intervals, or significance tests are given. This matters because differences of 0.02–0.3 pp are within plausible noise.
    • The cost is about 2.2 s per sample, versus 0.31 s for initialization.
    • Table 6 shows that nearly all of the gain comes from the pose update. For Panda XK, the articulation-only change gives 78.49→78.63, while the pose-only change gives 81.17. This undermines the central claim that structure-guided articulation–pose alternation is what helps. The paper also never reports joint-angle errors.
    • The refined results do not outperform the published SOTA (e.g., RoboTAG) in most settings.
  2. Soundness of the motivation and experimental protocol.

    • The Jacobian diagnostic uses only 14 keypoint projections. The actual objective uses 3D points and silhouettes, which may already resolve the ambiguity. The diagnostic therefore does not show that joint optimization of the real loss is ill-conditioned.
    • The joint-versus-decoupled comparison is on one dataset (XK, the best case), without variance. It is also confounded, because the decoupled objectives use different loss terms and the learning rates are untuned for the joint case.
    • The "no robot- or baseline-specific retuning" claim is unclear. The reference implementation and hyperparameters are Panda-specific (e.g., joint scales, an auxiliary seventh joint). It is not stated how hyperparameters were selected, and no validation split is described, which raises a risk of tuning on test data.
    • There is no oracle analysis (GT depth or mask) and no ablation of the DA3 and RobotSeg components. It is therefore unclear how much the method depends on foundation-model quality, which the paper itself flags as a limitation.
    • The method also lacks comparison with simpler refinement baselines, such as pose-only silhouette or render-and-compare refinement, or GISR.
Questions for authors
  1. How were the hyperparameters and acceptance thresholds chosen, and on which data? Please report results on a held-out split, or with hyperparameters tuned on a non-evaluation set.

  2. Can you report standard deviations or confidence intervals (e.g., bootstrap over images) for Table 1 and Table 3? Which of the small ΔADD-AUC values are significant?

  3. Given Table 6, how do you justify the claim that articulation updates matter? Please report joint-angle error before and after refinement, and a pose-only refinement baseline using the same losses.

  4. Can you show that the joint-versus-decoupled gap holds on other datasets and robots, with the learning rate and step budget tuned for the joint variant, and with identical loss terms?

  5. What is the total runtime including DA3 and RobotSeg? How do results change with GT depth or masks, or without each foundation model?

  6. Why does refinement hurt on some settings (e.g., DR, RoboPEPP KUKA DR vs. its published value)? Can acceptance rules prevent these degradations?

Reviewer_ZgZgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes SGAI, a training-free refiner for monocular robot state estimation. Starting from RoboPose or RoboPEPP outputs, it alternates bounded articulation and pose updates using depth and mask cues from foundation models. With one fixed configuration, it improves ADD-AUC on DREAM in 6/9 (RoboPose) and 7/9 (RoboPEPP) settings.

Critical strengths

The method is a plug-in, training-free refiner that uses one shared configuration across two initializers and three robots, and the paper also reports the settings where it hurts. Quantitative evaluation show improvement over baseline.

Critical weaknesses

I do not see much new insight for the ICLR audience unfortunately. The method is a training-free optimization that combines established ingredients: silhouette and depth alignment with off-the-shelf foundation-model cues, plus alternating (block-coordinate) updates of joints and pose. There is no learned component, and the paper does not offer a new understanding of the problem beyond the observation that decoupled updates are more stable than joint ones.

The gains are small, about +0.5 pp ADD-AUC on average, and appear to come mostly from the pose update. The evidence for these gains is also limited: the paper does not explain how the hyperparameters were chosen, all ablations are on Panda XK only (the setting with the largest gain), and there is no comparison with other refinement methods such as GISR.

Questions for authors

How were the hyperparameters chosen, and do the ablations hold on other settings and robots?

Reviewer_tVsSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SGAI to estimate robot articulation and robot-to-camera transformation. The authors try to decoule a single set of states (articulation angles, robot-to-camera transformation) into two components. For articulation, point-to-link consistency, silhouette consistency, boundary coverage objective terms are employed, and for transformation estimation, boundry coverage and projection anchor consistency are used.

This approach is a trainig-free approach, so it can be easily applied to a baseline (in this paper, this is applied to RoboPose and RoboPEPP, respectively), demonstrating the performance enhancement.

Critical strengths

This research explore how to overcom imperfect visual predictions by leveraging robot structure (i.e., the shape/appearance of the robot observed in the image). In particular, their problem definition is really clear so that it is easy to understand.

Critical weaknesses

I'm sorry to say that, but this paper has a few critical weakness as follows:

Weakness 1: I don't understand the the practical motivation that joint angles are unknown. While the authors assume that, in this situation, joint measurements are not available (by saying "without access to joint measurements"), this approach requires robot’s URDF, link geometry, and camera intrinsics. If this paper become more distinct and has an impact, it should work even if we have no information about URDF or link geometry.

Weakness 2: While the authors mentioned that "Visual foundation models offer dense depth and foreground cues for refinement, but these predictions are imperfect," but in the methodology, SGAI heavily relies on the quality of masks and point cloud quality. This approach's main argument is robustness against imperfection of vision foundational model, but as far as I feel, there's no thorough experiments that support it. Thus, validation of robustness is bit insufficient.

Weakness 3: while adding loss terms in Eqs. (4)--(5) can be considered a novelty, but most things are deployed from other papers, and alternating optimization itself is not a novel concept neither.

Questions for authors

[Q1] Question along weakness 1: What is the potential applications that needs articulation estimation from an external RGB image while assuming access to the robot’s URDF, link geometry, and camera intrinsics, but not its joint measurements?

  • The method also heavily relies on the strong assumption that the initial state estimate is already close to the true state, which limits its applicability when initialization errors are large.

[Q2] Question along Weakness 2: In general, in domain randomization (DR) setup, performance increase is not significant, and rather performance is degraded, as presented in Table 1. In this respect, I suspect SGAI manages the imperfections of visual cues. Futher analyses are needed.

[Q3] Question along Weakness 3: How can we guarantee the optimality of this alternating optimization? I think this concern is very relevant to the limitations that author raised (i.e., sensitivity of the initial conditions of the states)

[Q4] more through analyses about robustness should be analyzed. For example, it is meaningful to analyze the performance changes with weaker depth/segmentation models compared to DA3 and RobotSeg, respectively, or we can add some controlled noises (e.g., multipling wrong scales [0.95, 1.01] to depth, putting some noises, deliberately blocking some masks, etc.). I'm curious at what level of input degradation refinement stops improving the initial estimate or begins to make it worse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.