acceptodds
Withdrawn from ICLR 2027

ROLA-GS: Link-Canonical 3D Gaussian Splatting for Differentiable Robot Self-Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Differentiable robot rendering synthesizes robot images from joint configurations and enables joint-state optimization through image gradients. Although robot kinematics specifies rigid link motion, learned deformation can couple this motion with appearance variation. We introduce \method, a Gaussian visual self-model that separates rigid articulation from appearance modeling through fixed link assignments, exact relative forward kinematics, and pose-dependent appearance correction. Specifically, each Gaussian retains a fixed link assignment, allowing its center and covariance to move rigidly with the assigned link. A bounded appearance correction accounts for radiance variation without altering the transported geometry. This construction restricts each joint's spatial influence to its descendant links and provides differentiable rendering for visual inverse kinematics and selective control. Experiments demonstrate improved reconstruction across diverse robots and lower average errors in visual inverse kinematics and selective control. The results further show that rigid transport and appearance correction improve reconstruction, with rigid transport also improving forward-query efficiency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
robot self-modeling, 3D gaussian splatting, real2sim gap

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QQjDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ROLA-GS, a differentiable robot representation combining fixed Gaussian-to-link assignments, exact forward-kinematic transport, and bounded appearance correction. The representation preserves rigid link geometry and supports image-based joint-state estimation and selective control.

Critical strengths

Separating rigid articulation from appearance is well motivated and provides interpretable, structurally correct spatial derivatives. Evaluation covers diverse robot morphologies, with useful paired comparisons for visual IK and selective control.

Critical weaknesses

1,Incremental methodological novelty. Fixed link attachment, rigid Gaussian transport, and descendant-only spatial Jacobians follow naturally from known kinematics. The paper should more clearly distinguish its methodological contribution from integrating established components. 2,Comparison fairness remains uncertain. The adapted Part²GS/REArtGS++ baselines lack the supplied link assignments and FK transforms used by ROLA-GS, while the adapted 4D-GS exhibits training failures (Appendix A.9). These comparisons mix representation quality, prior availability, and adaptation quality. 3,Appearance modeling is insufficiently justified. Equation 16 depends on the Gaussian’s normal and viewing direction, which cannot generally capture shadow changes caused by independently moving links. Moreover, removing the bound improves PSNR from 35.67 to 36.19 dB, while its average IK benefit is only 0.06°.

Questions for authors

1,How were the method-specific optimization learning rates selected? Please use comparable validation-based tuning and report task error versus wall-clock time. The teaser’s 5.57× improvement measures iterations for one example, while Table 3 shows slower joint-state forward-plus-backward computation. 2,Appendix A.20 states that Table 20 and Figure 18 retain DrRobot checkpoints from before the camera-intrinsics correction. Please update this gradient comparison using corrected checkpoints. 3,Please clarify the implementation and contribution of the articulation-invariant “residual state” introduced in the Introduction, which is not explicitly defined in the subsequent formulation. 4,The revision could discuss Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splatting (NeurIPS 2025), particularly its treatment of photometric inconsistency and the resulting effects on reconstructed geometry. An adapted appearance-compensation baseline could help assess the proposed correction. 5,The adapted Part²GS/REArtGS++ baselines lack the supplied link assignments and FK transforms used by ROLA-GS, while the adapted 4D-GS exhibits training failures (Appendix A.9). These comparisons mix representation quality, prior availability, and adaptation quality.

Reviewer_F5wnOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses 3DGS to model a visual model of an articulated robot. Instead of relying learned groupings and motion such as the main comparable work Dr Robot, this work uses forward kinematics with fixed link groups to accurately model the motion with far fewer artifacts. As a robot moves, its specular artifacts should change, so this is captured via bounded appearance correction capable of taking into account the changes in the SH for motion, as well as including self-occluding artifacts. This is then evaluated photometrically as well as with joint state estimation.

Critical strengths
  1. The approach of using forward kinematics with grouped Gaussians is a very clean and wonderfully simple method to get these articulations, especially compared to the highlighted artifacts that arise with a lot of the learned methods. Usually this FK approaches struggle with changes in lighting/SH, especially when considering self-shadows and reflections, and this method nicely handles these changes.

  2. Very thorough ablation and testing campaign to isolate and motivate the impact of various components of the pipeline.

Critical weaknesses
  1. Contributions: The main weakness is the extent of the contributions/demonstrations. While the components are existing in other works, the combination seems like a very useful and new approach to this issue. That being said, these components, the relighting especially with self reflection (which from 4.4 Table 3 appears to hurt the overall reconstruction?), seem untested, or not properly evaluated in isolation or against the baselines. There are no numbers or figures showing they aid in the way are intended to.

  2. Applied evaluations: Building off the previous point, if DrRobot is the primary comparison, what of the additional experiments and comparisons they perform in 3.3? The comparisons for joint estimations are nice, but what about the CLIP poses, video based action sequences, etc. These additional experiments/comparisons would greatly strengthen the argument as at present, Table 1 only highlights SOTA metrics for 3 of the 7 robots.

Questions for authors
  1. Why is this method run at low resolution? How does the performance scale with image resolution, usually this scaling is quite good with 3DGS scenes (more dependent on the number of Gaussians being sorted rather than raw pixel count), so does this still hold up?

  2. What is the overall performance/speed of the system? I only see the per iter or forward/backwards pass speeds, what about total training time till convergence, or speeds of various components. Particularly with the visual IK, is this something that can be run in real, or does it need frozen sim time to match poses?

  3. In the field or applied settings are there any down stream tasks this is trying to tackle in particular?

  4. Do you have any independent evaluations of the self-model visual updates? The reflections or shadows parts of the robot may cast on the rest of its joints? And how far can this go, if we have a very low roughness, high metallic material, can we get actual smooth reflections off of it?

  5. I may have missed this, but how exactly are you training the splats for each link? Is it fully from observed motion sequences of the real robot (as 3.3 describes) or is there some priors from the source mesh/URDF (as Geometry-guided Gaussian reconstruction methods in 2 describes)?

Reviewer_AB4JOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

ROLA-GS proposes a differentiable Gaussian-based robot self-model that separates rigid robot articulation from appearance modeling. Each 3D Gaussian is assigned to a fixed robot link and transported using exact relative forward kinematics (FK), while a bounded, pose-dependent appearance correction captures radiance variations without deforming the underlying geometry. This structure also provides link-localized joint-state gradients for visual inverse kinematics (IK) and selective control. Experiments on real RGB data and MuJoCo simulations show improved reconstruction quality, lower visual IK and control errors than DrRobot, and faster forward rendering queries than learned deformation.

Critical strengths
  1. Strong structural inductive bias: Replaces learned deformation for robot articulation with exact FK and fixed link assignments, preserving rigid geometry and ensuring that each joint’s spatial influence is restricted to its descendant links.

  2. Broad quantitative evaluation: Tests reconstruction across multiple robot morphologies, real RGB observations and simulation, alongside visual IK and selective-control tasks. Reports a mean PSNR improvement of 0.55 dB over DrRobot across seven simulated robots under the 96-state protocol.

  3. Mechanistic ablations and practical benefits: Ablations isolate the effects of kinematic transport and appearance modeling. Exact FK improves reconstruction quality and makes forward queries 1.62× faster than learned deformation in the reported setting.

Critical weaknesses
  1. Limited conceptual novelty: The main contribution is a structured combination of 3D Gaussian Splatting, known robot kinematics, fixed part assignments, and appearance correction. The paper should more clearly establish what is fundamentally new relative to existing differentiable robot renderers, particularly DrRobot and related link-aware representations.

  2. Dependence on accurate prior information: The method assumes a known kinematic model, fixed link assignments, and calibrated cameras. Its robustness to imperfect kinematics, calibration errors, or inaccurate nominal geometry is not established.

  3. Limited real-world task validation: Real RGB experiments primarily validate reconstruction quality. Visual IK and selective control are evaluated on controlled benchmarks, largely in simulation; hardware closed-loop control remains future work. Thus, the practical impact on real robot control is not yet demonstrated.

  4. Trade-off in appearance modeling: The bounded appearance correction improves the separation between geometry and radiance, but the ablation shows that removing the bound yields better held-out PSNR. The benefit of the constraint beyond enforcing the intended modeling structure could be better justified.

Questions for authors

no

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.