acceptodds
Under review as a conference paper at ICLR 2027

: Action-Differentiable Neural Objects

Abstract

Learned 3D object representations are typically evaluated by reconstruction accuracy. However, accurate geometry does not guarantee that the representation provides useful gradients for optimizing physical interaction. We ask whether a learned object representation can serve directly as a differentiable interface from geometry to action. We introduce , Action-Differentiable Neural Objects, a structured neural representation that supports diverse articulated, skinned, and deformable object identities. Conditioned on sparse keypoints, provides continuous occupancy, signed-distance, and surface-normal queries at arbitrary 3D locations. The same neural object is used in two directions: geometry recontruction given keypints during representation learning, while after training, task gradients could flow backward through frozen geometry to optimize keypoints for specific interaction. achieves 94.2% simulated grasp success and 78.8% OOD real-world success, while representing up to 100 objects with a compact 1.9M-parameter shared model. These results suggest that neural object representations can be designed not only to reconstruct geometry, but also to expose useful derivatives for physical interaction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D Reconstruction, Object Representation, Robot Policy Training, Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_49KdOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes ∂Oθ, a shared, structured neural object representation conditioned on object structure and configuration. It extends a WorldString-style neural field with occupancy, signed-distance and surface-normal outputs, using a PCA-based decomposition of keypoint motion to distinguish persistent structure from changing configuration. Once trained, the neural field is frozen and used to propagate geometric gradients into dexterous-hand configurations; optimized trajectories then supervise a policy evaluated in simulation and on a real robot.

Critical strengths
  • The manuscript explicitly distinguishes network-parameter learning from configuration optimization.
  • The use of signed distances and spatial derivatives provides a reasonable mechanism for contact and penetration objectives.
  • The PCA-based structure/configuration decomposition is motivated by the limited availability of fine-grained kinematic annotations.
  • The method addresses a useful problem: a shared geometric model should be useful not only for reconstruction but also for downstream optimization.
  • The paper acknowledges that it differentiates through geometric queries and kinematic updates, not through full contact dynamics. This is an important scope distinction.
Critical weaknesses
  1. The method explicitly builds on WorldString's structured neural object formulation. Its extensions include multi-object conditioning, PCA-based structural descriptors, SDF and normal prediction, and downstream optimization. The manuscript should identify which of these changes are essential to actionability and which are engineering extensions.
  2. The method uses observed keypoint variation to estimate low-dimensional motion patterns and a complementary structural descriptor. This is sensible when configuration changes occupy a low-dimensional subspace. However, the manuscript does not establish how robust this assumption is for nonlinear articulation, discontinuous changes, sparse observations or objects whose geometry changes independently of their selected keypoints.
  3. The manuscript's three-stage training pipeline includes occupancy learning, SDF post-training and normal fine-tuning. The ablation shows that normal error falls from 49.11° after SDF post-training to 18.53° after normal fine-tuning on the rigid-object setting. This is evidence that gradient quality requires explicit attention. But the paper does not establish how accurately the learned gradients match gradients derived from ground-truth geometry, nor whether gradient quality predicts optimizer convergence and final grasp success.
  4. The appendix states that each method optimizes its own objective. ∂Oθ uses contact, penetration, finger/palm proximity, fingertip attraction, joint priors and closure terms. This makes the comparison difficult to interpret: improvements may result from a more effective objective or constraint configuration rather than the representation itself. A fair comparison should separate representation, objective and optimizer effects.
  5. The manuscript differentiates through neural geometry and kinematic updates, not through contact dynamics. Consequently, it should avoid implying that the model directly optimizes fully physical trajectories or captures force-dependent interaction dynamics. The current framing is mostly careful, but terms such as “action-differentiable” should be defined operationally.
Questions for authors
  1. What component of ∂Oθ produces the action improvement that cannot be obtained from WorldString with a separately trained SDF head?
  2. How closely do predicted SDF gradients align with ground-truth mesh or SDF gradients near contact?
  3. How sensitive is grasp success to PCA rank and keypoint perturbations?
  4. Does the method retain its advantage when all methods use the same objective, optimizer, initializations and stopping rule?
  5. What happens when the object is outside the observed configuration subspace?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 49Kd

Dear Reviewer 49Kd,

Thank you for the questions about contributions, gradient quality, and evaluation. We address them with additional controlled experiments and clarify the scope of each component.

W1/Q1. Contribution beyond an SDF head

Shared structural conditioning supports reconstruction across object identities; distance and normal training support action optimization through frozen geometry. We compare A: an SDF head trained on a frozen occupancy backbone; B: joint SDF/backbone training with accompanying training refinements; C: B plus normal supervision. All use identical grasp objectives and optimization settings, with three seeds and 30 trials per object per seed.

ConfigurationSuccessful graspsSuccess
A171/36047.50%
B310/36086.11%
C355/36098.61%

These results support the contribution of the training recipe beyond fitting a separate SDF head. The gain from B to C supports the value of normal supervision. Table 5a separately evaluates structural conditioning for shared reconstruction.

W3/Q2. Spatial and action gradients

Our normals are normalized spatial SDF gradients, so angular error against ground-truth normals measures gradient-direction accuracy. Table 5b reports a reduction from 49.11 to 18.53 degrees on 100 rigid objects after normal fine-tuning.

On the four grasp objects, we additionally compare gradients of the grasp objective with respect to QPose against those computed using ground-truth geometry. The table gives cosine similarity and, for C, the fraction of runs in which an update along the neural gradient decreases the evaluated objective.

ObjectABCDescent rate
Apple0.2180.8020.94098.89%
Pig0.5750.9150.974100%
Trophy-0.3380.7810.94995.56%
Duck0.3010.9790.985100%

Improved task-gradient agreement accompanies higher grasp success across these matched configurations.

W4/Q4. Common optimization protocol

The original system comparison preserves each method's native objective. We now also compare adaptations with a shared objective, optimizer, initializations, stopping rule, and budget, retaining each method's geometric queries: mesh-based Phong SDF for Fast-Grasp'D, point-to-mesh distances for DexGraspNet, and frozen neural fields for ours. Each method uses three seeds and 30 trials per object per seed. Four-object averages are:

MethodGT geometry lossPenetration (mm)Success
Fast-Grasp'D-common3.1982.75775.8%
DexGraspNet-common3.0013.81964.2%
Ours-common0.9671.17598.3%

Our advantage persists under the common protocol, complementing the original system comparison.

W2/Q3/Q5. PCA and keypoint robustness

PCA summarizes training keypoint variation to condition shared reconstruction. Grasp experiments use separately trained hand/object models without shape-code conditioning and optimize QPose through forward kinematics. PCA rank therefore pertains to the reconstruction setting. We will clarify this distinction in the paper.

Three-seed policy tests cover 2, 5, and 10 mm keypoint noise. At 10 mm, ID/OOD success is 100/98.33% for apple, 98.33/95% for pig, 88.33/85% for trophy, and 100/96.67% for duck, compared with clean rates of 100/100%, 100/100%, 95/88.33%, and 100/95%, respectively.

For reconstruction, we test configurations with large PCA residuals. Let k stack K keypoints, mu be the training mean, and C = I - U_r U_r^T be the saved complement projector. The residual in model voxels is

e(k) = 256 * ||(k - mu)^T C||_2 / sqrt(3K).

For each object, the common group comprises three held-out poses nearest the training median residual. The corner group comprises the three largest held-out residuals, each above the training 95th percentile; the groups are disjoint. Occupancy F1 (%) changes as follows:

ObjectCommonCorner
Laptop95.7994.62
Folding rule96.2396.81
Felidae90.2090.34
Female97.3096.58
Cube95.7293.21
Foam roll95.8193.52

These selected configurations retain strong occupancy accuracy. Soft objects show greater distance and normal sensitivity: normal error rises from 17.94 to 22.74 degrees for cube and 19.57 to 25.83 degrees for foam roll. Our scope is geometry captured by the selected keypoints and training coverage; this test quantifies robustness beyond typical training residuals.

W5. Meaning of action-differentiable

Task gradients flow through frozen neural geometry and forward kinematics to hand configuration variables. The term refers specifically to geometric and kinematic derivatives, with physical grasp stability assessed in simulation and on the robot. We will make this definition consistent throughout the paper.

Reviewer_keG6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on learned 3D object representations for physical interaction optimization. The authors argue for the need for a learned object representation that can serve directly as a differentiable interface from geometry to action beyond accurate reconstruction accuracy. To this end, they introduce ∂Oθ\partial O_\theta, Action-Differentiable Neural Objects. Evaluated on multi-object geometry reconstruction, simulated grasping, and real-world out-of-distribution robotic grasping benchmarks, ∂Oθ\partial O_\theta achieves 94.2% simulated grasp success and 78.8% OOD real-world success, while representing up to 100 objects with a compact 1.9M-parameter shared model.

Critical strengths
  1. The proposed ∂Oθ\partial O_\theta representation supports diverse articulated, skinned, and deformable object identities, and it can serve as a differentiable interface bridging geometry reconstruction and action optimization.

  2. The paper freezes trained neural object weights and propagates task gradients through geometric queries to optimize keypoints for physical interaction, and reports quantitative results across occupancy, SDF, and surface-normal geometric outputs.

Critical weaknesses
  1. This work addresses the task of dexterous robotic grasping; however, it fails to provide data on the runtime or computational efficiency of its neural field inference and grasp optimization processes. The absence of such timing statistics makes it impossible to determine whether the method can meet the practical requirements for online deployment on a real robot.

  2. The paper does not specify whether external pre-trained weights were reused; therefore, it is impossible to determine whether the performance gains stem from the method itself or from the transfer of pre-trained weights.

  3. In Figure 3, there are multi-colored dots in both the "XHand" and "Object code" sections. What do these dots represent?

  4. Many abbreviations are not defined upon their first appearance, which reduces the readability of the entire manuscript.

  5. This system relies heavily on high-quality, sparse keypoint inputs provided from an external source; it lacks any built-in module for predicting keypoints directly from raw sensor data. Consequently, without an external keypoint acquisition module, the entire pipeline cannot operate directly on raw images or point clouds. Furthermore, it remains to be determined how robust the system is to noise in these keypoints, and whether such noise might adversely affect geometry reconstruction or action optimization.

Questions for authors

Please see the "Critical Weaknesses".

1 comment
AuthorsOpenReview ↗

Response to Reviewer KeG6

Dear Reviewer KeG6,

Thank you for highlighting the deployment and presentation details. We clarify them below and provide additional robustness results.

W1. Runtime and deployment

Grasp optimization generates demonstrations offline. At deployment, the trained policy directly maps observed keypoints to actions. Timings for the duck setting are:

OperationMedianP95
Neural-field query (hand + duck)3.053 ms3.088 ms
Policy forward pass14.931 ms15.038 ms
Offline grasp optimization20.169 s20.549 s

Thus, iterative optimization is an offline data-generation cost; online computation uses policy inference. The reported policy latency measures the network forward pass, with sensing and communication contributing separately to the control cycle. We will add these timings to the implementation details.

Real-world execution videos of our method on all four evaluated objects are available on our anonymous demo page. All videos are played at their original speed (1×) to demonstrate the efficiency of our real-world deployment.

W2. Pretrained weights

Our neural object models and policies start from random initialization, without external pretrained weights. The MuJoCo-trained and Isaac Lab-trained policies also start from scratch and share our policy architecture and training budget. We will state this explicitly.

W3-W4. Figure and notation

The colored dots in Figure 3 denote structural keypoints; the pink bar denotes object code z. We will clarify the caption and expand abbreviations at first use: signed distance function (SDF), forward kinematics (FK), linear blend skinning (LBS), finite element method (FEM), and in-/out-of-distribution (ID/OOD).

W5. Keypoint sensing and robustness

Our interface takes observed keypoints, supplied by motion capture in the real-world setup. Our policy and the two simulator-trained policies share this sensing interface. Policy training includes keypoint perturbations. Additional simulation tests use three seeds and noise levels of 2, 5, and 10 mm; entries below report ID/OOD grasp success (%).

ObjectClean2 mm5 mm10 mm
Apple100/100100/100100/100100/98.33
Pig100/100100/100100/98.3398.33/95
Trophy95/88.3393.33/88.3390/83.3388.33/85
Duck100/95100/98.33100/98.33100/96.67

Separate reconstruction tests with 1% keypoint perturbations retain occupancy F1 of 91.70-97.53% across five objects. XHand's normal error changes from 10.19 to 11.99 degrees. Together, these results support tolerance to imperfect keypoints under the tested perturbations. We will clarify the sensing interface and include these results in the revision.

We welcome any further questions and would be happy to continue the discussion.

Reviewer_VpmUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a neural object representation that supports both object reconstruction and action optimization. The experimental results demonstrate that the proposed method can reconstruct diverse objects using a shared model and improve grasping success rates through action refinement.

However, the experimental evidence does not sufficiently support the paper's central claim that the proposed representation is particularly suitable for action. Furthermore, the manuscript does not fully answer why we need to use this representation on both object reconstruction and action refinement.

Critical strengths
  • The authors extend their previous single-object reconstruction approach to multiple objects while introducing only a modest increase in the number of model parameters.

  • By modeling both the hand and the object in a differentiable manner, the authors successfully optimize hand actions.

Critical weaknesses

Unclear advantages of the proposed object representation for action optimization

Although the Introduction suggests that the paper proposes an object representation particularly suitable for action optimization, the methodology appears to focus primarily on optimizing robot hand poses through a differentiable representation. Since similar optimization procedures could be implemented using various differentiable rendering or representation methods, it remains unclear what specific advantages the proposed representation offers over existing alternatives. Furthermore, it can be computed directly using object and hand meshes without relying on neural representations. Why do we need to use this representation on both object reconstruction and action refinement?

Limited contribution to object representation

The contribution to object representation appears to be relatively limited. The authors extend the object-wise representation of the previous work, WorldString, to multiple objects and improve reconstruction accuracy by introducing a normal-related loss. However, these modifications do not appear to be the key factors enabling the claimed action-differentiable representation. Instead, the proposed method seems to largely inherit the advantages of the original WorldString representation, making it unclear what additional benefits the proposed modifications provide for action optimization.

Additionally, the extension to world coordinates does not appear to account for rotations. (They only consider the global object position c) The authors should clarify how they handle rotations.

Insufficient explanation of keypoint extraction and motion analysis

The paper does not clearly explain how the keypoints are obtained, making it difficult to assess several aspects of the proposed method.

  • The proposed method relies heavily on object-specific keypoints. However, the paper does not discuss how reliably these keypoints can be extracted from unseen objects, raising concerns about the generalizability of the proposed approach.

  • Is the number of keypoints obtained from each observation consistent? In particular, is a one-to-one correspondence between the sets of (K_i^m) guaranteed? If not, how are the mean shape and offsets computed?

  • The PCA-based analysis appears to capture primarily linear translational motion. Can it also accurately characterize rotational motion, such as the movement of a door around its hinge?

  • It is difficult to understand how (Q_i) is computed from the explanations and equations provided in the paper. A more detailed derivation or explanation would be helpful.

Unclear reconstruction setup in real-world experiments

It is unclear whether the real-world experiments use reconstructed hand and object representations or ground-truth geometry. If reconstructed representations are used, the authors should explain how they are obtained, including the reconstruction procedure and the information available during reconstruction.

Questions for authors

Several important terms are not clearly defined.

  • What exactly does "configuration" refer to? Does it represent the 6D pose, mesh geometry, or articulated joint information?
  • What are the object keypoints, and how are they obtained?
  • What does the term "hand probe" refer to? Is it a linear projection, or does it involve using hand keypoints as probes?
  • What is the input format of an observation? Does the method assume RGB images, RGB-D data, or another type of input?**

Is the proposed method computationally efficient?

Table 4 reports only the number of optimization steps required for convergence. However, fewer optimization steps do not necessarily imply shorter convergence time. How much computation time is required per optimization step? How does the total optimization time compare with that of existing methods? The authors should provide wall-clock runtime comparisons to support any claims regarding computational efficiency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.