$\partial O_{\theta }$: Action-Differentiable Neural Objects

Learned 3D object representations are typically evaluated by reconstruction accuracy. However, accurate geometry does not guarantee that the representation provides useful gradients for optimizing physical interaction. We ask whether a learned object representation can serve directly as a differentiable interface from geometry to action. We introduce $\partial O_{\theta }$, Action-Differentiable Neural Objects, a structured neural representation that supports diverse articulated, skinned, and deformable object identities. Conditioned on sparse keypoints, $\partial O_{\theta }$ provides continuous occupancy, signed-distance, and surface-normal queries at arbitrary 3D locations. The same neural object is used in two directions: geometry recontruction given keypints during representation learning, while after training, task gradients could flow backward through frozen geometry to optimize keypoints for specific interaction. $\partial O_{\theta }$ achieves 94.2% simulated grasp success and 78.8% OOD real-world success, while representing up to 100 objects with a compact 1.9M-parameter shared model. These results suggest that neural object representations can be designed not only to reconstruct geometry, but also to expose useful derivatives for physical interaction.