- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The manuscript proposes ∂Oθ, a shared, structured neural object representation conditioned on object structure and configuration. It extends a WorldString-style neural field with occupancy, signed-distance and surface-normal outputs, using a PCA-based decomposition of keypoint motion to distinguish persistent structure from changing configuration. Once trained, the neural field is frozen and used to propagate geometric gradients into dexterous-hand configurations; optimized trajectories then supervise a policy evaluated in simulation and on a real robot.
Critical strengths
- The manuscript explicitly distinguishes network-parameter learning from configuration optimization.
- The use of signed distances and spatial derivatives provides a reasonable mechanism for contact and penetration objectives.
- The PCA-based structure/configuration decomposition is motivated by the limited availability of fine-grained kinematic annotations.
- The method addresses a useful problem: a shared geometric model should be useful not only for reconstruction but also for downstream optimization.
- The paper acknowledges that it differentiates through geometric queries and kinematic updates, not through full contact dynamics. This is an important scope distinction.
Critical weaknesses
- The method explicitly builds on WorldString's structured neural object formulation. Its extensions include multi-object conditioning, PCA-based structural descriptors, SDF and normal prediction, and downstream optimization. The manuscript should identify which of these changes are essential to actionability and which are engineering extensions.
- The method uses observed keypoint variation to estimate low-dimensional motion patterns and a complementary structural descriptor. This is sensible when configuration changes occupy a low-dimensional subspace. However, the manuscript does not establish how robust this assumption is for nonlinear articulation, discontinuous changes, sparse observations or objects whose geometry changes independently of their selected keypoints.
- The manuscript's three-stage training pipeline includes occupancy learning, SDF post-training and normal fine-tuning. The ablation shows that normal error falls from 49.11° after SDF post-training to 18.53° after normal fine-tuning on the rigid-object setting. This is evidence that gradient quality requires explicit attention. But the paper does not establish how accurately the learned gradients match gradients derived from ground-truth geometry, nor whether gradient quality predicts optimizer convergence and final grasp success.
- The appendix states that each method optimizes its own objective. ∂Oθ uses contact, penetration, finger/palm proximity, fingertip attraction, joint priors and closure terms. This makes the comparison difficult to interpret: improvements may result from a more effective objective or constraint configuration rather than the representation itself. A fair comparison should separate representation, objective and optimizer effects.
- The manuscript differentiates through neural geometry and kinematic updates, not through contact dynamics. Consequently, it should avoid implying that the model directly optimizes fully physical trajectories or captures force-dependent interaction dynamics. The current framing is mostly careful, but terms such as “action-differentiable” should be defined operationally.
Questions for authors
- What component of ∂Oθ produces the action improvement that cannot be obtained from WorldString with a separately trained SDF head?
- How closely do predicted SDF gradients align with ground-truth mesh or SDF gradients near contact?
- How sensitive is grasp success to PCA rank and keypoint perturbations?
- Does the method retain its advantage when all methods use the same objective, optimizer, initializations and stopping rule?
- What happens when the object is outside the observed configuration subspace?
Response to Reviewer 49Kd
Dear Reviewer 49Kd,
Thank you for the questions about contributions, gradient quality, and evaluation. We address them with additional controlled experiments and clarify the scope of each component.
W1/Q1. Contribution beyond an SDF head
Shared structural conditioning supports reconstruction across object identities; distance and normal training support action optimization through frozen geometry. We compare A: an SDF head trained on a frozen occupancy backbone; B: joint SDF/backbone training with accompanying training refinements; C: B plus normal supervision. All use identical grasp objectives and optimization settings, with three seeds and 30 trials per object per seed.
These results support the contribution of the training recipe beyond fitting a separate SDF head. The gain from B to C supports the value of normal supervision. Table 5a separately evaluates structural conditioning for shared reconstruction.
W3/Q2. Spatial and action gradients
Our normals are normalized spatial SDF gradients, so angular error against ground-truth normals measures gradient-direction accuracy. Table 5b reports a reduction from 49.11 to 18.53 degrees on 100 rigid objects after normal fine-tuning.
On the four grasp objects, we additionally compare gradients of the grasp objective with respect to QPose against those computed using ground-truth geometry. The table gives cosine similarity and, for C, the fraction of runs in which an update along the neural gradient decreases the evaluated objective.
Improved task-gradient agreement accompanies higher grasp success across these matched configurations.
W4/Q4. Common optimization protocol
The original system comparison preserves each method's native objective. We now also compare adaptations with a shared objective, optimizer, initializations, stopping rule, and budget, retaining each method's geometric queries: mesh-based Phong SDF for Fast-Grasp'D, point-to-mesh distances for DexGraspNet, and frozen neural fields for ours. Each method uses three seeds and 30 trials per object per seed. Four-object averages are:
Our advantage persists under the common protocol, complementing the original system comparison.
W2/Q3/Q5. PCA and keypoint robustness
PCA summarizes training keypoint variation to condition shared reconstruction. Grasp experiments use separately trained hand/object models without shape-code conditioning and optimize QPose through forward kinematics. PCA rank therefore pertains to the reconstruction setting. We will clarify this distinction in the paper.
Three-seed policy tests cover 2, 5, and 10 mm keypoint noise. At 10 mm, ID/OOD success is 100/98.33% for apple, 98.33/95% for pig, 88.33/85% for trophy, and 100/96.67% for duck, compared with clean rates of 100/100%, 100/100%, 95/88.33%, and 100/95%, respectively.
For reconstruction, we test configurations with large PCA residuals. Let k stack K keypoints, mu be the training mean, and C = I - U_r U_r^T be the saved complement projector. The residual in model voxels is
e(k) = 256 * ||(k - mu)^T C||_2 / sqrt(3K).
For each object, the common group comprises three held-out poses nearest the training median residual. The corner group comprises the three largest held-out residuals, each above the training 95th percentile; the groups are disjoint. Occupancy F1 (%) changes as follows:
These selected configurations retain strong occupancy accuracy. Soft objects show greater distance and normal sensitivity: normal error rises from 17.94 to 22.74 degrees for cube and 19.57 to 25.83 degrees for foam roll. Our scope is geometry captured by the selected keypoints and training coverage; this test quantifies robustness beyond typical training residuals.
W5. Meaning of action-differentiable
Task gradients flow through frozen neural geometry and forward kinematics to hand configuration variables. The term refers specifically to geometric and kinematic derivatives, with physical grasp stability assessed in simulation and on the robot. We will make this definition consistent throughout the paper.