acceptodds
Withdrawn from ICLR 2027

Rethinking Perceptual Representations for 6-DoF Grasping with Stereo Spike Streams

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most robotic grasping pipelines first convert sensory observations into explicit 3D representations such as point clouds, which is a computational step not found in biological intelligence. While effective in many standard settings, this reconstruct-then-act paradigm can become a bottleneck under asynchronous sensing, where high dynamic range and strict latency constraints make intermediate reconstruction costly and potential losses for downstream control. This paper explores a fundamentally different, neuro-inspired paradigm for 6-DoF grasp detection. We introduce SpikeGrasp, a framework that mimics the feedforward visuomotor pathway, processing raw, asynchronous events from stereo spike cameras, similarly to retinas, to directly infer grasp poses. Our model fuses these stereo spike streams and uses a recurrent spiking neural network, analogous to high-level visual processing, to iteratively refine grasp hypotheses without ever reconstructing the point cloud. To validate the approach, we construct a benchmark spanning clutter, textureless objects, severe illumination changes, and occlusion, and compare direct inference against reconstruct then grasp baselines. Across synthetic and real high-dynamic-range scenes, SpikeGrasp achieves stronger accuracy, data efficiency, and better robustness under challenging sensing conditions. Our results suggest that explicit point-cloud reconstruction is not universally necessary for embodied manipulation, and that direct learning in the asynchronous observation space is a promising alternative for low-latency robot perception and control, particularly in complex real environments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
6-DoF Grasp Detection, Neuromorphic Camera, Spiking Neural Networks, Neuromorphic Vision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_b9iCOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Robots often build a 3D map before deciding how to grab something. SpikeGrasp skips that step, using two specialized cameras that record rapid light-triggered signals to predict the gripper’s position and angle directly.

Critical strengths
  • The authors report 85% successful real-world grasps, versus 79.2% for the strongest tested competitor. Training on simulated data and then working on a physical robot is encouraging.
Critical weaknesses
  • Most evidence comes from simulation; the reported 240 real-world trials cover limited settings.
  • Some explanations contradict their tables. On page 26, the text claims switching to spike input improves one comparison, while the table shows performance dropping. These discrepancies undermine confidence in the explanation of why the method works.
Questions for authors

See above.

Reviewer_fyi3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether 6D grasping needs explicit 3D reconstruction and argues that with neuromorphic sensors it may not. Converting asynchronous spike streams into point clouds could discard information and become the failure point under high dynamic range, reflective surfaces or occlusion. It therefore learns grasps end to end from raw stereo spike streams. SpikeGrasp pairs stereo correlation with a recurrent spiking network that refines a latent scene representation, which a graspness-style head decodes into grasp poses. The authors also contribute a synthetic spike-based grasping benchmark. They conclude that the spike observation space is a viable alternative to geometry-first grasping.

Critical strengths
  • Grasping from spike cameras is unexplored, the paper asks a worthwhile question: is explicit 3D geometry the right intermediate representation for grasping with neuromorphic sensors? Thus the paper contributes both a full system and a dedicated benchmark.
  • The evaluation is well designed, and the synthetic benchmark randomizes transparent and specular materials, which specifically stress depth-based pipelines.
Critical weaknesses
  • Limited relevance and novelty for a representation-learning venue. Spike cameras are not widely adopted, so the practical impact depends on a sensor few groups can access. The work may fit a robotics or sensor-focused venue better. On the learning side, the architecture mostly combines existing components: RAFT-Stereo-style correlation with iterative refinement, plus a GSNet-style graspness head. The general representation-learning contribution is therefore incremental.

  • The paper does not use what makes spike cameras distinctive. All experiments use static scenes and fixed cameras. The spike stream is binned into a dense tensor, and stereo matching has no temporal axis. As a result, the sensor’s main advantage, ultra-high temporal resolution goes unused. The gains plausibly come from dynamic range, which HDR frame cameras also offer. Dynamic grasping, with moving objects or a moving camera, would be the natural test.

Questions for authors
  • What task-relevant information does the spike latent keep that an explicit depth map or point cloud discards? The paper argues that reconstruction loses temporal information. But with static scenes and fixed cameras, what temporal content remains beyond integrated intensity?
  • Is the advantage coming from the representation or from the sensor? The baselines receive depth or RGB produced by some conversion step or another camera. Their gap may reflect the quality of that input rather than any inherent limit of explicit geometry.
  • What role do spiking dynamics play in the refinement module? If I understand correctly, the SNN time steps correspond to refinement iterations, not sensor time. What does spiking add over a standard recurrent update in this setting?
Reviewer_QjR6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines learning to grasp objects using a parallel jaw robotic gripper with images generated by a spiking stereo event camera.

The paper proposes a neural architecture, proposes a loss, and provides a simulated dataset for training.

The proposed approach is compared to several baselines that use alternative sensors for obtaining the image data, and is shown to achieve better classification performance on held out sim data.

The real world experiments show the baselines perform comparably to the proposed approach on the isolated object scenes. In the cluttered and HDR scenes the proposed approach provides somewhat better performance.

Critical strengths

None

Critical weaknesses

(1) I had a very hard time following the paper and could not confidently replicate it. The details provided in the methods section have many undefined variables and the notation used in section 3.1, 3.2, 3.3, 3.4, and 3.5 don't connect. I have a vague idea of how one could connect these components, but a concrete overview is missing to the point I'm not confident the authors know what they actually did.

This is especially confusing when the definition of grasp quality is given in 3.5 and how this (1) is computed and (2) how it connects with the loss functions stated in the equation below is in no way stated.

(2) Additionally, the experiments provide no details of how they generated grasps in simulation other than saying "the benchmark is built from 88 object models from graspnet." Are these just using the same labels as that dataset presented in that paper and reskinning images to create the event camera data or something else? Without understanding the details of these experiments, it's quite hard to judge the results.

Questions for authors
  • Should N be T in the spike train S() exponent?

  • How does F() in 3.2 relate to f() in 3.1?

  • How is the feature map C passed into the recurrent neural network?

  • The figure has a label "spike context" this term is not used anywhere in the methods section, how is it computed?

  • How are the target grasp quality scores computed for training? In particular, where do the friction coefficient estimates come from?

  • How is the decomposition of the 6D pose parameterization quantified, the text only description is confusing? What are the explicit outputs from the model? This seems to contradict the Ri, ti, outputs described below in the same section.

  • How are the different loss terms computed? In particular how do they relate to the neural net outputs?

  • How were grasp labels generated for the simulated data? Are they simply labels for some scenes in Graspnet with new images generated or is an actual 3D simulation conducted?

  • How was inference performed for the baseline approaches on the real robot? Was anything done to reason about clutter?

  • Can a formal definition of what HDR scenes mean in the context of simulation be provided?

  • Additionally what does HDR mean in the context of the real world experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.