acceptodds
Withdrawn from ICLR 2027

Motion Before Events: IMU-Guided Optical Control and Timestamp-Aware Compensation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neuromorphic devices like event cameras enable low-latency visual sensing, but their information content depends strongly on image motion. Weak motion may leave important structures under-observed, whereas strong motion can produce excessive and redundant events. Inspired by active modulation in biological vision, we present Event-Inertial Retinal Modulation (EIRM), a bio-inspired framework that adaptively regulates optical excitation according to the current sensing state. EIRM uses a causal temporal policy to select discrete wedge-prism rotation speeds from recent visual, inertial, and event-derived observations, increasing modulation when natural motion is insufficient and reducing it when event excitation is already adequate. A complementary rule-based feedback loop regulates controlled camera motion from aligned-event activity. To assess whether the resulting events are informative rather than merely abundant, we introduce Target Event Observability (TEO), which jointly measures structural coverage, event purity, and temporal persistence. We further construct a hybrid benchmark combining controllable synthetic scenes, geometry-consistent reconstructions, and real-world visual–inertial trajectories. Experiments show that EIRM recovers useful structure under weak motion while suppressing unnecessary excitation under strong motion.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Event Cameras, Active Sensing, Optical Modulation, Visual-Inertial Sensing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PU4vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors present a method to add an optical element to an event camera to either motion stabilize or induce motion in the scene, in an attempt to produce an "optimal" event rate. The biological inspiration is the saccade.

This is achieved by providing a policy network with fixed-period buffered event frames. A CNN then proposes the magnitude of motion required. An IWE (image of warped events) is used to evaluate the outcome by measuring event occupancy.

Critical strengths

The paper is quite well presented in that it is easy to follow along what the authors are doing and why. There has been a good attempt at ablation and comparison to SoTA. The literature review seems pretty thorough to me, did not find much missing there.

Critical weaknesses
  • I find the premise of this paper to be flawed. In the first diagram, the authors differentiate between scenarios of “too sparse” or “too noisy”. In fact, event sensors sample the scene at an optimal rate, that is, proportional to the change occurring in the scene. Trying to artificially control the events produced by the scene through relative movement of the sensor, or through stabilization thus defeats the point of using an event sensor in the first place. In this case, why not just use a standard video camera? Or better, a hybrid event sensor/standard camera.
  • EIRM is not a generic framework for event rate control, since it assumes brightness consistency, which is not all the case in many everyday scenes. How would EIRM react to a sudden brightness change caused by lens flare, or a glint from a passing car, for example?
  • EIRM seems to be somewhat chicken-and-egg, since the observation window (control interval) is fixed length - thus the problem is not really solved for that interval which could still suffer from little data or too much.
  • In “Event-based motion segmentation by motion compensation” the authors of that paper discuss how relative motion impacts the integration time required for an IWE to be formed. I think that this might be interesting for the authors to consider, and should be cited here.
  • The comparison to benchmarks seems a bit pointless to be honest. The comparison is just to event sensor simulators and thus the results on the arbitrary evaluation metrics are meaningless. I feel like the only place to appropriately include these works in a benchmark, would be to compare simulation fidelity.
  • The method repeatedly describes a three-action control space:
  • A_p={0,180,360}\text{ RPM}, but, Appendix D.3, page 17, states “For the controlled optical-policy ablation reported in the main paper, the evaluated checkpoint uses the restricted two-action set 360 RPM.” What is going on there?
  • It’s not clear to me what role if any the IMU plays in this method. Because the motions in Blender are very synthetic sinusoids, so I do not think the model will learn anything about real world camera motions.
  • I know that it is a big ask, but this paper would have been way better with some sort of physical prototype to prove out the idea in the real world, on some qualitative data. It is nearly impossible for me to believe that this approach, trained an evaluatied only on synthetic data, would actually work. At least not without seeing some real world demo.
  • TEO (Eq 3-4) seem to not penalize background event redundancy. i.e two streams could have TEO=1 and one stream could have a bunch of background crap.
  • Some of the math here seems just gratuitous. Especially Eq 8-10, those equations add no value in terms of formalizing the problem and would be way clearer with just a sentence in plain English.
Questions for authors

Som small suggestions:

  • In “Adaptive Prism Excitation” the authors seem to have removed spaces before periods? Perhaps in an effort to reduce space, but it does make it distracting to read.
  • What are “dwell constraints”? If not important, maybe just write “Controller parameters can be found in the Appendix”/
Reviewer_2FwaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Authors present a framework that uses a state-based model to control the number of events triggered on an event camera by changing rotational speed of a wedge prism. This prevents issues such as having too few events (e.g., caused by little to no scene motion). Experimental results show that the inclusion of the prism helps capture information about the target object.

Critical strengths
  • The proposed idea to control the number of triggered events by changing the rotational speed of the prism is interesting at a high-level
Critical weaknesses
  • There is insufficient evidence that the policy-based approach is significantly better than simply adopting a fixed RPM. E.g., Fig. 3 suggests that keeping the prism fixed at 360 RPM is strictly better in terms of the proposed TEO metric and the number of events. Table 4 provides alternative evidence that adopting a strict 180 RPM policy performs well.
  • The paper identifies both weak and strong motion as problematic for event cameras (i.e., weak motion is non-ideal because little signal is captured and strong motion is non-ideal because event cameras have limited bandwidth). However, claims that the proposed approach "suppresses unnecessary excitation under strong motion" are overstated. The EIRM framework does not suppress events from triggering under strong motion, it can only choose whether to amplify (by spinning the prism) or not amplify (by not spinning the prism) the number of triggered events.
  • The novelty of the approach is limited. The use of a wedge prism has been previously done (as noted by the authors) and the benefit of adding the proposed policy-based feedback loop is questionable.
  • I find the claim that the benchmark is hybrid simulation-and-real-world to be overstated. Taking real scenes and rendering them in Blender is still a simulated (and importantly, not a hardware-validated) result.
Questions for authors
  • Additional context and clarifications about the design choices and intuition / motivation behind the proposed approach should be given. For example, no information is provided regarding the choice of the wedge prism or background about it. This raises questions like why a wedge prism was specifically chosen, what optical modulation does it perform, is this type of modulation optimal for certain scenes / types of movement, etc.? Furthermore, how were the 180 and 360 RPM settings chosen?
  • The paper states that data redundancy is the main problem with having too many events; however, I see the limited bandwidth of the event camera to be the problem here (i.e., the camera will necessarily drop events if the event stream is saturated).
Reviewer_3WVWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EIRM, an active event-sensing system that uses a rotating wedge prism to improve event observations. A neural network selects the prism speed based on visual, IMU, and event information, while a separate controller adjusts camera motion. The authors also introduce timestamp-aware event compensation and a new metric, TEO, to evaluate event quality.

Critical strengths
  1. The work considers the acquisition process as a complete system, including a rotating-prism mechanism, an event camera, an IMU, actuator control, optical calibration, and timestamp-aware event compensation. The presented prototype and the accompanying simulation and processing pipeline reflect appreciable engineering effort. Actively adjusting the sensing process to address weak relative image motion is a meaningful direction.

  2. The experiments provide useful evidence for the underlying optical-excitation and compensation mechanisms. The controlled motion regimes help illustrate when additional optical excitation is beneficial. In particular, active modulation recovers target structure under co-motion, where passive sensing provides little target information. The compensation ablation in Table 3(b) is also informative: it holds the generated event stream fixed and improves F1 from 0.5195 without compensation to 0.5696 with interval-level compensation and 0.5890 with per-event compensation. This provides reasonably clear evidence for the benefit of evaluating the inverse optical warp at individual event timestamps.

Critical weaknesses
  1. Limited real-world validation and unclear practical benefits. Most experiments rely on simulated or regenerated event data, despite the hardware prototype. The camera-motion controller is also questionable, as it changes the playback speed of recorded trajectories, which may not be possible in many real-world applications. Moreover, it is unclear whether the generated events can improve downstream tasks such as tracking, SLAM, or 3D reconstruction. I am also concerned about how switching between discrete prism speeds affects event quality and compensation accuracy in a real system.

  2. The benefit of the learned controller is not convincing. The improvement over Fixed-180 is very small (F1: 0.3907 vs. 0.3930), and Fixed-180 even performs better on some metrics. This makes the role of the neural network unclear. Although the paper includes some ablations, important comparisons are missing, such as removing IMU input, disabling camera-motion feedback, and replacing the network with a simple rule-based controller.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.