acceptodds
Under review as a conference paper at ICLR 2027

Geometry Before Fusion: Event-First State-Space Modeling for Event-Frame Stereo

11 views

Abstract

Event cameras remain informative under fast motion and extreme illumination but are sparse, whereas frame cameras provide dense appearance that degrades under the same conditions. Existing event–frame stereo methods fuse the two modalities before or during matching, so unreliable appearance can bias correspondence before any geometric reference exists. We propose to establish geometry before multimodal fusion: disparity is first estimated solely from stereo events with a state-space architecture, then refined by a single frame weighted by a reliability measure derived from event stereo consistency. On DSEC, our method is the most accurate among methods using at most one frame camera and approaches symmetric methods using two, while its event-only variant outperforms prior event-only methods. It also generalizes zero-shot to MVSEC and M3ED. Ablations show that fusion timing is critical: introducing the frame before matching degrades accuracy below the event-only model.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Event-based vision, stereo matching, event–frame fusion, state-space models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_fFGNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an event-first framework for stereo depth estimation using two event cameras and one frame camera. A state-space network first estimates disparity from stereo events alone. The paper focuses on when frame information should enter stereo estimation. Establishing event-only geometry first provides an appearance-independent reference for subsequent refinement. This gives the two modalities distinct roles and offers a sensible way to limit the influence of degraded frames on correspondence. The experiments include an event-only control with the same refinement capacity, helping distinguish the benefit of frame information from additional model capacity. Under blurred nighttime frames, gating reduces MAE from 0.560 to 0.412, while the matched event-only control achieves 0.414. This supports the gate’s practical value under the tested degradation.

Critical strengths

This paper proposes a Geometry-before-Fusion framework for Event–Frame Stereo. By adjusting the role and fusion timing of different modalities in stereo matching, event data is first used to establish geometric relationships independent of Frame Appearance, and then image information is introduced for parallax refinement. Specifically, Geometry-Structured Event States, Cross-view State Propagation and Coarse-to-Fine Epipolar Matching are first used to estimate the initial disparity, and then disparity refinement is completed through Reliability-Aware Frame Integration and Disparity-Conditioned State-Space Refinement. The proposed method has achieved competitive experimental results on serval benchmarks, and the overall experimental design is relatively complete.

Critical weaknesses
  1. Existing methods (such as EGEI-Net mentioned in the Related Work section) already utilize image-derived edge guidance to steer recurrent disparity updates. Furthermore, coarse-to-fine disparity refinement and reliability-aware fusion are also well-established strategies. The authors should clarify why the proposed geometry-before-fusion strategy represents a significant innovation in the context of event-frame stereo compared to existing multi-stage, confidence-guided fusion approaches.
  2. In Table 5, the early fusion configuration performs worse than the event-only Stage I (which uses only event data), while post-D_0 fusion demonstrates a notable performance improvement. The authors should further analyze whether the gain is attributed to the change in fusion timing or to the inherent advantages of matching within the event domain.
  3. In Section 3.2, the authors describe the reversible rearrangement operator within the Geometry-Structured Event States as lossless and reversible. However, if the dimensions of the feature map are not perfectly divisible by the neighborhood size, the defined rearrangement operation cannot guarantee reversibility. For instance, with an input resolution of 480 × 640, the width is not divisible by 3. The authors should explicitly explain how boundary regions are handled and how the lossless, reversible nature of this operation is maintained.
  4. While the paper highlights the proposed method's advantages in terms of parameter count, the results indicate a high computational cost. It would be helpful to provide additional measurements of actual inference latency, GPU memory usage, and throughput for a more comprehensive assessment of the method's computational efficiency and feasibility for practical deployment.
  5. Table 6 demonstrates that the reliability gate effectively improves matching performance when dealing with blurred frame inputs. However, the current experiments do not sufficiently validate the relationship between the frame gate and actual reliability. It is recommended to further analyze the gate's response under varying degrees of frame corruption or different levels of event consistency error.
Questions for authors
  1. Existing methods (such as EGEI-Net mentioned in the Related Work section) already utilize image-derived edge guidance to steer recurrent disparity updates. Furthermore, coarse-to-fine disparity refinement and reliability-aware fusion are also well-established strategies. The authors should clarify why the proposed geometry-before-fusion strategy represents a significant innovation in the context of event-frame stereo compared to existing multi-stage, confidence-guided fusion approaches.
  2. In Table 5, the early fusion configuration performs worse than the event-only Stage I (which uses only event data), while post-D_0 fusion demonstrates a notable performance improvement. The authors should further analyze whether the gain is attributed to the change in fusion timing or to the inherent advantages of matching within the event domain.
  3. In Section 3.2, the authors describe the reversible rearrangement operator within the Geometry-Structured Event States as lossless and reversible. However, if the dimensions of the feature map are not perfectly divisible by the neighborhood size, the defined rearrangement operation cannot guarantee reversibility. For instance, with an input resolution of 480 × 640, the width is not divisible by 3. The authors should explicitly explain how boundary regions are handled and how the lossless, reversible nature of this operation is maintained.
  4. While the paper highlights the proposed method's advantages in terms of parameter count, the results indicate a high computational cost. It would be helpful to provide additional measurements of actual inference latency, GPU memory usage, and throughput for a more comprehensive assessment of the method's computational efficiency and feasibility for practical deployment.
  5. Table 6 demonstrates that the reliability gate effectively improves matching performance when dealing with blurred frame inputs. However, the current experiments do not sufficiently validate the relationship between the frame gate and actual reliability. It is recommended to further analyze the gate's response under varying degrees of frame corruption or different levels of event consistency error.
Reviewer_gFKSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an event-frame stereo method built on one idea: establish geometry first, then fuse. Disparity is first estimated from the two event streams only, using a state-space architecture with explicit epipolar matching. A single left frame then refines it, and a reliability gate derived from event stereo consistency controls how much the frame is trusted.

The paper has two key insights. First, when the frame enters matters as much as how much appearance information is used, and early fusion can be worse than no frame at all. Second, the event-only disparity gives an appearance-independent reference for judging frame reliability. Experiments on DSEC and zero-shot MVSEC/M3ED show competitive results among methods using at most one frame camera.

Critical strengths
  1. The idea is simple and well motivated. Using a frame-independent geometry to decide how much to trust the frame is an interesting design, and the early-fusion finding is a useful insight.
  2. The method achieves strong results on DSEC and in zero-shot transfer, and it is much lighter than Bi-CMPStereo.
  3. The ablations are clear, especially the fusion-timing study and the blur experiment, where the gated version stays stable while the ungated one degrades.
Critical weaknesses
  1. The ``at most one frame camera'' comparison is not entirely fair. This method uses two event cameras, while Bi-CMPStereo uses one, so the gain from staging and the gain from the extra sensor are not separated. I encourage the authors to add a controlled comparison, such as dropping the right event view.
  2. Some ablation evidence is hard to interpret. Table 4 row D and the ``post-D0 fusion'' row in Table 5 look like the same setting. The early-fusion claim is also made against Stage-I, which has no refinement network, so I could not tell how much of the degradation comes from fusion timing itself.
  3. The robustness claim is a bit too strong. Only synthetic blur is tested, and several hand-set constants such as g_max, r_min have no sensitivity analysis.
Questions for authors

Authors are encouraged to address these comments:

  1. How is the early-fusion baseline built and trained, and does it still lose to an event-only model with the same refinement capacity?
  2. Why do Table 4 row D and Table 5 post-D0 fusion differ?
  3. How does the gate behave under other degradations, such as over-exposure, noise, misalignment, and how sensitive are results to r_min and g_max?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.