acceptodds
Withdrawn from ICLR 2027

Continuous - Time Voxel free Image - Event Semantic Segmentation.

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Semantic segmentation for autonomous driving and robotics remains challenging under fast motion, low illumination, and high dynamic range, where conventional RGB sensing becomes unreliable. Event cameras provide complementary measurements with high temporal resolution, yet most existing RGB-event segmentation methods first quantize asynchronous events into voxel grids, discarding fine-grained temporal structure and potentially introducing cross-modal misalignment. We present a voxel-free RGB-event segmentation framework that preserves event timing throughout the representation and fusion process. At its core is CT-EvSSM, a continuous-time event encoder that operates directly on raw events using per-event temporal embeddings and a selective state-space model whose discretization is conditioned on the true inter-event intervals. To integrate event features with RGB representations, we adapt a pretrained DINOv2-L/14 backbone into a multi-scale feature pyramid and introduce a dual-branch fusion mechanism that combines geometric cross-modal alignment with frequency-domain feature fusion and temporal state-space modeling. This design enables event information to be aligned and integrated with RGB features while retaining its asynchronous temporal structure. Extensive experiments on two widely used datasets, DSEC and DDD17, demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Event-based Vision, Semantic Segmentation, Multimodal Learning, State Space Models, Continuous-Time modelling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ixU8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CT-Net processes timestamped events with a time-gap-conditioned state-space encoder before mapping them onto a spatial grid, then fuses them with DINOv2 image features. The full system reports a substantial 4.33-point mIoU improvement over the published MambaSeg result on DSEC-Semantic. Yet the DDD17 improvement is only 0.78 points, with slightly lower pixel accuracy. More importantly, the only matched frozen-backbone DSEC comparison gives 76.48 mIoU for the voxel-free branch versus 76.50 for voxelization. The headline result therefore does not establish that continuous-time, voxel-free processing—rather than the stronger image backbone and other changes—produces the gain. The wording about selecting the “best-epoch test” result also requires clarification.

Critical strengths
  1. A clear temporal modeling hypothesis. Using measured inter-event gaps before spatial aggregation is a technically relevant idea for asynchronous event data. If its benefit is isolated, it could be more consequential than another fusion-module adjustment.
  2. Promising complete-system DSEC result. The reported 79.43 versus 75.10 mIoU is a sizable improvement worth investigating, although the present comparison does not establish which component produces it.
Critical weaknesses
  1. Define the new mechanism precisely. The distinctive step appears to be conditioning the state-space update on inter-event time gaps; the image backbone and parts of the fusion design build on existing components. Please state the exact technical difference from prior temporal event encoders and explain why that difference should improve segmentation.
  2. Directly test the voxel-free contribution. The matched frozen-backbone DSEC comparison is effectively tied, with voxel-free processing 0.02 mIoU points lower. Table 3 then changes both backbone fine-tuning and the event representation between key rows, so its larger gain cannot be attributed to either one. Please test voxelized versus continuous-time input with both frozen and fine-tuned backbones on both datasets, and ablate the time-gap term itself.
  3. Make the headline baseline comparison fair. The principal MambaSeg value is quoted from its publication, while CT-Net uses a large DINOv2 image encoder. Even the frozen-DINO voxel baseline exceeds that published DSEC value by 1.40 mIoU points. Please disclose the exact checkpoint and pretraining, and provide a matched-backbone or otherwise carefully qualified comparison before assigning the 4.33-point gain to the proposed event method.
  4. Establish whether the smaller gains are reliable. The joint fusion addition yields 0.64 and 0.65 mIoU points on the two datasets; the full-model DDD17 advantage is 0.78 points while pixel accuracy decreases by 0.07 points. Please report repeated-run variation, isolate the fusion components, and show per-class or difficult-condition results. These positive mIoU differences should not be treated as decisive without evidence that they exceed variation or deliver a meaningful practical benefit.

And the following two comments are very important for authors and need to be addressed before the next submission.

  1. The ablations do not establish the contribution of each component. Table 3 changes both DINOv2 fine-tuning and the event representation between key rows, so their effects cannot be separated. The matched frozen-backbone comparison gives 76.48 mIoU for voxel-free processing versus 76.50 for voxelization on DSEC-Semantic, while CMGA and SFF are evaluated only together, adding 0.64 and 0.65 points on the two datasets. Please evaluate CT-EvSSM against a matched voxel baseline, ablate its inter-event-time term, and report CMGA-only, SFF-only, and combined variants under the same training settings.
  2. The architecture figure does not adequately explain the proposed modules. Figure 1 shows CT-EvSSM and the fusion pathways largely as high-level boxes, making it difficult to trace how timestamped events are processed, scattered into spatial features, aligned by CMGA, and fused by SFF. Please expand or redraw the figure with module-level diagrams, feature dimensions, and the inputs and outputs of each stage. In particular, clarify how the temporal pathway operates after event features have been scattered onto a 2D grid.
Questions for authors
  1. Which split was used to select the reported epoch and tune hyperparameters? Does “best-epoch test” mean that test results influenced selection?
  2. Which exact DINOv2 checkpoint and pretraining were used, and are repeated-run results available for the smaller DDD17 and fusion-module gains?
Reviewer_H2ZSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The manuscript presents a model for performing RGB-event semantic segmentation in a voxel-free manner. Specifically, the manuscript presents a continuous-time event-based space state model to process raw asynchronous events based on true inter-event intervals rather than quantized temporal bins
Critical strengths
  • The idea of bypassing traditional voxelization to use a continuous-time state-space model conditioned on exact inter-event gaps is interesting to preserve fine-grained temporal dynamics. The ablations support that this approach has a beneficial impact on performance.
Critical weaknesses
  • The trade-offs of the method are not discussed. While the authors state the event branch scales linearly with the event budget the actual wall-clock inference time, memory consumption, and latency compared to voxel-based methods are omitted.
  • The presentation makes the manuscript difficult to follow. Particularly, the description on the CMGA and SFF blocks is not clear. Additionally, it is not clear how comparison in Table 2 is Fair if models use different backbones.
Questions for authors
  • Could you provide a discussion on the trade-offs that the model introduces regarding the actual wall-clock inference time, memory consumption, and latency compared to voxel-based methods?
  • Could you provide results for other models using DINOv2 as the backbone for a fair comparison?
Reviewer_udFeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CT-Net for RGB–event semantic segmentation. Its event encoder processes raw events with a selective state-space model, adding the inter-event time gap to the step-size prediction. The resulting features are fused with a DINOv2 feature pyramid through geometric alignment and frequency-domain fusion.

Critical strengths
  • Processing events before temporal binning is a reasonable direction for preserving fine-grained timing information.
  • The complete system achieves higher mIoU than the reported baselines on DSEC-Semantic and DDD17.
Critical weaknesses
  1. The technical contribution is limited. The main modification to the standard selective SSM is an additional time-gap term in the step-size prediction. The event embedding and DINOv2 pyramid adapter are straightforward designs, while CMGA and SFF are explicitly adapted from Evita. I do not see a sufficiently strong methodological contribution in this combination.

  2. The experiments do not establish the benefit of the proposed event modeling. The matched comparison in Section 6 gives 76.48 mIoU for CT-EvSSM versus 76.50 for voxelization on DSEC, showing no accuracy gain. The larger improvements are associated with DINOv2 fine-tuning and the added fusion module. Table 3 also changes backbone fine-tuning and event representation simultaneously, which prevents attributing that improvement to CT-EvSSM. There is no separate ablation of the explicit time-gap term. The results therefore support the complete system's performance, but do not substantiate the central claim that the proposed temporal modeling is responsible for the gains.

Questions for authors
  1. Could the authors provide a fine-tuned DINOv2 + voxel baseline with the same fusion and training settings as CT-EvSSM? A matched, separately trained RGB-only baseline would also help establish how much the event branch contributes.

  2. What does the explicit inter-event gap contribute beyond the timestamp already included in the event embedding? Please train a matched variant without the time-gap term while retaining the original embedding.

  3. Which parts of CMGA and SFF differ from Evita? Please distinguish the adopted components from the changes introduced in this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.