acceptodds
Under review as a conference paper at ICLR 2027

Label-free Efficient Event Encoder Pretraining from RGB Foundation Models

Abstract

Event cameras offer microsecond latency, high dynamic range, and freedom from motion blur, but labeled event data are scarce and costly to annotate, so event vision has not benefited from foundation-model-scale pretraining. Existing event pretraining methods either learn from events alone at substantial cost (up to hundreds of GPU-hours on large event corpora), or transfer semantics from RGB foundation models by converting events into frames for an image transformer with an appended temporal stage, leaving the recurrent, event-native backbones that are standard in event-based detection without pretraining. We propose a single-stage, label-free cross-modal pretraining method that keeps the event-native recurrent backbone intact. A frozen DINOv3 teacher processes synchronized RGB frames, while an event encoder, a Recurrent Vision Transformer (RVT) followed by DINOv3 transformer blocks, processes the event stream and is trained to match the teacher's global and patch-level features through cosine alignment and an auxiliary patch-level InfoNCE objective. No labels are used, and the RGB branch is discarded after pretraining, so inference is event-only. Pretraining requires 7k steps, 9 A6000 GPU-hours, and 1.2 EFLOPs on DSEC alone, i.e., 2.7 and 26 fewer FLOPs than GEP and TESPEC. On DSEC-Detection, our model reaches 46.3 mAP, improving over RVT trained from scratch (38.4) by +7.9 mAP and over the same architecture trained from scratch (36.6) by +9.7 mAP, with AP rising from 38.1 to 53.5. Transferring only the pretrained RVT backbone across sensors raises GEN1 from 47.2 to 50.2 mAP, and on DSEC semantic segmentation we obtain 65.68 mIoU, on par with GEP (65.22) and above TESPEC (62.77), both pretrained on substantially larger corpora. These results suggest that dense semantic supervision from an RGB foundation model, applied directly to an event-native backbone, is a compute-efficient alternative to large-scale event pretraining.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Event-Based Vision, Self-Supervised Learning, Representation Learning, Cross-Modal Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NmU1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an efficient method of pretraining an event encoder. The proposed method is straightforward: leveraging the aligned RGB-Event pairs, the method learns to distill the information from DINOv3 into a backbone specific to event streams. The pretraining is supervised by global cosine loss, patch cosine loss and patch InfoNCE loss. Experiments on several datasets demonstrate the representation ability of the resulting models under in-domain conditions, cross-dataset transferability, and cross-task generalization ability.

Critical strengths
  • Using paired RGB event data to transfer representations from RGB foundation models to event encoders is a promising idea. The paired images provide strong prior information that could be used to improve efficiency. Future work can build on this idea to scale up the training of event foundation models with reduced computational overhead.
  • The proposed method is simple but effective. Experiments on different datasets show significant improvements with the pretrained encoder under in-domain conditions (DSEC-Detection), cross-dataset transferability (GEN1), and cross-task generalization ability (DSEC-Seg).
  • This paper is clearly structured and well-written.
Critical weaknesses
  • Although experiments involve two different dense tasks, the scope of tasks validated was still limited, especially considering that training and evaluation were mainly based on DSEC dataset. For example, it remains unclear how the model performs on image recognition and depth estimation, which were covered in the GEP paper.
  • The evaluation of the model’s generalization ability can be improved. Although GEN1 is a new dataset that has not been seen during pretraining, the object categories (car, pedestrian) still fall within the in-domain scope. Therefore, it is unclear how the pretrained model transfers to categories that have not been seen during pretraining.
  • I am wondering how the proposed method performs compared to fusion-based methods, as the pretraining has gone through both modalities. Does the proposed method have an advantage even compared with fusion-based methods?
  • The pretrained models can be a good contribution to the community, while it remains unclear if they will be released.
Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer NmU1

We sincerely thank the reviewer for the positive assessment of our work and for recognizing the effectiveness and efficiency of RGB-guided event representation learning. We appreciate the constructive suggestions and have conducted additional experiments to further strengthen the evaluation of our method and address the concerns raised.


W1. Extension to Additional Downstream Tasks

To evaluate whether our learned representations generalize beyond semantic perception tasks, we conducted additional monocular depth estimation experiments on DSEC-Depth and MVSEC-Depth. Unlike object detection and semantic segmentation, depth estimation requires geometric understanding, providing a complementary evaluation of the learned representations.

As shown in the table below, our method achieves the best performance across all three metrics on DSEC-Depth, and the best δ₁ and Abs Rel on MVSEC-Depth. Furthermore, compared with the same architecture without our proposed pretraining, our method consistently improves all three metrics on both datasets. These results further demonstrate the effectiveness and generalizability of our RGB-guided pre-training across different downstream tasks and datasets. We will also update the manuscript to include these additional depth estimation results and a detailed analysis.

MethodBackboneDSEC δ₁ ↑Abs Rel ↓RMSE ↓MVSEC δ₁ ↑Abs Rel ↓RMSE ↓
Task-specific Supervised Models
E2DepthRecurrent CNN0.4090.39513.2580.3630.5277.894
EReFormerSwin-T + GRViT0.5240.29711.6080.3610.5188.423
Pretrained Models
ECDDPViT-S/160.5450.2639.4770.5130.4286.957
ECDPViT-S/160.5280.32411.4730.4760.4967.680
DepthAnyEvent-RViT-S/160.6910.1918.8800.4890.3656.465
GEPViT-S/16———0.5500.3856.600
ScaleEventViT-B/160.8720.1094.0320.5940.2805.891
Ours (scratch)RVT+DINOv30.8060.1474.6870.5920.3177.393
OursRVT+DINOv30.9230.0933.2000.6340.2776.430

W2. Generalization to Unseen Semantic Categories

We appreciate the reviewer's suggestion to evaluate generalization to unseen semantic categories. However, a direct evaluation remains challenging due to the limited availability of suitable event-based benchmarks. Specifically, we evaluate our method on three representative downstream tasks: object detection, semantic segmentation, and depth estimation. In event-based vision, established benchmarks for these tasks with high-quality annotations and standardized evaluation protocols remain limited in domain diversity, with most focusing on autonomous driving and road scenes. We therefore adopt these widely used benchmarks to ensure fair and direct comparisons with prior related work.

Meanwhile, the reviewer raises an important point that reflects a fundamental challenge in event-based vision. As discussed in our Introduction, event datasets remain substantially more limited in scale and diversity than RGB image datasets. This is precisely one of the key motivations for our work: leveraging the rich semantic knowledge of large-scale RGB foundation models to learn more transferable event representations with limited event data.


W3. Comparison with Fusion-Based Methods

RGB-event fusion methods and our approach operate under different inference settings. Fusion-based methods may achieve higher accuracy by combining complementary information from both modalities during inference, whereas all our reported downstream results are achieved using event inputs alone. Our research objective therefore differs from fusion-based approaches: we aim to utilize the rich semantic knowledge from RGB foundation models through RGB-guided pretraining to develop a more powerful event encoder, enabling downstream tasks to rely solely on event inputs without requiring paired RGB data.

Therefore, the key advantage of our design is modality independence during inference. When RGB observations are unavailable or unreliable, such as in low-light or high-speed scenarios, our model can still operate independently, whereas fusion-based methods that rely on both modalities may be limited. Moreover, training fusion-based models often requires substantial paired RGB-event data to achieve strong performance, yet such data remain scarce. In contrast, our approach can learn effective event representations with only limited paired RGB-event data by transferring rich semantic knowledge from RGB foundation models during pre-training.


W4. Release of Pretrained Models

Yes, we will publicly release our pretrained model checkpoints, training and evaluation code, and configurations to facilitate future research in event-based vision.

Thank you for your replies, the additional experiments on depth estimation look promising, I will keep my score.

Reviewer_EFRLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose LEEEP, a label-free method that pretrains event encoders from an RGB foundation model, with a frozen DINOv3 teacher on RGB frames and an RVT + DINOv3 student on events, trained by global cosine, patch cosine and patch InfoNCE losses. Experiments on DSEC-Detection, GEN1 and DSEC-Semantic demonstrate 46.3 mAP, 50.2 mAP and 65.68 mIoU.

Critical strengths
  1. The framework is conceptually simple and computationally efficient. It uses a single pretraining stage on modest hardware and discards the RGB branch at inference.
  2. The empirical gains are consistent across two tasks and two datasets. The improvement in high-IoU detection metrics suggests that the transferred representation particularly benefits localization.
Critical weaknesses
  1. Lack of support for the central claim. The paper argues that event-native recurrent backbones can absorb foundation-model semantics. The evidence is mixed: the RVT-only variant gains little in domain, and the full model transfers worse than the RVT baseline on GEN1.
  2. Limited methodological novelty. Cross-modal distillation from RGB models to event encoders is well established (EvDistill, ESS, ECDP, EventSAM, Depth AnyEvent, GEP), and the objectives are standard. The contribution is mainly an architectural combination and an efficiency study.
  3. Reproducibility. Results come from single runs, with no standard deviation reported.
Questions for authors

Please refer to the Weakness part.

2 comments
AuthorsOpenReview ↗

Response to Reviewer EFRL—W1

We sincerely thank the reviewer for the thoughtful evaluation of our work. We appreciate the constructive feedback and address the concerns regarding our central claim, methodological novelty, and reproducibility below.

W1. Clarification of the Central Claim Support and Interpretation of Experimental Results

We thank the reviewer for raising this important concern. We would like to clarify that our central claim is that event encoders incorporating event-native recurrent backbones can benefit from semantic knowledge transferred from RGB foundation models through cross-modal pretraining. The results highlighted by the reviewer are primarily from our architectural ablation study, whereas we would like to emphasize that the main evidence supporting our claim comes from the performance improvements achieved through RGB-guided pretraining across downstream tasks and datasets. Specifically, our RGB-guided pretraining improves DSEC object detection performance from 36.6 to 46.3 mAP (+9.7) under the same RVT+DINOv3 architecture. Furthermore, our learned representations improve GEN1 object detection performance from 47.2 to 50.2 mAP (+3.0) and achieve 65.68% mIoU on DSEC-Seg. To further strengthen this evidence, we have additionally conducted monocular depth estimation experiments on DSEC-Depth and MVSEC-Depth, five out of six state-of-the-art results have been achieved. Nevertheless, we would like to further clarify the reviewer's concerns regarding our architectural ablation results below.

First, regarding the effectiveness of RGB-guided pretraining on event-native recurrent backbones, our results demonstrate that the RVT encoder benefits from cross-modal pretraining. Since our pretraining is conducted exclusively on DSEC, the subsequent DSEC object detection fine-tuning is performed in an in-domain setting, where the RVT architecture may have limited room for further improvement in final detection accuracy. Apart from pretraining improves the final mAP from 38.4 to 38.8, its also benefits more in fine-tuning efficiency. Specifically, we found that the pretrained RVT encoder reaches comparable detection performance approximately 60k fine-tuning steps (17 hours on a single A6000 GPU) earlier than its non-pretrained counterpart. More importantly, when transferring to the unseen GEN1 dataset, the benefits of pretraining become more pronounced. The pretrained RVT-only encoder achieves 50.2 mAP, compared with 47.2 mAP when trained from scratch, yielding an improvement of +3.0 mAP. Together, these results demonstrate that event-native recurrent backbones can benefit from RGB foundation-model knowledge, both in downstream fine-tuning efficiency and cross-dataset transfer performance.

Second, regarding why the full RVT+DINOv3 model underperforms RVT-only on GEN1, we believe this is an important and worthwhile question that deserves discussion. The pretrained RVT-only encoder achieves 50.2 mAP, outperforming the pretrained RVT+DINOv3 encoder (45.3 mAP). This suggests that the architectural advantages of DINOv3 observed on DSEC do not necessarily translate into better cross-dataset transfer to GEN1. One possible explanation is that combining RVT with additional DINOv3 transformer blocks requires effective feature adaptation between the two components, which may be sensitive to differences between datasets. Moreover, the very lower spatial resolution of GEN1 may introduce a scale mismatch, potentially affecting cross-dataset transfer performance. Nevertheless, the full RVT+DINOv3 model continues to demonstrate strong performance across other downstream tasks, achieving 46.3 mAP on DSEC-Detection, 65.68% mIoU on DSEC-Seg, and state-of-the-art results on five out of six depth estimation metrics across DSEC-Depth and MVSEC-Depth. More Importantly, this architectural difference does not negate the effectiveness of RGB-guided pretraining itself, as demonstrated by the improvements of +9.7 mAP on DSEC with RVT+DINOv3 and +3.0 mAP on GEN1 with RVT-only. Compared with other downstream tasks, the effectiveness of cross-modal pretraining for event-based object detection remains relatively underexplored. Our work serves to advance the understanding of how RGB foundation-model knowledge can benefit event-based object detection tasks.

AuthorsOpenReview ↗

Response to Reviewer EFRL—W2&W3

W2. Methodological Novelty

Our main innovation lies in directly transferring rich semantic knowledge from an RGB foundation model into an event-native recurrent backbone through efficient, single-stage pretraining. Existing RGB-guided event pretraining methods (e.g., GEP, ECDP, and EventSAM) often adopt image-style backbones, such as ViTs, which can extract spatial features but lack explicit mechanisms for modeling temporal dependencies across consecutive event windows. However, such temporal modeling is particularly important for event streams, given their sparse, asynchronous nature and fine-grained temporal dynamics. In contrast, our framework integrates RVT's recurrent temporal modeling with DINOv3 transformer blocks into a unified event encoder, jointly optimized through feature alignment with a frozen RGB foundation model. Our state-of-the-art results across multiple downstream tasks further demonstrate the effectiveness of our proposed architecture in combining event-native temporal modeling with RGB foundation-model knowledge transfer.

Beyond our architectural innovation, our approach differs from existing methods in several important aspects. Most closely related to our work, GEP employs an image-style ViT event encoder and introduces an additional autoregressive pretraining stage to further capture temporal dependencies across consecutive event windows, resulting in a two-stage training pipeline with additional computational overhead. In contrast, our framework integrates temporal modeling and RGB-guided semantic alignment within a single pretraining stage, offering a simpler yet highly effective and computationally efficient design. Moreover, EvDistill, ESS, EventSAM, and Depth AnyEvent are early studies in the field of RGB-to-event knowledge transfer, most of which focus on models specifically designed for individual downstream tasks, such as ESS for semantic segmentation and Depth AnyEvent for monocular depth estimation. Building on these works, our work established a general-purpose event pretraining model that can effectively transfer RGB foundation-model knowledge across diverse downstream tasks, achieving superior performance in object detection, semantic segmentation, and depth estimation.


W3. Reproducibility and Statistical Reliability

Our reported results are based on multiple experimental runs. To assess their stability, we report the mean and standard deviation across several independent runs. On DSEC-Detection, our pretrained model achieved 46.30 ± 0.25 mAP, compared with 36.60 ± 0.48 mAP without pretraining. On GEN1, the pretrained RVT and RVT+DINOv3 models achieved 50.20 ± 0.28 and 45.30 ± 0.31 mAP, respectively. On DSEC-Seg, our method achieved 65.68 ± 0.18 mIoU. For depth estimation, our method achieved 0.923 ± 0.004 δ1 and 0.093 ± 0.003 Abs Rel on DSEC-Depth, and 0.634 ± 0.005 δ1 and 0.277 ± 0.008 Abs Rel on MVSEC-Depth. These results indicate consistent performance across runs. We also set random seeds and maintain consistent training and evaluation settings to ensure reproducibility and fair comparisons.


Additional Experimental Evaluation.

To further strengthen the empirical evidence supporting our framework, we have conducted additional monocular depth estimation experiments on DSEC-Depth and MVSEC-Depth. Our method achieves state-of-the-art performance on five out of six evaluation metrics across the two datasets. Specifically, compared with Depth AnyEvent-R, our method improves δ₁ from 0.691 to 0.923 and reduces Abs Rel from 0.191 to 0.093 on DSEC-Depth. On MVSEC-Depth, δ₁ improves from 0.489 to 0.634, while Abs Rel decreases from 0.365 to 0.277. These results further demonstrate the effectiveness and generalization capability of our learned event representations across different downstream tasks and datasets. We will include the complete experimental results and detailed analysis in the revised manuscript.

Reviewer_dtFJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LEEEP, a label-free pretraining method that transfers knowledge from an RGB foundation model to an event encoder. It uses a frozen DINOv3 teacher and an RVT+DINOv3 student, trained with global and patch-level feature alignment and InfoNCE loss. The authors report improvements on event-based object detection and semantic segmentation, with relatively low pretraining cost.

Critical strengths
  • The paper addresses a useful problem. Transferring knowledge from RGB foundation models to event backbones could reduce the dependence on labeled event data.

  • The downstream results are encouraging. The method achieves 46.3 mAP on DSEC-Detection and 65.68 mIoU on DSEC-Semantic, with clear improvements over the reported scratch baselines.

  • The relatively low pretraining cost is appealing. The paper also includes ablations on the student architecture, teacher model, and training objectives.

Critical weaknesses
  1. The generalization ability is not sufficiently demonstrated. One of the main motivations is to transfer generalizable representations from RGB foundation models to event encoders. However, pretraining is conducted on only 38 DSEC sequences, and most downstream experiments are also performed on DSEC. The only cross-dataset evaluation is object detection on GEN1, where the full RVT+DINOv3 model achieves 45.3 mAP, below both pretrained RVT-only (50.2) and RVT trained from scratch (47.2). These results provide limited evidence that the proposed framework learns generalizable event representations across datasets and tasks, which I consider an important part of the paper's claim.

  2. The methodological novelty is limited. Cross-modal distillation from RGB models to event representations has been explored in previous works such as ECDP and GEP. The proposed global/patch alignment and InfoNCE losses are also standard. The main difference appears to be combining RVT with pretrained DINOv3 blocks in a single-stage framework. The paper does not clearly establish what new representation learning insight this combination provides. More closely related work, such as Scaling Dense Event-Stream Pretraining from Visual Foundation Models (CVPR 2026), which also uses DINOv3 for event representation learning, should be discussed.

  3. The experimental results are mixed and do not convincingly justify the RVT+DINOv3 architecture. On DSEC, combining RVT with DINOv3 improves mAP from 44.2 to 46.3 over DINOv3 alone (Table 5). Moreover, the RVT+DINOv3 model trained from scratch achieves only 36.6 mAP, even worse than RVT alone (38.4), and the initialization of this baseline is not clearly explained. On GEN1, the full model also performs considerably worse than transferring RVT alone (45.3 vs. 50.2 mAP). These inconsistent results make it difficult to understand whether the proposed combination is actually beneficial and what each component contributes.

Questions for authors

Questions

  1. What is the actual contribution of the proposed RVT+DINOv3 architecture? Table 5 shows only a 2.1 mAP improvement over DINOv3-only (46.3 vs. 44.2). Meanwhile, the RVT+DINOv3 model trained from scratch achieves only 36.6 mAP, below RVT alone (38.4). How is the scratch baseline initialized, particularly the DINOv3 blocks? Could the authors compare against directly distilling DINOv3 features into RVT through a lightweight projection head, and isolate the benefit of cross-modal pretraining under identical initialization?

  2. How does the method differ from existing RGB-to-event pretraining methods? In particular, Scaling Dense Event-Stream Pretraining from Visual Foundation Models (CVPR 2026) also uses DINOv3 for event representation learning. Could the authors clarify the methodological differences and discuss how the results compare under different data and compute budgets?

  3. How well do the learned representations generalize? On GEN1, the full RVT+DINOv3 model achieves 45.3 mAP, below both pretrained RVT-only (50.2) and RVT from scratch (47.2). What explains this degradation? Could the authors provide additional cross-dataset evidence? Also, are the DSEC segmentation test sequences disjoint from those used for pretraining?

  4. How fair is the computational efficiency comparison? The reported pretraining cost is lower than GEP and TESPEC, but the models differ in architecture, parameter count, and pretraining data. Could the authors clarify whether all computational costs are measured consistently, including teacher inference, and provide a comparison of downstream performance against total training compute and model size?

4 comments
AuthorsOpenReview ↗

Response to Reviewer dtFJ—W1&Q3

We sincerely thank the reviewer for providing constructive feedback and raising several valuable questions. We have carefully addressed these concerns through additional experiments and analyses to further strengthen our work. Our detailed responses are provided below.


W1 & Q3. Generalization Across Tasks and Datasets

First, regarding generalization across tasks and datasets, although our RGB-guided pretraining uses only 38 sequences from DSEC-Detection, our experimental results could demonstrate effective transfer across multiple downstream tasks and datasets. To further strengthen the generalization evaluation, we have also conducted additional monocular depth estimation experiments on DSEC-Depth and MVSEC-Depth, extending our evaluation to geometric perception. Beyond object detection on DSEC-Detection and GEN1, our method achieves 65.68 mIoU on DSEC-Seg, demonstrating its effectiveness for semantic segmentation. We further clarify that the DSEC-Seg test sequences are strictly disjoint from the 38 DSEC-Detection training sequences used for pretraining, ensuring no sequence-level overlap between pretraining and segmentation evaluation. For the newly added depth estimation task, our method achieves state-of-the-art performance on five out of six evaluation metrics compared with the strongest baseline, including δ1 of 0.923 and Abs Rel of 0.093 on DSEC-Depth, and δ1 of 0.634 and Abs Rel of 0.277 on MVSEC-Depth. Detailed results are presented in the table below. These additional results, particularly on the unseen MVSEC dataset, provide further evidence of the transferability of our learned event representations across tasks and datasets.

MethodBackboneDSEC δ₁ ↑Abs Rel ↓RMSE ↓MVSEC δ₁ ↑Abs Rel ↓RMSE ↓
Task-specific Supervised Models
E2DepthRecurrent CNN0.4090.39513.2580.3630.5277.894
EReFormerSwin-T + GRViT0.5240.29711.6080.3610.5188.423
Pretrained Models
ECDDPViT-S/160.5450.2639.4770.5130.4286.957
ECDPViT-S/160.5280.32411.4730.4760.4967.680
DepthAnyEvent-RViT-S/160.6910.1918.8800.4890.3656.465
GEPViT-S/16———0.5500.3856.600
ScaleEventViT-B/160.8720.1094.0320.5940.2805.891
Ours (scratch)RVT+DINOv30.8060.1474.6870.5920.3177.393
OursRVT+DINOv30.9230.0933.2000.6340.2776.430

Second, regarding why the full RVT+DINOv3 model underperforms RVT-only on GEN1, the pretrained RVT-only encoder achieves 50.2 mAP, outperforming the pretrained RVT+DINOv3 encoder (45.3 mAP). This suggests that the architectural advantages of RVT+DINOv3 observed on DSEC do not necessarily translate into better cross-dataset transfer to GEN1. One possible explanation is that combining RVT with additional DINOv3 transformer blocks requires effective feature adaptation between the two components, which may be sensitive to differences in event characteristics across datasets. Moreover, GEN1 has a much lower spatial resolution than DSEC, potentially introducing a scale mismatch that affects cross-dataset transfer performance. Nevertheless, the full RVT+DINOv3 model demonstrates strong performance across other downstream tasks, outperforming representative prior methods such as GEP and ScaleEvent, and achieving state-of-the-art results. These results highlight the effectiveness of our RGB-guided pretraining framework across multiple downstream tasks and datasets. Compared with other downstream tasks, the effectiveness of cross-modal pretraining for event-based object detection remains relatively underexplored. Our work serves to advance the understanding of how RGB foundation-model knowledge can benefit event-based object detection tasks.

AuthorsOpenReview ↗

Response to Reviewer dtFJ—W2&Q2

W2 & Q2. Methodological Novelty and Comparison with Prior Work

Our main innovation lies in directly transferring rich semantic knowledge from an RGB foundation model into an event-native recurrent backbone through an efficient, single-stage pretraining framework. Existing RGB-guided event pretraining methods (e.g., GEP, ScaleEvent, and ECDP) often adopt image-style backbones, such as ViTs, as event encoders. However, event streams are inherently sparse, asynchronous, and characterized by fine-grained temporal dynamics, which differ fundamentally from conventional RGB images. While image-style ViT architectures can extract spatial representations from aggregated event inputs, they may not explicitly capture temporal dependencies across consecutive event windows without additional temporal modeling mechanisms. Our findings suggest that incorporating event-native backbones into cross-modal self-supervised learning can better capture event-specific spatiotemporal information, leading to improved downstream task performance. To this end, our framework integrates RVT's recurrent temporal modeling with DINOv3 transformer blocks, enabling RGB semantic knowledge transfer while preserving temporal continuity across event windows.

Regarding the reviewer's question on how our method differs from ScaleEvent, both methods leverage DINOv3 for RGB-to-event knowledge transfer but differ in their event encoder architectures and cross-modal knowledge transfer strategies. ScaleEvent directly employs a DINOv3-initialized ViT as its event encoder and introduces event activation masking and structure-aware distillation to preserve fine-grained semantic relationships. In our framework, a frozen DINOv3 serves as the primary source of RGB semantic guidance, while the event encoder is built around RVT's recurrent architecture, with subsequent DINOv3 transformer blocks enhancing its representations. We employ global- and patch-level feature alignment with contrastive learning to transfer RGB semantic knowledge into this event-native architecture. Overall, our method outperforms ScaleEvent on DSEC-Seg and achieves better results on five out of six depth estimation metrics across DSEC-Depth and MVSEC-Depth, while requiring lower pretraining computational cost. For a detailed comparison of results under different data and computational budgets, please refer to our response to Q4 below.

AuthorsOpenReview ↗

Response to Reviewer dtFJ—W3&Q1

W3 & Q1. Architectural Design and Ablation Analysis

First, regarding why we adopt the RVT+DINOv3 architecture and the contribution of each component, we conducted an architectural ablation study on DSEC object detection, as shown in the ablation study table. We evaluated three event encoder variants under the same RGB-guided cross-modal pretraining framework: RVT-only, DINOv3-only, and RVT+DINOv3. Notably, our RVT-only variant already corresponds to the reviewer-suggested baseline of directly distilling DINOv3 features into RVT through a lightweight projection head, without incorporating additional DINOv3 transformer blocks into the event encoder. The pretrained RVT-only and DINOv3-only variants achieve 38.8 and 44.2 mAP, respectively, while the full pretrained RVT+DINOv3 encoder achieves 46.3 mAP, outperforming both individual architectures. Therefore, we adopt the RVT+DINOv3 architecture for DSEC object detection tasks. Regarding the contribution of each component, RVT provides event-native spatiotemporal modeling, improving performance by 2.1 mAP over DINOv3-only (44.2 → 46.3), while DINOv3 transformer blocks facilitate richer semantic representation learning, improving performance by 7.5 mAP over RVT-only (38.8 → 46.3). Furthermore, under the same RVT+DINOv3 architecture, we observe the benefit of cross-modal pretraining, which improves detection performance by 9.7 mAP (36.6 → 46.3). These results highlight the complementary roles of RVT and DINOv3 transformer blocks, while also suggesting that their effectiveness depends on the interaction between architectural design and cross-modal pretraining.

Second, regarding the initialization and performance of RVT+DINOv3 without cross-modal pretraining on DSEC object detection, the RVT+DINOv3 model without cross-modal pretraining achieves 36.6 mAP (slightly below the RVT-only trained from scratch baseline at 38.4 mAP). For this experiment, we clarify that the DINOv3 transformer blocks are initialized from the official pretrained DINOv3 checkpoint and subsequently fine-tuned during downstream detection training, while the RVT backbone is randomly initialized. The RVT+DINOv3 model without cross-modal pretraining here is included primarily as an ablation test to evaluate the contribution of pretraining for main experimental results (46.3 mAP). The lower performance may arise because directly combining RVT with DINOv3 transformer blocks without pretraining requires effective additional feature adaptation between the two components, and simply increasing architectural complexity does not guarantee improved detection performance. This comparison further demonstrates that the substantial performance gain of pretrained RVT+Dinov3 model is primarily driven by our cross-modal pretraining rather than increased architectural complexity alone. RGB-guided pretraining may not only help bridge this feature adaptation gap but also elevate detection performance to a higher level, further demonstrating the effectiveness of our pretraining framework.

AuthorsOpenReview ↗

Response to Reviewer dtFJ—Q4

Q4. Computational Efficiency and Fair Comparison

To ensure a fair comparison, we normalize the computational costs of all compared methods to the equivalent of a single NVIDIA A6000 GPU. Specifically, as reported in the computational efficiency comparison table in our manuscript, Pre-training Time and EFLOPs measure the computational cost of the entire pretraining process, including RGB teacher inference for generating the representations used to guide the event student. Fine-tuning Time measures the time required to adapt the pretrained event encoder to DSEC-Seg, without any RGB teacher involvement, as only the event branch is used during downstream inference. Params reports the total number of parameters in the model used for DSEC-Seg inference. Finally, Total Time represents the sum of pretraining and DSEC-Seg fine-tuning time. These definitions and accounting boundaries are applied consistently across all compared methods.

In addition, we note that the compared methods differ in pretraining data. In particular, our method uses only DSEC for pretraining, whereas GEP uses Event-1.8M, ScaleEvent uses Image-Event-500K, and TESPEC uses 1Mpx, all of which are much larger than DSEC. This partly explains our lower pretraining cost, but it is also a key strength of our approach: achieving the best DSEC-Seg performance among the compared methods with limited paired RGB–event data, demonstrating our ability to learn strong and transferable event representations under a limited pretraining budget.

Building on the consistent evaluation setup described above, we further illustrate the trade-off between downstream performance, total pretraining compute, and model size through a direct comparison of the evaluated methods below.

MethodPre-training (h) ↓Fine-tuning (h) ↓Total Time (h) ↓Params (M)DSEC-Seg mIoU ↑
TESPEC22719.4246.44562.77
ScaleEvent10220.5122.576.7464.93
GEP230.323.32265.22
Ours92.411.48965.68

As shown in the table above, our framework achieves the highest DSEC-Seg mIoU (65.68) with substantially lower pretraining compute, despite having a larger model than GEP, ScaleEvent, and TESPEC. Specifically, our method requires only 9 A6000 GPU-hours for pretraining, 2.6×, 11.3×, and 25.2× less than GEP (23 h), ScaleEvent (102 h), and TESPEC (227 h), respectively. Even when including downstream fine-tuning, our total training time remains 2.0×, 10.7×, and 21.6× lower, respectively. Together with our results on object detection and depth estimation, these findings demonstrate that our approach can learn strong and transferable event representations with limited paired pretraining data and less computational resources.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response to Area Chair and Reviewers

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We have carefully addressed the concerns raised in our individual responses and uploaded a revised manuscript incorporating additional experiments, analyses, and clarifications. We summarize three key points below.


1. Additional Experiments and Generalization

To further evaluate the generalization of our method, we conducted additional monocular depth estimation experiments on DSEC-Depth and MVSEC-Depth, extending our evaluation beyond object detection and semantic segmentation. Our method achieves state-of-the-art results on five out of six depth estimation metrics compared with existing methods, including ScaleEvent, with Abs Rel of 0.093 on DSEC-Depth and 0.277 on MVSEC-Depth. These results provide further evidence of the effectiveness of our pretraining across different downstream tasks and datasets. We also appreciate Reviewer 3's acknowledgment of these additional experiments in the follow-up discussion.


2. Methodological Novelty

Our key contribution is demonstrating that event-native recurrent encoders can effectively benefit from rich semantic knowledge from RGB foundation models through single-stage, label-free cross-modal pretraining. Existing RGB-guided event pretraining methods (e.g., GEP and ScaleEvent) often adopt image-style backbones, such as ViTs, which can extract spatial features but lack explicit mechanisms for modeling temporal dependencies across consecutive event windows. However, such temporal modeling is particularly important for event streams, given their sparse, asynchronous nature and fine-grained temporal dynamics. In contrast, our framework integrates RVT's recurrent temporal modeling with DINOv3 transformer blocks into a unified event encoder, jointly optimized through feature alignment with a frozen RGB foundation model. Our state-of-the-art results across multiple downstream tasks further demonstrate the effectiveness of our proposed architecture in combining event-native temporal modeling with RGB foundation-model knowledge transfer.


3. Cross-Dataset Transfer to GEN1

We further discussed the different transfer performance of RVT and RVT+DINOv3 on GEN1. The pretrained RVT backbone achieves 50.2 mAP, outperforming RVT trained from scratch (47.2 mAP), while the full RVT+DINOv3 architecture achieves 45.3 mAP. This suggests that the architectural advantages of DINOv3 observed on DSEC do not necessarily translate into better cross-dataset transfer to GEN1. One possible explanation is that combining RVT with additional DINOv3 transformer blocks requires effective feature adaptation between the two components, which may be sensitive to differences between datasets. Moreover, the much lower spatial resolution of GEN1 may introduce a scale mismatch, potentially affecting cross-dataset transfer performance. Nevertheless, the improvement achieved by the pretrained RVT backbone alone demonstrates the value of transferring RGB foundation-model knowledge into an event-native recurrent encoder. Importantly, the full RVT+DINOv3 architecture continues to demonstrate strong performance across other downstream tasks, including object detection, semantic segmentation, and monocular depth estimation on DSEC.


We believe these additional experiments and clarifications further strengthen the evidence supporting our central contribution: efficiently transferring RGB foundation-model knowledge into event-native recurrent encoders for event-only downstream perception. We sincerely appreciate the reviewers' and Area Chair's time and consideration. We are still looking forward to the replies from the reviewers and welcome continued discussion. We would be happy to provide any additional clarifications if needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.