Label-free Efficient Event Encoder Pretraining from RGB Foundation Models

Event cameras offer microsecond latency, high dynamic range, and freedom from motion blur, but labeled event data are scarce and costly to annotate, so event vision has not benefited from foundation-model-scale pretraining. Existing event pretraining methods either learn from events alone at substantial cost (up to hundreds of GPU-hours on large event corpora), or transfer semantics from RGB foundation models by converting events into frames for an image transformer with an appended temporal stage, leaving the recurrent, event-native backbones that are standard in event-based detection without pretraining. We propose a single-stage, label-free cross-modal pretraining method that keeps the event-native recurrent backbone intact. A frozen DINOv3 teacher processes synchronized RGB frames, while an event encoder, a Recurrent Vision Transformer (RVT) followed by DINOv3 transformer blocks, processes the event stream and is trained to match the teacher's global and patch-level features through cosine alignment and an auxiliary patch-level InfoNCE objective. No labels are used, and the RGB branch is discarded after pretraining, so inference is event-only. Pretraining requires 7k steps, 9 A6000 GPU-hours, and 1.2 EFLOPs on DSEC alone, i.e., 2.7$\times$ and 26$\times$ fewer FLOPs than GEP and TESPEC. On DSEC-Detection, our model reaches 46.3 mAP, improving over RVT trained from scratch (38.4) by +7.9 mAP and over the same architecture trained from scratch (36.6) by +9.7 mAP, with AP${}_{75}$ rising from 38.1 to 53.5. Transferring only the pretrained RVT backbone across sensors raises GEN1 from 47.2 to 50.2 mAP, and on DSEC semantic segmentation we obtain 65.68 mIoU, on par with GEP (65.22) and above TESPEC (62.77), both pretrained on substantially larger corpora. These results suggest that dense semantic supervision from an RGB foundation model, applied directly to an event-native backbone, is a compute-efficient alternative to large-scale event pretraining.