Teaching Pretrained Event Encoders to Digest Any Representation

Pretrained event encoders have demonstrated strong transferability across a wide range of event-based vision tasks. Existing methods mainly focus on developing stronger pretraining architectures or distilling knowledge from powerful image encoders into the event domain, but typically rely on a fixed representation and overlook the differences among event representations. We observe that changing the input representation of a pretrained event encoder can substantially degrade downstream performance, even when the underlying event stream remains unchanged. We term this phenomenon representation lock-in. To address this issue, we present Omnivorous Event Encoder (OmniE${}^2$), a pretraining framework for adapting pretrained event encoders across diverse representations through global and dense feature alignment. To accommodate unseen representations, we further introduce the Cross-Representation Adapter (CRA), which uses a shared operator-conditioned generator to produce native input projections and low-rank corrections to early Transformer blocks while keeping the pretrained event encoder frozen, thereby enabling its semantic knowledge to transfer across representations. Experiments across diverse event representations and datasets show that OmniE${}^2$ substantially improves cross-representation transfer, achieving consistent gains in semantic segmentation, object classification, and depth estimation. https://anonymous.4open.science/r/OmniE2-3B61[CODE]