acceptodds
Under review as a conference paper at ICLR 2027

Teaching Pretrained Event Encoders to Digest Any Representation

Abstract

Pretrained event encoders have demonstrated strong transferability across a wide range of event-based vision tasks. Existing methods mainly focus on developing stronger pretraining architectures or distilling knowledge from powerful image encoders into the event domain, but typically rely on a fixed representation and overlook the differences among event representations. We observe that changing the input representation of a pretrained event encoder can substantially degrade downstream performance, even when the underlying event stream remains unchanged. We term this phenomenon representation lock-in. To address this issue, we present Omnivorous Event Encoder (OmniE), a pretraining framework for adapting pretrained event encoders across diverse representations through global and dense feature alignment. To accommodate unseen representations, we further introduce the Cross-Representation Adapter (CRA), which uses a shared operator-conditioned generator to produce native input projections and low-rank corrections to early Transformer blocks while keeping the pretrained event encoder frozen, thereby enabling its semantic knowledge to transfer across representations. Experiments across diverse event representations and datasets show that OmniE substantially improves cross-representation transfer, achieving consistent gains in semantic segmentation, object classification, and depth estimation. https://anonymous.4open.science/r/OmniE2-3B61[CODE]

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Event-based Vision. Cross-representation Adaptation. Pretrained Event Encoders. Feature Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TfkUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose OmniE2OmniE^2, a way to reuse an HTA-pretrained event encoder on other input representations. It works with a Cross-Representation Adapter (CRA) that builds the patch kernel and LoRA-style updates from hand-written channel descriptors. CRA is trained to match HTA features while the backbone stays frozen. Experiments on N-Caltech101, DSEC, DDD17 and MVSEC cover classification, segmentation, depth and retrieval. They show gains over a Hyper-RED baseline that copies the HTA patch kernel to each input channel.

Critical strengths
  1. Extensive experiments. The evaluation covers 3 tasks (classification, segmentation, depth), 5 dataset-task settings, 4 seen representations and 2 unseen ones. It also includes retrieval and qualitative results.
  2. Well-designed ablations. The adapter comparison uses parameter-matched controls, and ID-CRA isolates the effect of descriptor conditioning from model capacity.
Critical weaknesses
  1. The practical motivation is unclear. Every representation is computed from the raw events, and every dataset here provides them, so a user could simply convert to HTA. HTA wins on nearly every metric in Tables 2 and 3. Because the loss pulls everything toward HTA, the adapter can at best match it.
  2. The baselines are insufficient, and the comparison is not budget-matched. Both models are fine-tuned with the same small learning rates (1e-6 to 2e-5). OmniE2, however, first gets 60k extra CRA updates on the same training data, with a patch learning rate of 1e-4. It is therefore unclear whether the gains come from descriptor conditioning or simply from extra training of the input layer.
  3. The results may not be statistically reliable. All results come from a single run. The HTA input bypasses CRA, yet its results differ between rows (65.58 vs. 64.86 mIoU on DDD17), which suggests run-to-run noise as large as many of the reported gains.
Questions for authors

Please refer to the Weakness part.

1 comment

Thanks for the author's detailed reply. I have increased my rate

Reviewer_mrcGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies performance degradation when an event encoder is transferred to representations different from its pretraining input. It proposes OmniE2, whose cross-representation adapter (CRA) uses channel-level operator descriptors to generate input projections and low-rank updates to early Transformer blocks. Global and dense feature alignment use the frozen HTA encoder as a reference. The original encoder is frozen during CRA pretraining; generated weights are then merged before supervised fine-tuning of the full encoder and task head. Evaluation covers retrieval, classification, segmentation, and depth estimation, with additional fixed-head evaluation on seen representations and transfer experiments on held-out representations, Time Surface, and SBT-5.

Critical strengths

1 A practical problem supported by evidence across tasks. Event data have multiple common representations, making input mismatch a real obstacle to reusing pretrained encoders. In Table 2, mean segmentation mIoU over non-HTA representations improves from 63.16 to 64.15 on DSEC and from 58.06 to 60.93 on DDD17. Appendix Table 13 also shows adaptation gains with a single fixed HTA-trained classifier. These results support practical utility and indicate that the benefits do not depend entirely on subsequent full-model fine-tuning.

2 A concrete operator-conditioned design with relevant controls. CRA uses a common descriptor scheme to accommodate native channel counts and generate input projections and early-layer updates. The appendix details the descriptors and representation construction rules, and includes approximately parameter-matched Basis+LoRA and ID-CRA controls, as well as the larger Patch+LoRA model. This structured design is more interpretable than assigning unrelated parameters to each representation and provides a basis for testing the value of operator semantics.

Critical weaknesses

1 Inconsistent checkpoint selection confounds attribution of a small advantage. Appendix C.3, p. 22, lines 1148–1151 states that the new controls use their final pretraining checkpoints, whereas CRA retains its original checkpoint-selection protocol. In Table 5, CRA exceeds ID-CRA by only 0.26 percentage points in mean depth δ1 (79.36 versus 79.10), without variability across repeated runs. Operator semantics, unequal model-selection opportunities, and stochastic variation therefore cannot be separated. This directly affects the central evidence for the conditioning mechanism. A common checkpoint rule and repeated-run results are needed.

2 Unseen-representation results do not sufficiently isolate semantic generalization. Table 4 and Appendix C.5 evaluate zero-shot initialization followed by supervised fine-tuning against Hyper-RED without additional adapter pretraining. There is no equal-pretraining-budget shared-adapter or descriptor control, while the fixed-encoder, shared-head experiment in Table 13 covers only seen representations. The results establish useful initialization but do not separate operator conditioning from general multi-representation training and downstream supervision. Evaluation is also limited to one HTA-pretrained backbone and the existing descriptor vocabulary.

Questions for authors

1 Additional validation With identical checkpoint-selection rules and paired random seeds, does CRA still outperform ID-CRA and both LoRA controls? Please report means and variability across repeated runs, particularly for the 0.26-percentage-point difference in Table 5.

2 Additional validation How does a frozen encoder with the same HTA-trained classifier perform on held-out representations, Time Surface, and SBT-5? Relative to an equal-budget shared adapter or shuffled-descriptor control, how much gain is attributable to operator semantics?

  1. Which validation samples and metrics are used for CRA pretraining, downstream early stopping, and loss-weight selection? The paper excludes test samples from CRA checkpoint selection, but Table 9 only gives train/test sizes. Please specify the validation splits and selection rules at each stage.

4 Clarification Which paper results correspond to the released checkpoints? Can the CRA generators, actual training configurations, and evaluation entry points for unseen representations and ablations be provided for independent verification?

Reviewer_9WTXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents omnivorious event encoder (OmniE2), a pretraining framework for pretrained event encoders across diverse representations through global and dense feature alignment. They further introduce a Cross-Representation Adapter (CRA) that uses a shared operator-conditioned generator for native input projections and low-rank corrections to early Transformer blocks. The results show that the propresed OmniE2 achieves consistent improvements in semantic segmentation, object classification, and depth estimation. Overall, the problem is clearly motivated, the method is well targeted, and the empirical results support improved cross-representation transfer.

Critical strengths
  1. Pretrained event encoders are highly meaningful foundation models and are useful across a wide range of event-based vision tasks.
  2. The authors conduct extensive experiments, including in the appendix, which thoroughly demonstrate the effectiveness of the model.
Critical weaknesses
  1. Although the proposed pretrained event encoder improves performance across different tasks, the authors do not report the encoder’s inference time. For event cameras, low-latency inference is a critical metric. Please compare inference time across different pretrained encoders.
  2. Event streams are continuous, asynchronous, spatiotemporal sparse point signals, and their representation is closely related to the number of events and the event time window. Please provide ablation studies on how different event counts and window sizes affect downstream task performance.
  3. Event streams reflect intensity changes, and using events alone yields limited downstream performance. Jointly using events and frames is a very promising direction. Could the authors try a joint encoder using events and frames, enable asynchronous inference, and maintain high-frequency inference?
Questions for authors

Please respond to each comment in the Weaknesses section. In addition, please address the following questions:

  1. The selected tasks (i.e., classification, depth estimation, and depth estimation) belong to low-level vision tasks. Why did the authors not validate on high-level vision tasks such as object detection? Please discuss this in the discussion section.
  2. There are emerging works on event-based MLLMs, and this may become a trend. Can the proposed omnivorous event encoder (OmniE2) be used for MLLMs? Please outline the idea and discuss it in the discussion section.
1 comment
Reviewer_9WTXOpenReview ↗

Response to the Rebuttal

I thank the authors for their careful and thorough response to all my questions, including the additional experiments. These clarifications have improved my understanding of the paper. I therefore maintain my original score of weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.