acceptodds
Under review as a conference paper at ICLR 2027

Fidelity-aware Event Representation Learning

Abstract

Event cameras excel in dynamic sensing. Harnessing this potential hinges on constructing faithful, compact, and statistically stable representations from irregular events, a fundamental yet nontrivial challenge. Existing event representations face a persistent dilemma: information fidelity is traded for statistical tractability through progressively stronger structural constraints. To reconcile this tension, we propose a fidelity-aware event representation framework that distills asynchronous events into physics-grounded latents while preserving temporally structured event information. Raw events are first structured into conflict-free volumes using the coarsest temporal discretization observed to avoid intra-bin event collisions. We then employ a forward-physics transformation that integrates sparse differential signals into relative log-intensity trajectories. This transformation is exactly invertible by temporal differencing, reorganizing the information retained after discretization into a more coherent temporal form without further information loss, thereby bridging information fidelity and statistical tractability. A variational autoencoder subsequently maps these proxy trajectories into compact and distributionally regular latents that serve as model-ready representations. Extensive experiments demonstrate faithful and stable event latents that transfer across diverse domains and visual objectives, spanning semantic understanding, motion perception, and photometric reconstruction. These results support physics-guided latent learning as an effective approach to event representation. Code will be publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Event Representations, Event Latents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rptiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new event representation in which events are accumulated into voxels with many temporal bins, such as 100 or 300 bins. These sparse voxels are then converted into temporal accumulations and compressed into latent representations using a VAE. The authors claim that this representation improves performance on semantic segmentation, optical flow estimation, and video reconstruction.

Critical strengths
  • The finding that this representation can help models train more effectively would be valuable, if it can be properly validated.
Critical weaknesses
  • A major problem with this paper is its exaggerated language and serious overclaiming. After reading the actual method, I felt that much of my time had been wasted.

    • The phrase “structured into conflict-free volumes” apparently refers to accumulating events into temporal bins with a width of 100 μs, corresponding to 10,000 FPS. This relies on the empirical observation that no intra-bin collisions occur in the tested data. However, considering that event cameras can theoretically provide temporal resolutions on the order of 1×1061 \times 10^6 FPS, this design choice is questionable.
    • The “forward-physics transformation” is essentially just temporal accumulation. This terminology makes the method sound unnecessarily mysterious.
    • The paper uses terms such as “information fidelity” to imply that this representation preserves more information than ordinary voxel representations. However, in the final VAE encoding step, the information is still compressed into only one or three channels. Thus, the information is ultimately heavily compressed.
  • If the paper could sufficiently demonstrate the advantages of this representation, it would make a good contribution to the field. However, the experiments are not sufficiently convincing:

    • For the downstream tasks, the paper should demonstrate that the proposed representation consistently outperforms the original representation across different model architectures, rather than evaluating only one architecture per task. Simply listing results from different model architectures provides limited additional value for this purpose.
    • I have concerns about the sources and reliability of the data reported in the paper (see Questions).
    • The paper lacks ablation studies. It should provide quantitative experiments showing that the representation with the accumulation step improves downstream performance compared with directly feeding sparse voxels into the VAE, rather than only comparing the statistical properties and sensitivity of the two representations.

Minor Problems

These issues do not affect my review decision and are provided only for the authors’ consideration during revision:

  • Table 3 does not highlight the best results consistently with Tables 1 and 2.
  • Only Table 3 includes the note that “Methods marked with † provide public training code.”
Questions for authors
  • In Tables 1–3, the authors should clearly indicate:

    1. which results are directly copied from the original papers;
    2. which results are obtained by running inference with checkpoints provided by the original authors and evaluating them using a unified set of metrics; and
    3. which results are obtained by retraining and evaluating the methods in this paper.

    I have this concern because all the other numbers in Table 2 appear to be consistent with the results reported in the original STSC paper, while the performance reported for STSC itself is worse in this paper than in the original paper.

12 comments

Thank you for the careful review.

Q1. The phrase “structured into conflict-free volumes” apparently refers to accumulating events into temporal bins with a width of 100 μs, corresponding to 10,000 FPS. This relies on the empirical observation that no intra-bin collisions occur in the tested data. However, considering that event cameras can theoretically provide temporal resolutions on the order of (1 \times 10^6) FPS, this design choice is questionable.

R1: Microsecond timestamp precision describes how finely event times are recorded, not a million-frame-per-second acquisition rate or one event per pixel every microsecond. It should be distinguished from practical pixel-response latency.

As discussed in Sec. 3.1, our empirical 100 μs scale is comparable to reported pixel-response latencies of 100–220 μs for the IMX636 [1, p. 12] and approximately 100 μs for the DVS240 under the tested conditions [2, p. 9]. These values provide physical context, not a universal collision-free bound. We select 100 μs through event-restoration analysis across the evaluated windows, preserving event assignments and polarities while quantizing timestamps.

Downstream sensitivity. We compare 50, 100, and 200 μs with all other settings fixed within each task. At 200 μs, collisions occur and only the latest event per cell is retained. Acc denotes mean per-class accuracy; segmentation scores are percentages.

Bin widthDSEC-Semantic Acc ↑DSEC-Semantic mIoU ↑DSEC-Flow EPE ↓DSEC-Flow AE ↓HQF MSE ↓HQF SSIM ↑HQF LPIPS ↓
50 μs93.5871.810.6472.360.0380.6280.284
100 μs93.6471.840.6462.360.0380.6290.283
200 μs93.5271.530.6582.540.0390.6270.283

Halving the bin width to 50 μs doubles the bin count without improving the reported metrics; 200 μs worsens most metrics. These results assess sensitivity to the empirically selected resolution, rather than determine its selection. Since bin width changes both timestamp quantization and event retention, the ablation does not isolate collision effects. Finer sensor timestamp precision alone does not dictate a finer voxelization grid.

[1] Guo R, et al. EventLFM: Event camera integrated Fourier light field microscopy for ultrafast 3D imaging. Light: Science & Applications, 2024.

[2] Holešovský O, et al. Experimental comparison between event and global shutter cameras. Sensors, 2021.

Q2. The “forward-physics transformation” is essentially just temporal accumulation. This terminology makes the method sound unnecessarily mysterious.

R2: The operation is temporal accumulation; “forward-physics” describes its role in the representation, not a new mathematical operator.

Full trajectories. Rather than reducing a window to one polarity aggregate, we retain the cumulative state at every temporal bin. Event timing and polarity are encoded in trajectory changes, and the full trajectory is used for learned compression.

Explicit causal state evolution. Voxel grids encode temporal distributions through binning and interpolation. Our transformation makes cumulative state evolution explicit: each state incorporates the preceding state and current event increment, without depending on future events. The term refers to this time-directed construction motivated by the event-generation model.

Q3. The paper uses terms such as “information fidelity” to imply that this representation preserves more information than ordinary voxel representations. However, in the final VAE encoding step, the information is still compressed into only one or three channels. Thus, the information is ultimately heavily compressed.

R3: The VAE representation is intentionally compact and lossy, but latent channels and temporal bins serve different roles. A three-bin voxel grid merges events into three temporal intervals; our three-channel latent is learned from the full 300-bin cumulative trajectory under reconstruction supervision. Equal channel counts therefore do not imply equivalent temporal information processing.

We test this distinction on DSEC-Semantic with the same ScaleEvent/ViT-L/16 backbone, 30 ms windows, and training protocol, adapting only the input projection. Voxel grids use B=CB=C bins, while our latents use D=CD=C channels from a fixed B=300B=300 volume (Table 5).

Input channels31030100300
Voxel Grid69.6569.1668.0166.4264.36
Our Latent71.8471.8871.5671.0270.75

Metric: mIoU (%).

Increasing voxel bins from 3 to 300 reduces mIoU by 5.29 points, while our latents maintain 70.75–71.88 mIoU. The advantage is 2.19 points at three channels and 6.39 points at 300 channels, supporting the importance of how temporal information is organized and encoded.

“Information fidelity” refers to temporal-structure preservation under compression, evaluated through trajectory reconstruction in Figures 6 and 7. Figure 5(a) shows concentrated low-frequency DCT energy in the evaluated trajectories, consistent with accurate reconstruction at small latent capacities. Reducing latent channels limits learned coding capacity; reducing voxel bins directly coarsens temporal discretization. Our compression learns from the finely resolved trajectory, while the channel ablation demonstrates its downstream utility across coding capacities.

Q4. For the downstream tasks, the paper should demonstrate that the proposed representation consistently outperforms the original representation across different model architectures, rather than evaluating only one architecture per task. Simply listing results from different model architectures provides limited additional value for this purpose.

R4: We have added a paired evaluation beyond ScaleEvent and a complementary shared-backbone representation comparison.

Paired evaluation with an additional pretrained model. On DSEC-Semantic, the RGB-pretrained DINOv3 ViT-L/16 voxel-grid and latent variants share the same pretrained initialization, segmentation head, and training protocol, with input projections adapted as needed. Table 15 in Appendix A.4.1 reports the results alongside ScaleEvent for reference.

Segmentation pipelineVoxel Grid mIoU (%)Our Latent mIoU (%)Improvement
ScaleEvent / ViT-L/1669.6571.84+2.19
DINOv3 / ViT-L/1664.3170.29+5.98

The 5.98-point gain extends the evidence to another pretraining regime and segmentation pipeline. The ScaleEvent voxel-grid result comes from the official checkpoint, as documented in Appendix A.2.5; the matched DINOv3 experiment is the additional paired evaluation. Because both use ViT-L, these results do not yet establish transfer across distinct architecture families.

Complementary shared-backbone comparison. Table 7 compares representative encodings within the ScaleEvent/ViT-L/16 segmentation pipeline, adapting input projections as needed. Acc denotes mean per-class accuracy.

RepresentationRepresentation principleAcc (%) ↑mIoU (%) ↑
HATSAveraged time surfaces92.4366.41
TORETime-ordered recent-event histories92.5767.52
HistogramsPolarity-wise event accumulation92.7668.29
Event FrameEvent accumulation into frames92.6567.75
ESTDifferentiable event-tensor construction92.8369.14
Voxel GridTemporal interpolation and accumulation93.1069.65
E2VIDLearned event-to-video reconstruction92.5467.03
P-latent (Ours)VAE compression of the cumulative trajectory93.6471.84

Our latent achieves 71.84 mIoU, exceeding Voxel Grid by 2.19 points and E2VID by 4.81 points. This comparison broadens the representation baselines under a common downstream model and complements the additional DINOv3 evaluation.

Reviewer_sWBmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a task-agnostic framework for event representation learning. The framework first constructs a conflict-free temporal voxel volume, then applies a prefix-sum transformation to convert the sparse volume into a denser proxy, and finally uses a lightweight variational autoencoder to compress the proxy into a regularized latent tensor. The authors evaluate proxy reconstruction and latent statistics on DSEC and seven cross-domain datasets. They further test the learned representation by replacing the input representation in pipelines for semantic segmentation, optical-flow estimation, and event-to-video reconstruction. Compared with ScaleEvent, the method improves mIoU by 0.87 and 2.19 percentage points on DDD17-Seg and DSEC-Semantic, respectively, while comparison with STSC reduces EPE from 0.677 to 0.646. The paper is comprehensively written with rich content, and the results are encouraging.

Critical strengths
  1. The paper provides a comprehensive and coherent study of the proposed representation. It develops the method from the motivation and event-generation process through conflict-free voxelization, physics-based reorganization, and latent compression, and then evaluates the resulting representation from several complementary perspectives. The experiments examine proxy reconstruction, latent regularity, cross-domain behavior, downstream transfer, ablations, qualitative reconstructions, and computational cost. The study covers DSEC and seven cross-domain datasets, explores different temporal windows, latent dimensions, and KL-regularization strengths, and includes semantic segmentation, optical-flow estimation, and event-to-video reconstruction. Together with the implementation details in the supplementary material, these experiments make the evaluation broad and internally connected.
  2. The experiments can provide evidence for the advantages of the proposed design at both the representation and downstream levels. For semantic segmentation, mIoU improves by 0.87 and 2.19 points on DDD17-Seg and DSEC-Semantic. For optical flow, EPE decreases from 0.677 to 0.646 and the other reported error metrics also improve. For event-to-video reconstruction, the proposed representation improves MSE, SSIM, and LPIPS over the discrete voxel baseline on IJRR, MVSEC, and HQF. These results support the usefulness of the proposed representation in the tested pipelines, while the broader architectural generalization of the method remains an open question discussed below.
Critical weaknesses
  1. The downstream study evaluates one representative pipeline for each task, using ScaleEvent for semantic segmentation, STSC for optical flow, and V2V-E2VID for event-to-video reconstruction. However, the observed gains are established only within these three pipelines. The other methods reported in the tables provide useful benchmark context, but they do not show how the proposed representation performs when inserted into those architectures. A result from a substantially different backbone for at least one task, or a shared-backbone comparison among representative event representations, would therefore provide stronger evidence for broader transfer. The proposed system also introduces a pretrained encoder, so keeping the downstream network unchanged does not equalize total computation or pretraining resources. A learned-encoder control with matched capacity and pretraining would help clarify how much of the gain comes from the proposed organization itself. The existing V-versus-P latent-learning ablation is informative, but it does not answer this attribution question at the level of downstream accuracy.
  2. The supplementary material does include a useful efficiency analysis. Appendix reports complete autoencoder forward passes on a single NVIDIA A6000 with batch size 1 and float32 precision. These measurements support the feasibility of the autoencoder component on the tested GPU, but they do not establish the cost of the complete event-to-task pipeline. The paper does not separately report the time for voxelization and proxy construction, encoder-only inference, downstream prediction, data transfer, or peak memory, and it does not provide a same-hardware timing comparison with the voxel-grid baseline. The measured configurations also do not cover all of the actual downstream settings. In particular, the optical-flow pipeline encodes ten consecutive T10-B100-D1 segments, which are not included in the efficiency table. Since streaming requirements depend on both the window duration and the output update interval, the practical trade-off should be reported for the configurations used in the downstream experiments. The current results show that the autoencoder can run quickly on the tested GPU, but they do not yet establish complete real-time event-to-task operation or imply that such operation is infeasible.
Questions for authors
  1. Can the proposed representation be evaluated with at least one substantially different backbone or within a shared downstream model that compares representative event representations? A learned-encoder control with matched model capacity and pretraining would also help determine how much of the reported gain comes from the proposed organization rather than from the additional encoder and its pretraining. This would clarify whether the results in Tables 1–3 support transfer beyond the three tested pipelines.
  2. Beyond the complete-autoencoder measurements already reported, can the authors provide end-to-end latency, throughput, and peak memory from raw events to final predictions for the actual downstream configurations, together with a same-hardware voxel-grid baseline? The timing should identify the contributions from voxelization and proxy construction, encoder-only inference, downstream prediction, and data transfer. It should also include the ten consecutive T10-B100-D1 segments used by the optical-flow pipeline and state the event-window duration and output update interval. These measurements would establish whether the observed accuracy gains are compatible with complete real-time event-to-task operation.
4 comments

Thank you for the careful review.

Q1. The gains are demonstrated only within ScaleEvent, STSC, and V2V-E2VID; benchmark results from other methods do not establish broader transfer. A substantially different backbone or a shared-backbone comparison of representative event representations would strengthen the evidence. Moreover, keeping the downstream network unchanged does not equalize total computation or pretraining resources. A learned-encoder control with matched capacity and pretraining would help isolate the contribution of the proposed organization. The existing V-versus-P latent-learning ablation is informative, but it does not answer this attribution question at the level of downstream accuracy.

R1: We have added a shared-backbone representation comparison and a matched-encoder ablation to address effectiveness and gain attribution separately.

Shared-model representation comparison. Table 7 compares temporal-history, grid-based, and learned reconstruction representations on DSEC-Semantic using ScaleEvent/ViT-L/16, adapting input projections as needed. Acc denotes mean per-class accuracy.

RepresentationRepresentation principleAcc (%) ↑mIoU (%) ↑
HATS [1]Averaged time surfaces92.4366.41
TORE [2]Time-ordered recent-event histories92.5767.52
Histograms [3]Polarity-wise event accumulation92.7668.29
Event Frame [4]Event accumulation into frames92.6567.75
EST [5]Differentiable event-tensor construction92.8369.14
Voxel Grid [6]Temporal interpolation and accumulation93.1069.65
E2VID [7]Learned event-to-video reconstruction92.5467.03
P-latent (Ours)VAE compression of the cumulative trajectory93.6471.84

Our latent achieves 71.84 mIoU, exceeding TORE, Voxel Grid, and E2VID by 4.32, 2.19, and 4.81 points, respectively. This establishes effectiveness against alternative representation mechanisms within the same downstream model. E2VID provides an additional learned front-end comparison; the paired control below specifically matches encoder capacity and pretraining.

V-latent vs. P-latent. V-latent learns to encode and reconstruct sparse voxels, while P-latent learns their cumulative trajectories. All other settings, including encoder capacity, pretraining resources, and downstream training, are fixed within each task (Table 4).

Dataset / TaskMetricV-latentP-latent
DDD17 segmentationAcc (%) ↑91.0892.84
DDD17 segmentationmIoU (%) ↑57.3666.12
DSEC segmentationAcc (%) ↑91.6793.64
DSEC segmentationmIoU (%) ↑60.3171.84
DSEC optical flowEPE ↓0.7830.646
DSEC optical flowAE ↓2.872.36
IJRR reconstructionSSIM ↑0.5370.596
IJRR reconstructionLPIPS ↓0.2840.236
MVSEC reconstructionSSIM ↑0.2750.376
MVSEC reconstructionLPIPS ↓0.5810.490
HQF reconstructionSSIM ↑0.5460.629
HQF reconstructionLPIPS ↓0.3400.283

Cumulative reorganization improves every reported metric, including 8.76/11.53 mIoU points on DDD17/DSEC and 17.5% lower flow EPE. Since accumulation is invertible before compression, both variants start from the same discretized event information. Their downstream gap therefore supports the value of organizing that information for latent learning under matched capacity and pretraining. This extends the reconstruction ablation to task accuracy; sharing a downstream backbone alone does not equalize total resources across the broader comparison.

[1] Sironi et al. HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification. CVPR, 2018.

[2] Baldwin et al. Time-Ordered Recent Event (TORE) Volumes for Event Cameras. IEEE TPAMI, 2023.

[3] Maqueda et al. Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars. CVPR, 2018.

[4] Rebecq et al. Real-time Visual-Inertial Odometry for Event Cameras using Keyframe-based Nonlinear Optimization. BMVC, 2017.

[5] Gehrig et al. End-to-End Learning of Representations for Asynchronous Event-Based Data. ICCV, 2019.

[6] Zhu et al. Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion. CVPR, 2019.

[7] Rebecq et al. High Speed and High Dynamic Range Video with an Event Camera. IEEE TPAMI, 2021.

Q2. The appendix provides useful complete-autoencoder timings on a single NVIDIA A6000 (batch size 1, float32), but does not establish full event-to-task costs. It lacks separate measurements for voxelization, proxy construction, encoder-only inference, downstream prediction, and data transfer, alongside peak memory and a same-hardware comparison with the voxel-grid baseline. Measurements should cover the actual downstream configurations, particularly optical flow’s ten consecutive T10-B100-D1 segments, which are absent from the efficiency table. Streaming feasibility should also account for both window duration and output update interval. The current results demonstrate fast autoencoder execution on the tested GPU, but neither establish nor rule out complete real-time event-to-task operation.

R2: Following your suggestions, Appendix A.3.2 now reports CPU preprocessing, encoder-only inference, downstream network complexity and memory, and task-level processing costs (Tables 10–14). The measurements include the actual ten-segment optical-flow batch and a same-hardware voxel-grid comparison on DSEC-Semantic.

(1) Measurement scope. CPU preprocessing is measured on a dual-socket Intel Xeon Silver 4309Y system; GPU inference uses a single NVIDIA A6000 and FP32. The preprocessing and encoder tables cover multiple temporal configurations and spatial resolutions. Downstream deployment uses only the VAE encoder, excluding its reconstruction decoder.

We define processing latency as the sum of separately measured representation-construction, encoding, and downstream-prediction times. CPU–GPU transfer and routine data formatting are excluded consistently. Each task-level sample corresponds to one prediction; these component sums do not measure sustained end-to-end throughput.

(2) Task-level processing costs.

Task and representationVoxelization (ms)Proxy construction (ms)Encoder-only inference (ms)Downstream prediction (ms)Processing latency (ms)
DSEC-Semantic: Voxel Grid4.240——54.62058.860
DSEC-Semantic: Our latent14.2782.0305.85154.62076.779
DSEC-Flow: Our latent17.2731.8207.27774.302100.672

For DSEC-Semantic, all stages are profiled at 640 × 440, with T30-B300-D3 for our representation. Relative to the voxel-grid baseline, our latent adds 17.92 ms of processing and improves mIoU by 2.19 points.

For DSEC-Flow, a 100 ms window comprises ten consecutive T10-B100-D1 segments at 640 × 480. They are processed jointly and encoded as one batch, consistent with parallel segment processing in the voxel-grid pipeline. The encoder achieves 137.42 batches/s, or 7.277 ms for all ten segments. Combining this with the measured preprocessing and STSC prediction gives 100.672 ms per prediction, without extrapolating from single-segment FPS.

(3) Downstream prediction and memory.

Downstream networkInput resolutionParameters (M)GFLOPsGPU inference latency (ms)Peak GPU memory (GB)Peak host memory (GB)
ScaleEvent / ViT-L/16640 × 440239.09758.1754.6205.723.17
STSC640 × 4804.40701.2374.3023.5611.35
V2V-E2VID240 × 18010.7120.077.4101.514.52

These memory figures cover the downstream networks, excluding the upstream encoder. Video reconstruction is profiled at the downstream-network level; a complete task-level latency breakdown and whole-pipeline memory peaks are not established by these measurements.

(4) Streaming implications. An observation window specifies temporal context, whereas the output-update interval specifies how frequently predictions are required. If flow predictions are required every 100 ms, the summed 100.67 ms processing cost slightly exceeds that interval under the reported scope. Sustained operation additionally depends on transfer overhead and scheduling. The revised analysis therefore quantifies the processing trade-off without claiming complete real-time operation from encoder throughput alone.

Changes to the paper.

  1. Sec. 4.3 (Table 7) adds a shared-backbone comparison spanning temporal-history, grid-based, and learned reconstruction representations on DSEC-Semantic.
  2. Sec. 4.3 (Table 4) adds the V-latent/P-latent downstream control with matched encoder capacity and pretraining, separating the effect of cumulative reorganization from additional learning resources.
  3. Appendix A.3.2 (Tables 10–14) reports CPU preprocessing, encoder-only inference, downstream complexity and memory, and task-level processing costs, including the complete ten-segment optical-flow batch and a same-hardware segmentation baseline.
  4. Appendix A.3.2 specifies the timing and memory measurement scope and distinguishes observation windows, output-update intervals, and sustained streaming performance.

Thank you again for helping us make the gain attribution and computational trade-offs more explicit.

Thank you for your detailed rebuttal and additional experiments. I appreciate the substantial effort made to address my concerns.The new shared-backbone comparisons and matched V-latent/P-latent ablations provide convincing evidence supporting the proposed representation. I consider my first concern largely addressed. The additional efficiency measurements are also helpful. However, complete end-to-end latency, sustained throughput, and whole-pipeline peak memory remain insufficiently evaluated. Therefore, my second concern is only partially addressed. Overall, the rebuttal has strengthened the paper, although some limitations remain. I will maintain my original rating of Weak Acceptance. Thank you again for your careful responses.

Reviewer_zHzpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the problem of learning compact event-camera representations. A physics-guided representation learning framework is proposed that aims to address the trade-off between preserving fine-grained temporal information and obtaining representations suitable for efficient learning. The approach combines event reorganization with latent compression to produce compact representations that can be integrated directly into existing event-based frameworks. Experiments across multiple datasets and downstream tasks demonstrate the effectiveness and generalization of the proposed approach.

Critical strengths

Research Question

The paper addresses a relevant task for event representation learning.

  • Existing approaches represent event data in different forms, ranging from point-wise sets and pixel-level sequences to grid-based tensors, and often face a trade-off between preserving temporal information and making the data easier to process. Instead of discarding information to make event data easier to process, the paper proposes reorganizing it into a more meaningful structure based on the physics of event generation, while preserving fine-grained temporal information.

  • The key idea is that physically meaningful organization of events improves their statistical properties without sacrificing the information they contain. This is particularly useful for event cameras, where preserving temporal information is important for various downstream tasks such as motion estimation and video reconstruction.

Motivation and Design

The proposed method is simple in design and well-motivated.

  • The paper identifies two main problems with conventional event voxelization, that is, coarse temporal bins merge events and lead to information loss, while fine temporal bins increase sparsity and computational cost. A conflict-free voxelization is introduced which avoids merging events within temporal bins by conditioning that each voxel should contain at most one event.

  • The paper further identifies that the resulting conflict-free voxel representation is highly sparse, making it difficult to learn and compress efficiently. To address this, they introduce a forward-physics transformation that cumulatively integrates event polarities over time resulting in a more structured and spectrally compact representation.

  • Finally, the resulting physics-grounded representation remains high-dimensional despite its improved statistical properties. To address this, a lightweight variational autoencoder is used to compress it into a compact latent representation, using an event-aware reconstruction loss that focuses on nonzero regions of the accumulated representation in the previous step, together with KL regularization.

Experimental Validation

The latent event representations learned by the proposed method are thoroughly evaluated to demonstrate:

  • The preservation of the physics-grounded proxy with high fidelity across in-domain and cross-domain settings.
  • The retention of task-relevant event information, leading to improvements in various downstream tasks, including semantic segmentation, optical flow estimation, and video reconstruction, without requiring changes to the downstream architectures.
Critical weaknesses

Limited Technical Novelty

  • The technical novelty of the proposed method appears relatively limited. The conflict-free voxelization mainly relies on choosing a sufficiently fine temporal resolution, which is closely related to existing event voxelization approaches [1]. Similarly, the forward-physics transformation is essentially a cumulative sum of event polarities over time, which has already been explored [2]. While the motivation behind combining these steps for learning compact event representations is interesting, the individual components are relatively straightforward.

Conflict-free condition

  • The optimal 100 μs bin width is selected based on the event restoration analysis conducted on four datasets (MVSEC, DSEC, BS-ERGB, and M3ED). However, the subsequent experiments include several additional datasets, such as DDD17, RLED, ERFX170, GoPro, and SDSD, which were not considered when selecting this resolution. Does it also hold for these datasets? Are there any datasets with a different optimal bin width? What would be the impact on the downstream performance?

  • There is no ablation studying how different temporal resolutions (e.g. 50, 100, 200,etc) affect the learned representations or the downstream performance. Such an analysis would help understand whether avoiding event collisions actually contributes to the improvements.

Evaluation of original Event-Level Fidelity

  • The proposed method aims to preserve the fine-grained temporal information, but the evaluation is mainly focused on reconstructing the accumulated proxy rather than the original event volume. While Figures 6 and 7 show high reconstruction fidelity, it remains unclear how much of the original event information is preserved after VAE compression. An evaluation of event-level reconstruction fidelity would help validate the main claims of the paper.

Missing Comparison with CES

  • The paper would strongly benefit from a comparison with closely related work [3], which also aims to obtain compact event representations while preserving fine-grained temporal information. In particular, comparing event reconstruction fidelity at similar compression levels and downstream performance under the same architecture would provide stronger evidence for the effectiveness of the proposed representation.

[1] Zhu et al., CVPR 2019: Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

[2] Scheerlinck et al., ACCV 2018: Continuous-time Intensity Estimation Using Event Cameras

[3] Lin et al., IJCV 2025: Compressed Event Sensing (CES) Volumes for Event Cameras

Minor Issues

Fig. 7 is not discussed at all in any of the experimental sections. It should be added to the discussion in Section 4.1.

Questions for authors

See Weaknesses

7 comments

Thank you for the careful review.

Q1. Limited Technical Novelty. (1) The technical novelty of the proposed method appears relatively limited. The conflict-free voxelization mainly relies on choosing a sufficiently fine temporal resolution, which is closely related to existing event voxelization approaches [1]. (2) Similarly, the forward-physics transformation is essentially a cumulative sum of event polarities over time, which has already been explored [2]. (3) While the motivation behind combining these steps for learning compact event representations is interesting, the individual components are relatively straightforward.

[1] Zhu et al., CVPR 2019: Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion

[2] Scheerlinck et al., ACCV 2018: Continuous-time Intensity Estimation Using Event Cameras

R1: (1) Temporal resolution and latent dimensionality serve different roles. Voxel Grid [1] interpolates and accumulates event polarities across neighboring temporal bins. Our voxelization preserves individual event assignments and polarities at the selected collision-free resolution while quantizing timestamps. We then invertibly reorganize this volume into the full cumulative trajectory and learn a latent whose channel count is chosen independently of the source temporal resolution.

Controlled comparison with finer voxel grids. On DSEC-Semantic, we use the same ScaleEvent/ViT-L/16 backbone, 30 ms windows, and training protocol, adapting only the input projection to the channel count. For CC input channels, the baseline uses B=CB=C voxel bins, whereas our method produces D=CD=C latent channels from a fixed 300-bin volume (Table 5). Results are mIoU (%).

Input channels CC31030100300
Voxel Grid [1], B=CB=C69.6569.1668.0166.4264.36
Our Latent, B=300, D=CB=300,\ D=C71.8471.8871.5671.0270.75

Increasing voxel bins from 3 to 300 reduces mIoU by 5.29 points. This is consistent with Figure 2: coarse aggregation yields denser, statistically more regular inputs by merging temporal details, while finer grids produce sparser, less regular inputs. Our latents maintain 70.75–71.88 mIoU, with a 6.39-point advantage at 300 channels and 71.84 mIoU at just three channels. Thus, finer voxelization alone does not make the additional temporal detail useful under the tested training protocol; reorganization and latent learning address how that information is presented to the network.

(2) Our contribution concerns trajectory compression, not the accumulation operator. We acknowledge that cumulative integration is established, including in [2]. Its recursive filter maintains the latest per-pixel intensity estimate and update time (Sec. 4.2, Algorithm 1). Our framework explicitly retains every cumulative state within the observation window as the encoder input and reconstruction target. Temporal differencing exactly recovers the discretized event volume before learned compression.

We investigate whether these full trajectories provide an effective basis for compact representation learning. Figure 5 examines spectral compactness and temporal-shift behavior; Figures 6 and 8 evaluate reconstruction and latent learning; and the matched V-latent/P-latent comparison in Table 4 tests the downstream benefit across three tasks. This connects invertible reorganization, learned compression, and downstream utility, beyond the intensity-estimation objective of [2]. The related-work discussion explicitly acknowledges this relationship.

(3) The contribution is the representation principle and its validation. Thank you for recognizing the motivation. Our contribution is the representation principle: preserve finely discretized event information, reorganize it without additional loss, and then learn its compression. Conflict-free voxelization, invertible accumulation, and regularized latent learning serve distinct roles in addressing the tension between temporal fidelity and statistical tractability. This design is supported by more than the combination of familiar operators. Figure 5 analyzes trajectory compactness and shift stability, Figure 8 examines the benefit of reorganization for latent learning, and Figure 6 evaluates compression across capacities and domains. The controlled comparison in R1 (1) further shows that finer voxelization alone degrades segmentation performance, while our latents remain effective. Across three downstream tasks, our representation consistently improves over voxel-grid baselines under matched settings.

Q2.Conflict-free condition. (1) The optimal 100 μs bin width is selected based on the event restoration analysis conducted on four datasets (MVSEC, DSEC, BS-ERGB, and M3ED). However, the subsequent experiments include several additional datasets, such as DDD17, RLED, ERFX170, GoPro, and SDSD, which were not considered when selecting this resolution. Does it also hold for these datasets? Are there any datasets with a different optimal bin width? What would be the impact on the downstream performance? (2) There is no ablation studying how different temporal resolutions (e.g. 50, 100, 200,etc) affect the learned representations or the downstream performance. Such an analysis would help understand whether avoiding event collisions actually contributes to the improvements.

R2: We have extended the restoration analysis to the additional datasets and added a temporal-resolution ablation across all three downstream tasks.

Applicability across datasets. Appendix A.2.1, Figure 12 includes DDD17, RLED, ERFX170, GoPro, and SDSD. Among the tested candidates, the coarsest collision-free width is 200 μs for DDD17 and SDSD and 100 μs for RLED, ERFX170, and GoPro. The admissible width is therefore dataset-dependent; 100 μs satisfies the criterion across all evaluated windows and provides a conservative shared resolution.

Downstream sensitivity. We compare 50, 100, and 200 μs with all other settings fixed within each task. At 200 μs, collisions occur and we retain only the latest event per cell, discarding the others. This configuration does not satisfy the conflict-free condition. Table 6 summarizes the downstream ablation. Acc denotes mean per-class accuracy, and segmentation scores are percentages.

Bin widthDSEC-Semantic Acc ↑DSEC-Semantic mIoU ↑DSEC-Flow EPE ↓DSEC-Flow AE ↓HQF MSE ↓HQF SSIM ↑HQF LPIPS ↓
50 μs93.5871.810.6472.360.0380.6280.284
100 μs93.6471.840.6462.360.0380.6290.283
200 μs93.5271.530.6582.540.0390.6270.283

Relative to 100 μs, 50 μs doubles the temporal-bin count without improving the reported metrics, while 200 μs worsens most metrics. The shared resolution is selected through event-restoration analysis; this ablation evaluates its downstream sensitivity. Because bin width affects both timestamp quantization and event retention, the results do not isolate the contribution of collision avoidance alone.

Q3.Evaluation of original Event-Level Fidelity. The proposed method aims to preserve the fine-grained temporal information, but the evaluation is mainly focused on reconstructing the accumulated proxy rather than the original event volume. While Figures 6 and 7 show high reconstruction fidelity, it remains unclear how much of the original event information is preserved after VAE compression. An evaluation of event-level reconstruction fidelity would help validate the main claims of the paper.

R3: Thank you for raising this concern. Our evaluation targets the temporal structure of the retained event information in the cumulative domain, motivated by two observations.

Suitability for compression. Under matched settings, direct VAE learning from sparse voxels produces less stable optimization and poorer reconstruction than learning from cumulative trajectories (Figure 8). The DCT analysis in Figure 5(a) provides complementary evidence of the trajectories’ spectral compactness.

Limitations of point-wise event error. Moving an isolated event by one bin produces discrepancies at its original and shifted locations; a larger displacement can yield the same point-wise error despite greater temporal distortion. This limits strict point-wise error as a measure of timing fidelity and motivates evaluating the full cumulative trajectory.

The cumulative transformation preserves the discretized event information before compression. Figures 6 and 7 quantify and illustrate distortion of the reconstructed trajectories after VAE compression. Temporal differencing can amplify high-frequency reconstruction fluctuations, so trajectory fidelity does not establish equivalent fidelity in the event volume. Our evidence supports temporal-structure preservation under the reported trajectory metrics and downstream utility; an independent event-domain fidelity evaluation remains open.

Q4. Missing Comparison with CES. The paper would strongly benefit from a comparison with closely related work [3], which also aims to obtain compact event representations while preserving fine-grained temporal information. In particular, comparing event reconstruction fidelity at similar compression levels and downstream performance under the same architecture would provide stronger evidence for the effectiveness of the proposed representation.

[3] Lin et al., IJCV 2025: Compressed Event Sensing (CES) Volumes for Event Cameras

R4: Thank you for highlighting CES [3]. We agree that comparing reconstruction fidelity at matched storage budgets and downstream performance under identical architectures and training protocols would strengthen the evaluation.

As discussed in Appendix A.1.3, we could not locate a publicly available official implementation of CES and have not independently validated its encoding, sparse-recovery, and downstream training pipeline. Reproducing its fidelity results also requires verification of the scoring and aggregation procedures and sparse-recovery solver settings. These unresolved reproduction details currently prevent a reliable controlled comparison.

The expanded discussion clarifies the methodological relationship: CES compresses events through Fourier measurements and evaluates event recovery using sparse reconstruction. Our framework separates invertible cumulative reorganization from learned VAE compression, with fidelity evaluated through trajectory reconstruction. This distinction explains our design; establishing comparative advantages over CES requires the controlled quantitative evaluation described above.

Minor comment on Figure 7. Section 4.1 now discusses the qualitative pixel-wise trajectories and scene-level reconstructions in Figure 7 alongside the quantitative evaluation in Figure 6.

Changes to the paper.

  1. The method and related-work discussions distinguish our investigation of trajectory compression from the established voxelization and accumulation operators.
  2. Sec. 4.3 (Tables 4 and 5) adds a matched-encoder downstream ablation across all three tasks and a controlled comparison of voxel bins and latent channels.
  3. Sec. 3.1 and Appendix A.2.1 (Figure 12) extend restoration analysis to five additional datasets and clarify the empirical basis for the shared 100 μs setting; Sec. 4.3 (Table 6) evaluates downstream sensitivity to bin width.
  4. Secs. 3.2 and 4.1 clarify that accumulation is invertible before VAE compression and that Figures 6 and 7 evaluate fidelity in the cumulative-trajectory domain.
  5. Appendix A.1.3 expands the relationship to CES and explains the reproduction limitations preventing a controlled quantitative comparison.

Thank you again for helping us clarify the representation-level contribution and strengthen its experimental support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.