- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a new event representation in which events are accumulated into voxels with many temporal bins, such as 100 or 300 bins. These sparse voxels are then converted into temporal accumulations and compressed into latent representations using a VAE. The authors claim that this representation improves performance on semantic segmentation, optical flow estimation, and video reconstruction.
Critical strengths
- The finding that this representation can help models train more effectively would be valuable, if it can be properly validated.
Critical weaknesses
-
A major problem with this paper is its exaggerated language and serious overclaiming. After reading the actual method, I felt that much of my time had been wasted.
- The phrase “structured into conflict-free volumes” apparently refers to accumulating events into temporal bins with a width of 100 μs, corresponding to 10,000 FPS. This relies on the empirical observation that no intra-bin collisions occur in the tested data. However, considering that event cameras can theoretically provide temporal resolutions on the order of FPS, this design choice is questionable.
- The “forward-physics transformation” is essentially just temporal accumulation. This terminology makes the method sound unnecessarily mysterious.
- The paper uses terms such as “information fidelity” to imply that this representation preserves more information than ordinary voxel representations. However, in the final VAE encoding step, the information is still compressed into only one or three channels. Thus, the information is ultimately heavily compressed.
-
If the paper could sufficiently demonstrate the advantages of this representation, it would make a good contribution to the field. However, the experiments are not sufficiently convincing:
- For the downstream tasks, the paper should demonstrate that the proposed representation consistently outperforms the original representation across different model architectures, rather than evaluating only one architecture per task. Simply listing results from different model architectures provides limited additional value for this purpose.
- I have concerns about the sources and reliability of the data reported in the paper (see Questions).
- The paper lacks ablation studies. It should provide quantitative experiments showing that the representation with the accumulation step improves downstream performance compared with directly feeding sparse voxels into the VAE, rather than only comparing the statistical properties and sensitivity of the two representations.
Minor Problems
These issues do not affect my review decision and are provided only for the authors’ consideration during revision:
- Table 3 does not highlight the best results consistently with Tables 1 and 2.
- Only Table 3 includes the note that “Methods marked with † provide public training code.”
Questions for authors
-
In Tables 1–3, the authors should clearly indicate:
- which results are directly copied from the original papers;
- which results are obtained by running inference with checkpoints provided by the original authors and evaluating them using a unified set of metrics; and
- which results are obtained by retraining and evaluating the methods in this paper.
I have this concern because all the other numbers in Table 2 appear to be consistent with the results reported in the original STSC paper, while the performance reported for STSC itself is worse in this paper than in the original paper.
12 comments
Q3. The paper uses terms such as “information fidelity” to imply that this representation preserves more information than ordinary voxel representations. However, in the final VAE encoding step, the information is still compressed into only one or three channels. Thus, the information is ultimately heavily compressed.
R3: The VAE representation is intentionally compact and lossy, but latent channels and temporal bins serve different roles. A three-bin voxel grid merges events into three temporal intervals; our three-channel latent is learned from the full 300-bin cumulative trajectory under reconstruction supervision. Equal channel counts therefore do not imply equivalent temporal information processing.
We test this distinction on DSEC-Semantic with the same ScaleEvent/ViT-L/16 backbone, 30 ms windows, and training protocol, adapting only the input projection. Voxel grids use bins, while our latents use channels from a fixed volume (Table 5).
| Input channels | 3 | 10 | 30 | 100 | 300 |
|---|---|---|---|---|---|
| Voxel Grid | 69.65 | 69.16 | 68.01 | 66.42 | 64.36 |
| Our Latent | 71.84 | 71.88 | 71.56 | 71.02 | 70.75 |
Metric: mIoU (%).
Increasing voxel bins from 3 to 300 reduces mIoU by 5.29 points, while our latents maintain 70.75–71.88 mIoU. The advantage is 2.19 points at three channels and 6.39 points at 300 channels, supporting the importance of how temporal information is organized and encoded.
“Information fidelity” refers to temporal-structure preservation under compression, evaluated through trajectory reconstruction in Figures 6 and 7. Figure 5(a) shows concentrated low-frequency DCT energy in the evaluated trajectories, consistent with accurate reconstruction at small latent capacities. Reducing latent channels limits learned coding capacity; reducing voxel bins directly coarsens temporal discretization. Our compression learns from the finely resolved trajectory, while the channel ablation demonstrates its downstream utility across coding capacities.
Q4. For the downstream tasks, the paper should demonstrate that the proposed representation consistently outperforms the original representation across different model architectures, rather than evaluating only one architecture per task. Simply listing results from different model architectures provides limited additional value for this purpose.
R4: We have added a paired evaluation beyond ScaleEvent and a complementary shared-backbone representation comparison.
Paired evaluation with an additional pretrained model. On DSEC-Semantic, the RGB-pretrained DINOv3 ViT-L/16 voxel-grid and latent variants share the same pretrained initialization, segmentation head, and training protocol, with input projections adapted as needed. Table 15 in Appendix A.4.1 reports the results alongside ScaleEvent for reference.
| Segmentation pipeline | Voxel Grid mIoU (%) | Our Latent mIoU (%) | Improvement |
|---|---|---|---|
| ScaleEvent / ViT-L/16 | 69.65 | 71.84 | +2.19 |
| DINOv3 / ViT-L/16 | 64.31 | 70.29 | +5.98 |
The 5.98-point gain extends the evidence to another pretraining regime and segmentation pipeline. The ScaleEvent voxel-grid result comes from the official checkpoint, as documented in Appendix A.2.5; the matched DINOv3 experiment is the additional paired evaluation. Because both use ViT-L, these results do not yet establish transfer across distinct architecture families.
Complementary shared-backbone comparison. Table 7 compares representative encodings within the ScaleEvent/ViT-L/16 segmentation pipeline, adapting input projections as needed. Acc denotes mean per-class accuracy.
| Representation | Representation principle | Acc (%) ↑ | mIoU (%) ↑ |
|---|---|---|---|
| HATS | Averaged time surfaces | 92.43 | 66.41 |
| TORE | Time-ordered recent-event histories | 92.57 | 67.52 |
| Histograms | Polarity-wise event accumulation | 92.76 | 68.29 |
| Event Frame | Event accumulation into frames | 92.65 | 67.75 |
| EST | Differentiable event-tensor construction | 92.83 | 69.14 |
| Voxel Grid | Temporal interpolation and accumulation | 93.10 | 69.65 |
| E2VID | Learned event-to-video reconstruction | 92.54 | 67.03 |
| P-latent (Ours) | VAE compression of the cumulative trajectory | 93.64 | 71.84 |
Our latent achieves 71.84 mIoU, exceeding Voxel Grid by 2.19 points and E2VID by 4.81 points. This comparison broadens the representation baselines under a common downstream model and complements the additional DINOv3 evaluation.
Thank you for the careful review.
Q1. The phrase “structured into conflict-free volumes” apparently refers to accumulating events into temporal bins with a width of 100 μs, corresponding to 10,000 FPS. This relies on the empirical observation that no intra-bin collisions occur in the tested data. However, considering that event cameras can theoretically provide temporal resolutions on the order of (1 \times 10^6) FPS, this design choice is questionable.
R1: Microsecond timestamp precision describes how finely event times are recorded, not a million-frame-per-second acquisition rate or one event per pixel every microsecond. It should be distinguished from practical pixel-response latency.
As discussed in Sec. 3.1, our empirical 100 μs scale is comparable to reported pixel-response latencies of 100–220 μs for the IMX636 [1, p. 12] and approximately 100 μs for the DVS240 under the tested conditions [2, p. 9]. These values provide physical context, not a universal collision-free bound. We select 100 μs through event-restoration analysis across the evaluated windows, preserving event assignments and polarities while quantizing timestamps.
Downstream sensitivity. We compare 50, 100, and 200 μs with all other settings fixed within each task. At 200 μs, collisions occur and only the latest event per cell is retained. Acc denotes mean per-class accuracy; segmentation scores are percentages.
Halving the bin width to 50 μs doubles the bin count without improving the reported metrics; 200 μs worsens most metrics. These results assess sensitivity to the empirically selected resolution, rather than determine its selection. Since bin width changes both timestamp quantization and event retention, the ablation does not isolate collision effects. Finer sensor timestamp precision alone does not dictate a finer voxelization grid.
[1] Guo R, et al. EventLFM: Event camera integrated Fourier light field microscopy for ultrafast 3D imaging. Light: Science & Applications, 2024.
[2] Holešovský O, et al. Experimental comparison between event and global shutter cameras. Sensors, 2021.
Q2. The “forward-physics transformation” is essentially just temporal accumulation. This terminology makes the method sound unnecessarily mysterious.
R2: The operation is temporal accumulation; “forward-physics” describes its role in the representation, not a new mathematical operator.
Full trajectories. Rather than reducing a window to one polarity aggregate, we retain the cumulative state at every temporal bin. Event timing and polarity are encoded in trajectory changes, and the full trajectory is used for learned compression.
Explicit causal state evolution. Voxel grids encode temporal distributions through binning and interpolation. Our transformation makes cumulative state evolution explicit: each state incorporates the preceding state and current event increment, without depending on future events. The term refers to this time-directed construction motivated by the event-generation model.