- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Event-guided Neural Video Compression (ENVC), which uses event sensor data between RGB frames to aid in RGB video compression as an additional temporal signal. The approach uses synthetic events generated from video that has been temporally upsampled by a network like Super-SloMo. The method uses the events in an event-guided motion prior that reduces the amount of motion information that needs to be encoded and an event conditioned preduictor thet refines the temporal context used during encoding. Experiments show strong quantitative results.
Critical strengths
Strengths
- I like the quantitative comparison between the real and synthetic events. I agree that while there is difference, it does imply that synthetic events can be useful here.
- Very strong rate distortion performance. The paper reports savings of ~40% PSNR-RGB and ~67% LPIPS. Which is not too bad.
- I like that the paper acknowledges the failure cases, where the method does not do so well. Good honest reporting.
- The methodology is really documented in great detail, I do feel it would be possible to build this system even without the promised code, which will make that even easier.
Critical weaknesses
- I am rather skeptical of the approach here. Basically authors do a temporal upsampling using an existing CNN, use that to generate synthetic events and then use that as part of their compression codec. But as a reader I am given no reason to believe that events are a particular good representation of the information between frame t-1 and frame t. For example, I could just as well image a latent representation of the temporally upsampled frames to be used in the video codec - in fact I would expect such a learned representation to be much better and to be a better fit. As long as this stays in some synthetic domain, I don’t see why you would bother going to events as the intermediate representation. Now in a real-world camera setting where you have a frames + events sensor, then maybe I would see a use, since the events would be a very cheap, sparse, fast and low-latency way of encoding the change between the frames. But this does not seem to be the intended use at all. I really think this needs a comparison to some sort of latent representation to prove the idea, as well as some theory on why events would be the best representation here.
- Have authors seen “ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression”? ENCORE proposes using event streams as an auxiliary modality to improve motion estimation in learned RGB video compression. I find that there is a fair bit of overlap with this paper. Aside from the motivation, both use auxiliary motion information, have a learned rate-distortion framework, event guided flow correction…. Maybe at least it should mention it in the literature review and spell out the differences to this paper.
- Since this paper relies so heavily on synthetic events, the framing of how this is proposed to be used is not super clear to me. Do authors intend this to ever be a thing used on real systems (e.g DAVIS)?
- Events are assumed to be available to both encoder and decoder, but that cost is kept out of bitrate assumptions.
- Not enough component ablations. Ideally I would like to see ENVC vs ENVC without RMC, ECR and PCF
- The main experiment compares ENVC which has access to events against RGB-onyl codecs. Not too sure how useful that comparison is. Maybe an exciting codec can be minimally modified to see events as well as a better baseline.
Questions for authors
- To me the math in 3.1 seems gratuitous. It does not help with formalizing the problem and could just be replaced by one or two sentences in plain English. Which would make it easier to read. I quite prefer it when papers try to be as easy to understand as possible. Oftentimes many of the formulae in papers are somewhat unnecessary, I posit it is similar here. But up to you, I suppose that is a subjective preference.
8 comments
W4. Event Cost in Bitrate Accounting.
We appreciate the reviewer raising this point. As described in App. A, ENVC targets multimodal systems where events are already available for other purposes, such as event-based perception. This setting is consistent with prior compression studies that leverage auxiliary modalities as side information (Gnutti et al., 2024; Jiang et al., 2025; Sheng et al., 2026). Accordingly, our evaluation focuses on RGB compression under this shared-event setting.
We acknowledge that efficient event compression remains an important challenge. The G-PCC codec used in App. A.1 was originally developed for point cloud compression, rather than event streams. We use it to examine the impact of lossy compressed events on ENVC, not as an optimized event coding solution. While learned event compression is still relatively less explored than RGB video compression, developing dedicated event codecs is beyond the scope of this work. We will clarify this limitation and consider efficient event coding and joint RGB–event compression as future directions.
W5. Component Ablation Studies.
We appreciate this valuable suggestion. We are currently conducting additional ablation experiments by individually removing RMC, ECR, and PCF to evaluate their respective contributions. We will share the results in a follow-up comment once the experiments are completed.
W6. Comparison with an event-augmented baseline.
We acknowledge the reviewer's suggestion. In ENVC, events are not simply added as an auxiliary input to an existing codec. Instead, they guide residual motion coding (RMC) and temporal context construction and refinement (ECR, PCF). Meaningfully incorporating events into an existing codec therefore requires careful design of how they interact with motion coding and context modeling, which is precisely what ENVC addresses.
To further investigate this, the ablation experiments described in W5 examine the contribution of each component by individually removing RMC, ECR, and PCF. In addition, we are training a zero-event variant with the same architecture and parameter count to isolate the contribution of event guidance from model capacity. We will report these results in a follow-up comment alongside W5.
Q1. Mathematical Formulation in Sec. 3.1.
We thank the reviewer for the suggestion. We will simplify the mathematical formulation in Sec. 3.1 to improve readability in our revised version.
On the application of this method
So if this is intended for use with a real system combining RGB and event sensors, then it is going to be absolutely necessary to actual construct a benchmark with real data. I think if you can do this, then this paper can be quite strong. I don't think it is an impossible ask either, since such systems have been around for a very long time....
Note. W refers to the points raised under Critical Weaknesses, and Q refers to those under Questions for Authors, following the numbering in the review.
We appreciate the reviewer’s detailed comments and would like to clarify our intended setting.
(A) Intended Use: The RGB+event sensor setting described by the reviewer is precisely what ENVC targets. We consider systems where events are already captured and shared between the encoder and decoder, e.g., for downstream perception tasks. ENVC leverages these events as side information to reduce the additional RGB coding rate (Fig. 1c, App. A).
(B) Role of Frame Interpolation : Frame interpolation and event simulation are used only to construct paired RGB–event datasets, as standard video coding benchmarks lack paired event sensor streams (Sec. 3). They are not part of the proposed codec. In real-world deployment, events are directly captured by sensors rather than synthesized. We also evaluate ENVC on real sensor events (App. D).
(C) Comparison with Latent Representations : A learned latent representation derived from interpolated frames would generally require additional transmission if it depends on information unavailable to the decoder. In contrast, events directly capture brightness changes at high temporal resolution through a dedicated sensor, providing complementary temporal information beyond conventional RGB frames. ENVC is designed to exploit this information for more efficient motion coding and temporal context modeling.
We thank the reviewer for bringing ENCORE to our attention. Although both methods use events for neural video compression, they address different coding problems.
ENCORE leverages events only at the encoder to refine RGB-based motion estimation. The refined flow is then compressed using the existing motion codec, while the decoder remains unchanged. Thus, events serve as auxiliary information for improving motion estimation, rather than as shared information available during decoding.
In contrast, our ENVC considers events as complementary inter-frame observations shared by both the encoder and decoder. This enables a different coding strategy: both sides construct the same event-guided motion prior, allowing ENVC to encode only the remaining motion residual (RMC). Moreover, event-conditioned predictions and multi-scale features directly refine temporal contexts for conditional frame coding (ECR, PCF).
Therefore, ENCORE focuses on improving encoder-side motion estimation, whereas ENVC exploits shared event information to reduce the information that must be transmitted for both motion and frame coding. We will cite ENCORE and clarify this fundamental distinction.
We thank the reviewer for the question. ENVC is indeed designed for real-world systems equipped with RGB–event sensors, such as DAVIS, where events are already available for tasks such as event-based perception, while RGB video is also required (App. A).
Synthetic events are used solely to construct paired RGB–event datasets for training and benchmarking, since standard video coding datasets lack recorded events (Sec. 3). Event simulation is not part of the codec itself.
To examine real-world applicability, App. D evaluates ENVC on real sensor events. The remaining synthetic-to-real performance gap may partly arise from differences in event statistics across sensors and simulators. The limited availability of standardized paired RGB–event datasets further complicates systematic generalization evaluation. We will clarify the intended deployment scenario and these limitations in the revised manuscript, and explore improving real-sensor generalization as future work.