acceptodds
Under review as a conference paper at ICLR 2027

Event-guided Neural Video Compression

Abstract

Neural video codecs derive motion and temporal contexts mainly from RGB frames, leaving room for cross-modal guidance from complementary temporal observations. Event streams can provide such observations by recording brightness changes between frames. In this work, we propose an event-guided neural video codec (ENVC) that uses events shared by the encoder and decoder to improve RGB compression efficiency. For motion coding, ENVC forms an event- guided motion prior and codes the remaining motion residual. For frame coding, an event-conditioned predictor supplies multi-scale features for gated temporal context refinement. To support training and evaluation on standard video datasets, we synthesize paired RGB–event data and assess its predictive utility through comparisons with real events. Across six benchmarks, ENVC achieves average BD-rate savings of 39.13% using PSNR-RGB and 67.63% using LPIPS relative to DCMVC. Further analyses show that our gains persist on large-motion sequences and that ENVC effectively learns to integrate event information. These results demonstrate the potential of events as a complementary modality for reducing the RGB coding rate. Our model and code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural video compression, Event-based vision, Multimodal compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X3ymOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Event-guided Neural Video Compression (ENVC), which uses event sensor data between RGB frames to aid in RGB video compression as an additional temporal signal. The approach uses synthetic events generated from video that has been temporally upsampled by a network like Super-SloMo. The method uses the events in an event-guided motion prior that reduces the amount of motion information that needs to be encoded and an event conditioned preduictor thet refines the temporal context used during encoding. Experiments show strong quantitative results.

Critical strengths

Strengths

  • I like the quantitative comparison between the real and synthetic events. I agree that while there is difference, it does imply that synthetic events can be useful here.
  • Very strong rate distortion performance. The paper reports savings of ~40% PSNR-RGB and ~67% LPIPS. Which is not too bad.
  • I like that the paper acknowledges the failure cases, where the method does not do so well. Good honest reporting.
  • The methodology is really documented in great detail, I do feel it would be possible to build this system even without the promised code, which will make that even easier.
Critical weaknesses
  • I am rather skeptical of the approach here. Basically authors do a temporal upsampling using an existing CNN, use that to generate synthetic events and then use that as part of their compression codec. But as a reader I am given no reason to believe that events are a particular good representation of the information between frame t-1 and frame t. For example, I could just as well image a latent representation of the temporally upsampled frames to be used in the video codec - in fact I would expect such a learned representation to be much better and to be a better fit. As long as this stays in some synthetic domain, I don’t see why you would bother going to events as the intermediate representation. Now in a real-world camera setting where you have a frames + events sensor, then maybe I would see a use, since the events would be a very cheap, sparse, fast and low-latency way of encoding the change between the frames. But this does not seem to be the intended use at all. I really think this needs a comparison to some sort of latent representation to prove the idea, as well as some theory on why events would be the best representation here.
  • Have authors seen “ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression”? ENCORE proposes using event streams as an auxiliary modality to improve motion estimation in learned RGB video compression. I find that there is a fair bit of overlap with this paper. Aside from the motivation, both use auxiliary motion information, have a learned rate-distortion framework, event guided flow correction…. Maybe at least it should mention it in the literature review and spell out the differences to this paper.
  • Since this paper relies so heavily on synthetic events, the framing of how this is proposed to be used is not super clear to me. Do authors intend this to ever be a thing used on real systems (e.g DAVIS)?
  • Events are assumed to be available to both encoder and decoder, but that cost is kept out of bitrate assumptions.
  • Not enough component ablations. Ideally I would like to see ENVC vs ENVC without RMC, ECR and PCF
  • The main experiment compares ENVC which has access to events against RGB-onyl codecs. Not too sure how useful that comparison is. Maybe an exciting codec can be minimally modified to see events as well as a better baseline.
Questions for authors
  • To me the math in 3.1 seems gratuitous. It does not help with formalizing the problem and could just be replaced by one or two sentences in plain English. Which would make it easier to read. I quite prefer it when papers try to be as easy to understand as possible. Oftentimes many of the formulae in papers are somewhat unnecessary, I posit it is similar here. But up to you, I suppose that is a subjective preference.
8 comments

Note. W refers to the points raised under Critical Weaknesses, and Q refers to those under Questions for Authors, following the numbering in the review.

W1. Motivation for Events and Comparison with Latent Representation.

We appreciate the reviewer’s detailed comments and would like to clarify our intended setting.

(A) Intended Use: The RGB+event sensor setting described by the reviewer is precisely what ENVC targets. We consider systems where events are already captured and shared between the encoder and decoder, e.g., for downstream perception tasks. ENVC leverages these events as side information to reduce the additional RGB coding rate (Fig. 1c, App. A).

(B) Role of Frame Interpolation : Frame interpolation and event simulation are used only to construct paired RGB–event datasets, as standard video coding benchmarks lack paired event sensor streams (Sec. 3). They are not part of the proposed codec. In real-world deployment, events are directly captured by sensors rather than synthesized. We also evaluate ENVC on real sensor events (App. D).

(C) Comparison with Latent Representations : A learned latent representation derived from interpolated frames would generally require additional transmission if it depends on information unavailable to the decoder. In contrast, events directly capture brightness changes at high temporal resolution through a dedicated sensor, providing complementary temporal information beyond conventional RGB frames. ENVC is designed to exploit this information for more efficient motion coding and temporal context modeling.

W2. Relation to ENCORE.

We thank the reviewer for bringing ENCORE to our attention. Although both methods use events for neural video compression, they address different coding problems.

ENCORE leverages events only at the encoder to refine RGB-based motion estimation. The refined flow is then compressed using the existing motion codec, while the decoder remains unchanged. Thus, events serve as auxiliary information for improving motion estimation, rather than as shared information available during decoding.

In contrast, our ENVC considers events as complementary inter-frame observations shared by both the encoder and decoder. This enables a different coding strategy: both sides construct the same event-guided motion prior, allowing ENVC to encode only the remaining motion residual (RMC). Moreover, event-conditioned predictions and multi-scale features directly refine temporal contexts for conditional frame coding (ECR, PCF).

Therefore, ENCORE focuses on improving encoder-side motion estimation, whereas ENVC exploits shared event information to reduce the information that must be transmitted for both motion and frame coding. We will cite ENCORE and clarify this fundamental distinction.

W3. Intended Use with Real Event Sensors.

We thank the reviewer for the question. ENVC is indeed designed for real-world systems equipped with RGB–event sensors, such as DAVIS, where events are already available for tasks such as event-based perception, while RGB video is also required (App. A).

Synthetic events are used solely to construct paired RGB–event datasets for training and benchmarking, since standard video coding datasets lack recorded events (Sec. 3). Event simulation is not part of the codec itself.

To examine real-world applicability, App. D evaluates ENVC on real sensor events. The remaining synthetic-to-real performance gap may partly arise from differences in event statistics across sensors and simulators. The limited availability of standardized paired RGB–event datasets further complicates systematic generalization evaluation. We will clarify the intended deployment scenario and these limitations in the revised manuscript, and explore improving real-sensor generalization as future work.

W4. Event Cost in Bitrate Accounting.

We appreciate the reviewer raising this point. As described in App. A, ENVC targets multimodal systems where events are already available for other purposes, such as event-based perception. This setting is consistent with prior compression studies that leverage auxiliary modalities as side information (Gnutti et al., 2024; Jiang et al., 2025; Sheng et al., 2026). Accordingly, our evaluation focuses on RGB compression under this shared-event setting.

We acknowledge that efficient event compression remains an important challenge. The G-PCC codec used in App. A.1 was originally developed for point cloud compression, rather than event streams. We use it to examine the impact of lossy compressed events on ENVC, not as an optimized event coding solution. While learned event compression is still relatively less explored than RGB video compression, developing dedicated event codecs is beyond the scope of this work. We will clarify this limitation and consider efficient event coding and joint RGB–event compression as future directions.

W5. Component Ablation Studies.

We appreciate this valuable suggestion. We are currently conducting additional ablation experiments by individually removing RMC, ECR, and PCF to evaluate their respective contributions. We will share the results in a follow-up comment once the experiments are completed.

W6. Comparison with an event-augmented baseline.

We acknowledge the reviewer's suggestion. In ENVC, events are not simply added as an auxiliary input to an existing codec. Instead, they guide residual motion coding (RMC) and temporal context construction and refinement (ECR, PCF). Meaningfully incorporating events into an existing codec therefore requires careful design of how they interact with motion coding and context modeling, which is precisely what ENVC addresses.

To further investigate this, the ablation experiments described in W5 examine the contribution of each component by individually removing RMC, ECR, and PCF. In addition, we are training a zero-event variant with the same architecture and parameter count to isolate the contribution of event guidance from model capacity. We will report these results in a follow-up comment alongside W5.

Q1. Mathematical Formulation in Sec. 3.1.

We thank the reviewer for the suggestion. We will simplify the mathematical formulation in Sec. 3.1 to improve readability in our revised version.

Reviewer_X3ymOpenReview ↗

On the application of this method

So if this is intended for use with a real system combining RGB and event sensors, then it is going to be absolutely necessary to actual construct a benchmark with real data. I think if you can do this, then this paper can be quite strong. I don't think it is an impossible ask either, since such systems have been around for a very long time....

Reviewer_QYaUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ENVC, an event-guided neural video codec that uses events shared by the encoder and decoder to reduce RGB coding rates. It combines event-guided residual motion coding with gated multi-scale temporal context refinement. The authors construct synthetic RGB–event datasets and report substantial rate–distortion gains across six standard benchmarks. These gains concern the additional RGB coding rate under the shared-event setting, excluding event transmission costs.

Critical strengths
  1. The paper investigates a well-motivated compression setting in which event streams already available for downstream processing can also help reduce RGB coding costs.
  2. ENVC integrates event information into both motion coding and temporal context construction, providing a coherent framework for exploiting complementary temporal observations.
  3. The evaluation covers multiple standard benchmarks, additional 4K sequences, and both pixel-wise and perceptual quality metrics, supplemented by analyses of real versus synthetic events.
Critical weaknesses
  1. Although the authors state that the source code and pretrained model checkpoints will be publicly released upon acceptance, the submission does not provide anonymized access to these resources for reproducing the reported results. Providing an anonymized repository with the necessary code and checkpoints would strengthen reproducibility and positively influence my assessment.
  2. The experimental comparison omits ECVC and HyTIP, although both are cited in the paper. Including these methods under the same evaluation protocol would provide a more complete assessment of ENVC’s rate–distortion performance.
  3. ENVC has substantially more parameters than DCMVC (33.52M versus 21.02M, Table 3), so the reported RGB coding rate savings may partly reflect the increase in model capacity. The paper does not include comparisons under a matched parameter budget. Such comparisons would help isolate the contribution of the proposed event-guided motion coding and context refinement.
Questions for authors
  1. Could the authors provide anonymized access to the inference code together with pretrained model checkpoints, to enable reproduction of the reported results?
  2. How does ENVC compare with ECVC and HyTIP under the same evaluation protocol?
4 comments

Note. W refers to the points raised under Critical Weaknesses, and Q refers to those under Questions for Authors, following the numbering in the review.

W1. Code and Checkpoint Availability.

We appreciate the reviewer’s emphasis on reproducibility. We provide anonymized access to our inference code and pretrained model checkpoints through the following link: https://anonymous.4open.science/r/ENVC-52A8/

W2. Comparison with ECVC and HyTIP.

We thank the reviewer for suggesting additional comparisons. We have extended our evaluation baselines to include ECVC and HyTIP. Since the source code and pretrained checkpoints of ECVC are not publicly available, we use the RD results reported in the original paper for comparison, while HyTIP is evaluated under our evaluation protocol (96 frames, intra period 32, PSNR-RGB; Sec. 5.1). As shown below, ENVC outperforms both methods across all six benchmarks, achieving average BD-rate reductions of 33.4% over ECVC and 32.2% over HyTIP.

ComparisonUVGMCL-JCVHEVC-BHEVC-CHEVC-DHEVC-EAverage
ENVC vs. ECVC-22.4-46.7-45.6-47.0-33.8-4.8-33.4
ENVC vs. HyTIP-16.0-45.5-43.4-44.0-35.4-8.6-32.2

BD-rate (%) is evaluated using PSNR-RGB. Negative values indicate bitrate savings achieved by ENVC.

These results further demonstrate the effectiveness of ENVC against existing state-of-the-art neural video compression methods. We will include these comparisons and clarify the evaluation conditions in the revised manuscript.

W3. Model Capacity and Contribution of Event Guidance.

We appreciate the reviewer’s attention to the impact of model capacity on coding gains. To address this, we developed a lightweight variant, ENVC-L, by reducing the decoder channel widths of the Event-conditioned Frame Predictor (EFP) and performing Prediction-guided Context Fusion (PCF) at half resolution. ENVC-L was retrained under the same training schedule, with the remaining architecture unchanged.

As shown below, ENVC-L reduces the parameter count by 18.2% (33.52M to 27.42M), while achieving a 31.3% average BD-rate reduction over DCMVC across six benchmarks with comparable encoding and decoding speeds.

MetricDCMVCENVC (Original)ENVC-L
Parameters21.02M33.52M27.42M
Encoding FPS1.0610.3231.085
Decoding FPS1.2080.3861.240
Avg. BD-rate vs. DCMVC–-39.13%-31.3%

BD-rate (%) is evaluated using PSNR-RGB. These results demonstrate that substantial coding gains can be maintained with reduced model capacity. However, matching DCMVC's parameter budget would require further architectural modifications beyond the current lightweight design, followed by retraining and evaluation. Also, the ENVC-L implementation is available in our anonymized repository (https://anonymous.4open.science/r/ENVC-52A8/).

Q1. Reproducibility.

As addressed in W1 above, we provide an anonymized repository containing the inference code and pretrained model checkpoints.

Q2. Additional Baseline Comparisons.

This question is addressed in W2, where we present additional comparisons with ECVC and HyTIP across six benchmarks.

We thank Reviewer QYaU for the constructive feedback. We have addressed the reviewer's concerns in our rebuttal and would appreciate any follow-up on whether our response resolves the issues raised or whether any concerns remain.

Reviewer_QYaUOpenReview ↗

Response to Author

The authors' response has satisfactorily addressed my concerns. I will therefore increase my rating.

We sincerely thank the reviewer for the constructive feedback throughout the review process. We are glad that our responses have addressed your concerns, and greatly appreciate your time and thoughtful assessment.

Reviewer_uvMFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a method named event-guided neural video codec (ENVC), which introduced the event data into the neural codecs to improve the compression efficiency via learning some enc-dec common knowledge. The reported rate-distortion performances were encouraging that significant coding gains could be obtained with the introduction of event data plus neural coding methods. Extensive analysis were provided and conducted, the additional discussions on the relationship with event data and color video data.

Critical strengths
  1. It should be recognized that introducing the event data into NVC is the most salient part of this work. Although experts in video compression may challenge the use case and necessary of doing so, limited prior research in the field could be found.

  2. The proposed pipeline from the predictors to the residual coding is finely designed and the encouraging coding performances can be obtained with the underlying framework with its optimized tricks such as RDO, module improvements and proper injection of events.

  3. Plenty of analysis were simulated and presented to give a better understanding of this work.

Critical weaknesses
  1. The rationale of this research was not strongly expressed. From the very beginning, the authors claimed the relationship with event data and RGB data. In the first principle of video coding, the compression happens in the luma-chroma color spaces rather than the RGB color space. The motivation and origination of this research was not sufficiently presented and analyzed. Although the inspiring results were realized, the essential part of why this topic was proposed still remains unexplained. This does not quality the high bar of ICLR community of presenting novel know ledge and insights to the learning representations.

  2. Following 1, another drawback of this study is that sharing the event data in both encoder and decoder side. In fact, this is not practical (even those data were compressed). Video codecs have been emerged as the entrance of multi-modality understanding in the foundation model regime. Such heavy burdens of design harms the interface of video codecs with other models from the perspectives of deployment and storage cost. The analysis and discussion in both manuscript and appendix were not sufficient enough to persuade the video experts to the embrace such foundational change from conventional codec to multi-model data compression fashion. This research did not tough the core part of this design but devoted most of the efforts in the modeling tuning, ablations and some cherry pick experiments/analysis.

  3. Following 1 and 2, the most recent standard video codec, is evaluated on the High Dynamic Range (HDR) circumstances, e.g., input/output/internal bit depth are all 10-bits, with 2-Bytes for each pels (range from 0-1023). In such case, the event data associated with the RGB videos are completed different from the SDR videos. The gap was not mentioned throughout the manuscript. The event data itself is sensitive to many conditions, such as physical sensors, data formats, content variations etc.

  4. Following 3, regarding the simulation data generation and preparation part, the analysis and discussions were not sufficient enough. A) Plenty of intermediate frames were interpolated for each training sequences. By doing so, the training data were significantly increased with respect other DCVC-family models. The increasing of training data itself (together with back-propagation using longer videos) would definite increase the coding gain. The authors should consider about it. B) Such frame interpolation method might be effective in the video conferencing and television oriented coding scenarios since most of the background pets were static and limited changes could be observed.

  5. The model size and complexity of the proposed ENVC seemed to be a nightmare to be evaluated. Given Table 3, over 50% of params increasing with respect to the prior state-of-the-art NVCs brings the significant coding gain. From both the perspective of traditional video coding and learned video coding, such trade-off is an unacceptable one. The reviewer would like to recognize the idea of this study but the work presented in the current manuscript does not meet its optimal design and version. Large rooms of space for improving clearly exist and strongly stands.

  6. The authors have mentioned in the experiment section that the videos were processed frame by frame without some temporal attention and joint optimization. The reviewer would suggest to consider such spatial-temporal effect form the native video representation rather than abandoning the rich contextual information. The objective of more robust VCM models should be targeted at stronger analysis performance with the consideration of compression effects. If some tricks from other domain were directed borrowed, there is limited contribution to the VCM itself. And the VCM will just be another task in the computer vision related study, which should zero improvement to the signal processing and visual communication field.

Questions for authors
  1. In the appendix, the authors have provided extensive studies and simulations of ENVC. However, the most frustrating one is the generalization ability of ENVC. Given the current results, we could conclude that the proposed method could not be able to adapt to other types of event sensors. This calls back the weaknesses 3. The proposed codec might not be equipped with essential generalization factors that required by the video experts in standard organization group.

  2. The rate-distortions curves in Fig. 5/6/15 of the proposed ENVC is much much better than other DCVC-family models. Based on the eyebrow, over 100% BD-rate reductions could be obtained with some videos (for instance, ENVC used half bitrates realized similar quality with other models). This phenomenon only has two reasons 1) some flaws were obtained during BD-rate calculation. 2) The proponents are not aligned. The authors should revisit all simulation results. However, the reviewer believes that the large amount of rate saving was introduced by the implicit motion derivation using events (which is similar to DCVC-RT).

  3. Following 2, the authors are advised to include how were the quality of all decoded images assessed, for instances, the color space, the bit-rate for each frame. Similar details should also be reported for the anchors, such as the parameters of DCMVC, DCVC-family models. The quality and bpp for each frame of all 96 frames should also be provided.

  4. Although there are a lot of drawbacks existing in the current design, the reviewer still remains positive towards this work. More details should be provided in the revision by replacing some less salient ablation analysis, such that this work would be more informative. If some concerns were not well addressed, the reviewer would downgrade the score for the authors to offer them more time to improve this work before the design of ENVC well fully optimized.

  5. There has been a flaw ever since the earliest work in DVC since 2019. The test sequences used in HEVC/H.265 development should be named as JCT-VC Class A/B/C/D/E. Similarly, the sequences from H.66/VVC development should be named as JVET Class A1/A2/B/C. The authors should revise them.

5 comments

Note. W refers to the points raised under Critical Weaknesses, and Q refers to those under Questions for Authors, following the numbering in the review.

W1. Insufficient Research Motivation and Novelty.

We thank the reviewer for pointing out the need to better clarify our research motivation and the role of color spaces. We agree that conventional video codecs primarily operate in YUV color spaces. However, RGB-based coding is also widely adopted in neural video compression, as in DCVC and DCVC-TCM, while DCVC-DC and DCVC-FM support both RGB and YUV. ENVC also uses a combination of YUV420 and RGB distortions during training (App. B), and achieves an average BD-rate reduction of 53.51% in PSNR-Y (App. H). Furthermore, we additionally evaluated the reconstructed RGB frames in full-resolution YCbCr (4:4:4) using a 6:1:1 weighted PSNR metric, achieving a 42.7% average BD-rate reduction over DCMVC across six benchmarks.

More importantly, our motivation is not limited to a specific color space. Events capture brightness changes between sampled frames, providing complementary information at high temporal resolution. We investigate how this information can improve RGB compression by guiding both motion coding and temporal context construction. To this end, ENVC introduces event-guided residual motion coding and gated temporal context refinement. Our analyses in Sec. 5.3 and Apps. E-F further demonstrate the importance of learned event modulation and spatiotemporally aligned events. We will clarify this motivation and our contributions in the revision.

W2. Practicality of Shared Events and Deployment Overhead.

We appreciate the reviewer's concerns regarding the practical deployment of shared-event video compression. We agree that transmitting events solely for RGB compression can introduce substantial overhead. However, our work targets systems where events are already transmitted or stored for other purposes, such as robotic perception, while RGB video is also required (as further described in App. A). In this setting, the decoder can reuse the available events, and our objective is to reduce the additional RGB coding rate under shared-event conditions, rather than the total RGB-event bitrate.

Importantly, leveraging sensor information at both the encoder and decoder is an established setting in multimodal compression research (Lin et al., 2023; Gnutti et al., 2024; Jiang et al., 2025; Sheng et al., 2026). In particular, Gnutti et al. explicitly assume LiDAR depth maps are available at both coding ends, Jiang et al. consider point clouds transmitted beforehand and reused for RGB compression, and Sheng et al. utilize compressed depth information in RGB video coding. These studies support the practicality of exploiting auxiliary modalities already available to the decoder, rather than requiring them solely for RGB compression.

Also, we further examine the impact of event compression (App. A.1) and report the additional computational cost (Sec. 5.5). We also provide sequence-level results, including failure cases, across various sequences in App. C, rather than presenting only favorable examples. We acknowledge the remaining practical limitations and consider efficient event coding, lightweight event guidance, and joint RGB-event compression important directions for future work. We will clarify these assumptions and limitations in the revision.

W3. HDR Evaluation and Event Robustness.

We thank the reviewer for raising important questions about HDR applicability and event robustness. Our use of synthetic events is primarily motivated by the lack of paired event data in Vimeo-90K, which is the standard training set for NVC, and standard NVC benchmarks, allowing fair comparisons on the same RGB content as prior NVC works.

To address this concern, we additionally evaluate ENVC and DCMVC on seven paired SDR/HDR sequences from the Fraunhofer HHI 8K dataset [1] (1080p, 10-bit 4:2:0, 96 frames, intra period 32), without retraining or fine-tuning. HDR events are synthesized from display luminance derived from BT.2100 PQ signals. Compared with SDR, the average event density changed from 0.170 to 0.183 events/pixel/frame, with spatial correlations ranging from 0.43 to 0.87, indicating scene-dependent differences.

The additional RD results are summarized below:

Video ConditionEvaluation MetricBD-rate (↓) (vs. DCMVC)
SDR (BT.2020)PSNR-Y (SDR domain)-54.50%
HDR (BT.2100 PQ)PSNR-Y (PQ domain)-29.20%
HDR (BT.2100 PQ)wPSNR-Y-25.40%

ENVC outperforms DCMVC on six of seven HDR sequences. Although the gains are smaller than those on SDR, these results demonstrate that event guidance remains effective on HDR content without additional training. We will include these results in the revision and discuss HDR-aware training and robustness to different sensor characteristics as future directions.

W4. Synthetic Data Generation and Training Fairness.

We appreciate the reviewer's comments. We first clarify that interpolated frames are used solely for synthetic event generation, not as additional RGB training samples (Sec. 3.1, Fig. 2).

(A) Training Data and Fairness. ENVC uses the original RGB frames and follows DCMVC's 32-frame cascaded training protocol (App. B). To further isolate the contribution of events, we are currently training a zero-event variant under the same architecture and training schedule. We will share the results in a follow-up comment.

(B) Content Dependency. Since interpolation does not increase the RGB training data, the coding gains cannot be attributed to additional interpolated RGB samples. ENVC achieves gains across both low- and high-motion content, including video conferencing sequences (HEVC Class E: −7.76%, Tab. 2). Notably, large-motion sequences achieve greater savings than small-motion sequences (−56.60% vs. −47.96%, Sec. 5.4), highlighting the benefits of event guidance for motion coding.

W5. Model Complexity and Efficiency Trade-off.

We thank the reviewer for the valuable feedback on the computational complexity of ENVC. To address this concern, we developed a lightweight variant, ENVC-L, with two modifications:

1. Event-conditioned Frame Predictor (EFP): We reduce the decoder channel widths, decreasing EFP parameters from 12.40M to 6.30M.

2. Prediction-guided Context Fusion (PCF): We perform context fusion at half resolution and upsample only the learned correction, reducing computational overhead.

ENVC-L was retrained under the same training schedule, with the remaining architecture unchanged. As shown below, ENVC-L reduces the total parameters by 18.2% (33.52M to 27.42M), achieving 3.36× faster encoding and 3.21× faster decoding.

MetricDCMVCENVC (Original)ENVC-L
Parameters21.02M33.52M27.42M
Encoding FPS1.0610.3231.085
Decoding FPS1.2080.3861.240
Avg. BD-rate vs. DCMVC–-39.13%-31.3%

BD-rate (%) is evaluated using PSNR-RGB. Notably, ENVC-L achieves encoding and decoding speeds comparable to DCMVC while maintaining a 31.3% average BD-rate reduction across six benchmarks. We provide the ENVC-L implementation in our anonymized repository (https://anonymous.4open.science/r/ENVC-52A8/) and will include these results in the revised manuscript.

W6. Spatiotemporal Modeling and Compression-Specific Contributions.

We appreciate the reviewer’s comments on temporal modeling and contributions to video compression. We would like to clarify that ENVC explicitly models temporal dependencies through motion-aligned contexts constructed from previously reconstructed frames, propagated context features, and inter-frame events (Sec. 4 and Fig. 4). While we do not explicitly employ cross-frame temporal attention, these contexts provide temporal information for conditional coding. Moreover, ENVC performs joint optimization of all inter-frame modules under a single RD objective in Stage 4, followed by cascaded training with up to 32 frames (App. B). Our contributions are also specific to video compression: event-guided residual motion coding reduces motion coding costs, while gated context refinement improves conditional frame coding. The bit allocation analysis in App. G further shows reductions of 67.9% and 39.8% in motion and frame coding rates, respectively, compared with DCMVC. We will carefully clarify these aspects in the revision.

Q1. Generalization Across Event Sensors.

We thank the reviewer for highlighting the importance of cross-sensor generalization. As discussed in W3, our additional HDR experiments demonstrate robustness to different video characteristics without retraining. Furthermore, App. D evaluates ENVC on real sensor events, demonstrating its applicability beyond synthetic events. The remaining synthetic-to-real performance gap may partly arise from differences in event statistics across sensors and simulators. The limited availability of standardized paired RGB–event datasets further challenges systematic cross-sensor evaluation. We will clarify these limitations in the revised manuscript and explore more diverse event simulation and real-event adaptation as future work.

Q2. Reliability of RD Results and Sources of Coding Gains.

We thank the reviewer for raising concerns about the large RD gains. All codecs follow the same evaluation protocol (96 frames, intra period 32, PSNR-RGB, and estimated bpp; Sec. 5.1). To further support the reliability of our results, we provide frame-wise BD-rate analysis in Q3, and release per-frame PSNR/bpp results and baseline results in our anonymized repository.

Our additional analysis shows negligible I-frame differences (0 to +3.6%), but substantial P-frame savings (−35.7% to −67.0%), confirming that the gains primarily arise from inter-frame coding. Since both methods use the same intra codec, the small I-frame differences arise solely from input padding in evaluation.

Unlike DCVC-RT's implicit temporal modeling, ENVC employs event-guided residual motion coding and temporal context refinement. Our bit allocation analysis (App. G) further shows 67.9% and 39.8% rate reductions in motion and conditional frame coding, respectively, demonstrating that the gains extend beyond motion coding alone.

Q3. Evaluation Protocol and Per-frame RD Analysis.

We clarify that our evaluation protocol is described in Sec. 5.1, with additional PSNR-Y results in App. H. All codecs are evaluated under the same controlled protocol. To further improve transparency, we provide the following frame-wise BD-rate analysis, showing that the coding gains primarily arise from P-frames rather than I-frames. Detailed per-frame PSNR/bpp results for all 96 frames, along with baseline configurations and checkpoints used, are available at our anonymized repository.

DatasetsUVGMCL-JCVHEVC-BHEVC-CHEVC-DHEVC-EAvg
All frames-27.05-49.35-50.75-52.05-42.64-12.94-39.13
I-frame (0)0001.33.61.21
P-frames (1-31)-40.5-60.9-67-66.6-58-35.7-54.8
P 1-4-61.1-68.7-77.5-77.2-72.1-78.8-72.6
P 5-8-39.9-59.9-65.3-63.6-53.9-27.5-51.7
P 9-16-38.3-60.1-67-65.4-57.8-28-52.8
P 17-31-33.6-58.9-63.4-64-53.9-7.5-46.9

Q4. Manuscript Revision and Clarity.

We appreciate the reviewer's constructive feedback and positive assessment of our work. We will revise the manuscript to emphasize the key technical contributions, clarify the evaluation settings and practical assumptions, and incorporate the additional experimental results presented in this rebuttal.

Q5. Test Sequence Nomenclature.

We thank the reviewer for pointing this out. We agree and will correct the test sequence nomenclature to JCT-VC and JVET in the revised manuscript.

[1] B. Kowalewsky, “8K Berlin Test Sequences,” 2019. Available: https://www.hhi.fraunhofer.de/en/departments/vca/research-groups/video-coding-systems/8k-sequences.html

Comments on the paper

2 comments

Dear Reviewers, as the discussion deadline is approaching, please review the authors’ responses and join the discussion. Please share whether your concerns have been addressed and update your assessments as appropriate. Thank you for your time and contributions.

AuthorsOpenReview ↗

Final Remarks to All Reviewers, AC, and SAC

We sincerely thank the Area Chair and all reviewers for their constructive feedback and thoughtful discussions. During the rebuttal period, we have conducted extensive additional experiments to address the reviewer’s concerns and further substantiate the contribution of our paper. We summarize the key contributions recognized by the reviewers and the additional evidence provided in our responses.


Summarized Contributions

We appreciate that the reviewers acknowledge several important strengths of our work:

1. Novelty of Event-guided Video Compression. ENVC explores a complementary source of temporal information for neural video compression, introducing a shared-event coding framework that reduces the additional RGB bitrate when events are already available to both the encoder and decoder.

2. Compression-specific Architecture Contributions. ENVC integrates event information into both residual motion coding and temporal context refinement, rather than merely using events as auxiliary inputs. This provides a coherent framework for exploiting complementary temporal observations.

3. Strong Rate-Distortion Performance and Extensive Analysis. ENVC achieves average BD-rate reductions of 39.13% in PSNR-RGB and 67.63% in LPIPS over DCMVC across all evaluated benchmarks. The evaluations also include motion-dependent analyses, synthetic-to-real comparisons, and other failure cases.

We are encouraged by the reviewer’s positive assessments. Reviewer uvMF explicitly stated that “the reviewer still remains positive towards this work.” Reviewer QYaU confirmed that “The author’s response has satisfactorily addressed my concerns.” Reviewer X3ym also positively acknowledged our additional ablation studies, noting that “they would substantially strengthen the paper” and "this paper can be quite strong”.


Summary of Main Concerns Addressed during Rebuttal

1. Insufficient Architectural Ablations and the Contribution of Event Guidance (X3ym: W5, 26; uvMF: W4; QYaU: W3).

We conducted additional ablation experiments by individually removing RMC, ECR, and PCF. These modifications increased the BD-rate by 57.9%, 81.8% and 96.5%, respectively, demonstrating that each component contributes to our coding efficiency.

Furthermore, we retrained ENVC with zero-event inputs while preserving the architecture. This resulted in 221.2% BD-rate increase relative to original ENVC, confirming the importance of event information rather than model capacity alone.

2. Practical Scenario and Evaluation on Real Event Data (uvMF: W2, W3; X3Ym: W1, W3, W4).

We clarified that ENVC targets RGB-event systems where event observations are already available to both coding ends, and that synthetic events are used for dataset construction rather than as part of the codec.

To directly address concerns about real-world applicability, we additionally trained ENVC on the ERF-X170FPS real event dataset, achieving a 69.5% BD-rate reduction over DCMVC. This provides additional evidence that our event-guided compression is applicable beyond the synthetic event benchmarks, while acknowledging the remaining challenges of cross-sensor generalization.

3. Model Complexity and Comparisons with SOTA NVCs (uvMF: W5; QYaU: W2, W3; X3ym: Follow-up).

To examine the complexity-performance trade-off, we developed ENVC-L, reducing parameters from 33.52M to 27.42M (-18.2%). ENVC-L also achieves encoding and decoding FPS comparable to DCMVC while maintaining a 31.3% average BD-rate reduction.

We also extended the original comparisons to include ECVC (CVPR’25), HyTIP (ICCV’25), and DCVC-UF (CVPR’26). ENVC achieves average BD-rate reductions of 33.4%, 32.2%, and 60.5%.

4. Comparisons with Intermediate RGB Representations (X3ym: W1).

To address the remaining concern regarding the utility of intermediate temporal information, we conducted additional comparisons using interpolated RGB frames as auxiliary references.

Even when excluding the bitrates of these intermediate frames, baselines achieved BD-rate reductions of 27.2% and 25.2%, compared with 35.4% for ENVC. These results demonstrate that RGB-derived intermediate references also can be useful, but shared event observations provide effective complementary guidance.


Closing Remarks

We greatly appreciate the reviewer’s suggestions, which have motivated substantial improvements beyond the original manuscript. We believe the rebuttal significantly strengthens the empirical foundation of our work. While we acknowledge that further investigations remain valuable, we believe these can support the central contribution of ENVC: Exploiting shared event observations for more efficient motion and conditional frame coding.

We sincerely thank the Area Chair and reviewers for their time and consideration, and respectfully hope that these additional results will be taken into account in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.