acceptodds
Under review as a conference paper at ICLR 2027

Context Modeling for Efficient Loss-Resilient Neural Video Compression

Abstract

Rich context modeling gives modern neural video codecs strong rate-distortion performance, while resilience to packet loss has largely been bought by giving that modeling up. We pursue both at once. The obstacle we identify is receiver-side **entropy decoding**. Because the coding distribution is inferred from hyperprior, temporal, and spatial context, an erasure can cause the receiver to use a different distribution, corrupting symbols decoded from intact payloads and propagating damage to later frames. Holding retained payloads byte-identical and correcting only the entropy condition recovers **+5.64 dB at 2% target packet loss**. We formalize the property that excludes this failure, **entropy-condition locality**: the distribution used to decode a retained payload must not depend on what was erased elsewhere. Our codec, **SEAL (Sender-decided Entropy with Assured Locality)**, restores it by serializing the sender's final entropy decision instead of the dependencies that produced it. Sender-only staged refinement preserves rich conditional modeling, while the resulting scale is compressed into **packet-local entropy side information** bound to its residual payload. Both endpoints recover the same coding distribution from this information, without requiring the receiver to reproduce the sender's context. Locality settles interpretation, but the receiver must still reconstruct missing content from a reference state that can diverge from the sender's. **Dual-trajectory training** therefore optimizes reconstruction along a clean sender trajectory and a separate packet-erased receiver trajectory. With side-information cost included, **SEAL** lies on the measured rate-robustness frontier against **GRACE** at its reported rate and the evaluated **H.264/H.265+FEC** configurations under both independent and bursty loss, with gains across **UVG, MCL-JCV, and HEVC** sequences.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural video compression, Packet-Loss Resilience, Context-Adaptive Entropy Coding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fqL2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Context-conditioned neural video codecs such as the DCVC family achieve strong rate-distortion performance. Under packet loss, however, the receiver may infer a different entropy-coding distribution from damaged hyperprior, temporal, or cross-group context, which corrupts symbols decoded from intact payloads and propagates damage across frames, whereas existing loss-resilient codecs avoid this by giving up rich context modeling. The paper formalizes “entropy-condition locality” and proposes SEAL, which transmits the sender’s staged-refined entropy scale as packet-local side information bound to each residual unit so that both endpoints select identical CDFs, keeps staged context refinement on the sender side only, and trains along separate clean-sender and packet-erased-receiver trajectories.

Critical strengths

The paper isolates a concrete and underappreciated failure mode with a clean causal intervention in Section 2.3 and Appendix E.4. Holding the retained payload bytes and loss masks fixed and substituting only the sender CDF indexes recovers 5.64, 4.89, 3.69, with a positive effect on all seven videos.

An efficient loss-resilient codec is most convincingly supported by the unified MTU transport experiment, where all overhead is charged and the baseline uses real FEC with concealment.

Dual-trajectory training is supported by the 128 frame evaluation in Section 4.5, where SEAL’s loss-induced quality deficit grows more slowly than the control’s when each model is compared with its own clean run. The smaller drift of the receiver’s reference state matches the mechanism the training is designed to address.

Critical weaknesses

The claim that SEAL preserves the compression benefit of context modeling is not verified, because its clean efficiency is never compared with the original backbone or with a conventional codec without FEC. The design also turns context the receiver once shared for free into side information that costs bits, and the ablation in Section 4.3 suggests this cost is not small.

No experiment compares SEAL with a simpler way of obtaining locality on the same backbone, such as placing each hyperprior unit in the same packet as its residual. It therefore remains unclear whether transmitting a separately refined scale is necessary for the reported gains.

The main experiments configure the FEC baselines in ways that likely disadvantage them, including high fixed redundancy, disabled concealment, and burst matching that favors SEAL’s smaller packets. Under the fairer unified transport setup the advantage disappears at low loss, so the headline gains are likely overstated.

The main results cover only short sequences starting from a reliably delivered I-frame. This drift is also compared only against a weak control, so both long-session robustness and the specific value of dual-trajectory training remain uncertain.

Questions for authors

How does SEAL’s clean compression efficiency compare with the original DCVC-RT and with H.265 without FEC?

Evaluating longer sequences with periodic refresh or feedback would clarify how SEAL behaves in realistic long sessions.

4 comments

Thank you for your comments. Building on our first response, we ran two additional experiments, addressing your points on a simpler way to obtain locality and on the fairness of the FEC baseline configuration.

A simpler way to obtain locality. We implemented the scheme you suggested on the same DCVC-RT backbone: each hyperprior unit is placed in the same packet as its residual, with the same loss training and packetization as SEAL. The hyperprior can then no longer be lost on its own, but the receiver still has to infer the entropy-coding distribution using temporal context. Without packet loss, this scheme reaches 0.069 bpp at QP55, lower than SEAL's 0.073 bpp, because it does not need to transmit the scales. Under loss, however, once the temporal context drifts, even intact packets are decoded incorrectly: under GE 20% (L=20), 12.6% of the residual symbols at the receiver differ from those at the sender, compared with 0 for SEAL. Correspondingly, its average PSNR is 31.6 dB, 6.7 dB lower than SEAL, and 7.1 dB lower under IID 20%. In other words, SEAL spends about 6% more rate on transmitting the scales and gains about 7 dB in quality under loss. Placing each hyperprior unit in the same packet as its residual closes only the hyperprior path. For the receiver to always select the distribution the sender used, the scale determined at the sender must also be transmitted with each packet.

FEC baseline configuration. We agree that SEAL's advantage over FEC is small at low loss. We therefore further strengthened the H.265+FEC baseline under the unified MTU setup, where error concealment is already enabled: the redundancy is chosen separately for each loss rate to be optimal, both codecs use the same byte-level loss traces, and total rates include all overhead and are matched. At IID loss rates of 20% and below, the two achieve similar quality, within 0.6 dB. When loss is heavier or bursty, the redundancy FEC needs grows quickly, and FEC may fail to recover whole frames, so SEAL's advantage emerges: it leads by 4.7 dB under IID 40% and by 8.1 dB under GE 20% (L=20). In the revision, the unified transport setup will serve as the main experiment.

These results will be added to Section 4 and the appendix of the revision. We would be happy to clarify any remaining concerns.

A short note on why we use H.265+FEC as the reference, and how its configuration was set.

When we started this work, existing methods fell into two camps. DCVC-RT compresses very well on a clean channel but degrades quickly once packets are lost, while GRACE handles loss well but needs a much higher clean rate (Fig. 1). Practical delivery needs both, so we first looked for the existing method with the best overall balance between the two and used it as the reference. Among the methods we evaluated, H.265+FEC gave the best overall balance: on a clean channel GRACE needs about 81% more rate than it, and under loss the average BD-PSNR of GRACE and H.264+FEC relative to it in Table 2 is −0.26 dB and −0.85 dB, respectively. H.265 without FEC needs fewer bits on a clean channel, but it cannot recover lost data, so it cannot serve as a reference for loss-resilient schemes. The GRACE paper also uses H.265 with FEC as its main comparison.

The relatively high redundancy in the main experiments was not chosen to weaken the baseline either. Operating points were selected by a rate rule alone, without reference to any quality measure (Appendix D.6). A supplementary screening shows that, over the tested mixture of loss conditions, higher redundancy actually helps H.265+FEC: in the middle-rate region, raising α from 0.5 to 0.8 improves mean PSNR by 1.67 dB, and raising it to 1.0 adds another 1.38 dB. Only under mild loss such as IID 10% does lower redundancy do slightly better. This matches your observation that the advantage shrinks at low loss, which is why our previous comment selected the optimal redundancy separately for each loss rate.

How the methods compare at different rates can be seen in Fig. 7: the 0% loss endpoint of each panel gives the clean quality in the low, middle and high rate regions. The clean BD-rates in our first response were computed from the clean RD curves of the methods over their shared quality range, and summarize the average rate gap between the curves. The revision will plot the curves themselves.

Thank you for the detailed responses and additional experiments.

The rebuttal confirms rather than resolves my main concern. SEAL’s clean efficiency falls well behind both the original backbone and conventional codecs, yet the paper still claims to retain the compression benefit of context modeling.

The additional experiments do not fully address the locality and baseline concerns. They do show that SEAL’s advantage holds under heavy and bursty loss, and I appreciate this.

The promised changes are substantial. Because no revised manuscript was uploaded, I cannot judge how these changes would actually be reflected in the paper. I find the diagnosis of entropy-condition mismatch valuable and encourage resubmission with narrowed claims.

I will keep my score at 2.

Thank you for your reply, and for finding the diagnosis of entropy-condition mismatch valuable. A brief final note on how we arrived at this design, and why we agree with narrowing the claims.

Network delivery requires splitting each frame into packets, and we first tried two straightforward ways. If DCVC-RT is only split into packets for transport, the receiver still decodes from the whole-frame hyperprior and context, so clean efficiency is kept, but once any packet is lost the scales can differ between the two ends and even intact packets are decoded incorrectly; loss-aware training does not remove this. If each packet is instead coded fully independently, as in GRACE, decoding is robust, but the conditional modeling that makes the DCVC family efficient is largely given up. SEAL tries to sit between the two: each packet stays independently entropy-decodable, while the sender can still use context. This is what we meant by "retaining the compression benefit of context modeling".

The clean results show that this recovers only part of the benefit. Under packetization, SEAL does not yet close the gap to coding the whole frame as a single bitstream, as the original DCVC-RT does. We therefore agree that the current wording claims more than the results support, and we accept your suggestion to narrow it. The revision will revise the related statements in the abstract, introduction and contributions accordingly, state the clean compression cost explicitly in the limitations, and use the unified-MTU experiment as the main experiment.

Thank you again for the careful review; these comments make the paper's conclusions more accurate.

Reviewer_aJfuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SEAL, a resilient neural video codec. SEAL's main idea is the concept of entropy-condition locality, which allows neural video codecs to maintain rich, multi-stage conditional modeling at the sender without requiring the receiver to replicate fragile cross-packet dependencies over a lossy channel. Instead of forcing the decoder to re-derive CDF from potentially erased hyperpriors or diverged temporal states, the codec serializes the sender's finalized entropy decision into low-overhead, packet-local side information bound directly to each residual payload. This ensures that received payloads remain structurally decodable and byte-identical regardless of network erasures elsewhere, while a parallel dual-trajectory training framework optimizes the downstream neural synthesis network to gracefully reconstruct missing content and withstand temporal reference drift without feedback.

Critical strengths
  1. The entropy-condition locality as a design property that completely removes cross-packet dependency is interesting.

  2. The dual-path training is also interesting.

  3. The paper has experiments with actual bitstreams, container layouts, and transport headers. Testing under a unified 1500-byte MTU network emulation over IPv4/UDP with real Reed-Solomon comparisons.

Critical weaknesses
  1. Table 3 details parameter counts and kMAC/pixel computations, showing that SEAL manages to reduce decoder compute relative to the original DCVC-RT backbone. However, the paper explicitly lists wall-clock latency, peak memory footprint, and actual hardware execution bottlenecks as out of scope. Given the focus on real-time video delivery, these real-world metrics are highly relevant.

  2. The implementation is heavily coupled with the real-time architecture of DCVC-RT. While the mathematical definition of entropy-condition locality is universal, the empirical proof is demonstrated on a single family of codecs. It remains an open question how easily this packet-local side-information architecture integrates into non-autoregressive or transformer-based latent architectures.

Questions for authors

See all the weaknesses above, and additionally:

  1. How sensitive is SEAL's performance to different downsampling factors or channels? Does this coarse spatial allocation for the scale representation cause localized distribution mismatches?

  2. In the 128-frame evaluation (Section 4.5), you observe that SEAL significantly slows down deterioration but doesn't eliminate it completely Is this remaining decay entirely driven by the reconstruction module dealing with spatial reference drift in the DPB, or is there still an unaddressed leakage loop inside the conditional context pathways?

1 comment

Thank you for your comments. Our first response mainly addressed your two questions. Here we address the two weaknesses you raised: real-world runtime cost and generalization to other architectures.

Latency and memory. On the same A100 GPU, we measured the time and peak memory of SEAL and official DCVC-RT for encoding and decoding 1080p P-frames. We used UVG sequences with batch size 1, included entropy coding and packetization in the timing, and averaged over frames after warm-up. SEAL decodes in 9.4 ms per frame, versus 8.9 ms for official DCVC-RT; encoding takes 10.2 and 8.0 ms per frame, respectively. Peak memory is 2.21 and 2.27 GB, respectively. Although SEAL has lower decoder compute in Table 3, the receiver must additionally entropy-decode the scale side information in each packet and parse residuals packet by packet, so decoding takes about 0.5 ms longer. The slower encoding comes mainly from the sender-only staged refinement and the coding of scale side information. Both codecs stay well below the 33 ms per frame allowed for real-time 1080p transmission at 30 fps.

Generalization to other architectures. Entropy-condition locality only constrains the output of the entropy model: the coding distribution of each residual unit must be uniquely determined by information in the same packet. In SEAL, the sender derives the scale using any context, and the quantized scale is sent with each packet, so the receiver decodes and uses it directly. How the backbone extracts features, and how the entropy model captures context, therefore does not affect this property. What needs to change is only the step where the entropy model outputs the scale, and how residuals and side information are organized into packets. To our knowledge, leading neural video codecs, including DCVC-FM, DCVC-UF and perceptual codecs such as GLC-video and GVC-RT, adopt a conditional coding framework similar to DCVC-RT: entropy-model parameters are predicted from the hyperprior and temporal context, then refined group by group within a frame. Their receivers likewise have to infer distributions from context that may be damaged, so they can be modified in the same way, with the sender determining the scale and sending it with each packet. Dual-trajectory training also only requires differentiable reconstruction from the decoded symbols and does not depend on the specific structure of DCVC-RT.

We acknowledge that our experiments so far cover only the DCVC-RT backbone. With other architectures, locality still ensures that intact packets are correctly entropy-decoded, but how severe reconstruction drift becomes depends on how each architecture uses its reference state and needs to be evaluated separately. In the revision, we will add this analysis to the discussion and list cross-backbone validation as future work.

Reviewer_QBQAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SEAL, a neural codec which is resilient to packet loss and is based on the SOTA conditional codec DCVC-RT. The authors propose entropy-condition locality, a property which determines if the decoding of received bits could be using a mismatched distribution, due to the packet loss potentially propagated from different paths, i.e., the lost hyperprior and the drifted context. The proposed SEAL utilizes a coded decision for the distribution, by coding additional side information (as latent) instead of inferring the distribution parameters directly from the decoder side context, which could be drifted from the encoder side one. The scale for the distribution is decoded from the side latent, ensuring that the decoding of the latent is resilient. SEAL also uses sender-only staged refinement to keep the context modeling at the encoder side, and dual-trajectory training for the reconstruction drift. The method is evaluated on UVG, MCL-JCV and HEVC under diverse packet loss settings, and the results show that SEAL has better robustness and compression performance than GRACE and H.264/H.265+FEC.

Critical strengths
  • The paper addresses an important question that has rarely been considered in the neural coding literature: in conditional codecs, packet loss can corrupt the decoding of bits that were received correctly, since the receiver may select a different distribution from the lost hyperprior or the drifted context. The proposed entropy-condition locality is a simple and checkable property for this.
  • It proposes a new framework based on conditional coding, with a simple fix that enables loss-resiliency: the scale is coded and signaled based on the context at the encoder side, rather than the common approach in conditional coding methods, which utilizes the decoder side context, so the received bitstream is always decoded with the correct distribution.
Critical weaknesses
  • The clean performance is not sufficiently evaluated, so the compression performance impact caused by the proposed improvements for loss resiliency is not justified. From Fig. 1, DCVC-RT is beyond −60% while SEAL is at ~−30%, which suggests a large clean RD cost compared to the backbone. However, the clean setting configurations are not provided, e.g., which DCVC-RT is this (the official model?). Also, the anchor is H.265+FEC, which includes the FEC on a clean channel, so it is not a common anchor. Since the paper claims to retain the efficiency of context modeling, clean RD curves and BD-rate against a common anchor like HM/VTM (and original DCVC-RT) should be provided.
Questions for authors
  1. For the clean setting in Fig. 1, which DCVC-RT is used, and how are results computed? Could the authors provide the clean RD curves and BD-rate against HM/VTM and DCVC-RT?
  2. Is the proposed way to achieve locality better than simpler designs on the same backbone? E.g., a hyperprior z using context only at the encoder side, and coded and packed in the same way as s?
  3. The side information s only carries the scale. This is enough for locality, but is it optimal? The mean and the quantization step are still predicted from the receiver side context, which could be drifted, although they also do not affect the symbol decoding.
7 comments

Thank you for your comments. Regarding your second question, in our response to another reviewer we added a simpler comparison on the same backbone, which we also report here for your reference.

In this comparison, on the same DCVC-RT backbone, each hyperprior unit is placed in the same packet as its residual, with the same loss training and packetization as SEAL. z can then no longer be lost on its own, but the receiver still uses temporal context to obtain the scale. Without packet loss, it reaches 0.069 bpp at QP55, lower than SEAL's 0.073 bpp, because it does not transmit the scales. Under GE 20% (L=20), however, 12.6% of the residual symbols at the receiver differ from those at the sender, compared with 0 for SEAL, and its average PSNR is 6.7 dB lower than SEAL, and 7.1 dB lower under IID 20%.

This is consistent with the analysis in our first response: even when z arrives intact, if the receiver still relies on possibly drifted context to obtain the scale, the two ends may select different distributions, and intact packets may be decoded incorrectly. This comparison is not identical to the design you proposed. In your design, the sender forms z using context and both ends obtain the scale from z alone, which guarantees that the distributions agree, but z must then carry information previously supplied by context. We will add a comparison of the compression efficiency of this design and SEAL in the revision.

Thank you for the detailed response. I have a few follow-up questions and comments.

Q1:

  • Loss-trained DCVC-RT: is it fine-tuned from the official weights or trained from scratch? The same question applies to SEAL, since the response attributes part of the clean gap to training data.
  • I understand that the clean RD curves will be added in the revision, but there is still no full picture (RD curves + BD-rate) of SEAL against DCVC-RT and the baseline codecs under the clean setting, and the performance cannot be justified now.
  • Using the common-anchor numbers in the response, SEAL's clean performance seems far behind the stock DCVC-RT (more than 2x the rate), and also behind HM (~27% more rate) and the loss-trained DCVC-RT (~1.7x). I think the "efficient" claim does not hold under the current results.

Q2:

  • The same argument for z also applies to s. Both z and s can be formed with context at the (clean) sender and decoded without context at the receiver, so the mismatch between F(z or s, ctx) and G(z or s) applies equally to both.
  • From an information theory perspective, in terms of coding efficiency, a standalone z (formed with context at the sender) should in principle do at least as well as separate z and s. Thus, experiments are needed to support the advantage of having both z and s.
  • The added experiment uses temporal context at the receiver to obtain the scale, so it does not test the design of z vs z + s in a fair manner.

Other comments:

  • The clean BD-rate/etc. results use UVG with 32 frames only. At 120 fps, 32 frames span a short window (~0.27 s) where motion is small, so the conclusions may be biased. Correct decoding of the received packets (with the rest concealed) could benefit more in such settings.

Thank you for the follow-up questions.

Q1: training. Loss-trained DCVC-RT continues loss-aware training from the official weights. SEAL redesigns several modules, such as those for scale side information, and therefore has to be trained from scratch. About two-thirds of the video sources used for the official DCVC-RT training are no longer available. When we retrained DCVC-RT from scratch on the remaining data, its compression performance was about 25% worse than the official weights. To avoid weakening the baseline because of missing data, we let loss-trained DCVC-RT inherit the official weights, whereas SEAL could only be trained on the remaining one-third of the data. Part of SEAL's clean gap to official and loss-trained DCVC-RT therefore comes from training data rather than from the design itself.

Q1: direct clean comparison. We should have reported these comparisons directly in our first response rather than leaving the conversion to you. Since we cannot upload a revision at this stage, we first convert them through the common anchor and group the codecs by whether they are loss-resilient. Among codecs without loss protection, SEAL needs about 209%, 27% and 78% more rate on a clean channel than official DCVC-RT, HM and VTM, respectively, consistent with your conversion; against our reproduction of DCVC-RT trained from scratch on the same data, the gap is about 147%. Note that HM and VTM have no loss protection of their own; over a lossy channel they would also need FEC, which is the deployment H.265+FEC represents. Among loss-resilient schemes, SEAL's clean rate is about 61% lower than GRACE's, and about 56% and 30% lower than those of H.264+FEC and H.265+FEC, respectively (GRACE uses its reported rate). Loss-trained DCVC-RT needs fewer bits than SEAL on a clean channel, with SEAL needing about 72% more, but its quality under loss is clearly lower, as shown in the next paragraph. These are approximate values converted from rate ratios; directly integrated BD-rates and full RD curves will be given in the revision.

Q1: what "efficient" means. We agree that SEAL is not efficient when measured by clean efficiency. In the abstract and contribution 4, "efficient" refers to the rate-robustness frontier, i.e., the joint trade-off between compression performance and loss resilience. However, phrases in the introduction such as "keeps useful context modeling and its RD benefit" can be read as referring to clean efficiency, and we will correct them in the revision. The evaluation covers loss rates from 0 to 80%, and the clean channel is only one point in this range. The operating point we chose favors performance under loss and was not optimized specifically for clean efficiency. Under loss, SEAL achieves higher quality at a similar or lower rate. In Table 2, over ten loss conditions and six dataset groups, SEAL's average BD-PSNR against H.265+FEC is +4.87 dB, and at higher loss rates H.265+FEC does not reach SEAL's quality at any rate (Appendix A.6). Over IID 10%–80% loss, SEAL averages 37.4 dB at 0.073 bpp, versus 27.6 dB at 0.094 bpp for loss-trained DCVC-RT. Under GE 20% (L=20), SEAL leads the strengthened H.265+FEC by 8.1 dB. In the revision, we will state explicitly what "efficient" means, provide the full clean RD curves, and state the clean compression cost as a limitation.

Other comments: 32-frame UVG. In our response to Reviewer fqL2, we reported results on longer sequences. Over 128 consecutive frames under the unified MTU setup (about 1.07 s at 120 fps), SEAL leads H.265+FEC by 12.83 dB on average, compared with 9.06 dB over 32 frames. Over 256 frames (about 2.1 s) with a refresh every 64 or 32 frames, it still leads by 11.7 dB and 9.1 dB, respectively. The advantage does not shrink over longer windows. Regarding motion, the main experiments also include the lower-frame-rate MCL-JCV (24–30 fps, so 32 frames span about 1.1–1.3 s) and HEVC Class B (24–60 fps, about 0.5–1.3 s). In Table 2, SEAL's BD-PSNR against H.265+FEC on these two datasets is +4.37 dB and +5.15 dB, respectively, close to +5.00 dB on UVG.

We would be happy to clarify any remaining concerns.

Comments on the paper

1 comment
Area_Chair_wUmCOpenReview ↗

Gentle reminder, discussion closing soon

Dear Reviewer,

As the discussion period closes soon, we kindly encourage you to engage in the discussion and share any remaining thoughts or feedback.

Thank you for your time and valuable contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.