acceptodds
Withdrawn from ICLR 2027

PECV-RT: Real-Time Perceptual Enhancement of Compressed Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Perceptual Enhancement of Compressed Video (PECV) methods have achieved substantial success. However, most methods suffer from high computational complexity, thereby limiting their deployment in real-time scenarios. To address this issue, we present PECV-RT, a lightweight framework that focuses on three bottlenecks in prior work: suboptimal computation allocation across resolutions, costly global spatial context modeling, and redundant computation across overlapping temporal windows. Resolution-aware computation allocation optimizes the distribution of computation by shifting intensive global context modeling to low resolutions, while making local processing lightweight at high resolutions. Linear spatial attention reduces computational complexity from quadratic to linear in the number of spatial tokens. Cross-frame computation sharing limits repeated feature extraction and reuses temporal-attention keys and values across output frames. Experiments show that PECV-RT achieves state-of-the-art LPIPS performance with negligible PSNR degradation. PECV-RT enhances 1080p compressed video at 102 FPS on a consumer-grade GPU, delivering 15.7× the throughput of the fastest evaluated baseline. To our knowledge, it is the first method to perform perceptual enhancement of 1080p compressed video at over 60 FPS on such hardware.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Compressed Video Enhancement, Video Restoration, Perceptual Quality, Real-Time Inference, Linear Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bAk5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PECV-RT proposes a lightweight framework that focuses on three bottlenecks in prior work: suboptimal computation allocation across resolutions, costly global spatial context modeling, and redundant computation across overlapping temporal windows. Resolution-aware computation allocation optimizes the distribution of computation by shifting intensive global context modeling to low resolutions, while making local processing lightweight at high resolutions. Linear spatial attention reduces computational complexity from quadratic to linear in the number of spatial tokens. Cross-frame computation sharing limits repeated feature extraction and reuses temporal-attention keys and values across output frames. Experiments show that PECV-RT achieves state-of-the-art LPIPS performance with negligible PSNR degradation. It is the first method to perform perceptual enhancement of 1080 p compressed video at over 60 FPS on such hardware.

Critical strengths

It is the first method to perform perceptual enhancement of compressed video at real-time speed. A state-of-the-art method of Real-Time Perceptual Compressed Video Enhancement in terms of LPIPS. A good insight for perceptual enhancement of compressed video at real-time speed and practical application.

Critical weaknesses

The paper organization and writing should be improved. Such as the gap of figure and caption is too large. In figure 3, sub-figure (b) cannot found in the paper. The related work section should be out into the second section. The core contributions should be clarified in introduction section. Why author illustrate the figure 1 and figure 2 in the introduction section. The corresponding descriptions of Figure 1(b) and Figure 2 are missing. Some redundancy of motivations and presentations are obvious.

Why only conduct the experiments on QP=37 and 42? For other compressed video quality enhancement methods, 4 QPs or 5 QPs are adopted to verify the effectiveness of enhancement models. Besides, how to obtain the results of other compared methods on HM-16.5andVTM-17.0 conditions? STDR, STCF, TGAF are trained on MFQEV2 dataset and tested on JCT-VC sequences. Are the comparisons fair, given that the models were trained on different datasets? The more detailed settings should be provided and clarified. Could author provide the other perceptual metrices rather than only LPIPS for comprehensive comparisons? Could author the real-world compressed video quality enhancement results of the proposed method and other state-of-the-art methods for verifying the perceptual enhancement performance?

Why the proposed model cannot achieve the best PSNR performance? Compared SOTA method HFGAT, PECV-RT only has a small perceptual quality gain while has large PSNR degradation. For the visual results in Figure 8, the perceptual quality is hard to distinguished compared other methods, especially compared with TGAF. The more visual results should be provided and compared. Other complexity results of calss A-E of testing sequences should be provided rather than only have small resolutions.
Moreover, how to evaluate the temporal consistency of enhanced videos, the corresponding results should be provided and discussed. The more comprehensive comparisons and discussion should be provided for further verifying the effectiveness of the proposed method.

Some recent compressed video quality enhancement methods should be discussed and compared, such as: [1] CPGA: Coding priors-guided aggregation network for compressed video quality enhancement. CVPR2024 [2] MDEformer: Mixed difference equation inspired transformer for compressed video quality enhancement. TNNLS2024 [3] Rivuletmlp: An mlp-based architecture for efficient compressed video quality enhancement. CVPR2025 [4] Multi-frame deformable look-up table for compressed video quality enhancement. AAAI2025 [5] Perceptual quality enhancement for compressed video with high-frequency details and high-dimensional features. TIM2025 [6] Blind Quality Enhancement of Compressed Video via Fine-Grained Degradation-Guided Sequential Inference. TCSVT2026

Questions for authors

Please view Critical Weaknesses.

Reviewer_efJmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes an ultra-lightweight framework for video compression enhancement, dubbed PECV-RT, which achieves state-of-the-art LPIPS performance while enabling ultra-fast encoding and decoding.

Critical strengths
  1. The framework is highly efficient, achieving real-time encoding/decoding at 102 FPS on an NVIDIA 4090D.
  2. It delivers strong LPIPS performance.
Critical weaknesses
  1. PSNR performance decreases when applied to VTM-based compression, and shows only marginal improvement when applied to HM-based compression.
  2. Compared to HFGAT, this method results in a PSNR drop of nearly 0.2 dB, while offering only a slight additional gain in LPIPS.
Questions for authors

Q1: In Table 2, most video compression enhancement methods—including the one proposed here—exhibit a decline in PSNR. This seems to contradict the fundamental goal of video enhancement, as it appears that PSNR is being sacrificed to improve LPIPS. Furthermore, a 0.1 dB drop in PSNR is quite significant; specifically, this method shows a PSNR reduction of nearly 0.2 dB compared to HFGAT, with only a marginal improvement in LPIPS.

Q2: Given that HM and VTM are traditional, extremely slow video compression standards, and this work aims to provide a practical solution for video compression enhancement, the experiments should ideally be conducted using x265 or x264.

Reviewer_NB2EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PECV-RT enhances compressed video using nine consecutive input frames: the first provides a difference reference, and the remaining eight are enhanced jointly. The model combines lightweight high-resolution processing and detail skips with spatial linear attention and bidirectional temporal attention at H/16 resolution. Shared computation across outputs reduces inference cost. Training combines MSE and AlexNet-LPIPS. Evaluation uses 18 JCT-VC sequences encoded with HM-16.5 and VTM-17.0 in low-delay P configurations at QP37/42, reporting perceptual quality, computational cost, and 1080p throughput of 102.02 FPS on an RTX 4090D.

Critical strengths
  1. The efficiency gain is large while frame-level perceptual quality remains competitive. Table 4 increases 1080p throughput from HFGAT's 6.51 to 102.02 FPS and reduces MACs per pixel from 238,787 to 26,401. Tables 2–3 report the best mean LPIPS across all four codec/QP settings at the displayed precision. This is a useful quality–efficiency trade-off, even though it does not dominate the baselines in PSNR.

  2. Shared inference has a direct control experiment, and evaluation covers complete sequences. Disabling sharing while retaining the same outputs reduces throughput to 25.07 FPS in Table 5; Appendix C and Table 7 provide further component ablations. Appendix A evaluates all 7,962 P-frames from sequences of 150–600 frames, rather than selected short clips.

Critical weaknesses
  1. The real-time result establishes buffered throughput, not low per-frame latency. The bidirectional attention in Section 2.2 and eight-output processing in Figure 6 require the full block. Under Appendix B's timing convention, 102.02 FPS corresponds to about 78.4 ms per eight-frame call, not a response 9.8 ms after each frame arrives. For a stream arriving at 60 FPS, the first target frame would additionally wait about 116.7 ms for the other seven frames. This is an architectural latency estimate, not an end-to-end measurement. Buffered playback remains a valid application, but the paper needs to define that operating regime when making its real-time claim.

  2. The evaluation does not establish temporal stability across output blocks. Eq. (5) optimizes LPIPS, Tables 2–3 report frame-level metrics, and Figure 8 shows static crops. The output-position embeddings in Section 2.1 and changing frame groups in Section 2.3/Figure 6 make the transition between successive eight-frame blocks a specific concern. Appendix C includes temporal-attention and frame-independent ablations, but still reports no temporal consistency measure. The results therefore support frame-level perceptual improvement more directly than stable video enhancement; whether block boundaries introduce artifacts remains untested.

Questions for authors
  1. Is the intended setting playback with prebuffering? Please separate future-frame waiting time, GPU execution time per block, and time to the first output. For the HFGAT comparison, please also specify numerical precision, execution or compilation mode, batching, and timing boundaries. A causal redesign is not needed to answer this question.

  2. On the existing test outputs, is temporal error higher across block boundaries than within blocks? Please compare with decoded video and HFGAT on sequences containing motion and fine texture, using motion-compensated residuals with stated occlusion handling or temporal-change error relative to ground truth. A pronounced boundary effect would change my recommendation; the buffering requirement alone would not.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.