acceptodds
Under review as a conference paper at ICLR 2027

DecodeAlign: Post-Decoding Representation Alignment for Video Coding for Machines

Abstract

With the increasing use of videos for machine understanding, the objectives of human-oriented video coding differ from those of machine tasks, making the improvement of machine readability for compressed videos an important challenge. However, existing approaches for machine tasks often require modifications to the coding process, new bitstream designs, or joint optimization with machine models, limiting their applicability in existing video coding ecosystems. In this paper, we revisit VCM from a representation perspective and reveal that the machine readability of compressed videos depends not only on the information preserved during encoding, but also on the effectiveness of decoded content representations for downstream machine models, which is not constrained by the original coding process. Based on this observation, we propose DecodeAlign, a post-decoding representation alignment framework for compressed video machine tasks. DecodeAlign keeps the original codecs, bitstream structures, and downstream task models unchanged, and introduces an independent representation alignment network after decoding to reduce the representation discrepancy between decoded and original content representations in the representation space of machine models, thereby improving machine task performance. Experiments demonstrate that DecodeAlign effectively improves machine task performance across eight host codecs, achieving up to 45.6% BD-rate reduction, and generalizes across codecs according to compression distortion distributions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video coding for machines, video compression, representation alignment, post-decoding representation alignment, object detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wDY9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DecodeAlign, a post-decoding module for cfm. They use a frame-wise NAFNet to enhance the frames. It is trained on x265/DCVC-UF-compressed Vimeo-90k frames with a Charbonnier pixel loss and a cosine feature loss on the frozen Faster R-CNN (ResNeXt-101) backbone features. The proposed method splits the feature error of decoded frames into an irreducible term ("information availability") and a mismatch term ("representation availability"). This paper demonstrates that post-processing can reduce only the mismatch term. Experiments show up to a 45.6% BD-rate reduction.

Critical strengths
  1. The proposed method is a simple plug-in with consistent gains at low bitrates and can improve all kinds of codecs
Critical weaknesses
  1. The contributions overlap substantially with prior work that is not cited or is misdescribed.
  • Representation-space insight (Sec. 1, 3.1–3.2). This is the premise of [H]: Eq. 2 here is the intermediate-layer distortion of [H] (Eq. 10), and [H] proves that coding optimized for input fidelity is no better, and under mild conditions strictly worse, than coding optimized for task distortion (Thms. 3–4). Beyond [H], Sec. 3 only adds Prop. 1. That is the textbook MSE decomposition around E[Z∣YR]\mathbb{E}[Z\mid Y_R], and it gives no guarantee for a frame-domain GθG_\theta.
  • Post-decoding framework. Post-filters that adapt a human-centric codec to machine tasks already exist. [A] post-filters VVC-decoded images and reports ~49%/45% BD-rate gains for detection/segmentation. [B] trains post-filters after a VTM decoder with a feature-domain proxy loss. The paper cites [B] but lists it among methods that change "the coding process", and then claims that all existing approaches "adapt the coding process" (L048–050). Both statements are inaccurate as far as I understand.
  • Training objective. Training a restoration network with a loss on a frozen task network's backbone features, without labels, and transferring it to other task networks, is TTAC [C]. [H] likewise uses multi-level feature-MSE distillation on frozen Faster R-CNN/YOLOv3 features, and task-driven restoration is also well studied [D, E, F]. Fig. 5 shows generic DINOv2 features work about as well (+2.00 vs. +2.28 mAP on Faster R-CNN; +0.95 vs. +0.92 on YOLOv8m). The gain therefore looks like perceptual-loss [G] artifact removal rather than alignment specific to the target model.
  1. The experiments do not support the practicality and generality claims.
  • The only baseline is the unprocessed host output. There is no comparison with [A, B], TTAC [C], a strong compressed-video enhancement network, or even simply fine-tuning the detector on decoded frames.
  • Inference complexity is not provided. Needs to report runtime, MACs, or peak memory.

Reference:

[A] Ahonen et al., "Learned Enhancement Filters for Image Coding for Machines," IEEE ISM 2021.

[B] Zhang et al., "Competitive Learning for Achieving Content-specific Filters in Video Coding for Machines," ICIP 2024.

[C] Ehrlich et al., "Analyzing and Mitigating JPEG Compression Defects in Deep Learning," ICCV Workshops 2021.

[D] Liu et al., "When Image Denoising Meets High-Level Vision Tasks: A Deep Learning Approach," IJCAI 2018.

[E] Son et al., "URIE: Universal Image Enhancement for Visual Recognition in the Wild," ECCV 2020.

[F] Haris et al., "Task-Driven Super Resolution: Object Detection in Low-resolution Images," arXiv 2018.

[G] Johnson et al., "Perceptual Losses for Real-Time Style Transfer and Super-Resolution," ECCV 2016.

[H] Harell et al., "Rate-Distortion Theory in Coding for Machines and Its Applications," IEEE TPAMI, vol. 47, no. 7, pp. 5501–5519, 2025.

Questions for authors

Please see the weakness.

Reviewer_an7NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DecodeAlign improves object detection on compressed video through a learned post-decoding enhancement module. The module aligns features extracted from enhanced frames with those of the original frames, while keeping the codec, bitstream, and downstream detector unchanged. The paper also examines transfer across codecs and representation extractors. Experiments on eight host codecs report improved rate–detection performance, including transfer to an unseen detector.

Critical strengths
  • The post-decoding design is practically useful and relevant as it can operate with existing codecs and frozen downstream models.

  • Evaluation includes codecs excluded from training, an unseen YOLOv8m detector, and alternative DINOv2/CLIP alignment features. These provide meaningful evidence of transfer.

  • The loss ablation demonstrates that improved pixel reconstruction does not necessarily improve detection, which is unsurprising but still good to know. Feature alignment produces more consistent detection gains than pixel-only training.

Critical weaknesses
  • Unreported inference cost. The method adds a 29.2M-parameter network, but its latency and memory overhead are not reported. The bitrate savings therefore do not fully characterize the practical bandwidth–computation trade-off.

  • Novelty concerns. Zhang et al. (2024), already cited in the paper, describes post-decoding filters trained using feature differences between original and enhanced images. Thus, feature-supervised post-decoding adaptation itself is already established. DecodeAlign’s evaluation across eight codecs and its transfer findings are valuable, but the paper should more clearly distinguish these contributions from the existing approach. The current comparisons demonstrate improvements over unprocessed codec outputs, while leaving unclear how much additional benefit or understanding DecodeAlign provides over prior task-aware post-processing.

  • Unclear transfer interpretation. Figure 4 supports differences between within-family and cross-family transfer, but does not explicitly control compression severity across codecs. Since equal bitrate does not imply equal distortion or detection difficulty, part of the observed pattern could reflect different levels of corruption or performance headroom.

Limited sequence-level analysis. Results are averaged over 14 sequences. This establishes average improvements, but does not show how consistently individual sequences benefit or whether some experience degradation.

Questions for authors
  • At comparable host detection accuracy or reconstruction quality, does within-family transfer remain stronger than cross-family transfer? Can the existing rate points help separate distortion-family effects from compression severity?

  • What latency and memory overhead does DecodeAlign add to the complete decoding-plus-detection pipeline at the evaluated resolutions?

Reviewer_kF85OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed a method named DecodeAlign, which is a post-processing method for the codec compressed video frames to adapt to downstream visual analysis tasks. The proposed method basically learns a proxy representation for the decoded images before they were sent to the analysis tasks. The authors have tried their efforts in formulating the mapping procedure from the pristine images to the approximated images for the visual analytics. The experimental simulations revealed the effectiveness of the proposed method.

Critical strengths
  1. A novel understanding and formulation for the VCM in terms of post-processing.

  2. The performances of the proposed DecodeAlign is encouraging in the manuscript.

  3. The proposed approach is codec-agnostic and can be adapted to the any other vide codecs.

Critical weaknesses
  1. The major suggestion from where the reviewer stands, is that, this manuscript is more suitable to be submitted to the computer vision related journals rather than ICLR conference. Specifically, the VCM related papers are more machine-oriented design with engineering level modeling and optimization. The learning representations formulated in this paper is theoretically weak and other than some well-known conclusions from the visual analysis. Especially eq. 4-7 and related proof in Appendix, the conclusion only holds when the down stream tasks are closed-set classification tasks. When it comes to open set really world analytical visual models, the proof does not strongly stand from the assumption for eq. 4.

  2. From the implementation detail, the authors actually employed the NAFNet (ECCV2022) model to achieve the pixel-to-pixel mapping from the decoder to the input of downstream visual analysis models. The domain transfer and auxiliary feature-level guided adaptation for the VCM can be hardly recognized as a solid and inspiring method for the ICLR community.

  3. Regarding the experimental results, the authors should include how were the quality of all decoded images assessed, for instances, the color space, the bit-rate for each frame. Similar details should also be reported for the anchors, such as the parameters of VTM, VVenC, x265, DCVC-family models and SMC++ etc.

  4. The authors have mentioned in the experiment section that the videos were processed frame by frame without some temporal attention and joint optimization. The reviewer would suggest to consider such spatial-temporal effect form the native video representation rather than abandoning the rich contextual information. The objective of more robust VCM models should be targeted at stronger analysis performance with the consideration of compression effects. If some tricks from other domain were directed borrowed, there is limited contribution to the VCM itself. And the VCM will just be another task in the computer vision related study, which should zero improvement to the signal processing and visual communication field.

Questions for authors

Please see above, the reviewer would like suggest the authors to submit this work to IEEE TPAMI/TIP/TCSVT or Pattern Recognition. The topic is more encapsulated by those journal publication.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.