- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PreSTA, a receiver-side codec–MLLM interface that directly leverages temporally conditioned features from a predictive neural video decoder.
For selected P-frames, the method bypasses RGB reconstruction and the first two blocks of the pretrained MLLM's vision encoder. Instead, it maps intermediate decoder features into visual tokens through a trainable adapter and injects them directly into vision transformer block 2, without modifying the codec or MLLM.
Experiments demonstrate that PreSTA substantially outperforms the sparse reconstructed-RGB baseline and achieves performance comparable to a stronger 32-frame RGB-Fusion baseline, while significantly reducing visual-front-end computation and latency.
Critical strengths
Well-motivated research direction. Video contains substantial temporal redundancy, and reconstructing compressed video into RGB frames before re-encoding them through an MLLM vision encoder introduces redundant computation and latency. Addressing this inefficiency is a meaningful research direction.
Efficient codec–MLLM interface. The proposed method directly leverages intermediate features from a predictive video decoder and maps them into the pretrained MLLM's visual representation space through a trainable adapter, bypassing RGB reconstruction and the first two vision encoder blocks for selected P-frames.
Promising empirical results. The experiments demonstrate that intermediate predictive decoder features can replace RGB reconstruction and early visual encoding without substantially sacrificing accuracy, while significantly reducing visual-front-end computation and latency.
Critical weaknesses
Unclear end-to-end efficiency gains. The paper reports substantial reductions in visual-front-end computation and latency but excludes the latency of DCVC-UF predictive decoding. Consequently, the actual end-to-end speedup and practicality of the proposed method in real-time scenarios remain unclear.
Limited evaluation of out-of-domain generalization. The proposed method relies on intermediate features from the pretrained DCVC-UF codec, which may not generalize well to unseen video domains, such as medical or egocentric videos. The robustness of the proposed codec–MLLM interface under such domain shifts remains unexplored.
Questions for authors
End-to-end inference latency. Although the authors acknowledge the exclusion of DCVC-UF decoding latency in the conclusion, could they report the end-to-end inference latency, particularly the portion attributable to DCVC-UF? This would help assess the practical efficiency gains of the proposed method.
Out-of-domain generalization. Since DCVC-UF is pretrained on Vimeo-90K, could the authors evaluate the proposed method on substantially different video domains, such as egocentric videos? For instance, the Egocentric Navigation in the MVBench. This would help assess the robustness and generalizability of the proposed codec–MLLM interface.