acceptodds
Under review as a conference paper at ICLR 2027

Beyond RGB Reconstruction: Predictive State-to-Token Adaptation for Video MLLMs

Abstract

Recent advances in multimodal large language models (MLLMs) have improved video understanding, motivating more efficient visual interfaces. Conventional pipelines reconstruct compressed video into RGB frames and re-encode them, despite predictive decoders already producing temporally conditioned states. We introduce **PreSTA** (Predictive State-to-Token Adaptation), a receiver-side interface that maps these states into the intermediate visual space of a frozen MLLM. Under the same eight-group output budget as sparse RGB, PreSTA aggregates 32 P-frame states into compact visual groups, bypassing RGB reconstruction and early visual encoding for selected observations while preserving predictive state updates. A compact RGB teacher trained with privileged source-frame supervision guides the codec-domain adapter through feature- and semantic-level alignment; the codec, bitstream, and MLLM remain unchanged. Across four codec operating points on Video-MME and MVBench with Qwen3-VL-8B and Qwen3-VL-4B, PreSTA stays within 0.3 points of a stronger 32-frame RGB-Fusion baseline on both benchmarks and model scales, and improves Video-MME over the standard sparse Reconstructed RGB-8 pipeline by an average of 6.2 and 6.1 percentage points. Relative to RGB-Fusion, it achieves this with 49.2% fewer visual-front-end weights, 88.5% fewer MACs, and 93.4% lower measured latency from decoder outputs to visual block 2.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video compression, MLLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gAFFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PreSTA, a receiver-side codec–MLLM interface that directly leverages temporally conditioned features from a predictive neural video decoder.

For selected P-frames, the method bypasses RGB reconstruction and the first two blocks of the pretrained MLLM's vision encoder. Instead, it maps intermediate decoder features into visual tokens through a trainable adapter and injects them directly into vision transformer block 2, without modifying the codec or MLLM.

Experiments demonstrate that PreSTA substantially outperforms the sparse reconstructed-RGB baseline and achieves performance comparable to a stronger 32-frame RGB-Fusion baseline, while significantly reducing visual-front-end computation and latency.

Critical strengths

Well-motivated research direction. Video contains substantial temporal redundancy, and reconstructing compressed video into RGB frames before re-encoding them through an MLLM vision encoder introduces redundant computation and latency. Addressing this inefficiency is a meaningful research direction.

Efficient codec–MLLM interface. The proposed method directly leverages intermediate features from a predictive video decoder and maps them into the pretrained MLLM's visual representation space through a trainable adapter, bypassing RGB reconstruction and the first two vision encoder blocks for selected P-frames.

Promising empirical results. The experiments demonstrate that intermediate predictive decoder features can replace RGB reconstruction and early visual encoding without substantially sacrificing accuracy, while significantly reducing visual-front-end computation and latency.

Critical weaknesses

Unclear end-to-end efficiency gains. The paper reports substantial reductions in visual-front-end computation and latency but excludes the latency of DCVC-UF predictive decoding. Consequently, the actual end-to-end speedup and practicality of the proposed method in real-time scenarios remain unclear.

Limited evaluation of out-of-domain generalization. The proposed method relies on intermediate features from the pretrained DCVC-UF codec, which may not generalize well to unseen video domains, such as medical or egocentric videos. The robustness of the proposed codec–MLLM interface under such domain shifts remains unexplored.

Questions for authors

End-to-end inference latency. Although the authors acknowledge the exclusion of DCVC-UF decoding latency in the conclusion, could they report the end-to-end inference latency, particularly the portion attributable to DCVC-UF? This would help assess the practical efficiency gains of the proposed method.

Out-of-domain generalization. Since DCVC-UF is pretrained on Vimeo-90K, could the authors evaluate the proposed method on substantially different video domains, such as egocentric videos? For instance, the Egocentric Navigation in the MVBench. This would help assess the robustness and generalizability of the proposed codec–MLLM interface.

Reviewer_2diaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes PreSTA, a receiver-side interface that maps predictive video-decoder states into an intermediate visual space of a forzen MLLM. It aggregates 32 P-frame states into 8 groups using spatial folding and residual cross-attention, and then injects adapted tokens at some Qwen3-VL visual blcok. For training, it uses an RGB fusion teacher with source-frame supervision, followed by feature and hidden-state distillation.

For evaluation, the paper uses Video-MME and MVBench with two Qwen3-VL models. PreSTA shows matched performance compared with RGB-Fusion baseline, while reducing the measured visual-front-end latency from 273.9 to 18.0 ms.

Critical strengths
  • Video decoding is time-consuming, and the proposed method can reduce this cost significantly in its experimental setting. It is also easy to adapt to pre-trained video LLMs.

  • Experimental results show very promising performance. The compared baselines are meaningful (RGB-Fusion).

Critical weaknesses
  • The end-to-end speed-up is not studied, aside from the front-end efficiency.
Questions for authors
  • Why add a feature to the second block of VIT? Could we try block 1 or before block 1?

  • Compared with Bridge+multi-frame fusion, how much gain can PreSTA bring?

Reviewer_sfFNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

PreSTA adapts predictive video-decoder states into a frozen MLLM’s intermediate visual space. It aggregates 32 observations into eight visual groups, achieving accuracy close to RGB-Fusion while substantially reducing visual-front-end cost.

Critical strengths
  1. The receiver-side design preserves the codec, bitstream, and pretrained MLLM.

  2. RGB-Fusion provides a meaningful comparison with matched temporal observations and output-token budgets.

  3. Experiments cover two benchmarks, four codec operating points, and two model sizes, with useful component ablations.

Critical weaknesses
  1. The reported 93.4% latency reduction covers only the front end up to visual block 2. Although clearly disclosed, the practical end-to-end benefit remains unquantified.

  2. Generalization is limited to one codec and one MLLM family. Separate adapters are trained for each quality setting, and periodic intra refresh is untested.

  3. The contribution of privileged source-RGB supervision is not fully isolated for the student. Table 2 compares teacher accuracy, while disabling Stage 0 changes both teacher architecture and access to source frames.

  4. Methodological novelty is moderate: the approach combines established intermediate-feature adaptation, temporal folding, cross-attention, and distillation.

Questions for authors
  1. What are the end-to-end latency and time-to-first-token improvements, including predictive decoding and downstream inference, for short and long videos?

  2. Can one adapter support multiple or unseen quality settings? How sensitive is it to periodic intra refresh?

  3. With identical student training, how does distillation from reconstructed RGB-Fusion compare with the privileged Stage 0 teacher?

A useful and reasonably well-controlled contribution, with remaining questions about practical impact and generality. I would consider raising my score if the authors adequately address the concerns above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.