acceptodds
Withdrawn from ICLR 2027

RESUME: Recurrent State Updates from Motion and Residual signals for Efficient Video Language Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing video language models encode sampled RGB frames independently, so a long video must either exhaust the token budget or drop the changes between sampled frames. Codec-aware front-ends read the motion vectors and residuals that encoding already produced, but in their deployed form each predictive frame is still tokenized on its own: the tokens are a function of the current primitives, not of a carried reference. We argue that a more natural function is of both—the current primitives and a carried reference. A clip and its time reversal share the same frames and differ only in the order of changes—an axis that symmetric pooling discards by construction, and that is non-empty in the frozen vision features VideoLMs actually use—and the codec recurrence already composes those changes in order against a reference state. We introduce RESUME, a stateful codec representation: an anchor I-frame initializes a compact latent state, each subsequent predictive frame is consumed as an update to that state, and a shared readout exposes VideoLM-compatible tokens from the accumulated state. Codec prediction is thereby kept at the representation level and handed to the language model as a trajectory, not as a set of independent token groups. At the same per-predictive-frame token budget as prior codec-aware methods, a predictive frame enters the language model as a readout of what the front-end already knows, not as an encoding of the current primitives alone. Across ten benchmarks, the gains concentrate on temporal reasoning: on all three temporal benchmarks RESUME improves over both the RGB-frame baseline LLaVA-Video-7B (by , , and points on TempCompass, TOMATO, and MVBench) and the codec-based baseline CoPE-7B, while staying competitive on general and long-form QA. Frozen-transition tests further show anchor dependence, order sensitivity, and useful rollout behavior beyond the training horizon.

Primary area:
foundation or frontier models, including LLMs
Keywords:
video language models, video understanding, efficient video modeling, representation learning, recurrent state modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BgKjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

For encoding frames of long videos for Video Language models, RESUME initializes a codec-conditioned latent state from I-Frames and updates the state based on the unified motion and residuals to capture state transitions in the latents. Each I-Frame re-initializes the state so a single state is not carried throughout a long video. Aggregation between the groups is delegated to the video language model. RESUME shows improvements over the baseline on three temporal benchmarks and competitive performance on videoQA.

Critical strengths

The method effectively fuses motion vectors and residuals at shared spatial positions instead of concatenating them after processing them independently. The gated residual also contributes to corrections only where the prediction fails.

Critical weaknesses
  1. The recurrence does not learn the temporal information in the video, since the state regresses each readout onto a target that depends only on a given frame t (eqn 9). The trajectory component in the loss vanishes when the state matching is perfect and does not contribute any temporal information explicitly.
  2. The paper lacks a fair comparison with the baseline. In Table 1, RESUME should be compared against the VideoLM and CoPE-7B finetuned on the Stage 2 with similar frame sampling and no readouts. Unlike the inconsistent evaluation of ActivityNet-QA using Claude Opus 4.8 for RESUME and a different judge for the other methods, the above baselines should be evaluated using the same judge as RESUME.
Questions for authors
  1. Please provide clarification on what the state encodes complementary to the per-frame tagets since the loss is composed primarily of functions of per-frame embeddings?
  2. Since z has no spatial layout, how does the state represent spatial displacement?
  3. Please provide fair baselines as described in W2.
1 comment

The additional experiments have been completed, and the results are included in the revised supplementary material.

Reviewer_ZuiyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RESUME, a codec-aware representation for video language models. Instead of tokenizing each predictive frame's motion vectors and residuals independently, RESUME maintains a compact latent state within each group of pictures. The anchor I-frame's frozen vision features initialize the state, each predictive frame's fused motion-residual observation updates it through a gated cross-attention transition, and a shared head reads out 8 tokens per predictive frame from the current state. The transition is first trained on short rollouts to match frozen SigLIP features of the true frames, then attached to LLaVA-Video-7B with a new projector and fine-tuned on LLaVA-Video-178K. The paper reports results on ten QA benchmarks, a reversal probe on frozen features, and frozen-transition probes for anchor dependence, order sensitivity, and rollout length.

Critical strengths
  1. Clear motivation. The paper precisely identifies what independent predictive-frame tokenization discards: a codec update is only meaningful relative to a reference, and updates compose in order. Carrying a state within each group is a natural response.
  2. Consistent temporal-reasoning gains over the RGB baseline. RESUME improves over LLaVA-Video-7B on all three temporal benchmarks (+2.8 TempCompass, +5.1 TOMATO, +3.9 MVBench)
  3. Informative frozen-transition probes. Applying the same updates to different anchors, shuffling update order, and rolling out past the training horizon (Fig. 4) are good diagnostics for whether the transition actually carries state.
Critical weaknesses
  1. The ablation does not isolate the carried state. The central claim is that a carried state is better than independent predictive-frame tokens. Table 2, however, compares I-frames only against I-frames plus 256 readouts, which shows that adding codec-derived information helps (already established by CoPE). It does not show that carrying the state is what helps. In App. D, "I-frames only" is the same model with readouts dropped at evaluation time, a train/test mismatch that handicaps the baseline. In addition, the "+ readouts" input keeps the readouts of all 64 groups while retaining only 8 or 16 I-frames. Since each group's state is initialized from its anchor's features, the readouts implicitly carry content from the 48–56 dropped anchors. The gain may therefore reflect extra anchor coverage rather than tracked change. The claim that the gains are "attributable to the carried state alone" is not supported. The needed control is the same readout budget with the state reset before every update (a memoryless variant), trained and evaluated within the VideoLM. The paper runs this control only in the frozen probes. Table 2 also reports only Video-MME and LVBench, not the temporal benchmarks where the benefit is claimed.

  2. Gains over CoPE are small, and the comparison is not matched. Over CoPE-7B, the temporal gains are +0.5, +1.7, and +0.6. RESUME is 3.2 points below CoPE on LVBench and 1.4 below LLaVA-Video-7B on Video-MME. The paper attributes the Video-MME and LVBench drops to this training mixture, which means the mixture could equally explain the gains. It would be beneficial to add results with LLaVA-Video-7B fine-tuned on the same data and schedule without readouts, and CoPE trained with the same Stage 2 recipe.

  3. ActivityNet-QA scores are not comparable across rows. RESUME is scored by Claude Opus 4.8, while the baselines were scored by a different "language-model" judge- scores from different LLM judges are not comparable. The baselines should be rescored with the same judge, or RESUME with the same judge as the baselines.

  4. Efficiency is claimed but not measured. App. F plots how many hours of video a given token budget covers, but there is no latency or FLOP comparison. The cost of re-encoding every video to a fixed 30 fps I/P structure is not accounted for, and there is no accuracy comparison against LLaVA-Video-7B or CoPE at a matched total visual-token budget.

Questions for authors
  1. Can you report a memoryless variant (same readouts, state reset before each update) trained and evaluated in the VideoLM, including on TempCompass, TOMATO, and MVBench? In Table 2, can you also drop the readouts of groups whose I-frames are removed, so that both inputs see the same anchors?

  2. How does RESUME compare with LLaVA-Video-7B fine-tuned on LLaVA-Video-178K with the same schedule but without readouts, and with CoPE trained under your Stage 2 recipe?

  3. Can you rescore ActivityNet-QA for the baselines with the same judge?

  4. What are the end-to-end latency and FLOPs, including re-encoding, compared with LLaVA-Video-7B and CoPE, and how does RESUME compare at a matched total visual-token budget?

1 comment

The additional experiments have been completed, and the results are included in the revised supplementary material.

Reviewer_XJwDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a codec-based video representation framework that maintains a recurrent latent state between I-frames. The state is initialized from an I-frame and updated using motion vectors and residuals, with compact readouts passed to the language model. Experiments show improvements over LLaVA-Video-7B and CoPE on three temporal reasoning benchmarks, while results on general and long-form video QA are mixed.

Critical strengths

[+] The paper presents a clear motivation for preserving reference dependence and temporal composition in codec-based video representations.

[+] The proposed framework combines I-frame initialization, motion-residual fusion, and recurrent updates to produce compact visual tokens compatible with existing VideoLMs.

[+] The frozen-transition experiments provide useful evidence of anchor dependence, order sensitivity, and rollout behavior beyond the training horizon.

[+] RESUME improves over both LLaVA-Video-7B and CoPE on all three temporal benchmarks, suggesting benefits for temporal and motion reasoning.

Critical weaknesses

[-] The extension over CoPE appears relatively incremental, mainly introducing recurrent state into the codec-based visual encoder. Its contribution is not isolated under matched training settings. Comparing independent encoding, anchor-conditioned encoding with the state reset at each step, and recurrent encoding under the same codec inputs, token budget, and training setup would help establish the benefit of carrying state.

[-] Despite the emphasis on efficiency, RESUME does not report inference latency or memory usage. CoPE reports TTFT and end-to-end latency in Figure 4(a), but recurrent updates may reduce parallelism and increase latency within each GOP. A comparison with CoPE under matched hardware, input sampling, and token budgets would help establish whether RESUME preserves CoPE’s efficiency advantage.

[-] Figures 1 and 2 could be clearer. The dense annotations and fragmented layout make the data flow difficult to follow, particularly the distinction between independent tokenization and recurrent state updates. Simplifying the figures and highlighting the main architectural differences would improve readability.

[-] Minors:

  1. Equations (6) and (10) got out of bound in pages 5 and 6, respectively.
  2. Some sentences use awkward wording and are quite difficult for readers to understand, e.g., "reference dependence would be vacated" (L249) and "Those two cells track the Stage-2 mixture" (L376). These sentences could be expressed more directly.
Questions for authors

See weakness.

1 comment

The additional experiments have been completed, and the results are included in the revised supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.