- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
For encoding frames of long videos for Video Language models, RESUME initializes a codec-conditioned latent state from I-Frames and updates the state based on the unified motion and residuals to capture state transitions in the latents. Each I-Frame re-initializes the state so a single state is not carried throughout a long video. Aggregation between the groups is delegated to the video language model. RESUME shows improvements over the baseline on three temporal benchmarks and competitive performance on videoQA.
Critical strengths
The method effectively fuses motion vectors and residuals at shared spatial positions instead of concatenating them after processing them independently. The gated residual also contributes to corrections only where the prediction fails.
Critical weaknesses
- The recurrence does not learn the temporal information in the video, since the state regresses each readout onto a target that depends only on a given frame t (eqn 9). The trajectory component in the loss vanishes when the state matching is perfect and does not contribute any temporal information explicitly.
- The paper lacks a fair comparison with the baseline. In Table 1, RESUME should be compared against the VideoLM and CoPE-7B finetuned on the Stage 2 with similar frame sampling and no readouts. Unlike the inconsistent evaluation of ActivityNet-QA using Claude Opus 4.8 for RESUME and a different judge for the other methods, the above baselines should be evaluated using the same judge as RESUME.
Questions for authors
- Please provide clarification on what the state encodes complementary to the per-frame tagets since the loss is composed primarily of functions of per-frame embeddings?
- Since z has no spatial layout, how does the state represent spatial displacement?
- Please provide fair baselines as described in W2.
The additional experiments have been completed, and the results are included in the revised supplementary material.