- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ViSIL, an an information theoretic framework to quantifies multimodal video summaries. The idea is to compare the keyword prediction given full videos with multimodal summaries. Experiments on MVBench and LongVideoBench demonstrate ViSIL results are aligned with EQA accuracy.
Critical strengths
- The formulation avoids directly solving video-likelihood propabiilities by using caption C as a proxy. It uses \mathcal{I}(C;V|\hat{V}) instead. In this manner, the evaluatation of video caption transfers to a common caption-recovery objective. Besides, this method masks keywords to compute important token for efficiency.
Critical weaknesses
- Unfaithfal Experimental Results. Evaluating ViSIL with VQA accuracy cannot demonstrate that lower ViSIL represents better caption. This results can only prove that lower ViSIL represent better captions for QA. No other tasks, such as retrival, generation, recommandation, are included. Therefore, it is somehow over claimed.
- Limited baseline comparison. Resolution and length of videos are two key aspects for video analysis. However, experiments do not discuss about the influence of resolution and length. For example, for a 20 min video, 3 frames+text may get much lower performance than a full video. The efficiency analysis is related to resolution. For example, many recent agents, such as GPT 6, resize videos for more information and better analysis. Several single frames for experiments are not enough to draw a faithfal conclusion.
Questions for authors
- How does the video length influence the ViSIL results?The frame select strategy is limited by video length. How does the generalization of ViSIL across different length of input videos. For longer videos, its much more difficult for VLM to generate accurate caption C as well as to compute P(C|V).
- How does the resolution of video influence the ViSIL results? 1-frame, 2-frame, 3-frame are selected as baselines for both performance and efficiency analysis. Yet, it is not clear which strategy related to better caption quality. How about resizing video frames and concat them? Appendix B is also confusing. ViSIL scores cannot distinguish the difference between text and 3-frames. How can it evaluate the multimodel caption? In Table 5(b), text only is better than 1-image and 3-image on LongVIdeoBench.
- The ViSIL is aligned with VQA accuracy. However, different tasks may require different kinds of captions. I recommend to only claim that ViSIL is for VQA accuracy instead of general video captions.
- The captions are generated by GPT-4o. Does the findings maintain the same when replacing GPT-4o by much stronger VLM?