acceptodds
Withdrawn from ICLR 2027

ViSIL: Unified Evaluation of Information Loss in Multimodal Video Captioning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal video captioning condenses dense footage into a structured format of keyframes and natural language. By creating a cohesive multimodal summary, this approach anchors generative AI in rich semantic evidence and serves as a lightweight proxy for high-efficiency retrieval. However, traditional metrics like BLEU or ROUGE fail to quantify information coverage across disparate modalities, such as comparing a paragraph of text to a sequence of keyframes. To address this, we propose the Video Summary Information Loss (ViSIL) score, an information-theoretic framework that quantifies the video information not captured by a summary via vision-language model (VLM) inference. By measuring the information loss, ViSIL is a unified metric that enables comparison across multimodal summary formats despite their structural discrepancies. Our results demonstrate that ViSIL scores show a statistically significant correlation with both human and VLM performance on Video Question Answering (VQA) tasks. ViSIL also enables summary selection to optimize the trade-off between information loss and processing speed, establishing a Pareto-optimal frontier that outperforms text summaries by 7% in VQA accuracy without increasing processing load.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Video Summary, Video Captioning, Video-to-Text, Information Theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aYPYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ViSIL, an an information theoretic framework to quantifies multimodal video summaries. The idea is to compare the keyword prediction given full videos with multimodal summaries. Experiments on MVBench and LongVideoBench demonstrate ViSIL results are aligned with EQA accuracy.

Critical strengths
  1. The formulation avoids directly solving video-likelihood propabiilities by using caption C as a proxy. It uses \mathcal{I}(C;V|\hat{V}) instead. In this manner, the evaluatation of video caption transfers to a common caption-recovery objective. Besides, this method masks keywords to compute important token for efficiency.
Critical weaknesses
  1. Unfaithfal Experimental Results. Evaluating ViSIL with VQA accuracy cannot demonstrate that lower ViSIL represents better caption. This results can only prove that lower ViSIL represent better captions for QA. No other tasks, such as retrival, generation, recommandation, are included. Therefore, it is somehow over claimed.
  2. Limited baseline comparison. Resolution and length of videos are two key aspects for video analysis. However, experiments do not discuss about the influence of resolution and length. For example, for a 20 min video, 3 frames+text may get much lower performance than a full video. The efficiency analysis is related to resolution. For example, many recent agents, such as GPT 6, resize videos for more information and better analysis. Several single frames for experiments are not enough to draw a faithfal conclusion.
Questions for authors
  1. How does the video length influence the ViSIL results?The frame select strategy is limited by video length. How does the generalization of ViSIL across different length of input videos. For longer videos, its much more difficult for VLM to generate accurate caption C as well as to compute P(C|V).
  2. How does the resolution of video influence the ViSIL results? 1-frame, 2-frame, 3-frame are selected as baselines for both performance and efficiency analysis. Yet, it is not clear which strategy related to better caption quality. How about resizing video frames and concat them? Appendix B is also confusing. ViSIL scores cannot distinguish the difference between text and 3-frames. How can it evaluate the multimodel caption? In Table 5(b), text only is better than 1-image and 3-image on LongVIdeoBench.
  3. The ViSIL is aligned with VQA accuracy. However, different tasks may require different kinds of captions. I recommend to only claim that ViSIL is for VQA accuracy instead of general video captions.
  4. The captions are generated by GPT-4o. Does the findings maintain the same when replacing GPT-4o by much stronger VLM?
Reviewer_BbpFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ViSIL, a reference-caption-based metric for assessing information preservation across text-only, keyframe-based, and multimodal video summaries. It uses a detailed caption of the source video as a semantic proxy and estimates the relative recoverability of masked keywords when conditioning a VLM on either the video or a candidate summary. The authors evaluate associations with human and VLM video question answering (VQA), examine sensitivity to model choices, and explore ViSIL-guided summary selection under a token budget.

Critical strengths
  1. The problem of comparing information preservation across heterogeneous video summary formats is relevant. The paper is clearly written, and its common caption-recovery objective provides an intuitive, format-independent evaluation interface.

  2. The authors investigate the metric from several angles, including human and VLM VQA, evaluator and caption-generator sensitivity, and downstream summary selection. This breadth provides useful initial evidence and makes the limitations assessable.

Critical weaknesses
  1. The evidence for the metric's predictive validity and reliability is limited. The pooled correlations with VQA correctness are weak (Table 1: r = -0.178, -0.129, and -0.228 for MVBench, LongVideoBench, and human VQA, respectively). The paper acknowledges that neither VLM association remains significant after video-level clustering, while the human VQA analysis covers only four source videos. In addition, scores from two different evaluators exhibit weak agreement (Appendix E: Spearman rho = 0.186). These findings do not yet establish a robust or model-stable indicator of summary quality.

  2. ViSIL depends critically on the detailed reference caption, but the reliability of this proxy is insufficiently established. Reference captions are generated by a strong VLM and reduced to a small set of keywords, without a direct audit of their factual accuracy or coverage of important video events. Switching caption generators (Appendix F) and injecting artificial hallucinations (Appendix G) are informative sensitivity checks, but they do not establish the correctness or completeness of the original references; moreover, the hallucination penalty does not robustly persist across evaluator families. This uncertainty affects what the proposed information-loss score actually measures.

  3. The incremental practical value relative to simpler VLM-based evaluation is not convincingly demonstrated. The direct VLM-as-Judge comparison is mixed (Appendix H), without a consistent advantage for ViSIL. The pipeline also requires detailed video captioning, keyword extraction, and repeated VLM probability estimation, yet no end-to-end evaluation-cost comparison is provided. The reported seven-percentage-point gain from ViSIL-guided summary selection is an exploratory within-evaluation result without held-out validation (Appendix I); its token-budget analysis excludes the cost of producing and scoring candidates. These omissions weaken the claimed efficiency and deployment benefits, although preprocessing costs could be amortized in some applications.

Questions for authors
  1. Can the authors provide video-level or within-video analyses, preferably on more independent videos, showing that ViSIL reliably predicts summary usefulness after accounting for repeated observations? How stable are the rankings across evaluator models?

  2. Can the authors directly assess the factual accuracy and event coverage of the reference captions, and quantify how omissions or errors in these references affect ViSIL scores and summary rankings?

  3. How does ViSIL compare with a direct VLM judge in end-to-end evaluation cost and predictive utility, including reference generation and repeated scoring? Does its summary-selection gain persist on held-out videos?

Reviewer_M6CaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Video Summary Information Loss (ViSIL), a common evaluation score for video summaries containing text, keyframes, or both. A detailed caption serves as a textual proxy for the source video. ViSIL compares the caption’s conditional likelihood given the original video against its likelihood given a candidate summary, approximating these quantities through masked-keyword prediction by a VLM. Lower scores indicate better preservation of the captioned information. The paper evaluates associations between ViSIL and VQA correctness on selected subsets of MVBench and LongVideoBench, together with a human study. It also investigates summary-format trade-offs and reports a seven-percentage-point VQA improvement from ViSIL-based candidate selection at a compared token-budget operating point.

Critical strengths
  • The paper addresses the practical difficulty of comparing text-only summaries with summaries containing keyframes and text. Using caption recovery as a common task provides an understandable way to assess information conveyed through different modalities, beyond what text-overlap metrics can capture.

  • Measuring human VQA accuracy, response time, and confidence provides a useful perspective on summary utility. In particular, three-keyframe summaries achieve 78.57% accuracy compared with 80.36% for full videos, while reducing average response time from 85.23 to 65.94 seconds. Although the study covers few source videos, this is an informative observation about the potential usefulness of multimodal summaries.

  • The evaluator comparison, caption-generator ablation, hallucination experiment, and direct VLM-judge baseline investigate relevant failure modes. These analyses help identify where the metric’s behavior depends on its components. Reporting the negative and inconclusive results is valuable, even though the main-text conclusions should reflect them more consistently.

Critical weaknesses
  • The evidence does not sufficiently establish predictive validity across independent videos. Table 1 reports modest pooled correlations with correctness: (r=-0.178) on MVBench and (r=-0.129) on LongVideoBench. Section 5.1 further states that neither VLM association remains significant after video-level clustering. The human analysis includes only four source videos. In Appendix H, both ViSIL confidence intervals include zero, and the comparison does not establish an advantage over a direct VLM judge. These findings leave insufficient evidence that the proposed score reliably predicts summary utility beyond the evaluated observations, which is central to the contribution of an evaluation metric.

  • Evaluator reliability and hallucination robustness are overstated. Appendix E reports cross-evaluator Pearson and Spearman correlations of approximately 0.187 and 0.186, indicating weak agreement rather than reliable ranking consistency. All four matched downstream associations in Appendix E.1 are non-significant. Appendix G.2 also acknowledges that the hallucination penalty does not robustly persist with a GPT-4o evaluator, without providing quantitative results. Because ViSIL depends on both a textual proxy and evaluator predictions, these limitations directly affect its reliability. The main-text claims concerning reliable evaluators and strong hallucination robustness are therefore insufficiently supported.

  • The headline summary-selection gain lacks independent validation and a complete efficiency assessment. The reported improvement from 56% to 63% VQA accuracy is a seven-percentage-point difference, but Appendix I explicitly describes it as an exploratory within-evaluation comparison without held-out-video or independent-dataset validation. Consequently, it does not establish that the selection strategy generalizes to unseen videos. Furthermore, input-token counts characterize downstream input size rather than measured latency or total system cost, and the costs of constructing and scoring candidates are not established by this comparison. The claimed practical advantage therefore remains preliminary.

Questions for authors
  • What are the exact effect estimates and confidence intervals after accounting for video-level dependence? Does ViSIL reliably distinguish better and worse summaries of the same video, particularly at comparable token budgets, rather than primarily reflecting differences between source videos? Please distinguish this evidence from the pooled correlations.

  • How were the candidate pool, operating point, and trade-off parameter (\alpha) in Figure 3 determined? Were any choices informed by the reported VQA outcomes? Is there held-out-video evidence supporting the seven-percentage-point gain under comparable budgets? Please also distinguish downstream input-token savings from summary construction, scoring costs, and measured latency.

  • Can the authors provide quantitative results for the cross-family hallucination experiment in Appendix G.2, including uncertainty estimates? Given the weak cross-evaluator agreement in Appendix E, under which evaluator and caption-generation conditions do the authors consider summary rankings reliable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.