acceptodds
Withdrawn from ICLR 2027

Shape the Narrative: A Dataset for Evaluating Message-Driven Narrative Video Editing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video editing is fundamentally message-driven: even from the same source footage, the selected shots change depending on the narrative derived from messages the editor wishes to convey. Benchmarks for a closely related task, video summarization, reduce editorial intent to a single, message-agnostic notion of saliency and thus do not account for this diversity. For evaluating message-driven video editing, we present MEDit-Bench, a dataset and benchmark, which pairs long-form videos with multiple editing messages and multiple professionally produced edits per message, demonstrating that different messages yield substantially different edits from the same source. We define an automatic evaluation protocol based on temporal alignment metrics, and find that an LLM-as-a-judge preference, a natural proxy for narrative quality, is unreliable for this task due to severe position bias. We additionally annotate each message with ambiguity and contextfulness scores, and show that both dimensions negatively correlate with model performance, establishing video editing difficulty as a meaningful stratification factor. Experiments with state-of-the-art MLLMs and reinforcement fine-tuned baselines show that while strong models approach human temporal alignment at lenient thresholds, all models fall behind humans at stricter criteria. A human perceptual study further shows that professional human edits remain preferred over model outputs on every evaluated aspect.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Model, Video, Narrative Video Editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8iu8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies message-driven video editing, where a model is given a long source video and a natural-language message describing the intended narrative, and must select and order temporal segments to communicate that message.

To support this task, the authors introduce MEDit-Bench, which contains 60 long-form videos collected from LongVideoBench, VideoMME, and Blender Studio. Each video is paired with three author-created messages, resulting in 180 video-message pairs. Three professional editors independently edit each pair, producing 540 visual-only, cut-based reference edits. Editors are instructed to create 1–3 minute summaries, freely reorder shots, and ignore audio. Each message is also annotated for ambiguity and contextfulness.

The paper evaluates frozen commercial and open-source MLLMs by prompting them to directly predict a sequence of temporal cuts. It also introduces MEDitor, a GRPO-post-trained model initialized from Qwen3-VL. The post-training data consist of 2,931 GPT-5-generated message-cut pairs from 977 MLVU videos. The reward combines union-based temporal coverage with cut-level matching, encouraging both reference coverage and a reasonable number and placement of cuts. Performance is evaluated using temporal alignment metrics, LLM-based preference, and a human perceptual study.

Critical strengths
  1. MEDit-Bench provides multiple professionally produced edits for each video-message pair. This is a potentially useful resource for studying instruction- or message-conditioned video editing, particularly because multiple human references better reflect the subjectivity of editing than a single reference.

  2. The authors conduct a comprehensive human study covering several perceptual dimensions, including message transmission, cut order, shot continuity, contextual depth, and narrative pacing. This study helps reveal aspects of editing quality that cannot be captured by temporal overlap alone.

  3. The post-training objective is carefully designed. The reward ablation demonstrates that the coverage and cut-level matching terms are complementary, and that combining them improves temporal alignment while producing a more reasonable cut structure.

  4. The analysis of LLM-as-a-judge is transparent and informative. In particular, identifying severe position bias and order inconsistency is valuable because it cautions against using LLM preference as a reliable substitute for human evaluation in this task.

Critical weaknesses
  1. The writing and task positioning need to be improved. I found it difficult to follow the distinction among video editing, video summarization, and recap generation. The paper moves between these concepts without providing a sufficiently precise boundary, even though the proposed task operationally consists of selecting and reordering temporal segments to create a shortened video.

  2. The opening claim that “video editing is fundamentally message-driven” appears overly broad relative to the actual scope of the benchmark. MEDit-Bench is entirely visual-only: editors are explicitly instructed to ignore audio, and the models do not use dialogue, narration, music, sound effects, or audiovisual synchronization. These elements are often central to communicating a narrative in realistic video editing. The task would therefore be more accurately described as visual-only, cut-based, message-conditioned editing.

  3. It remains unclear whether the proposed task is fundamentally different from existing language-conditioned video summarization or recap generation. Several provided “messages,” such as “Breathtaking Swiss landscapes,” “History of the World Cup,” and “Transformation of ingredients,” resemble topic or theme conditions rather than communicative messages that express a specific proposition. The paper needs a clearer definition of what constitutes a communicative message and a stronger explanation of the additional capability required beyond existing language-conditioned summarization.

  4. The main MEDit-Bench experiments primarily compare general-purpose MLLMs and simple embedding-based retrieval baselines. Although SD-VSum is evaluated through cross-dataset transfer, this is not a controlled comparison because the methods use different training data, input semantics, and output formats. It would strengthen the paper to adapt or retrain representative language-conditioned summarization, recap-generation, or instruction-driven compilation methods using the same training data and evaluation protocol.

  5. More objective and scalable evaluation metrics are needed. The current automatic evaluation is almost entirely based on temporal overlap through R@θ, F1@θ, and mIoU. These metrics cannot directly determine whether an edit communicates the intended message, provides sufficient narrative context, or forms a coherent and well-paced story. This is especially problematic because the paper emphasizes that multiple temporally different edits may be valid for the same message.

Questions for authors
  1. Can you formally distinguish a “communicative message” from a topic, theme, query, or language instruction?
  2. What additional capability does MEDit-Bench evaluate beyond existing language-conditioned summarization benchmarks?
  3. How should the benchmark score an edit that communicates the intended message effectively but selects temporally different shots from all three reference edits?
  4. A comparison among the base model, SFT, GRPO, and SFT followed by GRPO would clarify whether the improvement comes specifically from reinforcement learning or more generally from task-specific pseudo-supervision.
  5. Could you include stronger task-specific baselines, particularly language-conditioned summarization, recap-generation, and instruction-driven compilation methods, under a controlled evaluation setting?
Reviewer_T8usOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MEDit-Bench, a benchmark for message-driven narrative video editing, containing 60 long-form videos, 180 editing messages, and 540 professional edits. The authors also propose MEDitor, a GRPO-based model trained on GPT-5-generated pseudo-labels. Experiments with various MLLMs demonstrate improvements in temporal alignment while revealing challenges in narrative coherence.

Critical strengths
  1. Interesting benchmark formulation. The paper illustrates how different narrative messages can lead to different edits of the same video. Collecting multiple professional edits per message is valuable for studying subjective narrative construction.

  2. Comprehensive empirical evaluation. The authors evaluate several strong MLLMs and provide useful analyses of temporal alignment, human preferences, and LLM-as-a-judge reliability.

Critical weaknesses
  1. Confused about the difference with the existing query-guided video summarization & terminology usage. The authors use the terminology such as "Narrative Video Editing", which from my understanding perspective, it should be something like editing narrative video based on the prompt, and it suddenly appears "Video Summarization", which makes me very confused.

Going through the whole paper, the proposed task seems closely related to query-focused, language-guided, and script-driven video summarization. The main difference appears to be the higher-level narrative intention in the text input, while the output remains a sequence of temporal cuts. It is unclear whether the proposed benchmark evaluates fundamentally different capabilities beyond existing text-conditioned summarization tasks, which needs more explanation and completely rewriting in terms of the terminology usage.

  1. Insufficient evaluation of MEDitor's narrative quality / qualitative results of MEDitor. Video summarization paper usually shows some qualitative results / comparisons to illustrate how well their method is doing. However, this paper completely misses that.

The qualitative comparisons (Figure 1) and human perceptual study focus on Gemini-3-pro rather than the proposed MEDitor model. Given that narrative coherence is the central motivation, improvements in temporal alignment alone may not demonstrate better storytelling. Qualitative and human perceptual comparisons between MEDitor, its pretrained backbone, professional edits, and other baselines would strengthen the contribution.

Questions for authors
  1. What fundamentally distinguishes the proposed task from existing text-conditioned video summarization beyond the abstraction level of the input message?

  2. Why do the qualitative results and human perceptual evaluation focus on Gemini-3-pro rather than MEDitor? Does MEDitor actually improve narrative coherence beyond temporal alignment?

  3. How well do the proposed temporal alignment metrics correlate with human judgments, given that different edits may convey the same narrative?

Reviewer_Cd2nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces message-driven narrative video editing as a task distinct from conventional video summarization. The central argument is that professional editing is not simply the selection of salient moments: shot selection and ordering depend on the message that the editor intends to communicate, and multiple valid edits may exist even for the same message. The authors introduce MEDit-Bench, which contains 60 long-form videos and 540 professional edits spanning multiple messages and multiple editors. The benchmark evaluates temporal alignment, message difficulty through ambiguity and contextfulness annotations, LLM-based preferences, and human perceptual judgments. The paper also trains MEDitor using GRPO with pseudo-labeled message-edit pairs generated by a stronger multimodal model.

Critical strengths
  • The paper motivates an important distinction between video summarization and narrative editing. The argument that the same footage can support substantially different edits depending on the intended message is convincing.

  • The benchmark design reflects this motivation well by including multiple messages per source video and multiple professional edits for each message, thereby capturing both message-conditioned diversity and annotator subjectivity.

  • The benchmark reveals that strict temporal alignment remains difficult even for strong multimodal models, providing evidence that precise message-conditioned editing is not solved.

Critical weaknesses
  • The main concern is the tension between the task motivation and the primary automatic metrics. The paper explicitly argues that different shot selections and narrative structures can validly communicate the same message, yet R@θ, F1@θ, and mIoU fundamentally reward temporal similarity to a finite set of reference edits. A valid but substantially different narrative may therefore receive a low score.

  • The paper itself acknowledges that temporal alignment does not capture narrative coherence, while the LLM-based preference metric is shown to suffer from substantial position bias. As a result, a reliable automatic metric for the central property of interest remains open.

  • The benchmark contains only 60 source videos. Given the strong dependence of narrative editing on genre, pacing, and source structure, broader coverage would strengthen claims about generality.

  • MEDitor is less novel than the benchmark contribution. It applies GRPO using pseudo-labeled message-edit pairs generated by a stronger MLLM. It would be useful to better disentangle gains from the proposed optimization formulation from the quality and possible biases of teacher-generated supervision.

Questions for authors

I find the task formulation compelling, but I am not yet convinced that the current automatic evaluation faithfully measures the narrative and communicative qualities that motivate the benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.