acceptodds
Under review as a conference paper at ICLR 2027

Generative co-evolution: rethinking weakly-supervised video highlight detection in the MLLM era

Abstract

Weakly-supervised video highlight detection (WSVHD) aims to localize highlight moments using only video-level labels. While Multimodal Large Language Models (MLLMs) have strong reasoning capabilities, adapting them to WSVHD remains underexplored and is severely bottlenecked by sparse supervision and complex topic semantics. Conventional supervised fine-tuning (SFT) struggles under this signal scarcity, whereas semi-supervised approaches are constrained by static unlabeled datasets and laborious manual curation that stall rapid model iteration. To address this problem, we introduce a novel generative-adversarial post-training framework that drives MLLM performance improvement for WSVHD without human-curated unlabeled data. Rather than isolated multiple instance learning, we introduce a generative model that co-evolves with the MLLM using an efficient video synthesis tool. Through parameter updates guided by Group Relative Policy Optimization (GRPO), the generative model moves beyond a static data generator and synthesizes realistic, topic-relevant training video features to increase the MLLM's discriminative capability for WSVHD. Extensive experiments demonstrate that our co-evolutionary paradigm significantly outperforms standard supervised fine-tuning and classical semi-supervised learning, establishing new state-of-the-art performance for WSVHD across three public benchmarks. Code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
weakly-supervised video highlight detection, multimodal large language models, semi-supervised learning, video understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vnJKOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a new method solving the problem that semi-supervised approaches' ability are constrained by static unlabeled datasets and financial and time cost of human labeling, by introducing a generative model improves together with th MLLM.

Critical strengths

The experiments are sufficient to prove that their methods outperform standard supervised and semi-supervised models; The problem they bring out and try to solve is very valuable because the co-evolution paradigm does not need dense human labeling and still achieves fully supervised results. Further, this paradiam may inspire other tasks constrained by labeling.

Critical weaknesses

None

Questions for authors

Do we need to change the critic model when the input video topic changes, during the inferences? Why is the perfomance on several topics significantly lower than the previous SOTA models?

1 comment
AuthorsOpenReview ↗

Author Response

We sincerely thank the reviewer for the positive evaluation and the insightful comments. The reviewer's sharp and constructive feedback clearly reflects deep expertise in the problem addressed in this manuscript. We are glad that you recognized the value of our co-evolution paradigm in circumventing dense human labeling.

Below, we address the two questions raised by the reviewer:

Response to Questions for Clarification: Question 1: "Do we need to change the critic model when the input video topic changes, during the inferences?"

Response: Yes. Following common practice in previous WSVHD studies, a critic model is trained for a particular topic or domain. This is consistent with previous WSVHD literature, where models are typically adapted or evaluated per domain/topic.

Question 2: "Why is the performance on several topics significantly lower than the previous SOTA models?"

Response: There are two primary reasons for this performance variation across specific topics:

Uniform Hyperparameters without Overfitting: We opted to use the exact same hyperparameters across all datasets and did not conduct heavy, specialized per-topic optimization of prompts and hyperparameters, prioritizing overall robustness and generalizability over topic-specific tuning.

Efficiency Trade-off in Video Synthesis: To ensure fast training loops during the GRPO phase—where each iteration requires synthesizing a group of videos—we utilized a relatively lightweight video generation model (TurboWan2.1-T2V-1.3B). We found that this model is not as competitive or visually rich as larger video synthesis models, but it was necessary for computational efficiency. We firmly believe that future advancements and improvements in video generative models will bring direct performance gains to our co-evolution framework.

We thank the reviewer once again for their supportive review and valuable time. Please feel free to ask any further questions as we continue to address misunderstandings and concerns raised by other reviewers during this discussion phase.

Reviewer_W6KHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a generative co-evolution framework for weakly supervised video highlight detection. A Director LLM generates scene descriptions, which are then rendered into synthetic clips by a frozen video generation model. The MIL-based Critic learns from weakly labeled real videos and synthetic clips through a supervised objective. The Director is alternately optimized through GRPO using realism and highlight-related feedback from the Critic. On TVSum and YouTube Highlights, the Critic uses frozen visual/audio encoder features with trainable attention and prediction modules; on MoSu, it uses the CLIP features provided by the dataset. Experiments report improved average performance on these three benchmarks, with ablations on TVSum comparing SFT, MIL, semi-supervised learning, and static versus adaptive synthetic-data generation.

Critical strengths
  1. Adaptive synthetic supervision offers a useful approach to addressing scarce weak supervision. The framework combines a Director trained through GRPO, a frozen video generator, and a MIL detector to adapt synthetic training data through task feedback. In the 100-clip setting, updating the Director improves TVSum Top-5 mAP from 0.778 to 0.800, while using only bag-level rewards yields 0.782 (Table 1). These ablations support the value of adaptive generation and highlight-specific rewards, providing an important argument in favor of acceptance.
Critical weaknesses
  1. Synthetic labels may primarily reflect topic relevance rather than highlight salience. Appendix A.5.2 explicitly excludes target-topic content from negative prompts, while positive prompts describe typical scenes within the topic. These generation conditions are directly used as segment-level labels, but the paper does not report independent label validation of the rendered content or a controlled comparison using non-highlight examples from within the same topic. Therefore, the additional supervision may reinforce topic-based shortcuts, and there is insufficient evidence that it teaches the model to distinguish between salient and ordinary moments within the same topic.

  2. The stability of the gains and their attribution across datasets remain insufficiently established. Based on the average scores reported in Tables 2-4, the gains over the strongest listed baselines are 1.3-1.9 percentage points, but no variability across runs or confidence intervals are reported; in category-wise comparisons, the method exceeds the strongest prior result in each category in only 5 of the 10 TVSum categories and 3 of the 6 YouTube categories. Ablations of key components, including MIL, Director updates, and rewards, are reported only on the 50-video TVSum dataset, while YouTube Highlights and MoSu lack corresponding MIL-only and frozen-Director controls. These omissions limit our confidence in the robustness of the gains and make it difficult to confirm whether co-evolution explains the improvements across datasets.

  3. The Critic described in the paper does not sufficiently support the central claim of MLLM post-training. The architecture described in Figure 2 and Appendix A.3 consists of frozen visual/audio encoders followed by trainable projection, self-attention, cross-attention, and prediction modules. According to the paper's description, no language backbone is described as participating in the Critic's prediction or update path. The SFT baselines that adapt a language backbone score 0.729/0.693, compared with 0.768 for MIL, but these configurations also differ in other respects and do not isolate the role of MLLM reasoning; the MoSu Critic uses the CLIP features provided by the dataset. Therefore, these experiments directly support the effectiveness of data augmentation for a detector built on pretrained multimodal features, but do not directly substantiate the paper's claim of MLLM post-training or the statement about unlocking MLLM cognitive priors in Section 4.3.

Questions for authors
  1. Clarification of synthetic-label validity: To what extent do the generated labels reliably reflect highlight salience rather than whether a clip belongs to a particular topic? Can the authors provide an independent assessment of label fidelity and a controlled comparison using non-highlight clips from within the target topic? This evidence would materially affect my assessment of the claimed fine-grained supervision.

  2. Evidence for robustness and attribution: Can the authors report results across multiple random seeds with uncertainty estimates, and extend the MIL-only controls and the ablations comparing static and updated Directors under otherwise matched conditions to YouTube Highlights and MoSu? These results would help clarify whether the gains are stable and can be consistently attributed to co-evolution.

  3. Clarification of the MLLM claim: According to Figure 2 and Appendix A.3, the Critic appears to extract features using frozen visual/audio encoders and train the subsequent attention, fusion, and prediction modules. Please confirm whether a language backbone participates in detection. If not, what exactly do "MLLM post-training" and "unlocking MLLM cognitive priors" mean in the paper? The current experiments demonstrate improved detector performance; what evidence supports attributing these improvements to the use of MLLM cognitive priors?

2 comments
AuthorsOpenReview ↗

Author Response

We sincerely thank the reviewer for the thorough reading, thoughtful assessment, and constructive feedback. We greatly appreciate your recognition of the potential of adaptive synthetic supervision and highlight-specific task feedback. In response to your comments, we have made revisions and added new experiments to directly address each of your concerns:

Question: "Clarification of synthetic-label validity: To what extent do the generated labels reliably reflect highlight salience rather than whether a clip belongs to a particular topic? Can the authors provide an independent assessment of label fidelity? This evidence would materially affect my assessment of the claimed fine-grained supervision."

Response: We thank the reviewer for this very insightful question. To provide an independent assessment of label fidelity and highlight salience, we conducted a new experiment using a separate, frozen critic to evaluate the highlight salience of the synthetic positive clips generated during training. Specifically, we use the pre-trained MIL+SSL model as an independent judge to predict the highlight reward for positive/highlight clips synthesized throughout the director's GRPO training. Because this critic is trained within each topic to distinguish salient highlight segments from ordinary non-highlight segments of the same topic (as demonstrated by its strong detection performance in Table 1), its predicted reward directly measures true highlight salience rather than mere topic membership. A higher reward therefore indicates greater highlight salience in the synthesized clip.

The results are added as Figure 5 in the revised manuscript. Across all topics in TVSum, the highlight reward predicted by this independent critic steadily increases throughout GRPO training. This upward reward trend verifies that during generative co-evolution, the director genuinely learns to synthesize highlight clips that are increasingly and significantly more salient, directly supporting our claim of effective fine-grained supervision.

Question: "Can the authors report results across multiple random seeds with uncertainty estimates, and extend the MIL-only controls and the ablations comparing static and updated Directors under otherwise matched conditions to YouTube Highlights and MoSu? These results would help clarify whether the gains are stable and can be consistently attributed to co-evolution."

Response: We thank the reviewer for this constructive suggestion. We have added the MIL-only controls and the ablations comparing static (no-director update) and updated Directors under matched conditions for both YouTube Highlights and MoSu in the revised manuscript (Tables 3 and 4). From these updated tables, we draw the same consistent conclusions as in Table 1, demonstrating that our proposed co-evolution framework is effective across datasets. In addition, we evaluated detector training across 5 random seeds to quantify stability. On the large-scale MoSu dataset, the average standard deviation of topic-level mAP is only 0.0080.008. On smaller datasets like YouTube Highlights and TVSum, the average standard deviations are 0.030.03 and 0.040.04, respectively—a level of variance consistent with prior open-source WSVHD studies such as MINI-Net. Thus, our method exhibits comparable stability to established baselines while achieving verified state-of-the-art performance on the newest and largest-scale benchmark.

Question: "Clarification of the MLLM claim: According to Figure 2 and Appendix A.3, the Critic appears to extract features using frozen visual/audio encoders and train the subsequent attention, fusion, and prediction modules. Please confirm whether a language backbone participates in detection. If not, what exactly do 'MLLM post-training' and 'unlocking MLLM cognitive priors' mean in the paper? The current experiments demonstrate improved detector performance; what evidence supports attributing these improvements to the use of MLLM cognitive priors?"

Response: We thank the reviewer for this careful observation and for pointing out this important distinction. We confirm that in our SFT video question-answering baselines, the full MLLM—including the language decoder—is actively used. In contrast, for our MIL-based Critic detector, the language decoder is not used during detection; instead, the pre-trained MLLM serves as a multimodal foundation backbone to extract rich visual and audio representations aligned during its multimodal pre-training. To avoid any ambiguity, we have revised the manuscript to clearly distinguish between these two settings and refined our phrasing around MLLM adaptation accordingly.

Furthermore, as also suggested by Reviewer iyYf, we added a new SFT experiment (Appendix 2.2) that utilizes the full MLLM (including the language backbone) supervised with segment-level pseudo-labels derived from our MIL Critic. This model achieves clear performance gains over the standard video-level SFT baseline, providing direct evidence that combining MIL-derived fine-grained supervision with the MLLM's full multimodal and language reasoning priors effectively unlocks stronger highlight detection capabilities.

Reviewer_iyYfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper replaces curated unlabeled video with generated video when adapting a multimodal model to weakly supervised highlight detection. The critic is a frozen Qwen3-VL-2B encoder sampled at 1 fps with attention and MIL heads, so the trained parameters are heads rather than the MLLM; a 0.6B Director writes scripts for Wan2.1, is tuned by GRPO on rewards read from the critic, and the critic trains on real bags plus one-second synthetic clips with an added fake class. TVSum reaches 0.802 top-5 mAP against 0.768 for the authors' MIL baseline and 0.773 for Mean Teacher with 25 curated videos per topic, with 0.670 on YouTube Highlights and 0.605 on MoSu.

Critical strengths
  1. The ablation separates synthetic data from director adaptation, which is the control the central claim needs. At 100 clips, a frozen director gives 0.778, the bag-only reward 0.782, and the full reward 0.800, with per-topic values in Table 11, so the effect of GRPO updating can be recomputed rather than taken on trust. Most work in this area reports only the final number.

  2. The system is specified at a scale others can rebuild: Qwen3-0.6B director, Wan2.1-1.3B at 16 frames and 480p, GRPO group size 16, 100 steps per phase, prompt templates and generated script examples. For a paper whose contribution is a training loop, this level of disclosure matters.

Critical weaknesses
  1. The generated data supplies neither segment-level nor independent supervision. Each synthetic clip is one second of 16 frames, features are sampled at 1 fps, so TkT_k = 1. The instance loss LinsL_{ins} in Section 3.4 sums over a per-segment label ck,tc_{k,t} that is never defined; only a per-clip ckc_k exists, and with one instance per clip the loss is clip-level BCE. Real videos still get only bag-level CE and the pairwise ranking term, so nothing trains the temporal head on where inside a video the highlight lies, and the "richer, fine-grained supervision" claim has no mechanism behind it. The reward is equally self-referential: rhlr_{hl} uses the critic's score while the critic is trained to reproduce cc on the same clips, so both sides move toward agreement, and Fig. 4 plots that agreement as if it were evidence of better clips. This decides the rating because the 0.022 gain over a frozen Director is the entire case for co-evolution, and self-consistency would produce the same number.

  2. The TVSum state-of-the-art claim does not survive Table 2. The RASL row lists 0.897, 0.657, 0.925, 0.583, 0.847, 0.830, 0.697, 0.833, 0.875, 0.890, which average 0.8034, not the printed 0.783. At 0.803 RASL is above the paper's 0.802. Either the per-topic cells or the mean is wrong and the reader cannot tell which, but as printed, Section 4.3 and the abstract's claim are unsupported. Other rows show the same pattern (KVS 0.395 vs 0.398, DSN 0.414 vs 0.424, GIF 0.434 vs 0.464), which suggests values were copied across papers without recomputation. This matters because the gain over the strongest prior method is the paper's main result; if the row is right, there is no gain on TVSum.

Questions for authors
  1. Table 2's RASL row does not average to its printed mean (0.8034 vs 0.783), and at 0.803 TVSum would no longer be state of the art. Which side is wrong, and can every row of Tables 2 and 3 be recomputed from its per-topic entries? If the per-topic values are correct, what supports the claim in Section 4.3?

  2. With one-second clips and 1 fps features, TkT_k = 1 for every synthetic sample, and ck,tc_{k,t} in LinsL_{ins} has no defined per-segment value. Is LinsL_{ins} in practice clip-level BCE? If so, which experiment shows that the Director provides segment-level rather than clip-level supervision, for example multi-second clips where only part of the clip is generated under the highlight condition?

  3. Both rgtfr_g^{tf} and rghlr_g^{hl} are read off the critic while the critic is trained on the Director's outputs. Please report real held-out mAP after each co-evolution round, not only the critic reward, and a run with many more Director steps. Does detection improve, plateau or degrade? This would separate better data from mutual reinforcement.

  4. How many rounds R were used, and how do 100 GRPO steps per phase relate to the 300 critic epochs in Table 10? Please also give λfλ_f, λiλ_i, the total number of generated clips, and GPU hours, next to the cost of collecting 25 unlabeled videos per topic.

  5. How is the supervised loss defined with a three-class head? Eq. 1 is binary CE on ŷ ∈ [0,1], while the critic outputs a softmax over non-highlight, highlight and fake. Is p(fake) supervised on real videos at all, and which quantity is ranked at inference, given that mAP uses instance scores?

  6. Please add MIL and Mean Teacher rows to Tables 3 and 4 and run the no-director variant on those datasets. On MoSu, Section 4.3 says CLIP features are used while A.3 describes Qwen3-VL features: which critic produced Table 4, and was the Director updated there?

  7. The SFT baseline is trained with target-domain videos as label 1 and out-of-domain videos as label 0, then asked at inference for the highlight probability of a segment inside a target-domain video. This teaches domain membership rather than where the highlight is. Can you report an SFT variant trained with same-domain negative segments or with MIL-derived segment labels? Note also that SFT tunes LoRA while the MIL baseline trains only heads, and the two see different inputs (16 frames versus 1 fps features).

  8. How much per-topic human effort do the Director prompts require, given hand-written subjects, shot types, style rules and five in-context examples for each of ten TVSum topics? How does that compare with the 25 curated unlabeled videos per topic the method is meant to replace?

6 comments
AuthorsOpenReview ↗

Part 1 of the author response (critical weakness 1)

We sincerely thank the reviewer for their time and meticulous review. We appreciate the questions raised, which help clarify the core methodology and strengthen the manuscript's arguments and contributions. While misunderstandings exist in the critical weakness section, clarifying them improves the overall presentation of this paper.

Our response in this discussion phase has two parts. The first part addresses the key misunderstandings and questions, especially in the critical weakness section. The second part adds the extra experimental results as requested to strengthen the manuscript and provide a revised manuscript.

Part 1 of our response:

Critical Weakness 1: Here are some misunderstandings about how the synthetic videos are used in the proposed method. We clarify them one by one below.

Misunderstanding 1: "The generated data supplies neither segment-level nor independent supervision."

Response: No. The synthetic videos supply both instance-level and bag-level supervision. TkT_k is not 1. Due to space limits, the implementation details for this part were placed in Appendix 5.2 at Line 1148 in the initial submission. As mentioned, "Each synthetic bag contains between 10 and 50 generated clips", and TkT_k is an integer randomly sampled from 10 to 50. The revised manuscript has moved this explanation back to the main text.

Misunderstanding 2: "The instance loss ck,tc_{k,t} in Section 3.4 sums over a per-segment label that is never defined. With one instance per clip the loss is clip-level BCE."

Response: Here ck,tc_{k,t} implies the tt-th element of ckc_k (vector sequence), indicating whether this synthetic instance is a highlight or not. The revised manuscript adds this explicit explanation. Again, the number of instances per synthetic video is not 1, but 10 to 50 as clarified above.

Misunderstanding 3: "Real videos still get only bag-level CE and the pairwise ranking term, so nothing trains the temporal head on where inside a video the highlight lies, and the 'richer, fine-grained supervision' claim has no mechanism behind it."

Response: This point is clarified based on the above responses. As mentioned in Line 1130 in the initial submission, 20% of the instances in one synthetic video bag are positive highlights, which explicitly trains the temporal head on where inside a video the highlight lies. This justifies our claim that "richer, fine-grained supervision" is introduced through synthetic videos.

Misunderstanding 4: "The reward is equally self-referential, rhlr^{hl} uses the critic's score while the critic is trained to reproduce on the same clips, so both sides move toward agreement."

Response: The reviewer raises concerns about the reward design, especially rhlr^{hl}. We clarify the following two points:

  1. The critic uses synthetic data to train and then rates the synthetic videos; does that make sense, and is the critic legitimate to perform such ratings?

Firstly, a key fact has been neglected: the weakly-labeled real videos. Note that the critic still has access to the weakly-labeled real videos during generative co-evolution. Even without any synthetic supervision, the baseline critic already possesses a degree of highlight discriminative power, meaning its rating on highlightness is somewhat useful, which justifies the reward rhlr^{hl}. With quality synthetic supervision, the critic becomes even stronger—as validated by our experimental results—making the reward even more legitimate.

Secondly, when the critic is used for rating, it is frozen and has no access to the newly generated synthetic videos. When new synthetic videos are under evaluation, they do not appear in the critic's training set.

  1. Further justification on the reward design rhlr^{hl}: Without rhlr^{hl}, the reward shares the spirit of the adversarial loss in semi-supervised Generative Adversarial Networks (GANs) (Dai et al., 2017), which is well-established. As noted in Line 276 in the initial submission, we propose rhlr^{hl} to push the director to go beyond simple realism for WSVHD. The contribution of rhlr^{hl} is experimentally supported in Table 1.

Question: "This decides the rating because the 0.022 gain over a frozen Director is the entire case for co-evolution, and self-consistency would produce the same number."

Response: Why does the reviewer conclude that self-consistency would produce the same number? In our experiments, the self-consistency variant (MLLM + MIL + SSL) underperforms the proposed approach by a clear margin.

AuthorsOpenReview ↗

Part 1 of the author response (critical weakness 2 and some questions)

Critical Weakness 2:

We really appreciate the reviewer’s careful review. The results of RASL are taken directly from the official publication (IEEE / arXiv). We do not have the right to modify or recompute reported results from previously published studies in our manuscript. However, the question raised is valid, though there is no open-source code for independent validation. In our view, the 0.783 result is more plausible because it appears in four distinct tables in the original paper (Table II, Table III, Table IV, and Table V), whereas topic-level results appear only once. Our correspondence with the authors of RASL indicates that 0.783 is the intended mean, and our claim on state-of-the-art performance on TVSum still holds. For KVS/DSN/GIF, the 0.424/0.464 results match those of the original publications (https://vcg.ece.ucr.edu/sites/default/files/2019-02/iccv_rameswar.pdf, https://arxiv.org/pdf/1605.04850). In addition, our approach outperforms KVS/DSN/GIF with a clear margin. We have added a note in the revised manuscript.

Regardless of the cause, typos in a referenced paper do not diminish our core contribution. In fact, as noted in responses to other reviewers, we did not try to heavily tune topic-specific hyperparameters solely in pursuit of a high score on a small 50-video dataset. In contrast to non-MLLM approaches like RASL, our work provides an in-depth study of MLLM adaptation for WSVHD and pursues a paradigm shift with generative co-evolution. Larger benchmarks such as YouTube Highlights and MoSu contain far more data, and our results consistently demonstrate state-of-the-art performance overall for WSVHD.

Response to questions for clarification:

Question: Table 2's RASL row does not average to its printed mean (0.8034 vs 0.783)... Which side is wrong?

Response: As mentioned above, we believe the 0.783 result is more plausible based on correspondence with the RASL authors.

Question: Is Lins\mathcal{L}_{\text{ins}} in practice clip-level BCE?

Response: No. This is explained above in detail (Tk∈[10,50]T_k \in [10, 50] instances per bag).

Question: How many rounds RR were used, and how do 100 GRPO steps per phase relate to the 300 critic epochs in Table 10?

Response: We simply use R=1R=1. As mentioned in Line 263, we alternate optimization between the Director and the Critic. The number of GRPO steps is not tied to how long the critic is trained. Once GRPO completes and synthetic data is sampled, the critic trains on a dataset combining weakly-labeled real videos and synthetic videos. The training epochs of the critic are independent of the GRPO steps.

Question: "How is the supervised loss defined with a three-class head? Eq. 1 is binary CE on y^∈[0,1]\hat{y} \in [0,1], while the critic outputs a softmax over non-highlight, highlight and fake. Is p(fake)p(\text{fake}) supervised on real videos at all, and which quantity is ranked at inference, given that mAP uses instance scores?"

Response: Eq. 1 is for the baseline. For the critic training in the generative co-evolution approach, the supervised loss is computed for the weakly-labeled real videos. The labels for these real videos are non-highlight/highlight. Thus, in the three-class cross-entropy loss, p(fake)p(\text{fake}) is multiplied by 0 for real videos. However, the logits for the fake class do participate in the softmax operation and influence the three-class probabilities. During inference, the bag-level predictions are not used, and instance scores are highlight predictions.

In fact, this true-false-fake formulation inherits the standard training design of semi-supervised Generative Adversarial Networks (GANs) in image classification, and "A Survey on Deep Semi-supervised Learning" is a great reference.

AuthorsOpenReview ↗

Part 1 of the author response (continuing answering questions)

Question: On MoSu, Section 4.3 says CLIP features are used while A.3 describes Qwen3-VL features: which critic produced Table 4?

Response: We emphasized in Line 339 in the initial submission that "To ensure our framework remains agnostic to specific visual/audio encoders, we bypass raw video processing and directly utilize the pre-extracted features provided by the authors." Because MoSu is large-scale, processing raw videos is time-consuming, so we used the official CLIP features for Table 4. Director updates and video generation follow the exact same pipeline as other datasets (see Figure 12 in the initial submission for the generated videos for MoSu). To extract features for synthetic videos on MoSu, we apply the official CLIP feature extractor, which we verified against the official benchmark features by checking feature similarity on training videos.

Question: "How much per-topic human effort do the Director prompts require, given hand-written subjects, shot types, style rules and five in-context examples for each of ten TVSum topics? How does that compare with the 25 curated unlabeled videos per topic the method is meant to replace?"

Response: To prepare the Director prompts, we first ask Doubao-Seed-2.1-pro to generate a draft version. Then we manually check and polish the prompts, which takes 10 minutes per topic. To curate unlabeled videos, we search for videos on YouTube and watch each video in full, which costs 5–15 minutes per video. Note that curating high-quality videos requires careful watching and quality checking. As mentioned in Line 245 in the initial submission, "presumed negative videos may inadvertently contain positive instances, introducing severe label noise that hinders training." We watch the videos carefully to make sure the collected videos are topic-relevant, and when these videos are used as negative videos, we try to ensure they contain very few positive instances regarding the target topic.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the author response

We sincerely thank all reviewers for their constructive feedback and insightful suggestions. We are grateful that the reviewers recognized our core contributions, which we summarize as:

  1. An in-depth study of MLLM adaptation for weakly-supervised video highlight detection (WSVHD).

  2. Demonstrating co-evolution: Showing how an MLLM can co-evolve with a generative model rather than relying solely on isolated post-training.

  3. Unlocking synthetic video utility: Demonstrating how synthetic videos benefit WSVHD and how to empower a video generator to continuously improve its highlight synthesis quality within this framework.

In response to your feedback, we have clarified potential misunderstandings regarding our architecture, added requested implementation details, and conducted additional experiments. We believe these additions and clarifications have significantly strengthened our paper and claims.

We welcome any further feedback and would be delighted to know if our revisions have successfully addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.