acceptodds
Under review as a conference paper at ICLR 2027

Advancing Video-text Pretraining with Multi-view Captions

Abstract

Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains underexplored. Existing web-scale datasets often provide only a single sparse caption per video that fails to capture rich spatiotemporal semantics, while directly using captioning models can generate noisy descriptions. We propose a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage. Starting from 10 million videos, our approach generates multi-view captions (MVC) through complementary summary and detailed captions, reasoning-based refinement, and semantic positive caption generation. To effectively exploit supervision at different granularities, we further introduce a granularity-aware text representation with separate CLS tokens for summary and detailed views. We pretrain video-text models using the resulting supervision corpus and evaluate them across standard, fine-grained and detailed text-to-video retrieval benchmarks. Our approach consistently improves both zero-shot and fine-tuned performance while using smaller pretraining corpora than existing methods, demonstrating the importance of rich and complementary textual supervision for video-text pretraining.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Video-Language Pretraining, Video-text Alignment, Video

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hGirOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the limitations of sparse and noisy captions in video-text pretraining. It proposes multi-view caption generation with reasoning-based refinement and a granularity-aware text representation using dual CLS tokens. Experiments demonstrate significant improvements in zero-shot and fine-tuned video-text retrieval across multiple benchmarks.

Critical strengths
  1. This paper proposes an effective multi-view captioning framework with complementary supervision and granularity-aware text representations.
  2. This paper achieves strong retrieval performance, particularly on fine-grained benchmarks, with fewer pretraining videos.
Critical weaknesses
  1. Prior works such as HACA[1] have already investigated hallucination correction using LLMs. Could the authors further discuss this work and clarify the novelty of their proposed supervision generation framework?
  2. In Tables 1–3, the compared methods use different pretraining datasets. For example, UMT uses a mixture of video-text and image-text data, while MVC is trained on InternVid-10M-FLT with only video-text data. The authors should explicitly clarify these differences in data composition and pretraining configurations to ensure a fair interpretation of the performance comparisons.
  3. The proposed method assigns equal weights to summary and detailed captions during both training and inference. However, Table 5 shows substantial performance differences when using these two types of captions independently. What motivates the equal weighting strategy, and could the authors provide an ablation study comparing different loss weights and similarity fusion weights to justify this design?

[1]Zhao, Lingjun, et al. "Can Hallucination Correction Improve Video-Language Alignment?." Findings of the Association for Computational Linguistics: ACL 2025. 2025.

Questions for authors

refer to weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer hGir

We thank the reviewer for the careful reading and constructive suggestions. Below, we clarify our contribution relative to HACA, detail the differences in pretraining data and configurations, and report new experiments on training and inference weights.

W1. Relation to HACA and novelty of the supervision generation framework

We agree that hallucination correction has been studied before, and we will explicitly discuss HACA [1] in the related work. HACA fine-tunes a video-language model to detect and correct hallucinated descriptions, improving that model's own alignment. Our goal is different: we construct pretraining supervision. Our framework:

  • generates complementary summary and detailed captions from multiple MLLMs;
  • refines the detailed captions through visual verification;
  • generates semantic positives.

The resulting corpus then supervises a separate video-text model. Our refinement is training-free: it uses an off-the-shelf reasoning MLLM and is applied once to 10M captions. Refinement is one component of the framework. Our novelty lies in constructing and jointly using complementary supervision views (Tables 4–6), not in hallucination correction itself. The two approaches address different stages of video-language learning and could be combined. Finally, our response to Reviewer YZJy shows on DREAM-1K and CaReBench human annotations that our refinement selectively removes unsupported content.

W2. Differences in pretraining data and configurations

We agree that equal sample counts do not imply matched conditions. For example:

  • UMT-B 5M combines WebVid-2M videos with 3M image-text pairs.
  • MVC-B 5M uses 5M InternVid-10M-FLT videos only.
  • Our video encoder is initialized from SMILE (Thoker et. al.) checkpoints, pretrained on Kinetics-400/700.

In the revision, we will add a table in the appendix listing, for each compared method:

  • video and image data sources and sizes;
  • caption source;
  • backbone and initialization;
  • number of frames.

We will also state in the results that Tables 1–3 compare complete pretrained systems and do not isolate the effect of caption supervision except for comparison with Baseline-B.

W3. Motivation for equal weights, and ablation of loss and fusion weights

Motivation. Table 5 shows that summary and detailed captions differ in quality when used alone, but that they are complementary: combining them outperforms either one. With dual-CLS, each view is aligned through its own text representation, so a weaker view does not degrade the other. We therefore give both views equal weight as an unbiased default that requires no tuning.

Ablation. We trained two additional ViT-B models on 1M videos with summary-to-detailed loss weights of 2/3:1/3 and 1/3:2/3. We also varied the inference fusion weights on the equal-weight checkpoint. We report zero-shot text-to-video R@1.

Table A. Training and inference use the same weights

Weights (S:D)MSRVTTMSVDANetDiDeMoLSMDCCARE-TCARE-SDREAM-DDREAM-ES2S-SS2S-W
1/2:1/2 (default)33.540.053.655.317.356.787.092.027.969.491.4
2/3:1/334.638.950.851.618.252.786.691.522.668.890.1
1/3:2/334.137.350.150.817.551.787.490.721.268.790.5

Equal weighting performs best on eight of eleven benchmarks. The largest gains over both unequal settings are on DiDeMo, CARE-T, and DREAM-E. Giving more weight to the summary view performs best on MSRVTT and LSMDC, while giving more weight to the detailed view performs best on CARE-S. These results compare configurations that use the same weights for training and inference.

Table B. Inference fusion weights on the equal-weight checkpoint

Fusion weights (S:D)MSRVTTMSVDANetDiDeMoLSMDCCARE-TCARE-SDREAM-DDREAM-ES2S-SS2S-W
1/2:1/2 (default)33.540.053.655.317.356.787.092.027.969.491.4
2/3:1/333.540.253.655.117.256.387.292.127.469.691.3
1/3:2/333.639.553.755.317.157.187.091.328.169.191.3

Performance is stable across fusion weights. With the checkpoint fixed, changing the fusion weights shifts R@1 by at most 0.7 points from equal averaging on any benchmark. The best weights vary across benchmarks, so we use equal averaging by default. We discuss HACA in the related works, and W2 and W3 in the Appendix A.8 and A2 respectively.

We hope these clarifications and additional experiments address the reviewer's concerns. We would be grateful if the reviewer would reconsider their score in light of this evidence, and we are happy to answer any further questions.

Dear Reviewer, thank you again for your review. We have provided additional experiments and revised the manuscript to address your concerns. As the discussion period draws to a close, we would appreciate your feedback on whether these additions address your main concerns and which issues, if any, remain unresolved. We would be happy to clarify further.

Reviewer_YZJyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper utilizes a MLLMs to generate both summary and detailed captions. The detailed captions are subsequently refined, and semantically positive captions are constructed to establish multi-view video-text supervision. The proposed method employs a dual-CLS representation, designed to distinguish between different levels of descriptive granularity, and its effectiveness is validated across standard, fine-grained, and detailed description retrieval tasks.

Critical strengths
  1. Valuable research problem. Focusing on the quality of text-based supervision offers a meaningful direction for improving video-text pre-training.
  2. Sound methodology. The overall design is coherent, and the effectiveness of the approach is supported by a range of experiments.
Critical weaknesses

1.There is a lack of direct evidence demonstrating that the caption refinement process actually improves quality or enhances semantic coverage. 2. It is difficult to disentangle the gains derived from the proposed design versus those resulting from additional supervision or inference-time processing. 3. The advantage in terms of video sample scale is not sufficient to draw conclusions regarding overall resource efficiency.

Questions for authors
  1. Could you provide an evaluation of caption quality based directly on the video content? Specifically, does the caption refinement process reduce hallucinations, and do the semantically positive captions effectively incorporate additional visual information?

  2. When the total number of captions and the text-training budget are held constant, does the multi-view supervision approach still offer advantages over standard text rewriting? Furthermore, what is the relative contribution of the dual-CLS granularity modeling versus the score fusion performed during inference?

  3. Does the term "data efficiency" in the paper refer exclusively to the number of videos used for pre-training? When accounting for the computational costs of caption generation and additional text encoding, how does the total resource investment compare to the baseline?

If these questions are addressed satisfactorily, I will consider increasing the scores accordingly.

4 comments
AuthorsOpenReview ↗

Response to Q1 : Reviewer YZJy

Video-grounded caption quality. We thank the reviewer for this suggestion. We agree that the most direct way to assess caption quality is a human check against the video content, but this does not scale to the millions of captions in our corpus, and InternVid does not provide human descriptions to compare against. We therefore use the following framework: we run our caption pipeline on two benchmarks whose videos come with detailed human annotations, and use these annotations as a video-grounded reference.

  • DREAM-1K [1]: 1,000 videos with dense human descriptions and annotated event lists;
  • CaReBench [2]: 1,000 videos with separate human annotations of spatial content (objects, attributes, scene) and temporal content (actions, events), each with a checklist.

For each video we generate D1D_1 (raw detailed from Qwen ), D1∗D_1^* (refined) and PP (semantic positives) exactly as for pretraining. We then compare them against the human annotations, which serve as video-grounded ground truth.

Protocol. We adapt the event-based evaluation "AutoDQ" from [1] ) to use the human-annotated event and object lists as references. In particular, An LLM judge (Claude Opus 5.5 ) is as used to:

  1. extracts atomic claims from each caption;
  2. label each claim as supported if the ground truth annotation confirms it, , contradicted if if the ground truth annotation conflicts with it or not mentioned if the ground truth annotation does not address it.;
  3. measures which annotated events/objects each caption covers.

Supported/contradicted/not mentioned measure how accurate a caption is; coverage measures how much of the annotated video content it describes.

(a) DREAM-1K (6,298 annotated events)

CaptionSupported (%)Contradicted (%)Not mentioned (%)Event coverage (%)
D1D_1 (raw)57.64.038.447.3
D1∗D_1^* (refined)63.33.433.343.1
D1∗+PD_1^*+P55.24.240.751.2
  • Refinement: raises the share of supported events from 57.6% to 63.3% and lowers contradictions. It is conservative, so it also drops some correct details, which lowers coverage (47.3% → 43.1%).
  • Positives:
    • They raise event coverage by +8.1 points, above even the raw captions.
    • 8.1% of annotated events, in 38% of videos, are described only by PP. These are mainly subject actions (33%), object interactions (28%) and fine-grained motion (27%).

(b) CaReBench, temporal (event claims; 6,942 annotated events)

CaptionSupported (%)Contradicted (%)Not mentioned (%)Event coverage (%)
D1D_1 (raw)66.37.326.433.5
D1∗D_1^* (refined)76.05.019.028.0
D1∗+PD_1^*+P61.46.931.738.9
  • Refinement: has its strongest effect on temporal content. Supported claims rise from 66.3% to 76.0%, and contradictions fall from 7.3% to 5.0%.
  • Positives:
    • They raise event coverage by +10.9 points.
    • 11.0% of annotated events, in 50% of videos, are described only by PP. These are mainly object interactions (44%) and fine-grained motion (39%).

(c) CaReBench, spatial (static claims; 10,032 annotated objects/attributes)

CaptionSupported (%)Contradicted (%)Not mentioned (%)Object coverage (%)
D1D_1 (raw)67.74.827.547.0
D1∗D_1^* (refined)69.54.526.042.8
D1∗+PD_1^*+P67.15.227.746.3
  • Refinement: gives a smaller precision gain for static content than for temporal content (67.7% → 69.5% supported).
  • Positives:
    • They add +3.5 points of object coverage.
    • PP adds little static content, by design: it describes visible actions and events rather than scene inventory.

(d) What does refinement remove?

Refinement deletes part of each raw caption. We check whether the deleted parts were correct or not:

% of claims that match the human annotationDeleted by refinementKept by refinement
DREAM-1K32.3%72.2%
CaReBench, temporal43.0%83.4%
CaReBench, spatial50.2%77.2%

Content that refinement deletes is much less often correct than content it keeps (32–50% vs. 72–83%). In other words, refinement mainly removes unsupported, likely hallucinated, details.

Summary

Across both benchmarks, refinement selectively removes content unsupported by human annotations, most strongly for actions and events. Semantic positives expand coverage of dynamic content (+8.1 and +10.9 points of event coverage), especially object interactions and fine-grained motion. These match their intended roles in our pipeline and the gains in Table 5. This discussion is now present in the Appendix A.5.

References [1] J. Wang, et al. Tarsier: Recipes for Training and Evaluating Large Video Description Models.
[2] Y. Xu, et al. CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval..

AuthorsOpenReview ↗

Response to Q3 : Reviewer YZJy

Q3: Data efficiency and total resource cost. We thank the reviewer for this question. In our paper, "data efficiency" refers to the number of pretraining videos, and we will make this explicit in the revision.

(i) Pretraining cost. Pretraining is identical to the baseline (same video encoder, videos, batch size, epochs and objectives), except for one extra text-encoder pass per video for the second caption (≤128 tokens). This overhead is small, as the video encoder dominates computation. At this near-identical cost, MVC-B outperforms Baseline-B (same 5M videos, original captions) by a significant margin (Tables 1,2,3 etc).

(ii) Comparison with existing captioning pipelines. We compare the number of model calls per video and any additional steps each pipeline requires:

PipelineModel calls per videoAdditional one-time stepsOutput per clip
InternVid [1]1 image-captioner call (Tag2Text) per sampled frame + 1 LLM call to merge them + 1 BLIP-2 call on the middle frameNone1 short caption
Panda-70M [2]8 teacher captioning models + 1 caption-selection pass = 9Human annotation of 100K videos; fine-tuning the selector model ; training a student captioner1 short caption
Ours (MVC)4 captions (2 MLLMs × summary/detailed) + 2 refinements + 1 call for all positives = 7None: all models are used off the shelf, with no human annotation or training6 complementary texts

Our pipeline needs fewer calls per video than Panda-70M, and none of its human annotation or model training. InternVid uses smaller image-level captioners, so its cost per video is lower. Most importantly, both prior pipelines end with a single short caption per clip (e.g 10 - 15 words on average for InternVid-10M-FLT). Ours produces six complementary texts per video in addition to the original caption: two summaries, two refined detailed captions (~60 words each) and two semantic positives. That is substantially richer supervision for the same video.

(iii) Inference cost. Dual-CLS averaging requires two text forward passes per query, which is negligible for short queries. As shown in our response to Reviewer 6c23 (Q2), a single head already outperforms the single-CLS baseline at identical inference cost.

Summary. Captioning is a one-time, offline cost. The captions are generated once and reused. Multi-caption supervision is well established for images, where datasets routinely pair each image with several captions, but large-scale video-text corpora still provide a single caption per clip. To our knowledge, our corpus is the first to close this gap at scale, and we will release it publicly to support future research on video-text pretraining. This discussion is now present in the Appendix A.6.

References

[1] Wang et al. InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation. ICLR 2024.

[2] Chen et al. Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers. CVPR 2024.

AuthorsOpenReview ↗

Response to Q2 : Reviewer YZJy

Q2: Multi-view supervision vs. standard text rewriting at a fixed caption budget, and dual-CLS vs. score fusion.

We thank the reviewer for this question. We address both parts with controlled experiments.

(i) Multi-view supervision vs. text rewriting. To compare at an identical caption budget, we build a text-rewriting baseline. For every video, a text-only LLM (Qwen3-30B) rewrites a single source caption (the raw Qwen3-VL-30B detailed caption D1D_1) into six captions that mirror our supervision exactly:

  • two summary captions;
  • two detailed captions;
  • two event-level positive captions.

Each rewritten caption is capped at the length of its counterpart in our corpus. The rewriter has no access to the video and is instructed to use only information stated in the source, without adding or correcting anything. The rewrites therefore match our captions in number, roles and length budget, but add no new visual information. We rewrite from the detailed caption D1D_1 because rewriting can shorten a caption but cannot faithfully expand one: starting from a short caption (e.g., the original caption OO) would require inventing details to produce detailed captions and positives. This makes the rewriting baseline strong, since its source is already a detailed, video-grounded MLLM caption.

Due to the limited rebuttal period, we run this comparison on a 512K-video subset of our 1M ablation set. Both models are trained on the same 512K videos with the same sampling scheme, schedule and single-CLS setup as our Table 5 ablations, and differ only in the captions. This keeps the number of videos, the number of captions and the text-training budget identical.

Supervision (6 captions per video, 512K videos)MSVDANetDiDeMoLSMDCCARE-TS2S-SAvg
Text rewriting of D1D_130.039.137.913.840.556.536.3
Ours: multi-view (S1,S2,D1∗,D2∗,P1,P2S_1, S_2, D_1^*, D_2^*, P_1, P_2)35.746.339.114.443.563.340.4

Our multi-view supervision outperforms text rewriting on all six benchmarks, by +4.1 R@1 on average, with the largest gains on ActivityNet (+7.2), Shot2Story (+6.8) and MSVD (+5.7). Rewriting changes the wording and structure of captions but cannot add information beyond its single source caption or correct it. Our framework adds new visual information through three sources: multiple MLLMs that each observe the video, visual verification during refinement, and video-grounded semantic positives. At an identical caption budget, the gains therefore come from this additional visual content, not from the number or diversity of captions. We are also trying to run this experiment at 1M scale, provided the compute becomes available on time.

(ii) Dual-CLS granularity modeling vs. score fusion at inference. We evaluate the dual-CLS model using each text representation alone, so no score fusion is used (see also our response to Reviewer 6c23, Q2):

Text representationMSVDANetDiDeMoLSMDCCARE-TCARE-SS2S-SS2S-WAvg
Single-CLS38.850.153.018.254.187.067.990.957.5
Dual-CLS, CLSs_s only39.850.853.517.955.886.668.891.158.0
Dual-CLS, CLSd_d only38.251.454.017.456.587.067.591.557.9
Dual-CLS, score fusion (default)39.451.354.318.756.487.068.591.358.4

Most of the improvement comes from dual-CLS training, not from score fusion. A single head already improves over the single-CLS baseline (58.0 / 57.9 vs. 57.5 average) at identical inference cost. Score fusion adds a further +0.4 on average over the best single head, and it matches or improves on the single-CLS baseline on all eight benchmarks.

We are adding both analyses to the paper and in the Appendix A.3.. We hope these experiments address the reviewer's concerns and would be grateful if the reviewer would reconsider their score in light of these results. We are happy to answer any further questions.

AuthorsOpenReview ↗

Multi-view supervision vs. text rewriting at 1M Scale

Multi-view supervision vs. text rewriting at 1M Scale

Supervision (6 captions per video, 1M videos)MSVDANetDiDeMoLSMDCCARE-TS2S-SAvg
Text rewriting of D1D_134.444.547.815.848.061.542.0
Ours: multi-view (S1,S2,D1∗,D2∗,P1,P2S_1, S_2, D_1^*, D_2^*, P_1, P_2)38.550.152.416.754.167.446.5

Similar to the 512K sample text, our multi-view supervision again outperforms text rewriting on all six benchmarks, by +4.5 R@1 on average, with the largest gains on ActivityNet (+5.6), Shot2Story (+5.9), and CARE-T (+6.1). This further validates the effectiveness of our proposed multiple-caption supervision pipeline compared to the standard text rewrite supervision. We will update these results in the manuscript accordingly.

Thank you again for your review. We have provided additional experiments and revised the manuscript to address your concerns. As the discussion period draws to a close, we would appreciate your feedback on whether these additions address your main concerns and would be grateful if you would reconsider the score in light of these results. We are happy to clarify further if required.

Reviewer_6c23OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first proposes a MLLM based supervision generation framework to produce improved captions through a three stage mechanism (use Qwen3-VL-30B and Tarsier2-7B to generate additional text, use Qwen3-VL-8B to correct captions and produce 2 positive text samples). Then their training pipeline uses separate CLS tokens for training with summary and detailed labels. During testing, they append two different CLS tokens to the original text query and average the similarity score from two predictions. The proposed method uses smaller pretraining data, and the zero-shot and fine-tuned retrieval results showed improved performance over the baseline and the existing methods across most benchmarks.

Critical strengths

The text generation pipeline proposed in the paper to generate the supervision signal introduces more information that improves the training process. The ablation study on Table 4 establishes the strength of this approach, since using the generated summary itself can improve the performance significantly. One of the key claims in this paper is that the proposed process improves text supervision, and the ablation result supports this claim.

Critical weaknesses

Most of the gains for the proposed method comes from introducing summaries from a framework that uses a newer MLLM (Table 4 shows large leap in performance for ANet, DiDeMo etc when using summary only). While the paper did compare some related works that used MLLMs to generate captions (for instance: UMT on Panda-5M, though it uses different videos and initialization), it is important to isolate the gains that simply came from using a newer MLLM over the gains that came from using this framework, since one of their key contributions lies in the effectiveness of the framework.

Questions for authors
  1. Can the authors utilize the generated Qwen 3 VL 30B outputs (unrefined S_1 + D_1 only) to train in the same 1M single-CLS setting and demonstrate how it performs against the results in table 5?
  2. Can the authors report how the dual-trained model performs with only CLS_s or CLS_d at test-time?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6c23

We thank the reviewer for their constructive suggestions. Both requested experiments are included in the revised paper, with changes marked in blue.

W1 / Q1. Isolating the effect of the newer MLLM

As requested, we trained the 1M-video single-CLS ablation model using only the unrefined Qwen3-VL-30B outputs (S1+D1S_1 + D_1), with the same videos, training schedule, and masking ratio as in Table 5. We report zero-shot text-to-video R@1 below.

SupervisionMSVDANetDiDeMoLSMDCCARE-TS2S-SAvg
Original web captions (OO)35.425.929.311.827.142.228.6
Qwen3-VL-30B only, unrefined (S1+D1S_1 + D_1)35.846.939.214.143.563.340.4
Summary only, both MLLMs (S1+S2S_1 + S_2)37.046.246.015.148.063.242.6
Multi-MLLM, unrefined (S1,S2,D1,D2S_1, S_2, D_1, D_2)36.648.548.615.950.165.944.3
+ Refinement (D∗D^*)37.549.649.716.151.866.745.2
+ Semantic positives (PP), single-CLS38.550.152.416.754.167.446.5
Ours, full (dual-CLS)39.451.354.318.756.468.548.1
  • Gains over raw Qwen supervision. The raw Qwen captions improve average R@1 by 11.8 points over the original web captions. Our framework improves over the raw Qwen model on all six benchmarks, by 6.1 points on average in the same single-CLS setting and 7.7 points with dual-CLS. The largest gains are on DiDeMo (+13.2 and +15.1) and CARE-T (+10.6 and +12.9).
  • Contribution of complementary caption views. The summary-only model in Tables 4 and 5 uses summaries from both Qwen3-VL and Tarsier2. It outperforms the raw Qwen-only model on four of six benchmarks. Combining unrefined summaries and detailed captions from both MLLMs improves on the raw Qwen-only model on all six benchmarks.
  • Contribution of subsequent components. Refinement, semantic positives, and dual-CLS each provide a further improvement across all six benchmarks.

These results isolate the improvements that come from our full framework, i.e., complementary views across MLLMs, refinement, and semantic positives, rather than just the newer MLLM captioner itself. We added the raw Qwen results as row 3 of Table 5 and discuss it in the "Contribution of Supervision Components" paragraph.

W2/ Q2. Dual-CLS with a single head at inference

We evaluated the dual-CLS model from Table 6 using each text representation separately and with the default score averaging. The single-CLS and dual-CLS models were trained with the same O+S+D∗+PO + S + D^* + P supervision. We report zero-shot text-to-video R@1 below.

Text representationMSVDANetDiDeMoLSMDCCARE-TCARE-SS2S-SS2S-WAvg
Single-CLS38.850.153.018.254.187.067.990.957.5
Dual-CLS, summary head only (CLS$_s$)39.850.853.517.955.886.668.891.158.0
Dual-CLS, detailed head only (CLS$_d$)38.251.454.017.456.587.067.591.557.9
Dual-CLS, score averaging (default)39.451.354.318.756.487.068.591.358.4
  • Dual-CLS Training with single-head inference. Both heads improve over the single-CLS baseline on average (58.0 for CLS$_s$ and 57.9 for CLS$_d$, compared with 57.5). CLS$_s$ improves on six of eight benchmarks. These results show that the gains extend to settings that use one text encoding per query, as in the single-CLS baseline.
  • Performance by query granularity. CLS$_s$ scores higher on MSVD and Shot2Story single-shot, which use short, single-event queries. CLS$_d$ scores higher on ActivityNet, CARE-T, and Shot2Story whole-video, which use longer or more detailed queries.
  • Score averaging. Averaging the two scores gives the highest average R@1 (58.4), matches or improves on the single-CLS baseline on all eight benchmarks, and stays within 0.4 R@1 points of the better head on each benchmark. We therefore use score averaging by default.

These results show that dual-CLS training improves retrieval even when only one representation is used at inference. We added these results and discussion to Table 7 and the "Dual-CLS at Inference" paragraph in the revised paper.

We hope these experiments address the reviewer's concerns and would be grateful if the reviewer would reconsider their score in light of these results. We are happy to answer any further questions.

Thanks to the authors for their detailed experiments. My concerns regarding the effectiveness of the pipeline (beyond simply using a newer MLLM caption) have been addressed, and I will be updating my score accordingly.

Dear Reviewer, thank you again for your review. We appreciate the constructive feedback and suggestions. It has clearly improved the manuscript, especially in isolating the contributions. We also appreciate the score update. Thank you.

AuthorsOpenReview ↗

Gentle reminder for score update

Dear Reviewer,

Thank you again for participating in the discussion. We really appreciate it. As the discussion period draws to a close, we would like to gently remind you that your score update has not yet been reflected.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful review. We appreciate the reviewers recognizing the value of improving text supervision (6c23, YZJy), the sound methodology and broad evaluation (YZJy), and the strong retrieval performance, particularly on fine-grained benchmarks (hGir).

During the rebuttal, we added controlled pretraining experiments, caption-quality evaluations against human annotations, and inference ablations. Reviewer 6c23 has confirmed that their concerns about the pipeline's effectiveness beyond using a newer MLLM have been addressed and stated that they will update their score. Reviewer hGir has also updated their score.

(1) Contribution beyond a newer captioning model (6c23)

  • We trained the requested raw Qwen3-VL-30B baseline using only its summary and detailed captions, with the same 1M videos and single-CLS setup as Table 5.
  • MVC improves over this baseline on all six benchmarks. In the single-CLS setting, the gains include +13.2 R@1 on DiDeMo and +10.6 on CARE-T.
  • Combining captions from both MLLMs, refining detailed captions, and adding semantic positives each improves performance across all six benchmarks.

(2) Caption quality and additional visual information (YZJy)

  • We applied our caption pipeline to DREAM-1K and CaReBench and evaluated the outputs against their detailed human annotations.
  • Refinement increases the share of supported claims and reduces contradictions. On CaReBench temporal, supported claims rise from 66.3% to 76.0%, while contradictions fall from 7.3% to 5.0%.
  • Refinement also reduces coverage. Claims it retains are more often supported than claims it removes, with support rates of 72–83% versus 32–50%.
  • Semantic positives increase annotated event coverage by +8.1 points on DREAM-1K and +10.9 on CaReBench temporal. They cover events absent from the refined caption in 38% and 50% of videos, respectively. Adding positives also lowers the share of supported claims relative to refinement alone, showing a trade-off between coverage and precision.

(3) Multi-view supervision versus text rewriting (YZJy)

  • We built a text-only Qwen3-30B baseline that rewrites the raw detailed caption into six captions matching MVC's roles and length limits.
  • Both models use the same 512K / 1M videos, caption sampling, training schedule, and single-CLS setup.
  • MVC improves on all six benchmarks by more than +4.1 R@1 on average. The largest gains are on ActivityNet, Shot2Story single-shot, and CARE-T.

(4) Dual-CLS training and inference fusion (6c23, YZJy)

  • Using either head of the dual-trained model alone improves average R@1 over the single-CLS baseline, from 57.5 to 58.0 with the summary head and 57.9 with the detailed head.
  • These evaluations use one text encoding per query and show that dual-CLS training provides gains with single-head inference.
  • Score averaging further raises average R@1 to 58.4 and matches or improves on the single-CLS baseline across all eight benchmarks.

(5) Training and fusion weights (hGir)

  • We trained two additional 1M-video models with summary-to-detailed loss weights of 2/3:1/3 and 1/3:2/3.
  • Among configurations using the same weights for training and inference, equal weighting performs best on eight of eleven benchmarks.
  • On the fixed equal-weight checkpoint, changing the fusion weights shifts R@1 by at most 0.7 points from equal averaging. The best weights vary across benchmarks, supporting equal averaging as the default.

(6) Related work, comparison settings, and resource costs (hGir, YZJy)

  • We clarified the relationship to HACA. MVC uses off-the-shelf models to construct supervision for a separate retrieval model, combining complementary caption views, visual refinement, and semantic positives.
  • We clarified differences in image and video data, caption sources, encoder initialization, and frame counts. Baseline-B provides the comparison using the same backbone and 5M InternVid clips.
  • We clarified that "data efficiency" refers to the number of pretraining videos. Our response details the one-time caption-generation pipeline, the additional text encoding during training, and the two text encodings used by default at inference.

The revised manuscript includes the new experiments and expanded discussions, with changes marked in blue.

As the discussion period draws to a close, we would be grateful if the reviewers and ACs could let us know whether these additions address their concerns and which issues, if any, remain unresolved. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.