acceptodds
Withdrawn from ICLR 2027

ViMix-14M: A Curated Video-Text Dataset with High-Quality Captions and Crawl-Free Access

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-video generation has surged in interest since Sora, yet open-source models still face a data bottleneck: there is no large, high-quality, easily obtainable video–text corpus. Existing public datasets typically require manual YouTube crawling, which yields low usable volume due to link rot and access limits, and raises licensing uncertainty. This work addresses this challenge by introducing ViMix-14M, a curated multi-source video–text dataset of around 14 million pairs that enables end-user crawl-free, download-ready access and provides long-form, high-quality captions tightly aligned to video. ViMix-14M is built by drawing from diverse openly licensed video sources, followed by unified de-duplication and quality filtering, and a multi-granularity, ground-truth-guided re-captioning pipeline that refines descriptions to better match actions, scenes, and temporal structure. We evaluate the dataset by multimodal retrieval, text-to-video generation, and video question answering tasks, observing consistent improvements over counterpart datasets. We hope this work can help remove the key barrier to training and fine-tuning open-source video foundation models, and provide insights into building high-quality and generalizable video-text datasets.

Primary area:
datasets and benchmarks
Keywords:
video-text dataset, text-to-video generation, vision-language models, video understanding, data curation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_X2pzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ViMix-14M assembles 13.7 million clips from seven existing sources and adds three caption granularities and per-video quality scores. Captions are generated using Qwen2.5-VL with source annotations where available. The paper evaluates caption quality through human preference, retrieval, generation, and question answering, and proposes crawl-free access through existing source distributions.

Critical strengths
  1. Combining multiple caption granularities with standardized quality metadata could make the resource useful across different applications. Users can select supervision detail and filtering criteria without repeating the annotation pipeline.

  2. The paper provides meaningful evidence that re-captioning improves descriptions. The 10,000-video human study reports an 88.7% preference for the new captions, and retrieval comparisons on the same videos show substantial improvements across several sources. These results support increased caption informativeness beyond selected qualitative examples.

Critical weaknesses
  1. The central accessibility contribution cannot be independently verified from the supplied manuscript. It identifies no concrete corpus release or source-to-segment download manifest; generic references to source datasets and hosting platforms do not establish reproducible crawl-free access to the assembled resource.

  2. The training value of the multi-source corpus remains insufficiently demonstrated. Fine-tuning uses only 15K InternVid clips, leaving the benefit of the mixture untested. Relative to pretrained FastVideo, Table 3 shows modest alignment gains alongside lower aesthetic and imaging scores, which limits the claim of broadly superior training supervision.

Questions for authors
  1. Can reviewers inspect the generated annotations and a versioned source-to-segment download manifest that reproduces the claimed crawl-free access?
  2. Under matched video counts, training budgets, and evaluation prompts, does a representative multi-source mixture outperform recaptioned InternVid alone?
Reviewer_kGFfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ViMix-14M is a dataset paper. The authors pool 13.7M clips (about 22.8K hours, mean length 6s) from seven openly licensed and stably hosted sources, re-caption everything with Qwen2.5-VL-7B at three prompt granularities (short, middle, long), and attach per-video VBench scores. The videos are not re-hosted, so the contribution is really the unified index plus the new annotations.

Critical strengths

The problem choice is the strongest part. Panda-70M, WebVid-10M, HowTo100M and HD-VILA-100M are all either withdrawn or effectively unusable because they are YouTube links, and a single index over stably hosted, openly licensed sources with an explicit per-source license table is something people would genuinely use.

Second, the metadata design is well thought out. One schema carrying duration, spatial size, five VBench scores and three caption levels, with metadata shipped instead of a hard filter, lets users impose their own quality trade-offs. The Fig. 7 correlation and the 7B vs 72B ablation then show that none of the expensive choices are needed.

Critical weaknesses

The dataset is not released, and access is the whole point of the paper. There is no URL, no dataset card, no join script, and the schema contains nothing that maps a segment to an actual source file. Appendix A says no metadata or caption data is re-hosted or redistributed either, which leaves the deliverable unclear. A resource paper whose artifact does not exist cannot be validated by the community.

Second, every headline comparison is confounded. Retrieval, CLAIR and the 88.7% human preference win all compare roughly 110 words against a 14-word label, and no length-matched baseline is given. The text-only VQA has GPT-5 writing the questions, producing the answers and judging them.

I would raise my rating, if the dataset would be released publicly and fully, including video data and metadata.

Questions for authors

Will the caption index be released, where, and under which license? Given the merged corpus inherits the most restrictive source license (a Qualcomm research license for ssv2, cc by nc for InternVid) and is therefore non-commercial, is that acceptable to the intended industrial audience?

Reviewer_Lsm5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ViMix-14M addresses limited access to richly annotated video data for open video models. Existing datasets often require unreliable crawling and provide captions that omit visual attributes, spatial relations, and temporal details. The dataset unifies 13.7 million videos from seven sources through existing download channels. A ground-truth-guided Qwen2.5-VL-7B pipeline generates short, medium, and long captions with progressively richer visual descriptions. Per-video quality metadata supports task-specific filtering rather than a fixed selection policy. Retrieval and generation evaluations demonstrate improved caption utility, including an InternVid Recall@1 increase from 76.5% to 90.2%, while full-corpus foundation-model pretraining remains unevaluated.

Critical strengths
  1. Strong practical relevance. The dataset addresses unreliable video acquisition and descriptions that omit important visual details, both of which constrain open video modeling.
  2. Clear pipeline correspondence. Annotation-guided recaptioning improves semantic specificity, multiple description lengths accommodate different tasks, and quality metadata supports flexible filtering.
Critical weaknesses
  1. Limited methodological novelty. Aggregation, pretrained-model recaptioning, and established quality metrics are conventional techniques, making the contribution primarily a dataset resource.
  2. Insufficient attribution of diversity benefits. InternVid supplies approximately 77% of the corpus, yet source-mixture ablations are absent, and richer descriptions alone do not establish factual or temporal accuracy.
  3. Incomplete validation at scale. Small retrieval pools and fine-tuning on 15,000 videos demonstrate caption utility but do not establish full-corpus training gains, scaling behavior, or the value of the complete mixture.
Questions for authors

Please see the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.