- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VidaForge is an infrastructure for constructing and comparing video pretraining data recipes through ingestion, segmentation, selection, annotation, and model-specific packaging. It retains processing records, intermediate outputs, and rejected examples so that recipe variants can reuse prior computation and remain traceable. A four-recipe study performs 24 from-scratch runs across Wan 2.1-1.3B and V-JEPA 2-1B, comparing a larger pool used once with smaller subsets used twice under matched update and processed-clip budgets. The paper reports higher mean downstream scores for the broader pool, a mismatch between validation-loss and downstream rankings, competitive operator throughput, and a 3.14-million-clip annotated dataset release.
Critical strengths
-
The infrastructure addresses the full path from data decisions to model evaluation. Separating saved quality scores from selection decisions and retaining rejected clips directly enables the Selected, Rejected, Mixed, and Random comparisons. Appendix B specifies identifiers, error records, and stage dependencies rather than presenting traceability only as a high-level aspiration. Supporting both generative and self-supervised training gives the workflow practical breadth.
-
The recipe study includes meaningful controls: a random subset of the same size, parent-video isolation from pretraining validation sets, three runs per recipe, and matched seeds for Mixed, Selected, and Rejected. Mixed exceeds Selected in all three matched runs for both VBench Total and SSv2. The reported means are 60.64 versus 59.74 on VBench and 16.256% versus 15.712% on SSv2. These results provide useful evidence within the explicitly limited early-pretraining regime.
Critical weaknesses
-
The generation study provides limited evidence about useful video-generation quality. Section 4.2 acknowledges that the models do not yet consistently generate coherent videos. Table 13 reports very low scores on several semantic dimensions, including Human Action of 1.56 and Multiple Objects of 0.14 for Mixed, while Mixed is not best on Overall Consistency or Temporal Style. The removal-of-Dynamic-Degree analysis is helpful, but does not establish that the roughly 0.9-point Total improvement over Selected reflects a perceptually meaningful gain. The scope should remain an early-training metric comparison, rather than a general recommendation about final data quality or mature generation models.
-
The released annotation resource is not semantically validated. Appendix E.1 defines annotation completeness through successful operations and schema checks, while Appendix I explicitly states that annotation accuracy has not been systematically assessed by human evaluation. Camera labels are also supplied to caption generation, so annotation errors could propagate across fields. Since fine-grained annotations are a major part of the dataset contribution, valid structure and full coverage do not establish their reliability for studying caption or camera-motion recipes. This leaves an important part of the resource's claimed research utility unverified.
Questions for authors
-
Clarification: What do the existing generated examples and per-dimension results support about the practical meaning of the VBench difference? Can the conclusion explicitly distinguish a higher early-stage aggregate score from improved coherent, prompt-aligned generation?
-
Clarification: Is there any existing semantic audit of camera labels, captions, or tags beyond schema validation? If none is available, which fields and recipe experiments should be treated as unvalidated in the release documentation?
4 comments
Response to Reviewer DuHq (2/2)
W1 / Q1: What the early VBench difference means
A: Thank you for asking whether the roughly 0.9 gain in VBench Total reflects a visible improvement. As noted in Sec. 4.2, the models do not yet consistently generate coherent videos that match the prompts. We therefore checked whether higher VBench scores also mean better visual quality at this early stage.
Fig. 16 and Tab. 13 show the scores for each dimension. Using the official weights, Imaging Quality and Aesthetic Quality account for about 52% of the 0.9 gain. Subject Consistency, Background Consistency, Temporal Flickering, and Motion Smoothness account for another 36%. We focused our visual checks on these six dimensions.
For each dimension, we selected two groups of videos with large score gaps: one favoring Mixed and one favoring Selected, giving 12 groups in total. Each group compares videos generated by Mixed, Selected, and Rejected from the same prompt. One author watched the original videos. DeepSeek V4.1 Flash, GPT-6.1-Terra, and GPT-6-Astra each reviewed image grids containing every frame, with anonymous video labels and no VBench scores. Each review compared the videos on the relevant dimension.
In 11/12 groups, the author preferred the higher-scoring video or saw no clear difference: 6 groups favored the higher-scoring video and 5 were tied. Dskv4.1, Terra, and Astra gave the same direction or a tie in 10/12, 8/12, and 12/12 groups, respectively; they strictly preferred the higher-scoring video in 6, 6, and 5 groups. These reviews support a visual basis for the early VBench gap: the inspected examples show visible differences in image quality and temporal stability that help explain the roughly 0.9 gain.
The video comparisons and human and model judgments are available in our anonymous repository (https://anonymous.4open.science/r/anno_ff-F4D8).
On a personal note, it meant a great deal to me that you took the time to examine the details in the appendix and understand the reasoning behind the design choices. Much of the work behind research infrastructure can be easy to overlook once a pipeline runs. Seeing that work understood and its limitations discussed with equal care was genuinely encouraging. Thank you.
Thank you for your response. I will maintain my score.
Many thanks for your response!
Thank you for getting back to us and for the care you put into reviewing our work.
Response to Reviewer DuHq (1/2)
Thank you for carefully examining our design and experimental controls, and for identifying where further evidence would strengthen the paper.
W2 / Q2: Annotation quality and Camera input to captions
A:Our semantic audits support the quality of Camera, Caption, and Tag annotations. Tests on CAPability [1], a benchmark for caption correctness and coverage, further show that Camera metadata improves both measures. The full audit results are available in our anonymous repository (https://anonymous.4open.science/r/anno_ff-F4D8)
We audit clips sampled from VidaForge-3M, following the annotation schemas (Appx. C.1–C.3) and generation prompts (Appx. D.1–D.3). We selected 1,000 clips to cover all 78 label values across Camera categories and audited both Camera and Caption on these clips. Since Camera labels guide caption generation, we check both annotations on the same clips. We selected another 1,000 clips to cover all 65 Tag categories. As stated in the paper, Camera uses Gemma-4-E4B-it, while Caption and Tag use Qwen3.6-27B-FP8.
We first tried training reviewers without a video-pipeline background, but the camera-motion and caption-level rules required substantial training. An author familiar with the project therefore reviewed 100 clips per annotation type, and GPT-6-Astra reviewed the other 900. Reviewers checked annotations against timestamped thumbnails of the full clip, viewed in time order.
For Camera and Tag, we checked whether each existing label matched the visual content. $\text{Label support} = \text{supported labels} / \text{all labels} $ with a definite judgment. About 2.4% of Camera labels could not be judged and were excluded from the denominator. Since subjects and actions are multi-label Tag fields, we also checked for missing main subjects or actions.
Our captions have four levels of detail, L0–L3. We audit L3, the most detailed level. We follow CAPability’s [1] prompts and scoring rules for its four video dimensions: Action, Event, Camera, and Count. DeepSeek-V4.1-Flash drafts rubric questions, such as what action occurs or how the camera moves, and reference answers for each clip. GPT-6-Astra checks these references against the video thumbnails, excluding incorrect or uncertain items. This gives 6,558 valid questions for 1,000 clips (6.56 per clip).
Let �1�, �2�, and �3� count correct, incorrect, and omitted items under the task rules. Precision (P) measures correctness, and Coverage (C) measures how much target content is correctly described:
$
P=N+1+N−1N+1,C=N+1+N−1+N0N+1.$
We average the four dimension scores equally and report percentages.
Both reviews show high Tag support and high L3 Precision and Coverage. In most clips (74%), the multi-label Tag fields cover the main subjects and actions.
We evaluate our multi-level caption schema and Camera input on all 2,192 CAPability videos using its official questions and reference answers. As described in the paper, L0 summarizes subjects, actions, and scenes; L1 adds event order; L2 adds temporal changes, relations, and background; L3 adds detail and camera descriptions. We generate Camera tags with our schema, then use them as input for Qwen to write L0–L3 captions. GPT-6.1-Terra scores them with the official evaluation prompts.
On CAPability, higher caption levels improve both correctness and coverage, supporting our multi-level design.
We use the same Qwen model and multi-level prompt, changing only Camera input. L3 results are:
Camera input improves both camera-description metrics and the four-dimension averages.
The repository (https://anonymous.4open.science/r/anno_ff-F4D8) also compares Gemma and Qwen Camera labels and reports per-dimension caption scores with and without Camera input on the 1,000 VidaForge-3M clips. It includes a GPT-6-Luna baseline using the same CAPability prompts.
[1] Liu, Zhihang, et al. “CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness.” NeurIPS, 2025.