- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes using the generative uncertainty of text-to-video (T2V) diffusion models as a self-supervised signal to learn semantic video similarity. The core observation is that T2V models generate highly consistent videos for general concepts but produce diverse and uncertain outputs for specialized or niche concepts. Exploiting this variance, the authors train a mask over pretrained video embeddings (such as VideoMAE and V-JEPA) to filter out dimensions that vary significantly under specialized prompts while retaining those that are stable under general prompts. The learned sparse masks are then evaluated on human alignment, out-of-distribution (OOD) detection for generated samples, and video retrieval tasks.
Critical strengths
By utilizing the generative uncertainty output by text-to-video (T2V) models as a self-supervised signal, the paper provides a novel and intuitive perspective for learning video semantic similarity without manual annotations.
The method only requires training a simple feature mask (soft or binary) on top of frozen pre-trained features (such as V-JEPA and VideoMAEv2). Its training efficiency is extremely high, taking only about 1 minute to complete on a single H100 GPU, making it highly convenient to integrate into existing architectures as a plug-and-play module.
The paper conducts extensive evaluations across multiple downstream tasks, including human perception alignment (ConViS-Bench), out-of-distribution (OOD) detection for generated videos, and video retrieval (UCF101, HMDB51), with results consistently outperforming unmasked baseline features.
Critical weaknesses
When constructing "specialized prompts," the method strictly limits them to the domains of clinical surgery and cellular microscopy. Therefore, the mask is essentially trained to distinguish everyday general concepts from specific medical/microscopic domains. However, in the downstream OOD detection evaluation, two of the four OOD datasets are exactly MedVidBench (medical surgery) and HAM10000 (dermatoscopic images). This raises a severe evaluation fairness issue: the mask might have merely learned to identify and isolate medical domain features rather than acquiring a universally applicable "semantic similarity" or "generative uncertainty." This potential data leakage significantly undermines the credibility of the OOD experimental results.
The training objective (Equation 1) strictly enforces a target similarity of for video pairs generated from the same specialized prompt. This implicitly assumes that multiple videos generated by the model share "zero" semantic similarity simply because the generator struggles with a complex concept. In reality, however, two "fluorescent bacteria biofilm" videos generated by the model inevitably still share a massive amount of semantic overlap (e.g., similar color distributions, microscopic textures, etc.). Forcing their distance to be maximized in the representation space might destroy these originally highly valuable fine-grained semantic features within these specific domains.
The training of the mask solely uses a synthetic dataset comprising 600 videos (200 prompts 3 random seeds), partitioned into an 80% training set and a 20% validation set. Given the high dimensionality of the pre-trained feature space (e.g., 1024 dimensions for V-JEPA 2.1-L), training the mask with only about 480 video pairs (160 prompts) carries an extremely high risk of overfitting. A low-dimensional subspace learned from such limited prompt diversity is hard to prove robust across broader open-world videos.
Questions for authors
How can the authors prove that the feature mask has learned generalized "generative uncertainty" rather than merely acting as a domain-specific filter that down-weights "medical/microscopic" related features? If OOD detection were evaluated solely on datasets entirely unrelated to the specialized domains from the training phase (e.g., keeping only UVEB and WikiArt), or if the mask were retrained on completely different specialized domains (e.g., astrophotography, industrial manufacturing), would the current performance improvements still hold?
Equation 1 forces the similarity of specialized prompt video pairs to be . Have the authors experimented with smoother penalization methods (e.g., using an InfoNCE-style contrastive loss, or assigning a non-zero, low-similarity soft label) to avoid destroying reasonable semantic overlaps in specialized domains?
Given that the training scale is only 600 synthetic videos, how sensitive is the learned feature mask to the selection of these specific 200 prompts? If a large language model were used to generate a completely new set of 100 general prompts and 100 specialized prompts, how high would the overlap be between the newly selected binarized feature subspace (binary mask) and the current version after retraining?
2 comments
Sensitivity to prompt selection. We agree this is important to test. To do so, we generated two additional training datasets with the same video generator and three videos per prompt. The new set uses a completely new set of 100 general and 100 specialized prompts. These prompts come from the same domains as the original ones (animals, scenery, and lifestyle for general prompts; clinical surgery and cellular microscopy for specialized prompts) and are generated with the same procedure, but none of them appears in the original set. This gives a total of 600 videos. The ×10 set enlarges the original dataset tenfold to 6000 videos.
We use these two datasets to retrain the masks with identical hyperparameters. The table below reports the agreement of the resulting masks with the original mask, measured by the Spearman correlation between the soft masks and between the binarized masks ( for two independent masks).
The mask trained on the new prompts is strongly correlated with the original mask, and the binary masks agree far beyond chance. With ten times more data, the mask moves even closer to the original one. Together, these results indicate that the learned subspace is largely preserved across prompt sets rather than specific to the 200 original prompts.
| Encoder | Training set | Spearman (soft) | Spearman (binary) |
|---|---|---|---|
| V-JEPA 2.1-L | New set | 0.804 | 0.482 |
| V-JEPA 2.1-L | ×10 set | 0.893 | 0.575 |
| VideoMAEv2-B | New set | 0.672 | 0.482 |
| VideoMAEv2-B | ×10 set | 0.809 | 0.552 |
We thank the reviewer for recognizing the simplicity and efficiency, no need for manual annotation and extensive evaluations of our method. We recognize the value of concerns on prompt category, labels and dataset scale and followed the suggestions in the following experiments:
Specialized sets with different domains: To test whether the mask learns generative uncertainty rather than filtering out medical or microscopic features, we retrained it on two specialized domains unrelated to every benchmark: astronomy and industrial manufacturing. The new training set has 50 prompts per domain, each length-matched to one of the paper's 100 general prompts, which are unchanged. Training uses the same objective and hyperparameters as the paper.
For V-JEPA 2.1-L, the mask trained only on astronomy and manufacturing videos improves AUROC and AUPR on all four benchmarks with both detectors, for both binary and soft masks. This includes UVEB and WikiArt, which share no domain with the training data, as the reviewer suggested. With DDPM-OOD, MedVidBench AUROC rises from 68.11 to 86.92, and HAM10000 FPR@95 falls from 51.60 to 34.80. Since no medical or microscopic content was seen during training, these gains cannot come from down-weighting such features. For VideoMAEv2-B, the gain transfers only partly: UVEB improves with both detectors, MedVidBench stays at the unmasked level, and HAM10000 and WikiArt drop by about 1 AUROC point. We read this as a property of the encoder. VideoMAEv2-B's features of generative uncertainty appear to be more tied to the training domains, while those of V-JEPA 2.1-L are not.
Bold: better than the unmasked encoder.