acceptodds
Under review as a conference paper at ICLR 2027

Generative Uncertainty as a Self-supervised Signal for Semantic Similarity Learning

Abstract

Evaluating semantic similarity between videos is a fundamental challenge in computer vision, essential for tasks ranging from out-of-distribution (OOD) detection to video retrieval. However, defining and labeling video similarity is notoriously difficult and expensive due to the complex spatio-temporal nature. In this paper, we propose a novel self-supervised approach that leverages generative uncertainty from text-to-video (T2V) diffusion models to learn semantic similarity without human annotations. Our method is based on the observation that T2V models produce consistent outputs for familiar concepts but exhibit high variance and uncertainty when prompted with specialized concepts. We utilize this behavior to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA. Specifically, we learn a mask over these embeddings using purely generated data, encouraging the model to retain features that remain consistent across generations of general concepts while discarding those associated with generative noise or uncertainty. Experimental results across three key tasks demonstrate that our learned feature subspaces consistently outperform original pretrained features and baseline feature selection methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
similarity learning, generative uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YtrtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes using the generative uncertainty of text-to-video (T2V) diffusion models as a self-supervised signal to learn semantic video similarity. The core observation is that T2V models generate highly consistent videos for general concepts but produce diverse and uncertain outputs for specialized or niche concepts. Exploiting this variance, the authors train a mask over pretrained video embeddings (such as VideoMAE and V-JEPA) to filter out dimensions that vary significantly under specialized prompts while retaining those that are stable under general prompts. The learned sparse masks are then evaluated on human alignment, out-of-distribution (OOD) detection for generated samples, and video retrieval tasks.

Critical strengths

By utilizing the generative uncertainty output by text-to-video (T2V) models as a self-supervised signal, the paper provides a novel and intuitive perspective for learning video semantic similarity without manual annotations.

The method only requires training a simple feature mask (soft or binary) on top of frozen pre-trained features (such as V-JEPA and VideoMAEv2). Its training efficiency is extremely high, taking only about 1 minute to complete on a single H100 GPU, making it highly convenient to integrate into existing architectures as a plug-and-play module.

The paper conducts extensive evaluations across multiple downstream tasks, including human perception alignment (ConViS-Bench), out-of-distribution (OOD) detection for generated videos, and video retrieval (UCF101, HMDB51), with results consistently outperforming unmasked baseline features.

Critical weaknesses

When constructing "specialized prompts," the method strictly limits them to the domains of clinical surgery and cellular microscopy. Therefore, the mask is essentially trained to distinguish everyday general concepts from specific medical/microscopic domains. However, in the downstream OOD detection evaluation, two of the four OOD datasets are exactly MedVidBench (medical surgery) and HAM10000 (dermatoscopic images). This raises a severe evaluation fairness issue: the mask might have merely learned to identify and isolate medical domain features rather than acquiring a universally applicable "semantic similarity" or "generative uncertainty." This potential data leakage significantly undermines the credibility of the OOD experimental results.

The training objective (Equation 1) strictly enforces a target similarity of y=0y=0 for video pairs generated from the same specialized prompt. This implicitly assumes that multiple videos generated by the model share "zero" semantic similarity simply because the generator struggles with a complex concept. In reality, however, two "fluorescent bacteria biofilm" videos generated by the model inevitably still share a massive amount of semantic overlap (e.g., similar color distributions, microscopic textures, etc.). Forcing their distance to be maximized in the representation space might destroy these originally highly valuable fine-grained semantic features within these specific domains.

The training of the mask solely uses a synthetic dataset comprising 600 videos (200 prompts ×\times 3 random seeds), partitioned into an 80% training set and a 20% validation set. Given the high dimensionality of the pre-trained feature space (e.g., 1024 dimensions for V-JEPA 2.1-L), training the mask with only about 480 video pairs (160 prompts) carries an extremely high risk of overfitting. A low-dimensional subspace learned from such limited prompt diversity is hard to prove robust across broader open-world videos.

Questions for authors

How can the authors prove that the feature mask has learned generalized "generative uncertainty" rather than merely acting as a domain-specific filter that down-weights "medical/microscopic" related features? If OOD detection were evaluated solely on datasets entirely unrelated to the specialized domains from the training phase (e.g., keeping only UVEB and WikiArt), or if the mask were retrained on completely different specialized domains (e.g., astrophotography, industrial manufacturing), would the current performance improvements still hold?

Equation 1 forces the similarity of specialized prompt video pairs to be y=0y=0. Have the authors experimented with smoother penalization methods (e.g., using an InfoNCE-style contrastive loss, or assigning a non-zero, low-similarity soft label) to avoid destroying reasonable semantic overlaps in specialized domains?

Given that the training scale is only 600 synthetic videos, how sensitive is the learned feature mask to the selection of these specific 200 prompts? If a large language model were used to generate a completely new set of 100 general prompts and 100 specialized prompts, how high would the overlap be between the newly selected binarized feature subspace (binary mask) and the current version after retraining?

2 comments

We thank the reviewer for recognizing the simplicity and efficiency, no need for manual annotation and extensive evaluations of our method. We recognize the value of concerns on prompt category, labels and dataset scale and followed the suggestions in the following experiments:

Specialized sets with different domains: To test whether the mask learns generative uncertainty rather than filtering out medical or microscopic features, we retrained it on two specialized domains unrelated to every benchmark: astronomy and industrial manufacturing. The new training set has 50 prompts per domain, each length-matched to one of the paper's 100 general prompts, which are unchanged. Training uses the same objective and hyperparameters as the paper.

For V-JEPA 2.1-L, the mask trained only on astronomy and manufacturing videos improves AUROC and AUPR on all four benchmarks with both detectors, for both binary and soft masks. This includes UVEB and WikiArt, which share no domain with the training data, as the reviewer suggested. With DDPM-OOD, MedVidBench AUROC rises from 68.11 to 86.92, and HAM10000 FPR@95 falls from 51.60 to 34.80. Since no medical or microscopic content was seen during training, these gains cannot come from down-weighting such features. For VideoMAEv2-B, the gain transfers only partly: UVEB improves with both detectors, MedVidBench stays at the unmasked level, and HAM10000 and WikiArt drop by about 1 AUROC point. We read this as a property of the encoder. VideoMAEv2-B's features of generative uncertainty appear to be more tied to the training domains, while those of V-JEPA 2.1-L are not.

DetectorEncoderMaskUVEB AUROC ↑UVEB AUPR ↑UVEB FPR95 ↓MedVidBench AUROC ↑MedVidBench AUPR ↑MedVidBench FPR95 ↓HAM10000 AUROC ↑HAM10000 AUPR ↑HAM10000 FPR95 ↓WikiArt AUROC ↑WikiArt AUPR ↑WikiArt FPR95 ↓
2XDMV-JEPA 2.1-LUnmasked78.4273.7757.2074.6674.1876.0070.8061.2163.2063.1055.8265.20
2XDMV-JEPA 2.1-LBinary83.9782.4952.4084.9485.8766.4077.6769.9858.0068.3762.5065.20
2XDMV-JEPA 2.1-LSoft83.3481.6451.6084.5685.3967.2076.5468.2258.8067.4861.4266.00
2XDMVideoMAEv2-BUnmasked71.0868.5077.6078.5980.7382.4068.3862.4278.4057.7851.0582.00
2XDMVideoMAEv2-BBinary71.8668.7974.4078.7680.5282.0067.5460.9381.6056.8350.2583.20
2XDMVideoMAEv2-BSoft71.5268.5176.4078.4780.2983.6067.4960.9481.2056.6250.1484.00
DDPM-OODV-JEPA 2.1-LUnmasked76.3770.3163.6068.1168.4786.0087.6787.9551.6069.8065.0382.40
DDPM-OODV-JEPA 2.1-LBinary87.6285.9850.8086.9286.7564.8094.0794.7134.8078.0772.7272.40
DDPM-OODV-JEPA 2.1-LSoft86.8684.9650.4086.5586.2162.4093.5694.2733.2076.4870.4872.40
DDPM-OODVideoMAEv2-BUnmasked75.5073.3383.2077.0179.4679.6086.2388.3767.6063.8863.1689.20
DDPM-OODVideoMAEv2-BBinary76.8475.2384.0076.9979.2881.6085.4787.7271.6062.8561.3288.80
DDPM-OODVideoMAEv2-BSoft76.7275.1482.0076.6878.9981.2085.6687.8170.0062.5161.0089.60

Bold: better than the unmasked encoder.

Sensitivity to prompt selection. We agree this is important to test. To do so, we generated two additional training datasets with the same video generator and three videos per prompt. The new set uses a completely new set of 100 general and 100 specialized prompts. These prompts come from the same domains as the original ones (animals, scenery, and lifestyle for general prompts; clinical surgery and cellular microscopy for specialized prompts) and are generated with the same procedure, but none of them appears in the original set. This gives a total of 600 videos. The ×10 set enlarges the original dataset tenfold to 6000 videos.

We use these two datasets to retrain the masks with identical hyperparameters. The table below reports the agreement of the resulting masks with the original mask, measured by the Spearman correlation ρ\rho between the soft masks and between the binarized masks (ρ=0\rho=0 for two independent masks).

The mask trained on the new prompts is strongly correlated with the original mask, and the binary masks agree far beyond chance. With ten times more data, the mask moves even closer to the original one. Together, these results indicate that the learned subspace is largely preserved across prompt sets rather than specific to the 200 original prompts.

EncoderTraining setSpearman ρ\rho (soft)Spearman ρ\rho (binary)
V-JEPA 2.1-LNew set0.8040.482
V-JEPA 2.1-L×10 set0.8930.575
VideoMAEv2-BNew set0.6720.482
VideoMAEv2-B×10 set0.8090.552
Reviewer_Vwz2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper learns a mask based on frozen video embeddings (e.g., V-JEPA, VideoMAEv2). This paper propose that existing video embedding only measure general embedding and contain irrelevant or even detrimental information. Thus the learnable mask aims to suppress it. Specially, the training data is automated generated by text-to-video generation: videos from the same "general" / "specialized" prompt are pushed to similar or dissimilar pairs. The masked features are evaluated on human-similarity alignment, ood detection and video retrieval.

Critical strengths

1 The idea that use light-weight model to remove irrelevant or even detrimental information is interesting. 2. The ablation study against other feature masking strategies and training objectives shows its advantage.

Critical weaknesses
  1. This paper lacks of critical comparisons with strong embedding models like those modern multimodal embedding model family such as vlm2vec or video-text encoders. Since I suspect these current instruction following model can tackle the task-specific embedding very well. It is therefore unclear whether the gains matter in practice. Please provide the comparisons, or explain why they are not included.

  2. A single concept-agnostic mask cannot justify concept-conditioned similarity. ConVis-Bench evaluates similarity along main action, subject, action order, but the result show the method applies one global mask to all. By contrast, instruction-following embedding models can condition similarity on the queried context.

Questions for authors

see above

2 comments

We thank the reviewer for suggesting missed embedding models tailored for this specific task. We extended our experiments to include them as well, further showing generalization of our method:

Relation to multimodal embedding models. We want to clarify that our aim is not to propose a novel state-of-the-art encoder for semantic similarity. Rather, our method improves the embeddings of individual models by learning a subspace of their features that improves the original model on similarity-based tasks. We now show how our method can do so for the two different models.

Multimodal embedding models. We can apply our method also to the video-text models, since our masking pipeline is not tied to a specific encoder. We applied our method to X-CLIP and evaluated on ConViS-Bench. The table below shows that the soft mask improves the average Spearman's ρ\rho from 45.9645.96 to 50.1050.10, demonstrating that our method is applicable beyond video encoders and can also improve multimodal encoders.

Instruction-following embedding models. Moreover, we can improve the instruction-following embedding models the reviewer mentions. For this, we train masks for VLM2Vec, prompting the models to create embeddings for the different tasks of ConViS-Bench. For instance, to obtain embeddings for the main actions in a video. The table below shows that the soft mask improves the average Spearman's ρ\rho from 53.4353.43 to 54.5754.57 for VLM2Vec. Also here, our mask improves the results, demonstrating its broad efficacy. We added these results to Appendix F.

EncoderMethodMain ActionMain SubjectsMain ObjectsLocationActions OrderAverage
X-CLIPBaseline (no mask)50.344.251.439.744.245.96
X-CLIPBinary mask53.1‾\underline{53.1}49.3‾\underline{49.3}55.4‾\underline{55.4}43.7‾\underline{43.7}47.5‾\underline{47.5}49.79‾\underline{49.79}
X-CLIPSoft mask53.249.955.744.147.650.10
VLM2VecBaseline (no mask)48.254.751.163.749.453.43
VLM2VecBinary mask50.0‾\underline{50.0}53.651.865.550.4‾\underline{50.4}54.28‾\underline{54.28}
VLM2VecSoft mask50.454.5‾\underline{54.5}51.865.3‾\underline{65.3}50.854.57

Dear Reviewer Vwz2, we hope our responses have addressed the concerns raised in your review. As the discussion period is about to end, we would appreciate knowing whether you have any remaining questions or concerns that we could clarify.

Reviewer_Myx9OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised method for learning video semantic similarity from synthetic videos generated by text-to-video models. The method builds on the observation that general prompts tend to produce more consistent generations, whereas specialized prompts lead to greater variability. The authors generate multiple videos from each prompt and assign similarity labels to video pairs according to whether the prompt belongs to the general or specialized category. A feature mask is then learned over frozen pretrained video embeddings to emphasize dimensions that are more useful for semantic similarity.

The method is evaluated on human similarity alignment, out-of-distribution detection for generated videos, and video retrieval, using different video encoders and text-to-video generators. The reported results show improvements in many experimental settings, although the gains are not consistent across all metrics and model configurations.

Critical strengths

The idea of using differences in generative consistency to learn a feature mask for video semantic similarity is interesting. The method is also simple, since it only learns a mask over pretrained video representations without updating the video encoder. The resulting feature subspace can be used across different downstream tasks, and the binary masks retain only a fraction of the original feature dimensions.

The experiments cover three text-to-video generators, two pretrained video encoders, and several downstream tasks. Table 1 shows improved alignment with human similarity judgments for both encoders, with particularly noticeable gains for V-JEPA 2.1-L. The OOD detection results are also promising in several settings. Moreover, the comparisons with other feature-selection methods in Table 4 suggest that the improvements are not simply due to reducing feature dimensionality.

Critical weaknesses

My main concern is whether generative uncertainty itself is the supervisory signal learned by the method. In Section 3.2, video pairs generated from general prompts are assigned a similarity label of 1, while pairs from specialized prompts are assigned 0. These labels are determined by the prompt categories rather than the measured variability of the generated videos. Furthermore, Appendix B.1 shows that the specialized prompts come from clinical surgery and cellular microscopy, while general prompts are sampled from VBench. Although Figure 3 shows higher feature-space diversity for specialized prompts, it remains unclear whether the learned mask captures semantic consistency or exploits differences between these predefined domains. The comparison with a standard similarity objective in Table 4 does not fully resolve this issue, because the construction of negative pairs also changes. The theoretical analysis in Appendix A provides limited additional support: Proposition 2 follows directly from the definitions of semantic variability and similarity, but does not explain why binary labels derived from prompt categories should lead to generalizable semantic similarity features.

The novelty claim in Section 2 is also somewhat broad. StableRep (NeurIPS 2023) already uses multiple generations from the same text prompt to construct a self-supervised contrastive learning objective. Although the present paper differs by using generative consistency to select feature dimensions in pretrained video representations, the distinction from previous work should be made clearer. In addition, some experimental conclusions are stronger than the results support. In Table 2, the binary mask increases FPR@95 on all four OOD datasets when using VideoMAEv2-B with DDPM-OOD, even though AUROC improves. In Table 3, both binary and soft masks reduce retrieval mAP for VideoMAEv2-B on HMDB51. These results do not invalidate the overall findings, but they indicate that the benefits are not consistent across all encoders, tasks, and evaluation metrics.

Questions for authors

Could the authors provide a more direct evaluation of whether generative uncertainty is responsible for the learned similarity features? In particular, would it be possible to construct continuous or binary similarity targets from the actual variability of videos generated for each prompt, rather than assigning labels based only on whether the prompt is general or specialized? Such a comparison would help clarify the role of the proposed supervisory signal.

Would similar improvements be observed if prompts with high and low generative variability were selected from closely related semantic domains? Since the current general and specialized prompt sets also differ substantially in content, this experiment would help determine whether the learned mask captures semantic consistency rather than domain-specific characteristics.

Could the authors provide a more controlled comparison with the standard similarity objective in Table 4? Since the current comparison changes both the labels and the construction of negative pairs, it is difficult to isolate the benefit of the generative-consistency labeling strategy. A comparison using the same pool of generated video pairs with different labeling rules would be informative.

Finally, how was the binarization threshold of 0.5 in Eq. (1) selected? Although the paper analyzes different thresholds in its feature-preservation experiments, it would be useful to know how sensitive the main human-alignment, OOD, and retrieval results are to this choice.

2 comments

We thank the reviewer for appreciating the idea and simplicity of our approach as well as the evaluations and ablations proposed in the paper. We found the comment concerning the prompt category particularly insightful and addressed the remaining concerns to strengthen the paper:

Generative uncertainty as the supervisory signal. We thank the reviewer for this suggestion. Our setup, however, assumes that we don't have an exact measure of variability for the generated videos, as this would require a notion of similarity, thus introducing an external human or model judge in the loop. Nevertheless, to validate that generative uncertainty is indeed the signal learned by our method, and therefore responsible for the learned similarity features, we tested the suggested alternative to label our videos. Specifically, each pair is assigned a label corresponding to its variability according to a model that measures how similar the generated videos are. To do so, we embed the videos with a separate encoder, VLM2Vec. Each pair then uses the mean cosine similarity of its prompt's video generations as its target. We learn masks for both encoders and report results on all experiments in the table below.

EncoderObjectiveMaskConViS-Bench ρ\rho ↑2XDM AUROC ↑DDPM-OOD AUROC ↑UCF101 mAP ↑HMDB51 mAP ↑
V-JEPA 2.1-LUnmasked—31.1471.7575.4940.8217.84
V-JEPA 2.1-LHard label, y=0y=0Binary35.4481.7389.6541.3418.40
V-JEPA 2.1-LHard label, y=0y=0Soft36.0480.70‾\underline{80.70}89.30‾\underline{89.30}42.3718.99
V-JEPA 2.1-LVLM2Vec targetBinary35.72‾\underline{35.72}73.3679.6942.98‾\underline{42.98}19.37‾\underline{19.37}
V-JEPA 2.1-LVLM2Vec targetSoft34.5473.1679.4843.8019.83
VideoMAEv2-BUnmasked—44.9068.9675.6695.5246.94
VideoMAEv2-BHard label, y=0y=0Binary46.1070.4577.6995.61‾\underline{95.61}46.39
VideoMAEv2-BHard label, y=0y=0Soft46.04‾\underline{46.04}70.23‾\underline{70.23}77.40‾\underline{77.40}95.6546.66‾\underline{46.66}
VideoMAEv2-BVLM2Vec targetBinary40.2867.6573.2894.6442.84
VideoMAEv2-BVLM2Vec targetSoft44.9068.9575.6595.5146.94

The results show that the variability label gives worse results than the generative uncertainty label on all experiments, except for video retrieval with V-JEPA 2.1-L. For example, with V-JEPA 2.1-L, the average OOD AUROC drops from 81.7381.73 to 73.3673.36 (2XDM) and from 89.6589.65 to 79.6979.69 (DDPM-OOD).

Overall, the generative uncertainty label is a more consistent supervisory signal in our experimental setup, confirming its usefulness.

General and specialized prompts from closely related domains. We agree this is important and ran an experiment where general and specialized prompts come from the same categories to verify the learned mask captures semantic consistency rather than domain-specific characteristics. For this, we kept the 100 general prompts of our main dataset and created a new set of 100 specialized prompts from the same categories (animals, scenery, and lifestyle), following the same protocol as our main dataset. For example, animal prompts include "spider making a web" (general) and "echidna consuming carpenter ant" (specialized). The remaining settings are unchanged. We show the results in the table below.

EncoderRepresentationConViS-Bench ρ\rho2XDM AUROCDDPM-OOD AUROCUCF101 mAPHMDB51 mAP
V-JEPA 2.1-LUnmasked31.1471.7575.4940.8217.84
V-JEPA 2.1-LOriginal binary mask35.4481.7389.6541.3418.40
V-JEPA 2.1-LOriginal soft mask36.0480.7089.3042.3718.99
V-JEPA 2.1-LSame-category binary mask36.4675.3484.3444.2920.35
V-JEPA 2.1-LSame-category soft mask36.1075.8885.0644.1620.20
VideoMAEv2-BUnmasked44.9068.9675.6695.5246.94
VideoMAEv2-BOriginal binary mask46.1070.4577.6995.6146.39
VideoMAEv2-BOriginal soft mask46.0470.2377.4095.6546.66
VideoMAEv2-BSame-category binary mask46.7669.7876.9195.6847.15
VideoMAEv2-BSame-category soft mask46.4469.0175.9295.6947.30

The results show that the mask learned from generative uncertainty when general and specialized prompts come from the same categories still improves performance on all tasks. Hence, the learned mask does not rely on the two prompt sets coming from different categories.

Controlled comparison against the standard similarity objective. We would like to clarify that the labeling rule and pair construction are inherently linked. Our specialized generated pairs are semantically similar but differ in generative consistency. Treating these pairs as negatives (y=0) while keeping the same pair pool would already apply our uncertainty-based labeling and therefore would no longer represent the standard similarity objective. Conversely, the standard similarity objective would label these pairs as similar (y=1), ignoring the generative inconsistency targeted by our method. Therefore, changing their labels from (y=1) to (y=0) would effectively reproduce our proposed strategy rather than provide a separate similarity baseline.

Effect of the binarization threshold. We selected the 0.50.5 as the most natural threshold and keep it fixed across all of our experiments. Still, as suggested, we evaluated several other binarization thresholds for all models and tasks. The table below summarizes the results and shows that no single threshold consistently performs best in every setting. Please see Appendix G for the full analysis. However, 0.50.5 gives good results overall, and we therefore select it as a reasonable and conventional default, rather than tuning it separately for each model or task.

EncoderSparsity (%)ConViS-Bench Avg. ρ\rho2XDM AUROCDDPM-OOD AUROCUCF101 mAPHMDB51 mAP
V-JEPA 2.1-L (baseline)—31.1471.7575.4940.8217.84
V-JEPA 2.1-L (θ=0.5\theta=0.5)67.835.4481.7389.6541.3418.40
V-JEPA 2.1-L (θ=0.7\theta=0.7)71.934.8881.5489.1840.5417.80
V-JEPA 2.1-L (θ=0.3\theta=0.3)62.535.6080.3189.1541.7618.73
VideoMAEv2-B (baseline)—44.9068.9675.6695.5246.94
VideoMAEv2-B (θ=0.5\theta=0.5)58.046.1070.4577.6995.6146.39
VideoMAEv2-B (θ=0.7\theta=0.7)72.146.1869.7676.5895.5445.15
VideoMAEv2-B (θ=0.3\theta=0.3)44.746.0870.2677.5795.6846.90

Novelty wording. We have adjusted the corresponding claims in the Related Work section, lines 104-114, and the Experiments section, lines 353 and 356-358, to more clearly distinguish our contributions from the suggested related works.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.