- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces EgoCrash, a synthetic benchmark of 20,000 egocentric traffic-event videos across 56 categories. The authors fine-tune Wan2.1-T2V on 2,000 real accident videos, mainly from MM-AU, and generate new videos from LLM-augmented descriptions. They then filter the videos manually and annotate them with VLM-generated descriptions, accident labels, optical flow, entity trajectories, and QA pairs. The paper also defines an open-vocabulary video-text retrieval task. For this task, it proposes MRA-VLM, built on X²-VLM. MRA-VLM fuses RGB and optical-flow features with a gated cross-attention module. At inference, it enhances queries with prototypes from seen categories when the two retrieval directions disagree. MRA-VLM outperforms CLIP4Clip, DRL, X-CLIP, and X²-VLM on both synthetic and real test sets.
Critical strengths
- Important and timely problem. Real accident videos are rare and unevenly distributed. The paper directly studies whether synthetic videos can fill this gap, i.e., whether or not models trained only on synthetic data generalize to unseen categories and real videos.
- Large dataset with rich annotations. With 20,000 videos in 56 categories, EgoCrash is larger than existing egocentric accident datasets. It provides descriptions, accident labels, optical flow, trajectories, and QA pairs.
- Transparent discussion of dataset limitations. Appendix openly reports the category imbalance. It also notes that the semantic coverage of descriptions is measured only lexically, and that flow and trajectories are estimated rather than ground truth.
- Consistent improvements. On the synthetic test set (Table 2), MRA-VLM outperforms all baselines in both retrieval directions and on both seen and unseen categories. The additional inference cost is small, about 9%.
Critical weaknesses
- Weak evidence for synthetic-to-real transfer. On real videos, the overall gain over X²-VLM is only +0.3 to +0.45 points in R@1 (Table 3), although the gains on the seen and unseen subsets are larger (+2 to +4 points). In addition, important baselines are missing, such as training on real data only, combining real and synthetic data, or using pretrained models zero-shot. The condition for 0% training data in Fig. 3(a) is also not explained. As a result, the central claim that synthetic supervision transfers to real accidents is not well supported.
- Annotations and test ground truth are mostly generated by LLMs and VLMs. Category labels are assigned by a VLM from the text descriptions alone. The paper mentions manual audits and sample-based checks, but reports neither their scale nor their results. In addition, the descriptions for the real test set are generated with the same pipeline, which may favor models trained on similar text.
- Limited novelty and effectiveness of the method. The three components, namely gated RGB–flow cross-attention fusion, prototype-based query enhancement, and score fusion triggered by bidirectional inconsistency, are similar to existing techniques. Examples include two-stream RGB–flow fusion (Simonyan & Zisserman, 2014), reciprocal-neighbor re-ranking (Zhong et al., 2017), and test-time score normalization for retrieval (e.g., QB-Norm; Bogolin et al., 2022). Moreover, without the inference-time modules, MRA-VLM is worse than X²-VLM on most metrics (e.g., 21.52 vs. 22.15 in text-to-video R@1 from Table 4 and Table 2), so most of the gain comes from inference-time prototype retrieval.
- Outdated baselines and concerns about evaluation reliability. All baselines in Table 2 date from 2022–2023 . The paper does not compare with recent video-language models such as InternVideo2 (ECCV 2024), LanguageBind (ICLR 2024), VideoCLIP-XL (EMNLP 2024), or Perception Encoder (NeurIPS 2025). In addition, some numbers in Table 4 appear duplicated: the overall scores of “w/o R” match exactly the unseen scores of “w/o P&R”. Also, the number of categories in the real test set differs between the text (56) and Table 6 (58).
- Limited quality and length of the synthetic videos. Each video is only about 4 seconds long (41 frames at 10 fps). Apart from manual screening, there is no quantitative evaluation of whether the generated videos depict the described maneuvers and interactions correctly.
Questions for authors
- How does MRA-VLM, trained on EgoCrash, compare with (a) the same model trained on real MM-AU data (excluding the test set), (b) training on real and synthetic data together, and (c) recent pretrained models (e.g., InternVideo2, VideoCLIP-XL) used zero-shot? What does “0%” mean in Fig. 3(a)? (could change my assessment)
- What are the scale and results of the manual audits of category labels and descriptions (e.g., the number of samples checked and the error rate)? Can the authors report results on a subset of the real test set with human-written descriptions? (could change my assessment)
- How do recent video-language models, such as InternVideo2, LanguageBind, VideoCLIP-XL, or Perception Encoder, perform when fine-tuned on EgoCrash? (could change my assessment)
- Please check Table 4. The overall scores of “w/o R” (T2V R@1/mAP = 26.87/38.80, V2T R@1/mAP = 24.25/35.43) are identical to the unseen scores of “w/o P&R”. Is this coincidence? In addition, the notation is inconsistent: the caption defines the uncertainty-guided score fusion as “U”, but the row names use “R” (“w/o R” and “w/o P&R”). In addition, Table 6 lists 58 categories, while the text says 56. How are the MM-AU categories mapped to the seen and unseen splits? (clarification)
- Is there evidence that the gate in the Maneuver-Aware Fusion Module suppresses ego-motion responses, e.g., an analysis of gate values? (clarification)
5 comments
Q2: Insufficient quantitative validation of LLM/VLM-generated annotations, including the scale and error rates of manual audits for category labels and descriptions, and potential evaluation bias from machine-generated captions without human-written comparisons.
We thank the reviewer for raising these important concerns. As detailed in Appendix A.1, five annotators screened 48,000 generated videos, with each video independently reviewed by two annotators and disagreements resolved by a third. Based on visual quality, temporal consistency, traffic-interaction plausibility, and description alignment, 28,000 videos were excluded, retaining 20,000 samples (41.7%). Following video screening, we further verified the generated annotations. Accident-occurrence labels were cross-checked using independent video- and text-based predictions, with disagreements resolved through human inspection. Fine-grained category labels, initially assigned from textual descriptions, were also manually reviewed and corrected against the corresponding video content. Specifically, 20,000 category labels were inspected, with 2,487 errors identified and corrected (12.4% initial error rate). The finalized reviewed labels achieved 100% agreement with human adjudications. For video descriptions, we conducted five rounds of random-sampling-based verification, reviewing approximately 2,000 annotations per round and manually correcting identified inconsistencies. An additional random audit of 2,000 descriptions after these corrections yielded a consistency rate of 98%.
To assess potential bias from machine-generated descriptions in the real-world evaluation, we additionally collect independently written human descriptions for 200 MM-AU test videos, covering 130 seen and 70 unseen samples. We evaluate MRA-VLM and competing baselines using both machine-generated and human-written descriptions on the same video subset, keeping model checkpoints, gallery composition, and evaluation metrics identical. MRA-VLM consistently outperforms the strongest baseline in T2V retrieval across both settings and all evaluation splits. With human-written captions, it improves overall T2V R@1 and mAP by 15.00% and 13.07%, respectively. These results provide complementary evidence of the effectiveness of synthetic visual supervision for real-world accident retrieval and its generalization to unseen categories.
We clarify the annotation verification procedures and audit statistics in Section 3.2 and Appendix A.2, and add the human-written caption evaluation in Appendix B.4 (Table 12).
Q3: Limited methodological novelty and unclear component effectiveness, particularly the similarities to existing RGB–flow fusion and retrieval refinement techniques, and the dominant contribution of inference-time prototype augmentation.
We thank the reviewer for this constructive comment. Our contribution lies in combining motion-aware representation learning with seen-category prototype augmentation and uncertainty-guided retrieval for open-vocabulary accident understanding, rather than introducing these operations as entirely new techniques.
Unlike conventional two-stream architectures that combine independently extracted RGB and motion features, our Maneuver-Aware Fusion Module uses cross-attention to extract motion cues relevant to visual appearance, followed by feature-wise gating to regulate their contribution. This design addresses the challenge of distinguishing accident-related motion from potentially unreliable optical-flow responses in egocentric videos. Table 4 shows that replacing this module with simple addition reduces overall T2V R@1 from 27.59% to 22.03% and V2T R@1 from 25.10% to 20.30%. Unlike reciprocal-neighbor re-ranking, which exploits neighborhood relationships to refine retrieval rankings, our method constructs visual and textual prototype banks exclusively from seen training categories and uses them to enhance query representations. Reciprocal inconsistency serves as a trigger for selectively incorporating the prototype-enhanced matching scores, rather than directly determining a neighborhood-based ranking. Unlike QB-Norm, which addresses retrieval-score bias through normalization, our method combines base and prototype-enhanced matching using a query-dependent weight derived from bidirectional disagreement.
We acknowledge that inference-time refinement contributes substantially to the overall improvement. This is an intended component of MRA-VLM, designed to exploit semantic and interaction patterns learned from seen categories when retrieval is ambiguous. Importantly, the complete framework improves unseen-category T2V and V2T R@1 from 30.47% and 27.63% for to 34.74% and 32.34%, respectively. Importantly, these drops are comparable to those caused by removing MAFM rather than overwhelmingly larger. This suggests that the final performance does not arise solely from inference-time prototype augmentation; instead, training-time motion-aware representation learning and inference-time ambiguity refinement make complementary contributions. The refinement is particularly useful for open-vocabulary generalization. Removing P&U decreases unseen-category T2V R@1 from 34.74% to 26.87% and V2T R@1 from 32.34% to 24.25%, corresponding to drops of 7.87 and 8.09 points, respectively. We interpret this result as evidence that seen-category prototypes provide useful semantic and interaction priors when an unseen query is ambiguous, rather than simply memorizing seen-category labels.
We clarify the distinctions from conventional RGB–flow fusion and existing retrieval refinement methods in the Related Work and Method sections, and further discuss the respective contributions of training-time fusion and inference-time refinement in Section 5.2.
We sincerely thank the reviewer for their constructive feedback. We have carefully addressed the reviewer's concerns below. The revised manuscript has been uploaded, with all changes highlighted in blue for easy reference.
Q1: Insufficient evidence for synthetic-to-real transfer, including marginal overall improvements despite larger seen/unseen gains, missing real-only, synthetic-real joint training and zero-shot baselines, and an unclear 0% training setting in Figure 4(a).
We thank the reviewer for these constructive comments. We acknowledge that the overall improvements over X²-VLM in Table 3 are modest (+0.45 and +0.45 percentage points in T2V and V2T R@1, respectively). However, our primary objective is to investigate whether synthetic video-text supervision supports both open-vocabulary generalization to unseen accident categories and transfer to real-world videos, rather than solely achieving large overall performance gains. In particular, MRA-VLM achieves larger improvements on unseen real-world accidents, outperforming the strongest conventional retrieval baselines by 4.16 and 3.88 percentage points in T2V and V2T R@1, respectively. Furthermore, Figure 4(a) shows that increasing synthetic training data consistently improves real-world retrieval performance, providing complementary evidence of the utility of synthetic supervision.
Training Data Comparisons. We appreciate the reviewer's suggestion. However, the original MM-AU dataset does not provide sufficiently complete video-text annotations for training our instance-level bidirectional retrieval model. We note that the same pipeline could in principle annotate an MM-AU training split we did not do so because pairs generated y a pipeline trained on Egocrash would not constitude an independent real-world baseline, and manually validating them at thr scale needed for training was not feasible within the rebuttal period. While we constructed video-text annotations for the 2,000-video real-world test set to enable evaluation, conducting (a) and (b) would additionally require annotating the MM-AU training portion at scale and manually validating these newly generated pairs to ensure supervision quality and consistency with EgoCrash. We were unable to complete this large-scale annotation and verification process within the rebuttal period. We therefore do not claim that the zero-shot experiments replace the requested real-only or joint-training comparisons.
Specifically, we add them as a complementary evaluation of whether EgoCrash supervision provides benefits beyond those already available from pretrained video-language models. We strengthen our evaluation by including recent pretrained video-language models, such as InternVideo2[1] and Perception Encoder[2], under a zero-shot setting without fine-tuning on either EgoCrash or MM-AU. All models are evaluated using the same real-world test set and retrieval protocol. Compared with zero-shot evaluation, training on EgoCrash substantially improves real-world retrieval performance (Table~10). Specifically, InternVideo2-Stage2-1B achieves T2V and V2T R@1 improvements of 14.50 and 9.90 percentage points, respectively, while Perception Encoder gains 3.30 and 5.70 points. Notably, these improvements also extend to unseen accident categories, demonstrating the potential of EgoCrash synthetic supervision to enhance real-world retrieval and generalization to previously unseen accident categories.
Clarification of Figure 3(a). The 0% setting represents the pretrained retrieval model evaluated on the real-world test set without fine-tuning on EgoCrash. As the proportion of EgoCrash training data increases, performance on real-world videos improves. This trend provides direct evidence that synthetic training data contributes to better real-world retrieval performance, even without target-domain fine-tuning. We also add the discussion in Section 5.2 .
cite: [1] Wang et al., InternVideo2: Scaling Foundation Models for Multimodal Video Understanding, ECCV 2024.
[2] Bolya et al., Perception Encoder: The Best Visual Embeddings Are Not at the Output of the Network, NeurIPS 2025.