acceptodds
Under review as a conference paper at ICLR 2027

Synthetic Accidents as Probes for Open-Vocabulary Egocentric Driving

Abstract

Fine-grained understanding of egocentric traffic accidents is essential for safe autonomous driving and requires large-scale, diverse training data. Synthetic supervision offers a scalable way to expand training coverage beyond what can be readily obtained from real accidents. To investigate whether such supervision can support generalizable accident understanding, we first introduce Ego-view Crashes (EgoCrash), a large-scale synthetic video-text benchmark comprising 20,000 egocentric accident videos across 56 fine-grained traffic-event categories, with rich multimodal annotations covering accident descriptions, question-answer pairs, motion cues, event categories, and entity trajectories. We further formulate open-vocabulary video-text retrieval for traffic accident understanding, where models retrieve accident videos and free-form descriptions bidirectionally and generalize from seen events to category-disjoint unseen events and real-world accident videos. To address this task, we propose maneuver-aware retrieval-augmented video-language matching (MRA-VLM). It learns motion-sensitive video representations by combining RGB appearance and optical-flow cues, and further enhances inference with prototype-based retrieval from seen events and uncertainty-guided score fusion, improving fine-grained matching and unseen-event generalization. In extensive experiments on EgoCrash, MRA-VLM consistently outperforms strong baselines in both retrieval directions and across both seen and unseen category splits, and models trained exclusively on synthetic data also transfer to real accident videos, demonstrating the potential of synthetic video-text supervision for generalizable accident understanding.

Primary area:
datasets and benchmarks
Keywords:
Egocentric Traffic Accident Understanding, Synthetic Video-Text Dataset, Open-Vocabulary Video-Text Retrieval, Vision-Language Matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2qGHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EgoCrash, a synthetic benchmark of 20,000 egocentric traffic-event videos across 56 categories. The authors fine-tune Wan2.1-T2V on 2,000 real accident videos, mainly from MM-AU, and generate new videos from LLM-augmented descriptions. They then filter the videos manually and annotate them with VLM-generated descriptions, accident labels, optical flow, entity trajectories, and QA pairs. The paper also defines an open-vocabulary video-text retrieval task. For this task, it proposes MRA-VLM, built on X²-VLM. MRA-VLM fuses RGB and optical-flow features with a gated cross-attention module. At inference, it enhances queries with prototypes from seen categories when the two retrieval directions disagree. MRA-VLM outperforms CLIP4Clip, DRL, X-CLIP, and X²-VLM on both synthetic and real test sets.

Critical strengths
  1. Important and timely problem. Real accident videos are rare and unevenly distributed. The paper directly studies whether synthetic videos can fill this gap, i.e., whether or not models trained only on synthetic data generalize to unseen categories and real videos.
  2. Large dataset with rich annotations. With 20,000 videos in 56 categories, EgoCrash is larger than existing egocentric accident datasets. It provides descriptions, accident labels, optical flow, trajectories, and QA pairs.
  3. Transparent discussion of dataset limitations. Appendix openly reports the category imbalance. It also notes that the semantic coverage of descriptions is measured only lexically, and that flow and trajectories are estimated rather than ground truth.
  4. Consistent improvements. On the synthetic test set (Table 2), MRA-VLM outperforms all baselines in both retrieval directions and on both seen and unseen categories. The additional inference cost is small, about 9%.
Critical weaknesses
  1. Weak evidence for synthetic-to-real transfer. On real videos, the overall gain over X²-VLM is only +0.3 to +0.45 points in R@1 (Table 3), although the gains on the seen and unseen subsets are larger (+2 to +4 points). In addition, important baselines are missing, such as training on real data only, combining real and synthetic data, or using pretrained models zero-shot. The condition for 0% training data in Fig. 3(a) is also not explained. As a result, the central claim that synthetic supervision transfers to real accidents is not well supported.
  2. Annotations and test ground truth are mostly generated by LLMs and VLMs. Category labels are assigned by a VLM from the text descriptions alone. The paper mentions manual audits and sample-based checks, but reports neither their scale nor their results. In addition, the descriptions for the real test set are generated with the same pipeline, which may favor models trained on similar text.
  3. Limited novelty and effectiveness of the method. The three components, namely gated RGB–flow cross-attention fusion, prototype-based query enhancement, and score fusion triggered by bidirectional inconsistency, are similar to existing techniques. Examples include two-stream RGB–flow fusion (Simonyan & Zisserman, 2014), reciprocal-neighbor re-ranking (Zhong et al., 2017), and test-time score normalization for retrieval (e.g., QB-Norm; Bogolin et al., 2022). Moreover, without the inference-time modules, MRA-VLM is worse than X²-VLM on most metrics (e.g., 21.52 vs. 22.15 in text-to-video R@1 from Table 4 and Table 2), so most of the gain comes from inference-time prototype retrieval.
  4. Outdated baselines and concerns about evaluation reliability. All baselines in Table 2 date from 2022–2023 . The paper does not compare with recent video-language models such as InternVideo2 (ECCV 2024), LanguageBind (ICLR 2024), VideoCLIP-XL (EMNLP 2024), or Perception Encoder (NeurIPS 2025). In addition, some numbers in Table 4 appear duplicated: the overall scores of “w/o R” match exactly the unseen scores of “w/o P&R”. Also, the number of categories in the real test set differs between the text (56) and Table 6 (58).
  5. Limited quality and length of the synthetic videos. Each video is only about 4 seconds long (41 frames at 10 fps). Apart from manual screening, there is no quantitative evaluation of whether the generated videos depict the described maneuvers and interactions correctly.
Questions for authors
  1. How does MRA-VLM, trained on EgoCrash, compare with (a) the same model trained on real MM-AU data (excluding the test set), (b) training on real and synthetic data together, and (c) recent pretrained models (e.g., InternVideo2, VideoCLIP-XL) used zero-shot? What does “0%” mean in Fig. 3(a)? (could change my assessment)
  2. What are the scale and results of the manual audits of category labels and descriptions (e.g., the number of samples checked and the error rate)? Can the authors report results on a subset of the real test set with human-written descriptions? (could change my assessment)
  3. How do recent video-language models, such as InternVideo2, LanguageBind, VideoCLIP-XL, or Perception Encoder, perform when fine-tuned on EgoCrash? (could change my assessment)
  4. Please check Table 4. The overall scores of “w/o R” (T2V R@1/mAP = 26.87/38.80, V2T R@1/mAP = 24.25/35.43) are identical to the unseen scores of “w/o P&R”. Is this coincidence? In addition, the notation is inconsistent: the caption defines the uncertainty-guided score fusion as “U”, but the row names use “R” (“w/o R” and “w/o P&R”). In addition, Table 6 lists 58 categories, while the text says 56. How are the MM-AU categories mapped to the seen and unseen splits? (clarification)
  5. Is there evidence that the gate in the Maneuver-Aware Fusion Module suppresses ego-motion responses, e.g., an analysis of gate values? (clarification)
5 comments

We sincerely thank the reviewer for their constructive feedback. We have carefully addressed the reviewer's concerns below. The revised manuscript has been uploaded, with all changes highlighted in blue for easy reference.

Q1: Insufficient evidence for synthetic-to-real transfer, including marginal overall improvements despite larger seen/unseen gains, missing real-only, synthetic-real joint training and zero-shot baselines, and an unclear 0% training setting in Figure 4(a).

We thank the reviewer for these constructive comments. We acknowledge that the overall improvements over X²-VLM in Table 3 are modest (+0.45 and +0.45 percentage points in T2V and V2T R@1, respectively). However, our primary objective is to investigate whether synthetic video-text supervision supports both open-vocabulary generalization to unseen accident categories and transfer to real-world videos, rather than solely achieving large overall performance gains. In particular, MRA-VLM achieves larger improvements on unseen real-world accidents, outperforming the strongest conventional retrieval baselines by 4.16 and 3.88 percentage points in T2V and V2T R@1, respectively. Furthermore, Figure 4(a) shows that increasing synthetic training data consistently improves real-world retrieval performance, providing complementary evidence of the utility of synthetic supervision.

Training Data Comparisons. We appreciate the reviewer's suggestion. However, the original MM-AU dataset does not provide sufficiently complete video-text annotations for training our instance-level bidirectional retrieval model. We note that the same pipeline could in principle annotate an MM-AU training split we did not do so because pairs generated y a pipeline trained on Egocrash would not constitude an independent real-world baseline, and manually validating them at thr scale needed for training was not feasible within the rebuttal period. While we constructed video-text annotations for the 2,000-video real-world test set to enable evaluation, conducting (a) and (b) would additionally require annotating the MM-AU training portion at scale and manually validating these newly generated pairs to ensure supervision quality and consistency with EgoCrash. We were unable to complete this large-scale annotation and verification process within the rebuttal period. We therefore do not claim that the zero-shot experiments replace the requested real-only or joint-training comparisons.

Specifically, we add them as a complementary evaluation of whether EgoCrash supervision provides benefits beyond those already available from pretrained video-language models. We strengthen our evaluation by including recent pretrained video-language models, such as InternVideo2[1] and Perception Encoder[2], under a zero-shot setting without fine-tuning on either EgoCrash or MM-AU. All models are evaluated using the same real-world test set and retrieval protocol. Compared with zero-shot evaluation, training on EgoCrash substantially improves real-world retrieval performance (Table~10). Specifically, InternVideo2-Stage2-1B achieves T2V and V2T R@1 improvements of 14.50 and 9.90 percentage points, respectively, while Perception Encoder gains 3.30 and 5.70 points. Notably, these improvements also extend to unseen accident categories, demonstrating the potential of EgoCrash synthetic supervision to enhance real-world retrieval and generalization to previously unseen accident categories.

Clarification of Figure 3(a). The 0% setting represents the pretrained retrieval model evaluated on the real-world test set without fine-tuning on EgoCrash. As the proportion of EgoCrash training data increases, performance on real-world videos improves. This trend provides direct evidence that synthetic training data contributes to better real-world retrieval performance, even without target-domain fine-tuning. We also add the discussion in Section 5.2 .

cite: [1] Wang et al., InternVideo2: Scaling Foundation Models for Multimodal Video Understanding, ECCV 2024.

[2] Bolya et al., Perception Encoder: The Best Visual Embeddings Are Not at the Output of the Network, NeurIPS 2025.

Q2: Insufficient quantitative validation of LLM/VLM-generated annotations, including the scale and error rates of manual audits for category labels and descriptions, and potential evaluation bias from machine-generated captions without human-written comparisons.

We thank the reviewer for raising these important concerns. As detailed in Appendix A.1, five annotators screened 48,000 generated videos, with each video independently reviewed by two annotators and disagreements resolved by a third. Based on visual quality, temporal consistency, traffic-interaction plausibility, and description alignment, 28,000 videos were excluded, retaining 20,000 samples (41.7%). Following video screening, we further verified the generated annotations. Accident-occurrence labels were cross-checked using independent video- and text-based predictions, with disagreements resolved through human inspection. Fine-grained category labels, initially assigned from textual descriptions, were also manually reviewed and corrected against the corresponding video content. Specifically, 20,000 category labels were inspected, with 2,487 errors identified and corrected (12.4% initial error rate). The finalized reviewed labels achieved 100% agreement with human adjudications. For video descriptions, we conducted five rounds of random-sampling-based verification, reviewing approximately 2,000 annotations per round and manually correcting identified inconsistencies. An additional random audit of 2,000 descriptions after these corrections yielded a consistency rate of 98%.

To assess potential bias from machine-generated descriptions in the real-world evaluation, we additionally collect independently written human descriptions for 200 MM-AU test videos, covering 130 seen and 70 unseen samples. We evaluate MRA-VLM and competing baselines using both machine-generated and human-written descriptions on the same video subset, keeping model checkpoints, gallery composition, and evaluation metrics identical. MRA-VLM consistently outperforms the strongest baseline in T2V retrieval across both settings and all evaluation splits. With human-written captions, it improves overall T2V R@1 and mAP by 15.00% and 13.07%, respectively. These results provide complementary evidence of the effectiveness of synthetic visual supervision for real-world accident retrieval and its generalization to unseen categories.

We clarify the annotation verification procedures and audit statistics in Section 3.2 and Appendix A.2, and add the human-written caption evaluation in Appendix B.4 (Table 12).

Q3: Limited methodological novelty and unclear component effectiveness, particularly the similarities to existing RGB–flow fusion and retrieval refinement techniques, and the dominant contribution of inference-time prototype augmentation.

We thank the reviewer for this constructive comment. Our contribution lies in combining motion-aware representation learning with seen-category prototype augmentation and uncertainty-guided retrieval for open-vocabulary accident understanding, rather than introducing these operations as entirely new techniques.

Unlike conventional two-stream architectures that combine independently extracted RGB and motion features, our Maneuver-Aware Fusion Module uses cross-attention to extract motion cues relevant to visual appearance, followed by feature-wise gating to regulate their contribution. This design addresses the challenge of distinguishing accident-related motion from potentially unreliable optical-flow responses in egocentric videos. Table 4 shows that replacing this module with simple addition reduces overall T2V R@1 from 27.59% to 22.03% and V2T R@1 from 25.10% to 20.30%. Unlike reciprocal-neighbor re-ranking, which exploits neighborhood relationships to refine retrieval rankings, our method constructs visual and textual prototype banks exclusively from seen training categories and uses them to enhance query representations. Reciprocal inconsistency serves as a trigger for selectively incorporating the prototype-enhanced matching scores, rather than directly determining a neighborhood-based ranking. Unlike QB-Norm, which addresses retrieval-score bias through normalization, our method combines base and prototype-enhanced matching using a query-dependent weight derived from bidirectional disagreement.

We acknowledge that inference-time refinement contributes substantially to the overall improvement. This is an intended component of MRA-VLM, designed to exploit semantic and interaction patterns learned from seen categories when retrieval is ambiguous. Importantly, the complete framework improves unseen-category T2V and V2T R@1 from 30.47% and 27.63% for X2−VLM{X^2-VLM} to 34.74% and 32.34%, respectively. Importantly, these drops are comparable to those caused by removing MAFM rather than overwhelmingly larger. This suggests that the final performance does not arise solely from inference-time prototype augmentation; instead, training-time motion-aware representation learning and inference-time ambiguity refinement make complementary contributions. The refinement is particularly useful for open-vocabulary generalization. Removing P&U decreases unseen-category T2V R@1 from 34.74% to 26.87% and V2T R@1 from 32.34% to 24.25%, corresponding to drops of 7.87 and 8.09 points, respectively. We interpret this result as evidence that seen-category prototypes provide useful semantic and interaction priors when an unseen query is ambiguous, rather than simply memorizing seen-category labels.

We clarify the distinctions from conventional RGB–flow fusion and existing retrieval refinement methods in the Related Work and Method sections, and further discuss the respective contributions of training-time fusion and inference-time refinement in Section 5.2.

Reviewer_wVAUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EgoCrash, a synthetic egocentric traffic-event video-text benchmark with 20,000 videos and multimodal annotations. It formulates category-disjoint video-text retrieval and proposes MRA-VLM, combining RGB-flow fusion with prototype-augmented matching. The method is evaluated on seen and unseen synthetic events and transferred directly to real MM-AU videos.

Critical strengths

The paper provides a concrete synthetic-data pipeline and a useful retrieval-oriented benchmark formulation. Sections 3.1-3.3 specify a sizeable synthetic collection, category-disjoint splits, captions, flow, trajectories, and QA annotations, while the instance-level retrieval protocol is precisely defined in Appendix B.1. This is a practically relevant framing for studying whether synthetic accident data can support language-grounded understanding beyond a fixed closed-set label space.

The proposed method is evaluated with a coherent set of component ablations.

Critical weaknesses

The paper’s empirical evidence does not yet convincingly establish the claimed synthetic-to-real advantage.

the overall gains over the strongest reported baseline are only 0.30 percentage points for T2V R@1, from 12.65 to 12.95, and 0.45 points for V2T R@1, from 10.40 to 10.85. No seed-wise variation, confidence intervals, or statistical test is reported, so it is impossible to determine whether these narrow overall differences are meaningful rather than run-to-run noise. This matters because direct transfer to real accidents is a headline claim in the abstract and conclusion. Reporting repeated-run statistics for all methods, and comparing against a strong driving-specific retrieval method, would substantially increase confidence.

The benchmark’s annotation reliability is insufficiently quantified for a dataset paper whose contribution rests on rich semantic supervision. Section 3.2 states that captions are generated by GPT-4o and Gemini and selected by Qwen-VL, event labels are assigned by Qwen2.5-VL based on captions, and QA answers and labels receive only sample-based human checks or review of disagreement cases. The paper provides no held-out human annotation study, error rate, agreement statistic, or stratified assessment by category, despite the captions and labels defining both the paired retrieval task and the seen/unseen evaluation. The current procedure may be reasonable, but the evidence does not establish the fidelity of the released annotations. A modest independently annotated subset with caption-consistency, accident-occurrence, and category-label agreement would resolve this concern.

Questions for authors

were all methods run with multiple random seeds under identical tuning budgets? Please report mean and variation or confidence intervals for the real-world results.

What category mapping or filtering produces the reported seen/unseen real-world results in table 3 ?

6 comments

We sincerely thank the reviewer for their constructive feedback and for recognizing the contributions of EgoCrash, particularly its synthetic data construction pipeline, well-defined retrieval benchmark, and systematic component ablations. We have carefully addressed the reviewer's concerns below. The revised manuscript has been uploaded, with all changes highlighted in blue for easy reference.

Q1: Limited synthetic-to-real gains and insufficient statistical validation, with additional concerns about evaluation fairness, driving-specific baselines, and the strength of transfer claims.

We thank the reviewer for raising this important concern. We conduct additional experiments over 3 independent random seeds for MRA-VLM, using consistent training and hyperparameter settings wherever applicable. We report the mean and standard deviation in Table 2 and 3. All of the results in two tables show the observed improvements are stable and not sensitive to random initialization.

Driving-specific Baselines. We thank the reviewer for the suggestion. We examined recent driving-specific retrieval methods, including CARIM (ICCV 2025)[1] and STRIVE-D (NeurIPS 2026) [2], but found no publicly available complete implementations or pretrained checkpoints. STRIVE-D releases event annotations and visualization tools, but not the full retrieval pipeline. Moreover, both methods adopt retrieval settings different from our instance-level bidirectional video-text matching task, making direct reproduction and comparison non-trivial. We therefore include InternVideo2-Stage2-1B under the same EgoCrash evaluation protocol (see Appendix B.3) and acknowledge the lack of a directly comparable driving-specific baseline as a limitation.

Our primary objective is to investigate whether synthetic video-text supervision supports both generalization to category-disjoint unseen accident events and transfer to real-world videos. All retrieval models are trained exclusively on synthetic EgoCrash data using seen categories and evaluated directly on real-world videos without further training. Compared with the strongest baseline for each metric, MRA-VLM improves unseen T2V R@1 from 11.18 to 15.34 (+4.16 points) and mAP from 20.38 to 25.77 (+5.39 points), while unseen V2T R@1 improves from 10.03 to 13.91 (+3.88 points) and mAP from 19.46 to 22.69 (+3.23 points). Furthermore, Figure 4(a) shows consistent improvements in real-world retrieval performance as the amount of synthetic training data increases, providing complementary evidence of the transferability of synthetic supervision.

We incorporate the multi-seed evaluation and additional baseline comparisons into Section 5.1 (Table 2 and 3) and Appendix B.3 (Table 9 and 10). We also clarify the scope of our synthetic-to-real transfer claims in the Section 5.1.

cite: [1] Ki et al., CARIM: Caption-Based Autonomous Driving Scene Retrieval via Inclusive Text Matching, ICCV, 2025.

[2] Yao et al., Driving Video Retrieval for Complex Queries with Structured Grounding, arXiv:2606.09109, 2026.

Q2: Insufficient quantitative validation of semantic annotation reliability, including caption consistency, accident-occurrence accuracy, fine-grained category-label correctness, and human agreement across categories.

We thank the reviewer for this important suggestion. As detailed in Appendix A.1, five annotators screened 48,000 generated videos, retaining 20,000 samples after quality control. We further conducted five rounds of random-sampling-based verification of the generated descriptions, reviewing approximately 2,000 annotations per round and manually correcting identified inconsistencies. An additional post-correction audit of 2,000 descriptions yielded a consistency rate of 98%. Accident-occurrence and fine-grained category labels were also manually inspected and corrected. Specifically, 20,000 category labels were inspected, with 2487 errors identified and corrected (about 12.4% initial error rate).

To further assess the reliability of our automatic annotation pipeline, we randomly sample 200 videos from the MM-AU real-world test set, covering both seen and unseen accident categories. Importantly, we do not use the original MM-AU annotations in this evaluation. Instead, we apply the same VLM-based annotation pipeline used for EgoCrash to generate video descriptions, accident-occurrence labels, and fine-grained category labels. Two human annotators then independently annotate these videos without access to either the automatically generated labels or the original MM-AU labels. We compare the automatically generated annotations against the independent human judgments to assess their reliability. We report caption consistency, accident-occurrence and category-label accuracy, and inter-annotator agreement, together with category-stratified results.

Table 1 presents the quantitative results. The evaluation yields 90.0% human-confirmed accident occurrence, 89.0% category-label agreement, and 80.0% caption consistency, with an overall inter-annotator agreement of 95.5%. Results are slightly lower for unseen categories, particularly in category-label agreement (85.7% versus 90.8% for seen categories). For the 200 real-world test videos, the category and accident-occurrence labels are retained from MM-AU and mapped to the 56 category Egocrash taxonomy for splitassignment, while descriptions and optical flow are regenerated with the Egocrash pipeline. The 200-video study in Table 1 separately measures the pipeline's own error rate by regenerating labels and descriptions from scratch, without access to the MM-AU annotations. The additional evaluation provides quantitative evidence of the annotation pipeline's reliability, while also indicating that generating accurate fine-grained descriptions remains more challenging. Metric

Table 1 : Human Annotation Quality Verification on Real-video-200

MetricOverallSeenUnseen
Number of samples20013070
Caption consistency (%)80.080.878.6
Accident-occurrence agreement (%)90.090.888.6
Category-label agreement (%)89.090.885.7
Human inter-annotator agreement (%)95.596.294.3

Table 2: Per-category Agreement Between VLM-generated and Human-verified Annotations

N denotes the number of samples, Correct denotes the number of matching annotations, and Agr. denotes agreement (%).

CategoryNCorrectAgr. (%)CategoryNCorrectAgr. (%)
ego-car hits a car181794.4ego-car hits a cyclist2150.0
car hits car1717100.0car hits telegraphpoles22100.0
truck hits car131292.3ego-car hits other road facilties2150.0
ego-car hits a truck131292.3ego-car hits trees22100.0
ego-car hits a curb121083.3ego-car hits a pedestrian22100.0
car hits a curb88100.0cyclist falls down22100.0
car scratches truck7685.7truck hits trees11100.0
car hits pedestrian6583.3motorbike hits other road facilities100.0
truck hits truck66100.0truck hits telegraphpoles11100.0
car hits motorbike66100.0motorbike hits motorbike100.0
ego-car hits a crossing car55100.0truck hits a curb100.0
ego-car is overtaken by a car55100.0motorbike scratches truck11100.0
ego-car hits large roadblocks5360.0ego-car hits a crossing pedestrian11100.0
ego-car hits a crossing motorbike55100.0motorbike hits large roadblocks11100.0
car is out of control44100.0truck hits large roadblocks11100.0
motorbike falls down44100.0car hits trees11100.0
car hits cyclist44100.0truck hits pedestrian11100.0
truck is out of control33100.0truck hits other road facilities11100.0
ego-car hits small roadblocks road pothholes33100.0truck hits motorbike11100.0
ego-car is out of control33100.0motorbike hits a curb11100.0
ego-car is overtaken by a truck3266.7ego-car hits a crossing cyclist11100.0
ego-car hits a motorbike33100.0ego-car has failure of components11100.0
no accident3133.3ego-car hits telegraphpoles11100.0
motorbike scratches car22100.0car has failure of components11100.0
pedestrian falls down2150.0motorbike hits pedestrian11100.0
car hits other road facilities22100.0cyclist hits cyclist11100.0
truck has failure of components2150.0truck hits cyclist11100.0
car hits large roadblocks200.0motorbike hits cyclist100.0

Overall agreement: 178/200 (89.0%).

Per-category Annotation Reliability. We further evaluate category-level agreement on the 200-video subet (Table 2). The VLM-generated labels agree with human annotations on 177 samples (89%), covering 56 categories. Frequently represented categories show high agreement, such as ego-car hits a car (94.4%) and car hits car (100%). Lower agreement occurs in some categories, particularly no accident (33.3%). Results for categories with few samples should be interpreted cautiously.

We will report somecases in 200 human-annoated set in this link: https://anonymous.4open.science/r/Synthetic-Accidents-as-Probes-for-Open-Vocabulary-Egocentric-Driving-41DA/

Reviewer_VNB4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoCrash, a synthetic dataset of 20,000 first-person accident videos across 56 categories, annotated with accident descriptions, QA pairs, tracks, event categories, and optical flow to address the lack of real crash data. Leveraging this dataset, the authors formulate accident understanding as an open-vocabulary video-text retrieval task and propose MRA-VLM, a model that integrates RGB frames with optical flow and prototype retrieval to match crash videos with descriptive text. The method is evaluated on both synthetic seen/unseen categories and real-world dashcam records from MM-AU.

Critical strengths
  1. Using synthetic simulation to generate rare, safety-critical driving events and hazardous corner cases is well-motivated and addresses a fundamental bottleneck in autonomous driving safety.
  2. EgoCrash provides solid scale (20k videos, 56 categories) and rich annotations, including captions, QA pairs, tracks, and optical flow, making it a useful resource for future research.
  3. Evaluating on category-disjoint unseen events and direct transfer to real-world videos is an effective way to test generalisation capabilities.
  4. Formulating accident understanding as an open-vocabulary video-text retrieval task is a novel and interesting angle for traffic accident benchmarks, which traditionally focus only on classification or anticipation.
Critical weaknesses
  1. Inaccurate & Incomplete Dataset Comparison: The marking of AC (Accident Category) as absent for DADA-200 and MM-AU is not correct. Both include fine-grained annotations 54 and 58, respectively. Also there are still missing 2 major dates such as DoTA [1] for real-world scenarios, and DeepAccident [2] for synthetic data, as both provide rich object tracking and fine-grained categories.
  2. Outdated Baselines: Evaluating solely against 2022-2023 (CLIP4Clip [2022], X-CLIP [2022], DRL [2022], and X2X^2-VLM [2023]) CLIP-based baselines are insufficient. Modern video-language foundation models (e.g., InternVideo2, Video-LLaVA, LanguageBind, or ViCLIP) are missing.
  3. Limited Visuals & Simulation Artefacts: Lack of visualisation and examples of annotation from the dataset. More importantly, the quality of a few visualisations looks poor in Figure 7. No visible physical destruction during the crash, cars appearing from other cars and from the fence, motorbikes staying still after the accident, and inconsistent vehicles / objects. These are generally important to catch cues leading to real accidents. Authors should provide a few videos or more images as supplementary to convince about quality.
  4. Contradictory Cross-Category Results: In cross-category evaluation, baselines maintain stable performance on a much larger distractor gallery, whereas MRA-VLM drops drastically. This unexplained discrepancy casts doubt on either evaluation consistency or the method’s robustness.
  5. Questionable Synthetic-to-Real Transfer: In Table 3 (overall), MRA-VLM improves over X2X^2-VLM by merely +0.30% in T2V R@1 (12.95% vs. 12.65%) and +0.45% in V2T R@1. Such marginal differences fall within standard evaluation noise and do not substantiate claims of effective synthetic-to-real transfer.

Minor:

  1. Move Figure 4 to the main manuscript; it is cited there and essential for illustrating the dataset construction pipeline for the readers.
  2. Typo in Appendix A.3: Correct "DATASET STATIC ANALYSIS" to "DATASET STATISTICAL ANALYSIS".

[1] Yao, Yu, et al. "DoTA: Unsupervised detection of traffic anomaly in driving videos." IEEE transactions on pattern analysis and machine intelligence 45.1 (2022): 444-459.

[2] Wang, Tianqi, et al. "Deepaccident: A motion and accident prediction benchmark for v2x autonomous driving." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 6. 2024.

Questions for authors
  1. Could you clarify why 'AC' is marked as absent for DADA-2000 and MM-AU in Table 1 when both already have fine-grained categories (54 and 58, respectively)? Also, how does EgoCrash compare in scale, tracking, and modalities against other key accident benchmarks like DoTA [1] and DeepAccident [2] dataset?
  2. The evaluated baselines are primarily older CLIP-based models from 2022–2023. Have you benchmarked or compared MRA-VLM against more recent video-language models (e.g., InternVideo2, Video-LLaVA, ViCLIP, …)?
  3. Can the authors provide uncompressed video clips or more images in the supplementary material to demonstrate that the visual quality and physical plausibility of the synthetic crashes are sufficient despite the artefacts visible in Figure 7. ?
  4. Could you explain why the baselines barely drop in the larger gallery with extra distractors, while MRA-VLM takes such a big hit? Were all models evaluated using the exact same gallery setup?
  5. In Table 3, the overall synthetic-to-real gain over X2X^2-VLM is +0.30% in T2V R@1. Have you conducted multi-seed runs with error bars or significance testing to confirm this gain is not simply random evaluation variance?
5 comments

We sincerely thank the reviewer for their constructive and insightful feedback, as well as their recognition of the motivation underlying EgoCrash, its rich annotations, and the importance of evaluating generalization to unseen accident categories. We have carefully addressed the reviewer's concerns and have uploaded revised versions. All revisions are highlighted in blue to facilitate review.

Q1: Inaccurate AC annotations for DADA-2000 and MM-AU, missing comparisons with DoTA and DeepAccident, and insufficient discussion of dataset scale, tracking, and modalities.

We thank the reviewer for pointing out these inaccuracies and the missing datasets in our comparison. We acknowledge that the original Table 1 incorrectly indicated the absence of accident-category annotations in DADA-2000 and MM-AU. We have corrected these entries and expanded Table 1 to include DoTA and DeepAccident. Regarding the comparison, EgoCrash contains 20,000 videos, compared with 4,677 in DoTA, while DeepAccident adopts a multi-view, scenario-based structure that is not directly comparable in terms of video counts. All three datasets provide tracking annotations, but their research focuses differ: DoTA emphasizes real-world traffic anomalies, DeepAccident targets multi-view V2X perception and prediction, whereas EgoCrash focuses on egocentric accident understanding with video descriptions, question-answer pairs, optical flow, and accident-category annotations. We have updated Table 1 and the Related Work section (Lines 142 and 146) to clarify these differences.

Q2: Baselines are insufficient and Modern video-language foundation models are missing.

We thank the reviewer for this valuable suggestion. We agree that comparisons with recent video-language foundation models are important for a more comprehensive evaluation of MRA-VLM. To strengthen the comparison with modern video-language foundation models, we add InternVideo2-Stage2-1B[1] and Perception Encoder (671M)[2], two representative large-scale models with strong video-text alignment capabilities and support for similarity-based retrieval. Video-LLaVA [3] is a generative model and does not yield a retrieval embedding under our protocol; LanguageBind [4] and ViCLIP [5] are covered by the same video-language pretraining family as InternVideo2, which we include as the stronger representative. These models were selected to provide challenging foundation-model baselines under our bidirectional retrieval protocol. We evaluate them using the same data splits, candidate galleries, and T2V/V2T metrics, with results reported in Appendix B.3 (Tables 9–10).

we report two settings: synthetic-seen-to-unseen, synthetic-to-real. Both foundation models outperform MRA-VLM under the same evaluation protocol, highlighting the advantages of large-scale pretrained video-language representations while providing stronger baselines for assessing our benchmark. InternVideo2-1B achieves the strongest performance across these settings. More importantly, synthetic training also improves retrieval performance on unseen accident categories in the real-world test set. For InternVideo2-1B, unseen-category T2V R@1 increases from 25.21% to 40.30% (+15.09 points), while V2T R@1 improves from 28.12% to 35.04% (+6.92 points). Corresponding mAP gains reach 13.47 and 12.34 points, respectively. Perception Encoder also benefits from synthetic training, achieving unseen-category improvements of 2.72 and 5.58 points in T2V and V2T R@1. These consistent gains suggest that EgoCrash provides transferable supervision beyond the accident categories observed during training, supporting its utility for open-vocabulary accident retrieval in real-world scenarios.

cite: [1] Wang et al., InternVideo2: Scaling Foundation Models for Multimodal Video Understanding, ECCV 2024.

[2] Bolya et al., Perception Encoder: The Best Visual Embeddings Are Not at the Output of the Network, NeurIPS 2025.

[3] Lin et al., Video-LLaVA: Learning United Visual Representation by Alignment Before Projection, EMNLP 2024.

[4] Zhu et al., LanguageBind: Extending Video-Language Pretraining to N-Modality by Language-Based Semantic Alignment, ICLR 2024.

[5] Wang et al., InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (ViCLIP), ICLR 2024.

Q3: Limited visual examples and concerns about simulation artifacts and physical plausibility in Figure 7. Additional images or uncompressed videos are requested to demonstrate dataset quality.

We thank the reviewer for raising these important concerns regarding the visual quality and physical plausibility of EgoCrash. We have carefully re-examined the examples in Figure 7&8 and provide the following clarifications. (1) Physical damage. Most collisions in Figure 7&8 occur at relatively low speeds, where visible vehicle deformation may be minimal. We acknowledge that EgoCrash does not fully reproduce real-world collision mechanics, particularly vehicle deformation and complex post-impact dynamics, due to the limitations of current video generation models. However, EgoCrash focuses on short-duration accident understanding, accident recognition, and video-text retrieval rather than detailed physical simulation. Therefore, the absence of visible damage alone does not invalidate an accident or its corresponding annotation.

(2) Object appearance and clipping. The apparent emergence of vehicles from other vehicles is caused by temporary occlusion by foreground objects. Similarly, the vehicle appearing to pass through a fence actually enters through an opening. We will provide consecutive frames and original videos to clarify these cases.

(3) Post-impact behavior. In some examples, the limited displacement of motorbikes after a collision may be explained by low-speed contact or their initially stationary state. Thus, remaining nearly stationary does not necessarily indicate physically implausible behavior. Nevertheless, we acknowledge that current video generation models may not accurately reproduce complex post-impact dynamics, such as realistic displacement, rotation, or falling motions.

(4) Object consistency. As the ego-vehicle moves forward, the visible road scene continuously changes, and surrounding vehicles may enter or leave the camera's field of view. We acknowledge that some generated videos may exhibit minor changes in vehicle appearance or geometry, particularly around collision events. However, these visual artifacts do not necessarily affect the underlying accident semantics. In cases where the accident participants remain identifiable, the collision event is preserved, and the accident category and textual description remain consistent, such artifacts primarily affect visual fidelity rather than annotation reliability. To further clarify these observations, we will supplement Figure 7&8 with consecutive frames and additional original video examples, allowing a more direct assessment of the collision sequences, object interactions, and temporal consistency.

All videos in EgoCrash undergo quality inspection during dataset construction, and only those meeting our quality criteria are retained. To further address the reviewer's concerns, we additionally assess the prevalence and potential impact of simulation artifacts in the curated dataset. Specifically, we randomly select 200 videos and examine them for visual consistency, collision plausibility, and post-impact behavior. One human annotators then independently review these videos. The results show that 38% exhibit noticeable simulation artifacts, while 0% contain artifacts that affect accident interpretation or annotation reliability. ALL visible simulation artifacts do not materially affect accident interpretation. These findings suggest that although minor simulation artifacts may remain, their impact on accident semantics and annotation reliability is limited.

We have expanded the supplementary materials with original-resolution video clips, additional key frames, and annotation visualizations covering different accident categories. We have updated Figure 7&8 and the corresponding discussion in Appendix B.6. These examples illustrate complete accident sequences, including pre-impact interactions, collisions, and post-impact behavior, allowing direct inspection of both visual quality and annotation consistency. We acknowledge that the current simulation-based accident generation does not fully reproduce real-world collision dynamics, particularly vehicle deformation and complex post-impact interactions. We have clarified these limitations in the revised manuscript and outlined potential improvements in Appendix B.7.

We will provide more videos in this link:https://anonymous.4open.science/r/Synthetic-Accidents-as-Probes-for-Open-Vocabulary-Egocentric-Driving-41DA/ .

Q4: Inconsistent cross-category retrieval results, with MRA-VLM showing a substantial performance drop on a larger distractor gallery while baselines remain stable, raising concerns about evaluation consistency and model robustness.

We thank the reviewer for raising this concern. We rechecked the evaluation pipeline and confirm that all methods are evaluated using exactly the same query/gallery construction within each setting. As clarified in Appendix B.1 (Table 7), the real-world evaluation contains 1,302 seen pairs, 698 unseen pairs, and their union of 2,000 pairs for the overall setting, with one positive match per query. Thus, every method in Table 3 is evaluated on the same 1,302/698/2,000 query-gallery sets, respectively.

We also agree that the drop of MRA-VLM in Table 3 is larger than that of the conventional baselines. Specifically, when moving from the seen split to the overall setting, its T2V R@1 decreases from 15.76 to 13.05 (-2.71 points), compared with 13.85 to 12.65 (-1.20) for X$^2$-VLM and 12.99 to 11.78 (-1.21) for DRL. In contrast, on the synthetic test set (Table 2), the corresponding reductions of MRA-VLM and X$^2$-VLM are comparable (-7.95 vs. -8.15 points).

Q5.Limited synthetic-to-real improvements and insufficient statistical validation, including multi-seed evaluation and significance testing.

We thank the reviewer for raising this important concern. We evaluate MRA-VLM and the competing methods over 3 independent training seeds under the same experimental protocol. The results are reported as mean ± standard deviation in Table 2 and 3. We would like to clarify that our primary objective is to investigate whether synthetic video-text supervision can support generalization to real-world accident understanding, rather than to claim substantial improvements over existing methods in the real-world setting. All retrieval models are trained exclusively on synthetic EgoCrash data and evaluated directly on held-out real-world videos, without real-world retrieval training. In Table 3, MRA-VLM improves unseen-category T2V and V2T R@1 by 4.16 and 3.88 percentage points over the strongest conventional retrieval baselines, respectively, demonstrating its effectiveness under the combined challenges of unseen-category generalization and synthetic-to-real domain shift. Furthermore, Figure 4(a) shows that increasing synthetic training data consistently improves real-world retrieval performance, providing additional evidence of the transferability of synthetic supervision. We add the multi-seed evaluation results and statistical analysis in Section 5.1 (Table 2and 3). We also clarify the scope of our synthetic-to-real transfer claims in Section 5.1.

Importantly, the overall evaluation is not constructed by simply adding distractors to a fixed set of seen queries: both the query set and candidate gallery are changed to the union of seen and unseen samples. Therefore, the difference between category-specific and overall performance cannot be attributed solely to gallery size. Nevertheless, the larger relative drop of MRA-VLM on the real-world set indicates that our method is more sensitive to cross-category ambiguity when it is combined with synthetic-to-real domain shift. We now explicitly acknowledge this behavior as a limitation rather than attributing it to evaluation inconsistency.

We have revised Appendix B.1 to make the query/gallery construction explicit and added the above quantitative comparison and limitation discussion to Section 5.1.

Q6. Move Figure 4 and fix typo.

The constrution pipeline figure (previously Figure4 inAppendix A.2) is now Figure 2 in Section 3. We have also corrected the Appendix A.3 heading from 'Dataset Static Analysis' to 'Dataset Statistical Analysis'.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers: Summary of Revisions

We are grateful to all reviewers for their constructive feedback. The questions and concerns raised in the reviews have substantially improved our paper. We have addressed each comment point by point in our detailed responses. The revised manuscript has been uploaded, with all changes highlighted in blue. The major revisions include:

  1. Expanded baseline comparisons: Added InternVideo2-Stage2-1B and Perception Encoder, including zero-shot and EgoCrash fine-tuning evaluations.

  2. Strengthened dataset quality assessment: Provided additional original-resolution videos, consecutive frames, and annotation visualizations, together with supplementary artifact analysis during the rebuttal.

  3. Improved annotation reliability assessment: Added manual verification statistics, independent human annotation evaluation, per-category agreement analysis, and human-written caption retrieval experiments.

  4. Strengthened experimental evaluation: Added three-seed results, clarified query/gallery construction, and provided further cross-category and synthetic-to-real analyses.

  5. Clarified methodological contributions: Expanded discussions of MAFM and prototype-based retrieval refinement, supplemented component effectiveness analyses, and corrected ablation results.

  6. Improved dataset documentation: Clarified the EgoCrash–MM-AU category mapping, seen/unseen splits, dataset statistics, and video-text semantic alignment.

  7. Improved manuscript presentation: Updated figures, tables, supplementary videos, and related discussions throughout the manuscript.

Clarification on EgoCrash's Reliability and Intended Scope.

First, we have narrowed our claims: EgoCrash is not a physically accurate collision simulator, but a source of synthetic accident-related visual-semantic supervision for open-vocabulary retrieval and real-world generalization. We explicitly acknowledge remaining artifacts, including geometric inconsistencies, occasional vehicle interpenetration, and imperfect collision responses.

Second, accident validity was assessed using full available video clips rather than isolated frames. Close vehicle proximity alone was insufficient; annotators considered collision evidence and temporal cues such as deceleration, trajectory changes, and post-impact responses. We have revisited the Figure 7 examples, acknowledged their limitations, and provided original videos for inspection.

Third, the 200-video artifact diagnostic was conducted as an exploratory check during the rebuttal in response to Reviewer VNB4, not as a formal experiment in the original manuscript. This exploratory check was performed by a single annotator. Given its limited scale and single-annotator nature, we do not treat these results as a definitive estimate of artifact prevalence or semantic reliability across the dataset.

Finally, experiments with two independent foundation models demonstrate EgoCrash's transferability beyond MRA-VLM. On real-world unseen categories, EgoCrash training improves InternVideo2-1B's T2V/V2T R@1 by 15.09/6.92 percentage points and Perception Encoder's by 2.72/5.58 points, respectively, with consistent mAP gains. These results provide independent evidence that EgoCrash offers transferable accident-related visual-semantic supervision beyond our proposed model.

Taken together, these results suggest that despite inevitable generation artifacts, EgoCrash retains useful accident-related semantic information for retrieval training and evaluation. Moreover, its unified evaluation protocol enables meaningful comparisons of the relative retrieval performance of different models. We therefore position EgoCrash as a practical benchmark for accident-related semantic retrieval and transfer, rather than a physically faithful collision simulator or a definitive measure of absolute real-world capability.

We sincerely thank the reviewers for helping us clarify both the limitations and contributions of EgoCrash.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.