acceptodds
Under review as a conference paper at ICLR 2027

MMHU: A Large-Scale Multimodal Benchmark for Human Behavior Understanding in Autonomous Driving

Abstract

Humans are integral components of the transportation ecosystem, and understanding their behaviors is crucial to facilitating the development of safe driving systems. Although recent progress has explored various aspects of human behavior—such as motion, trajectories, and intention — a comprehensive benchmark for evaluating human behavior understanding in autonomous driving remain unavailable. In this work, we propose MMHU, a large-scale benchmark for human behavior analysis featuring rich annotations, such as human motion and trajectories, text description for human motions, human intention, and critical behavior labels relevant to driving safety. Our dataset encompasses 57k human motion clips and 1.73M frames gathered from diverse sources, including established driving datasets such as Waymo, in-the-wild videos from YouTube, and self-collected data. A human-in-the-loop annotation pipeline is developed to generate rich behavior captions. We provide a thorough dataset analysis and benchmark multiple tasks—ranging from motion prediction to motion generation and human behavior question answering—thereby offering a broad evaluation suite. Our dataset will be released to promote further human-centric research in this vital area of autonomous driving.

Primary area:
datasets and benchmarks
Keywords:
Autonomous Driving, Human Behavior

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_QqyTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MMHU, a large-scale dataset for human behavior understanding in driving scenes. It contains approximately 57K human motion clips and 1.73M frames. The annotations include 3D motion, trajectories, text descriptions, and 13 behavior labels. The authors evaluate several downstream tasks and show that training with MMHU improves baseline performance.

Critical strengths
  1. MMHU provides aligned video, motion, language, and behavior annotations at a substantial scale. This is a useful resource for studying human behavior in driving scenes. Building the dataset also represents considerable collection and annotation effort.

  2. The experiments cover several tasks and a range of baseline models. The improvements on 3DPW and JAAD provide evidence that MMHU can benefit existing methods.

Critical weaknesses
  1. The methodological novelty is limited. Text annotation mainly combines existing pose-description tools with LLM/VLM prompting. Behavior VQA uses templated binary questions. The paper provides little evidence of how much the specific pipeline designs contribute.

  2. The annotation and fine-tuning settings are not fully clear. The paper should explain the relationship between the annotation VLM and the model evaluated in Table 5. It should also separate the gains from human annotations and VLM-generated labels.

  3. Some aspects of dataset quality need further validation. The reported 97% agreement only covers the human-labeled subsets. The quality of automatic labels and text descriptions remains less clear. Motion-label validation is mainly qualitative, and the choice of the 13 behaviors needs more justification.

Questions for authors
  1. Which model generates the pseudo-labels, and which model is evaluated in Table 5? Please include comparisons using human annotations only, pseudo-labels only, and both.

  2. Does adding PoseScript-based descriptions improve caption quality or downstream performance? A comparison with directly prompting a VLM with video frames would help.

  3. Could the authors report an independent quality assessment of the automatic labels and captions? Per-class behavior results and quantitative motion-label validation would be useful.

  4. How were the 13 behaviors selected? Were domain experts involved, and what criteria were used to include or exclude behaviors?

  5. See weaknesses.

Reviewer_WG5AOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MMHU, a human behaviour focused benchmark for autonomous driving. The authors curate 57k human instances (1.73M frames) from Waymo, Youtube driving videos and self collected footage, and annotate each instance with SMPL motion and trajectory, low and high level text descriptions, and 13 binary "critical behaviour" labels (crossing, using phone, riding bike, etc.). Behaviours are discovered by prompting a VLM, then labelled with a human-in-the-loop pipeline: 9.5k instances by humans, 47k by a Qwen2.5-VL-3B fine-tuned on the human labelled subset, with an 840 instance human labelled test set. The paper benchmarks motion prediction, text-to-motion generation and behaviour VQA, and shows that adding MMHU to training improves PhysMoP on 3DPW, TrEP on JAAD, MotionDiffuse FID and Qwen2.5-VL on behaviour VQA.

Critical strengths
  • A unified benchmark with motion, trajectory, text and behaviour labels in a traffic context is highly essential in the field of autonomous driving, especially to understand human behaviour and characteristics. Existing driving datasets cover either generic perception tasks or a single human aspect, and the authors have done a good job of identifying this gap.
  • The authors found a bounded solution to curate 57k human instances with a lot of variety in terms of behaviour in public traffic situations. The transfer experiments (Tab. 6, 7, 8) show that MMHU helps on external test sets, which is the right kind of evidence for a benchmark paper.
  • The paper is easy to follow, Fig. 2 gives a clear picture of the pipeline and Tab. 1 positions the dataset well against related work.
  • The effort in sourcing open source videos and self collected data to fit to a traffic context is commendable. The open sourced dataset is a good start for the AD field to explore human behaviour beyond the binary crossing label.
Critical weaknesses
  • Missing frame completion. The missing/occluded frame correction in extracted SMPL on Waymo, Youtube or self collected data follows an "angle-based" interpolation (Eq. 1), and missing trajectory is done via "linear interpolation" (Eq. 3). There is skepticism in the quality of these filled frames. The paper would benefit massively by showing the statistics here, i.e. % of interpolated frames and the max gap length, and by flagging these frames in the release. Eq. 1 also applies slerp on the full concatenated axis-angle vector rather than per joint. An occlusion-aware recovery such as GLAMR (already cited) or a learned motion infiller (VPoser style prior) would offer significantly higher benefits than interpolation.
  • Critical behaviours. The identified critical behaviours are mentioned as 13 categories, however the core research question of "what aspects of human behaviour in AD is critical" is unanswered. The paper explores VLM routes and "identifies" 13 categories, but these come from a VLM that is not specified, on n clips from Waymo only (1.7 of 78 hours). Whether these are "available" or essentially critical is under explored. Fig. 3b shows the distribution dominant in crossing (~48%), while wheelchair is 0.11%. Labels are multi-label per instance, but co-occurrence between categories (e.g. phone + crossing) is never reported, and the paper does not answer this clearly.
  • Label quality of the VLM labelled subset. The majority of the dataset (MMHU-V, 47k instances) is labelled by the fine-tuned Qwen2.5-VL-3B, however how good this annotator actually is on the human labelled data is not shown anywhere. The 97% agreement in Sec. D.6 is only between human annotators on the human labelled subset, it says nothing about the VLM labels. This is a big portion of the dataset to leave unverified. Similarly, the test set is only 840 instances, and with wheelchair being 0.11% of the data, the 100.0 F1 scores in Tab. 3 are most likely coming from one or two samples. The paper would benefit from reporting the annotator accuracy and the per class counts in MMHU-T.
  • Release and back-tracking. Youtube will be released as reference links with labels, which is fine, but for the Waymo data there is no mention of segment/object IDs or even the source Waymo scene, to back-track which scene corresponds to the extracted motion. Releasing the annotation pipeline (prompts, annotator weights, configs etc) would benefit the community and further solidify MMHU.
Questions for authors
  • What fraction of frames is interpolated and what is the maximum gap length? Will interpolated frames be flagged in the release?
  • Which VLM and how many clips were used to discover the 13 behaviours, and why only Waymo?
  • What is the accuracy / per class F1 of the annotator model on MMHU-T, and how many positives per class does MMHU-T contain?
  • Will Waymo segment/object IDs be released with the motion data?
Reviewer_dWXWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a new human behavior benchmark in automated driving domain. The new benchmark integrates multiple data sources of pedestrian videos, and is annotated with comprehensive labels like motion and trajectory, text-based human behavior descriptions at low- and high- levels, and 13 critical behaviors related to automated driving decision making. The annotation is largely machine and VLM-based with some human supervisions. The authors have listed potential tasks that can be supported by the dataset with baseline experiment results reported.

Critical strengths

The dataset is relatively large in size and has rich behavior labels in pedestrian behavior modeling domain. Pedestrian dataset is generally much limited compared to general automated driving dataset, due to the difficulties to record large number of pedestrians and annotate pedestrian behaviors carefully. The proposed dataset can potentially contribute to the research area with additional annotated pedestrian encounters.

Critical weaknesses

Ambiguity is the core challenge in pedestrian behavior and intention prediction. The current dataset seems to have deterministic labels for observable behaviors, but did not capture how the observed behavior can be interpreted differently when inferring the intention of the pedestrians or informing different driving decisions. The corresponding confidence is also not benchmarked. Those labels are more useful to support driving decisions and guarantee safety margins during vehicle-pedestrian interactions.

The dataset seems to be ego-centric, which can be good to support VQA and intention estimation, but may not be ideal to support driving decisions or trajectory predictions.

Questions for authors

Data annotation quality is not reported clearly in the paper. For all the automatic annotations and the human-on-the-loop annotations, it is not clear how the label accuracy is measured and what accuracy level has been achieved. Please report the annotation quality control methods, measurements, and the achieved annotation quality levels.

1 comment
AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer dWXW,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #18883

Reviewer_xxbmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MMHU, a multimodal benchmark for human behavior understanding in autonomous driving. The authors develop a human-in-the-loop annotation pipeline and evaluate several tasks, including motion prediction, motion generation, and behavior VQA. Experiments also show that training with MMHU improves existing models. However, concerns about data splitting and the reliability of class-level evaluation weaken the benchmark's credibility. Addressing these issues would substantially strengthen the contribution.

Critical strengths

The dataset addresses an important problem in autonomous driving. Its combination of motion, trajectory, language, and behavior annotations supports multiple research tasks.

The dataset is relatively large and collected from diverse sources. The annotation pipeline provides a practical approach to generating multimodal labels.

The experiments cover several tasks and demonstrate improvements when incorporating MMHU into model training.

Critical weaknesses

Potential data leakage. The dataset splitting strategy includes sampling different instances or clips from the same source videos into different subsets. This may introduce strong visual and temporal overlap between training and testing data. A strictly video-disjoint evaluation is needed to establish generalization.

Limited test set and class-level analysis. The test set contains only 840 instances, while some behaviors are extremely rare. Several methods achieve 100% F1 on individual classes, but the number of positive test samples is not reported. This makes the reliability of class-level results difficult to assess.

Limited justification of behavior categories. The 13 behavior categories are largely identified through VLM-based analysis. Their relevance to driving safety is not systematically established. In addition, recognizing an observed behavior, such as crossing the street, is different from predicting a pedestrian's future intention.

Limited evaluation of downstream driving benefits. While the paper demonstrates improvements on several human behavior tasks, its evidence of benefits to autonomous driving is mainly a qualitative planning example. More quantitative evaluation of downstream driving performance would strengthen the practical contribution.

Questions for authors

Can the authors provide results using a strictly video-disjoint train/test split?

Can the authors report the positive sample counts and uncertainty estimates for each behavior class in MMHU-T?

How were the 13 behaviors determined to be safety-critical, and can their usefulness be evaluated in downstream driving tasks?

2 comments
AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer xxbm,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #18883

Reviewer_xxbmOpenReview ↗

Response to Authors

Thank you for the response. My concern on Q3 remains:

The paper does not define what makes a behavior “critical.” No definition or selection criteria are given in the paper. The paper never mentions that transportation experts validated the behaviors. There is no record of how many experts took part, what their background is, how they validated the behaviors, or whether any candidate behaviors were rejected. In addition, it seems nothing shows that the behaviors the VLM identified were checked for correctness.

Please clarify these points and add them to the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.