- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces MMHU, a large-scale dataset for human behavior understanding in driving scenes. It contains approximately 57K human motion clips and 1.73M frames. The annotations include 3D motion, trajectories, text descriptions, and 13 behavior labels. The authors evaluate several downstream tasks and show that training with MMHU improves baseline performance.
Critical strengths
-
MMHU provides aligned video, motion, language, and behavior annotations at a substantial scale. This is a useful resource for studying human behavior in driving scenes. Building the dataset also represents considerable collection and annotation effort.
-
The experiments cover several tasks and a range of baseline models. The improvements on 3DPW and JAAD provide evidence that MMHU can benefit existing methods.
Critical weaknesses
-
The methodological novelty is limited. Text annotation mainly combines existing pose-description tools with LLM/VLM prompting. Behavior VQA uses templated binary questions. The paper provides little evidence of how much the specific pipeline designs contribute.
-
The annotation and fine-tuning settings are not fully clear. The paper should explain the relationship between the annotation VLM and the model evaluated in Table 5. It should also separate the gains from human annotations and VLM-generated labels.
-
Some aspects of dataset quality need further validation. The reported 97% agreement only covers the human-labeled subsets. The quality of automatic labels and text descriptions remains less clear. Motion-label validation is mainly qualitative, and the choice of the 13 behaviors needs more justification.
Questions for authors
-
Which model generates the pseudo-labels, and which model is evaluated in Table 5? Please include comparisons using human annotations only, pseudo-labels only, and both.
-
Does adding PoseScript-based descriptions improve caption quality or downstream performance? A comparison with directly prompting a VLM with video frames would help.
-
Could the authors report an independent quality assessment of the automatic labels and captions? Per-class behavior results and quantitative motion-label validation would be useful.
-
How were the 13 behaviors selected? Were domain experts involved, and what criteria were used to include or exclude behaviors?
-
See weaknesses.
Thank you for your time and feedback - happy to discuss further
Dear Reviewer dWXW,
Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.
Thank you again for your time and consideration.
Best regards, Authors of Submission #18883