- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces MMA-MTU, a benchmark for systematically evaluating multi-modal large language models (MLLMs) on micro-action understanding. The benchmark assesses five tasks: Recognition, Temporal Reasoning, Grounding, Detection, and Description. It uses a multi-domain dataset with many videos and over 122,000 queries, including both label-based and label-free queries describing observed movements.
Critical strengths
- The main strength is the construction of a single benchmark that evaluates models on multiple aspects of video understanding, from recognizing individual events to describing the sequence as a whole.
- This approach allows for assessment of how models handle different types of understanding within the same framework.
- Incorporation of Label-Free Queries: Including label-free queries, which use natural language descriptions instead of action names, encourages models to rely on visual evidence rather than memorized labels, providing a more practical evaluation of general understanding.
Critical weaknesses
- The introduction of 12 subtasks and a vast array of evaluation metrics (e.g., [email protected] for Grounding, mAP for Detection, BLEU-4 for Description) is extensive. While comprehensive, the sheer volume of related yet distinct metrics risks diluting focus on the core scientific contribution and making the overall landscape difficult for readers to fully digest.
- Because the entire benchmark is built upon and annotated using the same process (the Motion-Evidence-Guided MLLM Annotation Tool) and anchored to the MMA-82 dataset, the results might reflect the capability of the annotation pipeline as much as the inherent limitations of current MLLMs. Clarifying the distinction between benchmark capability and annotation-process capability would be beneficial.
Questions for authors
- Since the five tasks are connected but show varied correlation strengths (e.g., Recognition vs. Grounding), could the authors propose a methodology or weighted scoring mechanism that quantifies how severely a failure in one early task (e.g., Recognition) degrades performance on a later, dependent task (e.g., Description)?
- Regarding the label-free queries, the authors note that the generation process still requires a paired label-based query to anchor the description. Can the authors elaborate on the theoretical limitations of this process when attempting to model truly unseen or highly colloquial natural language descriptions that might deviate significantly from the annotated video evidence?