acceptodds
Under review as a conference paper at ICLR 2027

MMA-MTU: Benchmarking Multi-Task Micro-Action Understanding

Abstract

Multimodal large language models (MLLMs) have shown strong capabilities in video understanding, yet existing benchmarks largely focus on daily activities and general actions. Their ability to perceive and understand micro-actions, which are brief, low-amplitude, and spatially localized, remains insufficiently assessed. Beyond recognizing action categories, micro-action understanding requires reasoning about temporal relations, locating action boundaries, recovering relevant events, and describing how movements unfold. Our work addresses this evaluation gap through five key contributions: **1) Multi-task benchmark.** We introduce **MMA-MTU**, a benchmark that jointly evaluates Recognition, Temporal Reasoning, Grounding, Detection, and Description on shared video evidence. **2) Large-scale, multi-domain data.** MMA-MTU contains 4 domains, 11,180 untrimmed videos, and 122,845 queries covering 82 fine-grained categories, and 5 tasks with 12 subtasks. It provides dedicated training, validation, and test splits for instruction tuning and model learning. **3) Paired queries and fine-grained descriptions.** MMA-MTU includes 34,876 label-free queries paired with label-based counterparts, replacing action names with descriptions of observable movements. It also provides motion-evidence-guided event descriptions that capture how individual actions are performed. **4) Extensive evaluation.** We evaluate **36** MLLMs under the same protocol. The results expose a clear divide between different tasks: the best Temporal Reasoning accuracy reaches 66.62%, whereas the best Grounding average R@1 and Detection average mAP reach only 26.65% and 5.44%, respectively, showing that success on one ability does not reliably transfer to the others. Supervised fine-tuning significantly improves performance across all 5 tasks. **5) Downstream exploration.** A preliminary emotion-recognition study further explores how predicted micro-action information can support human emotion understanding.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Micro-Action, Multimodal large language models, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_QuXdOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MMA-MTU, a benchmark for systematically evaluating multi-modal large language models (MLLMs) on micro-action understanding. The benchmark assesses five tasks: Recognition, Temporal Reasoning, Grounding, Detection, and Description. It uses a multi-domain dataset with many videos and over 122,000 queries, including both label-based and label-free queries describing observed movements.

Critical strengths
  • The main strength is the construction of a single benchmark that evaluates models on multiple aspects of video understanding, from recognizing individual events to describing the sequence as a whole.
  • This approach allows for assessment of how models handle different types of understanding within the same framework.
  • Incorporation of Label-Free Queries: Including label-free queries, which use natural language descriptions instead of action names, encourages models to rely on visual evidence rather than memorized labels, providing a more practical evaluation of general understanding.
Critical weaknesses
  • The introduction of 12 subtasks and a vast array of evaluation metrics (e.g., [email protected] for Grounding, mAP for Detection, BLEU-4 for Description) is extensive. While comprehensive, the sheer volume of related yet distinct metrics risks diluting focus on the core scientific contribution and making the overall landscape difficult for readers to fully digest.
  • Because the entire benchmark is built upon and annotated using the same process (the Motion-Evidence-Guided MLLM Annotation Tool) and anchored to the MMA-82 dataset, the results might reflect the capability of the annotation pipeline as much as the inherent limitations of current MLLMs. Clarifying the distinction between benchmark capability and annotation-process capability would be beneficial.
Questions for authors
  • Since the five tasks are connected but show varied correlation strengths (e.g., Recognition vs. Grounding), could the authors propose a methodology or weighted scoring mechanism that quantifies how severely a failure in one early task (e.g., Recognition) degrades performance on a later, dependent task (e.g., Description)?
  • Regarding the label-free queries, the authors note that the generation process still requires a paired label-based query to anchor the description. Can the authors elaborate on the theoretical limitations of this process when attempting to model truly unseen or highly colloquial natural language descriptions that might deviate significantly from the annotated video evidence?
Reviewer_rKe4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes a micro-action understanding benchmark MMA-MTU and evaluates a large number of MLLMs on the proposed 5 tasks. The benchmark builds on the previous dataset MMA-82-Det, and covers 5 main tasks: Recognition, Temporal Reasoning, Grounding, Detection, and Description. The extensive evaluation with existing open-source MLLMs (and one proprietary model) show current approaches likely are insufficient, and fine-tuning on the proposed training set give a non-trivial improvement across several tasks.

Critical strengths
  • Multi-task action understanding is a field which can benefit from larger-scale benchmarks. This proposed dataset would be useful for other researchers and contributors to explore and build/evaluate on.
  • The evaluation on MLLMs is extensive and thorough. Furthermore, the fine-tuning experiment shows that the proposed training set would be useful for others to train their video understanding methods on.
Critical weaknesses
  • One improvement that can be made is the evaluation of non-MLLM-based approaches for some of these action tasks that are less free-form. There is an extensive literature on Recognition and Detection, which can potentially shed some light onto why mAP on Detection is low (i.e. is this purely the difficulty of the problem, or is it that current MLLMs are just unable to solve this problem).
Questions for authors
  • Currently, the reader needs to go to the appendix to understand what the sub-tasks present in the dataset are (i.e. the main text/figures mainly have the abbreviations like VLMR, CGMR). Would it be possible to bring these subtasks into the main-text so that the work is self-contained without the requirement of an appendix?
Reviewer_9ZKROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces MMA‑MTU, a multi‑task benchmark for evaluating multimodal large language models on fine‑grained micro‑action understanding. It unifies recognition, temporal reasoning, grounding, detection and description over shared untrimmed videos, and includes paired label‑based / label‑free queries. The authors conduct large‑scale zero‑shot evaluation over 36 MLLMs, reveal significant capability gaps across tasks, and show its utility for supervised fine‑tuning and exploratory affect recognition.

Critical strengths
  1. This benchmark unifies five inter‑related tasks (recognition, temporal reasoning, grounding, detection, description) on the same set of untrimmed videos, covering the full pipeline of micro‑action comprehension. Empirical results clearly demonstrate that model performance does not transfer well across tasks, which highlights the necessity of multi‑task assessment instead of relying on single‑task metrics.

  2. The paired query design enables controlled study of whether MLLMs can understand actions from natural movement descriptions rather than only predefined category labels. Evaluating 36 diverse proprietary and open‑source MLLMs yields solid empirical observations about current bottlenecks of MLLMs on subtle human motion, providing valuable community reference.

Critical weaknesses
  1. This benchmark is built upon the existing MMA‑82‑Det dataset, so it inherits its category system, video distribution and potential annotation noise. Most queries are template‑generated rather than naturally human‑written, which may lack real‑world ambiguity and diverse colloquial expressions. Besides, grounding task suffers severely from imbalanced negative samples, weakening its ability to test models’ capability of rejecting non‑existent actions.

  2. There is no cross‑dataset generalization experiment. All test data comes from the same source distribution, and we cannot know how well conclusions transfer to unseen camera settings, populations and action styles. Meanwhile, automatic metrics for the description task cannot reliably judge factual correctness of fine‑grained motion details; human evaluation for proposition‑level motion facts is absent.

Questions for authors

Q1. Since MMA‑MTU inherits MMA‑82‑Det, how do the authors quantify annotation noise inherited from the original dataset? Besides, only 5 negative “No Match” samples exist for the Grounding subtask. Could you explain how this severe imbalance influences the reliability of Grounding evaluation results?

Q2. There are no cross‑dataset generalization experiments in the paper. Can you provide more evidence or analysis to show whether the benchmark’s main conclusions hold for videos from fully independent data sources? Moreover, why is proposition‑level human evaluation for the description task not included to verify factual correctness of fine‑grained motion details?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.