- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the BARISTA benchmark, which provides an integrated set of object masks, tracking information, attributes, relationships, Hand-Object Interaction (HOI), and procedural behavior information for 185 coffee-making egocentric videos. Based on this, six zero-shot evaluation tasks are constructed to assess a wide range of abilities, from object-centric visual recognition to high-level procedural understanding of VLMs, and to analyze performance differences between tasks.
Critical strengths
-
It offers rich annotations, including 610,000 instance masks and 2.48 million typed relations, and its strength lies in the ability to evaluate a wide range of tasks from object recognition to high-level inference within the same video. Unlike existing datasets that focus on individual annotations, an integrated scene graph allows for the comparison of multiple capabilities
-
Evaluate various VLMs and select some specialist baselines for six tasks, and analyze cross-task failure, performance by machine type, and the robustness of LLM judges. These experiments are useful for understanding the task-specific characteristics and limitations of VLMs, going beyond simple performance comparison
Critical weaknesses
-
Generalizability : Although this paper proposes a benchmark for evaluating compositional visual understanding, it only addresses the single domain of coffee production, raising questions about the generalizability of the evaluation results and their broad applicability.
-
Limited methodologial solution : The main contribution lies in integrating existing annotation techniques with evaluation tasks, while the new methodological contribution is limited.
-
Insufficient validation on annotation quality : Although it is generated automatically by the LLM, human validation has not been reported, so further verification of the accuracy of the ground truth is needed
Questions for authors
see weakness