acceptodds
Under review as a conference paper at ICLR 2027

BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding

Abstract

Scene understanding is central to general physical intelligence, and video is a primary modality for capturing both state and temporal dynamics of a scene. Yet understanding physical processes remains difficult, as models must combine object localization, hand-object interactions, relational parsing, temporal reasoning, and step-level procedural inference. Existing benchmarks usually evaluate these capabilities separately, limiting diagnosis of why models fail on procedural tasks. We introduce BARISTA, a densely annotated egocentric dataset and benchmark of 185 real-world coffee-preparation videos covering fully automatic, portafilter-based, and capsule-based workflows. BARISTA provides verified per-frame scene graphs linking persistent object identities to masks, tracks, boxes, attributes, typed relations, hand-object interactions, activities, and process steps. From these graphs, we derive zero-shot language-based tasks spanning phrase grounding, hand-object interaction recognition, referring, activity recognition, relation extraction, and temporal visual question answering. Experiments reveal strong variation across task families and no consistently dominant model family, positioning BARISTA as a challenging diagnostic benchmark for procedural video understanding.

Primary area:
datasets and benchmarks
Keywords:
Egocentric Video Dataset, Dense Video Annotation, VLM Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LxEhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the BARISTA benchmark, which provides an integrated set of object masks, tracking information, attributes, relationships, Hand-Object Interaction (HOI), and procedural behavior information for 185 coffee-making egocentric videos. Based on this, six zero-shot evaluation tasks are constructed to assess a wide range of abilities, from object-centric visual recognition to high-level procedural understanding of VLMs, and to analyze performance differences between tasks.

Critical strengths
  1. It offers rich annotations, including 610,000 instance masks and 2.48 million typed relations, and its strength lies in the ability to evaluate a wide range of tasks from object recognition to high-level inference within the same video. Unlike existing datasets that focus on individual annotations, an integrated scene graph allows for the comparison of multiple capabilities

  2. Evaluate various VLMs and select some specialist baselines for six tasks, and analyze cross-task failure, performance by machine type, and the robustness of LLM judges. These experiments are useful for understanding the task-specific characteristics and limitations of VLMs, going beyond simple performance comparison

Critical weaknesses
  1. Generalizability : Although this paper proposes a benchmark for evaluating compositional visual understanding, it only addresses the single domain of coffee production, raising questions about the generalizability of the evaluation results and their broad applicability.

  2. Limited methodologial solution : The main contribution lies in integrating existing annotation techniques with evaluation tasks, while the new methodological contribution is limited.

  3. Insufficient validation on annotation quality : Although it is generated automatically by the LLM, human validation has not been reported, so further verification of the accuracy of the ground truth is needed

Questions for authors

see weakness

Reviewer_5yRpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BARISTA, a dataset and benchmark built from 185 egocentric coffee-preparation videos totaling 4.4 hours. A semi-automated annotation pipeline combines human annotation, mask propagation, and human review to link objects, attributes, relations, hand-object interactions, and activities through persistent object identities. These annotations support six evaluation tasks: phrase grounding, hand-object interaction recognition, region description, activity recognition, relation extraction, and temporal visual question answering. Zero-shot experiments with five VLMs and two task-specific baselines reveal differing strengths across tasks, with Gemini 3 Flash strongest on grounding and region description, Gemma 4-31B on relation extraction, and Qwen3.5-27B on activity recognition and temporal VQA. The paper also evaluates 200 matched events across tasks to examine co-occurring failures.

Critical strengths
  1. Rich, aligned annotations. Persistent object identities connect masks, attributes, relations, interactions, and activities within the same videos. The evaluation on 200 matched events makes concrete use of this alignment, allowing task performance and co-occurring failures to be compared on the same evidence rather than unrelated examples.

  2. Annotation quality checks. The pipeline includes human review and correction, supplemented by independent re-annotation of six sampled videos. The reported agreement provides supporting evidence for annotation reliability, an important consideration for a densely annotated benchmark.

Critical weaknesses
  1. Limited scope and incremental contribution. The dataset contains only 185 coffee-preparation videos totaling 4.4 hours (about 86 seconds per video), offering limited domain and temporal diversity. Several annotation types and task families have substantial precedents: EPIC-KITCHENS VISOR provides hand/active-object masks and contact relations; Ego4D annotates hand-object interactions and object states and benchmarks interaction/action anticipation; HD-EPIC provides fine-grained action annotations and video QA; and AGQA generates compositional temporal questions from scene graphs, with AGQA-Decomp additionally evaluating consistency across composed questions and subquestions. These datasets do not reproduce BARISTA's exact annotation bundle, but the paper does not clearly establish what distinctive experiments or findings its integration enables. The pipeline also combines established annotation and propagation techniques without clearly demonstrating improved scalability.

  2. Insufficient support for the central claims. Defining six tasks on the same videos does not itself establish a test of compositional understanding: the tasks are answered independently, and many primarily assess local perception. The matched-event analysis examines co-occurring errors but does not establish that successful answers require composing these abilities. The narrow domain and short temporal windows also provide limited evidence for broader procedural video understanding.

  3. Limited model coverage. The evaluation omits the strongest frontier models and uses low thinking effort for the closed models. Consequently, the results do not establish how challenging the benchmark remains for current leading systems, weakening claims about present model limitations and the benchmark's ability to distinguish their capabilities.

Minor concern: The temporal VQA rubric in Appendix A.3.3 allows action-transition scoring to ignore temporal anchors and emphasizes whether contact targets are mentioned. This may reward identifying the involved objects without correctly recovering their interaction order, weakening the interpretation of these scores as temporal understanding.

References for the comparisons above:

Questions for authors
  1. What specific experiment or finding does BARISTA's unified annotation enable that existing resources such as VISOR, Ego4D, HD-EPIC, and AGQA cannot support? What evidence demonstrates that the annotation pipeline scales beyond this small coffee-preparation domain?

  2. How is compositional understanding operationally defined and directly tested here, given that the six tasks are answered independently? What evidence distinguishes the measured capabilities from local perception and short-term change recognition, and supports broader procedural understanding?

  3. Can you provide results from stronger frontier models and higher thinking budgets to establish the benchmark's remaining difficulty and whether the reported capability gaps persist?

  4. How does temporal-task performance change with single-frame, endpoint-only, shuffled-frame, and text-only inputs? These controls would clarify whether the questions require temporal evidence rather than static cues or language priors.

Reviewer_c3oLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this work, the authors propose a dataset, BARISTA, of 185 ego-centric coffee preparation videos with rich, temporally dense annotations, applied to various downstream object-focused and semantic tasks, like phrase grounding, relation extraction, etc.

Critical strengths
  • The rich annotations for the provided videos are a valuable asset for the community.
Critical weaknesses
  • Novelty: Recently, there have been benchmarks like VSTAT [1], Flat-Pack Bench [2], etc., that have also evaluated the performance of MLLMs on multi-step procedural videos and state-level tracking. BARISTA’s potential distinction is its shared, dense annotation structure across several prediction tasks. The authors should explain what this enables beyond previous benchmarks, then demonstrate that benefit.
  • Small size: With only 185 videos and missing samples, the dataset might have limited diversity. L245 mentions a train/test split. Yet, the training applications mentioned in L471-477 remain untested.
  • Matched Diagnostic claim is weak: As per the results on L1226-1230, it seems that the matched diagnostic result only holds reliably for Qwen3.5 9B, not Qwen3.5-27B and Gemma 4-31B.
  • No evidence of samples to judge quality: Representative raw and annotated videos are unavailable in the supplement, which prevents direct assessment of temporal annotation quality and recording diversity.

[1] Benchmarking Visual State Tracking in Multimodal Video Understanding. Yu et al., 2026. https://vision-x-nyu.github.io/vstat-site/

[2] Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly. Chetan et. al., CVPR 2026. https://flat-pack-bench.github.io/

Questions for authors
  1. Distinct Contributions: Which analyses or capabilities does BARISTA enable beyond VSTAT and Flat-Pack Bench? Can you demonstrate some concrete examples?
  2. Dataset Diversity: Can the authors share some additional statistics to assess dataset diversity, such as the number of unique coffee machines tried, the number of distinct scenes, the number of unique recorders, etc.?
  3. Split and training: Which videos contribute to each reported benchmark? How does this relate to the 148/37 split? What evidence supports the proposed training uses?
  4. Diagnostic validity: How do you link a process-level reasoning error to a perceptual error? Does supplying correct boxes or relations improve downstream answers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.