- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EgoDual, a benchmark for granularity-conditioned procedural prediction from egocentric video. The benchmark evaluates when to intervene and what step to predict under coarse subtask-level and fine action-level requests, using matched visual observations. It contains 7,017 scored query points from 93 videos. This paper also trains separate timing and content adapters and analyzes model performance across granularities, temporal distances, and transition types.
Critical strengths
-
This paper investigates a meaningful problem in procedural video understanding: the requested level of guidance affects both what should be predicted and when an intervention is appropriate. Evaluating different granularities under identical visual observations provides a useful setting for studying this capability.
-
The experiments evaluate multiple VLMs and provide analyses beyond aggregate accuracy, including granularity sensitivity, timing precision, and transition types. These analyses reveal potentially useful limitations in current models, particularly in anticipating subtask transitions and precisely identifying intervention times.
Critical weaknesses
-
The content evaluation protocol has a significant validity issue. The prompt asks whether the assistant should predict the next event, rather than explicitly requesting a description of the next step. As a result, some zero-shot models produce Yes/No responses instead of step predictions, leading to near-zero scores. The paper also reports that adding a simple format instruction substantially improves output validity, yet the original results are retained. A clearer instruction should be applied consistently across all models before drawing conclusions about their procedural prediction capabilities. Additionally, the LLM-based content judge lacks reported agreement with independent human evaluations, further limiting the reliability of the content results.
-
The benchmark construction and task definition require stronger validation. Despite containing 7,017 evaluation queries, the benchmark is derived from only 93 videos, raising concerns about task diversity and representativeness. Although the annotations are manually reviewed, detailed annotation guidelines and inter-annotator agreement are not provided.
Questions for authors
-
The benchmark contains only 93 videos. How diverse are the underlying procedures, participants, and environments? How were the videos selected, and how representative are the resulting evaluation queries? Can the authors provide video-level uncertainty estimates to assess the robustness of the reported results?
-
Can the authors provide detailed annotation guidelines, particularly for subtask boundaries, action onsets, and transition categories? How many annotators were involved, how were disagreements resolved, and was inter-annotator agreement measured? Will the complete benchmark annotations and evaluation data be publicly released?
-
Can the authors re-evaluate the zero-shot baselines with clear and consistent next-step generation instructions? How do the results and model rankings change after addressing the formatting failures? Can the authors also validate the LLM judge against independent human assessments?