acceptodds
Under review as a conference paper at ICLR 2027

EgoDual: Multi-Level Task Planning from Egocentric Video

Abstract

A system that guides a procedural task from video, whether a proactive assistant coaching a person or a planner instructing a robot, must decide not only what comes next, but when to intervene. The right level of guidance depends on who carries it out: a person can act on the next subtask, while a robot policy needs the next concrete action. These levels follow different timelines. Within a subtask, the next action is imminent while no subtask boundary is near. At a boundary, the opposite holds. A granularity-agnostic policy cannot be correct at both. We introduce granularity-conditioned procedural planning, in which the requested level (subtask or action) is an explicit input that determines both what the agent predicts and when it intervenes. We build hierarchical supervision over subtasks and actions, and introduce EgoDual, a dual benchmark of 7,017 queries from 93 egocentric procedural videos in which coarse and fine requests share the same visual input. Across four VLMs, zero-shot models are largely insensitive to granularity. Our dual supervision learns distinct coarse and fine intervention policies, raising the granularity separation score from 0.45 to 0.68, and improves action-level prediction. Our analysis further shows that subtask-level anticipation is bottlenecked by predicting the next procedural state, not by generating text at the right level of abstraction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Procedural Video, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m7GzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EgoDual, a benchmark for granularity-conditioned procedural prediction from egocentric video. The benchmark evaluates when to intervene and what step to predict under coarse subtask-level and fine action-level requests, using matched visual observations. It contains 7,017 scored query points from 93 videos. This paper also trains separate timing and content adapters and analyzes model performance across granularities, temporal distances, and transition types.

Critical strengths
  1. This paper investigates a meaningful problem in procedural video understanding: the requested level of guidance affects both what should be predicted and when an intervention is appropriate. Evaluating different granularities under identical visual observations provides a useful setting for studying this capability.

  2. The experiments evaluate multiple VLMs and provide analyses beyond aggregate accuracy, including granularity sensitivity, timing precision, and transition types. These analyses reveal potentially useful limitations in current models, particularly in anticipating subtask transitions and precisely identifying intervention times.

Critical weaknesses
  1. The content evaluation protocol has a significant validity issue. The prompt asks whether the assistant should predict the next event, rather than explicitly requesting a description of the next step. As a result, some zero-shot models produce Yes/No responses instead of step predictions, leading to near-zero scores. The paper also reports that adding a simple format instruction substantially improves output validity, yet the original results are retained. A clearer instruction should be applied consistently across all models before drawing conclusions about their procedural prediction capabilities. Additionally, the LLM-based content judge lacks reported agreement with independent human evaluations, further limiting the reliability of the content results.

  2. The benchmark construction and task definition require stronger validation. Despite containing 7,017 evaluation queries, the benchmark is derived from only 93 videos, raising concerns about task diversity and representativeness. Although the annotations are manually reviewed, detailed annotation guidelines and inter-annotator agreement are not provided.

Questions for authors
  1. The benchmark contains only 93 videos. How diverse are the underlying procedures, participants, and environments? How were the videos selected, and how representative are the resulting evaluation queries? Can the authors provide video-level uncertainty estimates to assess the robustness of the reported results?

  2. Can the authors provide detailed annotation guidelines, particularly for subtask boundaries, action onsets, and transition categories? How many annotators were involved, how were disagreements resolved, and was inter-annotator agreement measured? Will the complete benchmark annotations and evaluation data be publicly released?

  3. Can the authors re-evaluate the zero-shot baselines with clear and consistent next-step generation instructions? How do the results and model rankings change after addressing the formatting failures? Can the authors also validate the LLM judge against independent human assessments?

Reviewer_PjpdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies procedural assistance from egocentric video when the requested granularity is given as an explicit input. The task is to predict both when to intervene (SPEAK/SILENT) and what to say next, conditioned on video context and a granularity token. The authors build EgoDual, a benchmark that evaluates every query under both granularities on the same observation with prompts differing in one token. Methodologically, they freeze a VLM backbone (Qwen3-VL-8B, Gemma3-4B) and attach two separately trained adapters: a timing adapter and a content adapter. Experiments use the EgoDual benchmark and compare four zero-shot VLMs against the two fine-tuned backbones.

Critical strengths
  1. The paper introduces granularity-conditioned procedural planning, making task granularity an explicit input token to dynamically adapt both intervention timing and guidance content.
  2. It demonstrates that separating the timing and content adapters resolves multi-task gradient dominance issues, improving both intervention accuracy and granularity-dependent control.
Critical weaknesses
  1. Absolute performance is far from usable. Fine-grained ROC-AUC is 0.612 and content accuracy is around 0.2–0.3.
  2. The comparison set is narrow. Only four zero-shot VLMs are compared, all instruction-tuned general-purpose models with no streaming-video or proactive-assistant baselines.
  3. The ranking objective does not yield consistent gains. Table 1 shows that adding L_gran helps Gemma3-4B slightly but hurts Qwen3-VL-8B on fine ROC-AUC (0.612→0.595) and GranSep (0.675→0.661), so the ranking term is not a reliable improvement and its inclusion lacks clear justification.
  4. Content results rest on an unvalidated judge. All content figures depend on a Qwen3-VL-235B judge with no human agreement study and a backbone shared with the reference-label generator, so judge bias may be comparable in magnitude to the signal.
  5. The evaluation is disconnected from real utility. There is no qualitative examples, no user study, and no downstream task success measure.
Questions for authors

Please refer to the weaknesses.

Reviewer_xzDoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates granularity-conditioned procedural planning from egocentric video, addressing intervention timing ("when to speak") and next-step prediction ("what to say") across coarse and fine granularities. The authors construct EgoDual, a dual-granularity benchmark derived from 93 egocentric procedural videos where identical visual prefixes are paired with either coarse or fine requests differing by a single prompt token.

Critical strengths
  1. The paper identifies an important disconnect in procedural guidance, which highlights that intervention boundaries and instruction contents differ substantially between human assistance and robot.
  2. EgoDual rigorously isolates the effect of granularity by presenting identical visual observations and task contexts, changing only the granularity token and introducing the rank-based GranSep for offset.
Critical weaknesses
  1. Content generation is evaluated using Qwen3-VL-235B, the same model family that generated the benchmark annotations, introducing potential family-level stylistic bias. Furthermore, zero-shot models are penalized severely due to minor token-formatting mismatches rather than semantic errors.
  2. The pairwise ranking objective improves Gemma3-4B but consistently degrades performance on the Qwen3-VL-8B. The authors discard it in their default setup without providing an explanation.
  3. Main content evaluations are restricted to STRONG transitions that retain the in-hand object, masking WEAK transitions involving tool or object switches. Performance drops significantly on WEAK transitions, limiting the benchmark's fidelity to realistic multi-tool workflows.
Questions for authors
  1. How about evaluate gradient-balancing approaches (such as GradNorm or PCGrad) to retain a unified model before deciding to split timing and content into separate adapters?
  2. How can the model be improved on WEAK transitions, where predicting the next object cannot rely on object continuity?
  3. How does the timing adapter perform when evaluated under varying temporal tolerance windows around the true onset query?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.