acceptodds
Withdrawn from ICLR 2027

EgoSkill: Rethinking What to Store and How to Reason in Long-Form Egocentric Video Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-form egocentric video question answering (VQA) requires reasoning over evidence that may be sparse, distributed across distant moments, and heterogeneous across question types. We introduce EgoSkill, a fully training-free framework that factorizes this problem into two complementary inference-time components: what to store and how to execute. For what to store, we propose Dynamic Hierarchical Memory (DH-Mem), a query-agnostic representation of long-form egocentric video. DH-Mem represents local events as temporally grounded substeps and semantically composes related substeps, even when they are temporally non-contiguous, into higher-level procedural steps and a video-level goal. For how to execute, we introduce Evidence Construction Skills (EC-Skills), reusable query-aware procedures that adapt evidence retrieval, visual processing, tool use, and verification to the requirements of each question type. Together, these components realize a store-execute factorization, with DH-Mem providing query-agnostic video context and EC-Skills constructing query-specific evidence for frozen MLLMs, without task-specific parameter updates. Across HD-EPIC, EgoSchema, and QAEgo4D, EgoSkill consistently improves the corresponding frozen MLLM baselines, demonstrating the effectiveness of factorizing long-form egocentric VQA into query-agnostic representation and question-dependent evidence construction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_RmWsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a training-free framework for long-form egocentric video question answering. The proposed Dynamic Hierarchical Memory (DH-Mem) organizes temporally grounded substeps into higher-level procedural steps based on semantic similarity, allowing non-contiguous video segments to be grouped together. The Evidence Construction Skills (EC-Skills) component employs predefined procedures to adapt evidence retrieval, visual processing, tool use, and answer verification to different question types. Experiments on HD-EPIC, EgoSchema, and QAEgo4D demonstrate consistent improvements over frozen MLLM baselines.

Critical strengths
  1. One interesting insight of this paper is that some procedural steps consist of temporally non-contiguous segments that can be semantically grouped together.
  2. The paper is clearly presented, and the figures and tables effectively convey the core ideas.
Critical weaknesses
  1. I think the contributions are somewhat limited. The paper presents the overall pipeline as one contribution, while counting its two components as separate contributions. Moreover, although the paper considers temporally non-contiguous action segments, hierarchical representations of procedural steps are not a new concept, as seen in prior work such as the Ego4D Goal-Step dataset.
  2. The EC-Skills in this paper rely on predefined mappings from question types to inference procedures, rather than capabilities learned through RL or embedded in the model harness. This raises concerns about their generalizability, particularly to unseen question types or more open-ended scenarios.
  3. Some of the benchmarks used in the experiments seem somewhat outdated, particularly EgoSchema and QAEgo4D. The authors could extend the evaluation to more recent benchmarks, such as EgoLifeQA and MM-Lifelong. Additionally, several baselines in Table 1 are relatively old, including VideoLLaMA and LLaVA-Video, both from 2024. The specific version of Gemini used in the comparison is also not clearly specified.
  4. The necessity of incorporating specialized models such as SAM3 and Depth Anything 3 is not sufficiently justified. These tools introduce additional computational overhead and installation requirements, which may limit the practicality of the proposed framework. It would be helpful to evaluate whether the performance gains justify these additional costs.
Questions for authors
  1. How much of the performance improvement can be specifically attributed to grouping temporally non-contiguous segments into the same procedural step compared with continuous merging?
Reviewer_p2WwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EgoSkill, a training-free framework for long-form egocentric video QA that separates memory construction from question-dependent reasoning. It proposes Dynamic Hierarchical Memory (DH-Mem) to organize temporally distant but semantically related activities (Fig. 1), and Evidence Construction Skills (EC-Skills) to adapt retrieval, tool use, and visual processing to different questions (Fig. 2). EgoSkill improves frozen MLLM performance across three benchmarks (Tables 1 and 2), demonstrating the benefits of structured memory and task-specific evidence construction.

Critical strengths
  1. Structured memory and evidence construction. Separating query-agnostic memory from query-dependent reasoning is a useful design primitive (Fig. 1). DH-Mem can group non-contiguous but related activities (Fig. 3), while ablations demonstrate contributions from both memory organization and evidence construction (Tables 3–4).

  2. Strong improvements without post-training/ fine-tuning. EgoSkill consistently improves frozen MLLMs across different backbones (Tables 1–2). On HD-EPIC, Qwen3-VL-8B improves from 37.7% to 56.1%, and Qwen3.5-27B from 41.7% to 59.6% (Table 1).

Critical weaknesses
  1. Limited evaluation on hour-long videos. The evaluation includes EgoSchema (Table 2), whose approximately three-minute videos can fit within modern MLLM context windows and whose original human-performance benchmark has been surpassed by current MLLMs. Given the limited technical novelty and reliance on pretrained models, stronger benchmarking is important. Evaluating on HourVideo (Chandrasegaran et al., 2024) and LVBench (Wang et al., 2024) would better establish effectiveness on extended videos requiring complex temporal and visual reasoning.

  2. Lack of direct evidence-retrieval evaluation. Although the paper proposes better memory organization and question-relevant evidence construction, the quantitative evaluation relies primarily on downstream QA accuracy. Figure 3 provides a qualitative example of procedural-step grouping, but no systematic evaluation of evidence-retrieval accuracy. The LV-Haystack benchmark by Ye et al. (CVPR 2025) provides human-annotated reference keyframes for this purpose. Such evaluation is important to establish whether EgoSkill retrieves the correct evidence rather than merely benefiting from additional tools and inference-time computation.

===

Ye, Jinhui, et al. “Re-thinking Temporal Search for Long-Form Video Understanding.” CVPR 2025

Chandrasegaran, Keshigeyan, et al. "Hourvideo: 1-hour video-language understanding." NeurIPS 2024

Questions for authors

Please see the weaknesses list.

Reviewer_fSvxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EgoSkill, a training-free framework for long-form egocentric VQA. It separates the problem into dynamic hierarchical memory, which organizes local observations into semantically related procedural steps, and evidence construction skills, which adapt retrieval, visual inputs, tools, and verification according to question type. Experiments show improvements over frozen MLLM baselines.

Critical strengths
  1. Strong improvements across three benchmarks.
  2. Effective separation of memory and execution.
Critical weaknesses
  1. EC-Skills are heavily handcrafted and benchmark-specific. Skill selection relies on predefined mappings, manual prompts, sampling rules, and specialized tools, which limit generality.
  2. The skill-development protocol is unclear. It is not sufficiently specified whether skill assignments and configurations were developed entirely on data disjoint from the reported test sets.
Questions for authors
  1. How were the EC-Skill assignments and configurations developed? Were all design decisions made using data completely disjoint from the reported evaluation sets?

  2. Can the authors quantify performance when skill selection is automatic or transferred to unseen question types without benchmark-specific mappings?

  3. How much of the improvement comes from DH-Mem itself versus specialized tools and manually designed EC-Skills under a comparable inference budget?

Reviewer_rE2LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoSkill, a training-free framework for long-form egocentric video question answering. It combines Dynamic Hierarchical Memory (DH-Mem), which groups temporally separated observations into semantically related procedural steps, with Evidence Construction Skills (EC-Skills), which adapt retrieval, visual inputs, tool use, and candidate verification to different question types. The framework is evaluated on HD-EPIC, EgoSchema, and QAEgo4D with multiple frozen backbones.

Critical strengths

The empirical improvements are substantial, and the method is clearly presented and easy to follow. Both DH-Mem and EC-Skills are well motivated, and the analysis provides useful support for the design choices.

Critical weaknesses

My main reservation concerns the methodological novelty. The results are strong, but I found it difficult to identify a distinctive technical contribution beyond combining semantic organization of video captions with question-specific inference procedures. The store–execute formulation is intuitive, yet the paper could explain more clearly what new insight this separation provides and why the particular design is essential.

A similar concern applies to EC-Skills. Providing specialized OCR, geometry tools, denser frame sampling, or candidate-wise verification is a reasonable way to improve performance on the corresponding question types. However, these gains are also fairly expected when the system receives additional task-specific processing and resources. The current results therefore demonstrate the effectiveness of the overall pipeline more clearly than they establish the novelty of the proposed approach. I would find the contribution more convincing if the paper showed what its memory structure and skill framework enable beyond a carefully engineered combination of existing components.

Computational cost is another unresolved issue. Some skills use denser sampling, separate calls for each answer choice, or an additional 32B narration model. Comparisons under matched inference budgets would help distinguish the contribution of the framework from the benefit of additional computation.

Questions for authors

See the weakness mentioned above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.