acceptodds
Under review as a conference paper at ICLR 2027

EgoSCOPE: Seeking What Is Missing via Query-Sufficient Evidence Routing for Long-Horizon Video Understanding

Abstract

Egocentric video understanding supports episodic memory, personal assistance, and embodied reasoning. In long-horizon recordings, answering a query requires combining observations scattered across time, viewpoints, and changing object states under a limited evidence budget. Existing approaches build long-video representations and structured memories or select query-relevant observations through ranking, diversity modeling, and adaptive search. However, individually relevant or diverse observations may still leave essential query relations unsupported, while repeated evidence consumes the remaining budget. Inspired by human problem solving, we propose Egocentric Spatiotemporal Closure over Progressive Evidence (EgoSCOPE), a fixed-budget router that progressively seeks evidence for unresolved query demands. EgoSCOPE represents each query's evidence requirements as a transient Query Demand Graph composed of query-specific relations from explicit semantic categories. A lightweight set encoder recurrently updates relation support and conflict, using marginal closure and necessity–sufficiency signals to prioritize unresolved demands and suppress redundant or conflicting evidence before a single frozen task-head call. Across Ego4D NLQ, EgoLifeQA, and VSI-Bench, EgoSCOPE consistently improves temporal localization, long-term memory, and spatial reasoning over representative task-specific systems and matched evidence-selection baselines, while increasing relation closure and reducing evidence duplication. These results demonstrate that explicitly tracking unresolved query demands enables more sufficient evidence composition under fixed budgets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Video Understanding, Multi-modal Learning, Spatiotemporal Closure

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TnfdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on long video reasoning under a fixed evidence budget. The overall logic is complete, and the method modules are mainly designed around the core challenge of "finding missing evidence". Its main contribution is the valuable integration of mechanisms and the reconfiguration of the problem. The overall structure of the paper is clear, the technical description is relatively clear, and the experiments are relatively reasonable and complete.

Critical strengths

The research motivation is reasonable, and the description of the challenges is logically consistent. The key information in the long video may indeed be scattered across different times, events and perspectives. Simply relying on the ranking of relevance is prone to generating duplicate evidence. The paper progresses from "relevant evidence" to "joint evidence sufficient to answer the question", and the problem definition is highly reasonable.

The methods have a certain degree of relative innovation and have a good correspondence with the core challenges. The uniqueness of the paper lies in treating the query-side requirement graph as the dynamic control state of evidence routing, rather than merely using the graph as video memory or scene representation.

The overall structure of the paper is clear, and the technical description is relatively complete. The connection between the problem definition, methods, training objectives, experiments and appendices is natural. The algorithm flow and main formulas can support reproduction. Figures 1 and 2 express the motivation and overall architecture of the method in a relatively intuitive way, and the qualitative cases also help to illustrate how the method handles the temporal sequence and cross-event relationships.

Critical weaknesses

The motivation is valid, but the statement regarding the inadequacies of the existing methods is somewhat absolute. The paper generalizes the existing methods as correlation or diversity selection, which may give the impression that existing research generally lacks the ability to model relationships. In fact, existing work has already covered logical dependencies, scenario graphs, problem decomposition, and adaptive search, so the paper needs to more accurately distinguish "existing relationship modeling" from "the relationship state used in this paper for dynamic evidence routing".

The key argument experiments still have room for improvement. The paper needs to further discuss the impact of graph parsing errors, query complexity, candidate pool quality, and evidence budget changes on the method. The current experiments have well demonstrated that "the method is effective under the given protocol", but the discussion on the method's scope of application, failure scenarios, and actual deployment costs is still insufficient.

The chart information is rich, but some expressions can be further simplified. The overall charts can support the main conclusions, and qualitative cases also have explanatory power. Some tables simultaneously carry the main results, matching control, and routing statistics, with high information density. It is recommended to strengthen the way of highlighting the main conclusions and reduce the cost for readers to distinguish between different experimental protocols.

The ablation study is basically complete, but it lacks necessary discussion experiments. The current ablation mainly proves that the removal of each module will decrease performance, but it does not fully answer whether these modules are superior to simpler alternative solutions. For example, whether ordinary problem decomposition, no type of relationship state, or simple coverage goals can achieve similar effects, there is still a lack of direct verification.

Questions for authors
  1. Please clarify the core differences of this paper compared to existing graph-based, logical search, and iterative retrieval methods.

  2. Please explain why a complete necessity, sufficiency, and conflict mechanism is required. It should be demonstrated why these modules are necessary relative to simple relevance, coverage, or diversity goals, and provide comparisons of stronger alternative solutions or error cases.

  3. Please provide a more precise positioning of the shortcomings of the existing research. The paper should acknowledge that existing work has introduced scenario graphs, logical dependencies, and adaptive search, and further explain that this paper addresses the more specific problem of "relationship state-driven fixed-budget evidence closure".

  4. Please describe in which scenarios the method may fail. For example, when the problem analysis is inaccurate, when the required relationship exceeds the predefined types, when the candidate pool lacks key evidence, or when the problem requires open-ended reasoning, how the performance and behavior of the method change.

Reviewer_TBakOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Under a fixed evidence budget, relevance-based selection picks redundant observations and diversity-based selection can still miss the relations a query needs. EgoSCOPE parses each query into a transient Query Demand Graph of typed relations, tracks accumulated support and conflict per edge with a noisy-OR state, and at each of K steps scores candidates by marginal closure, necessity, task gain, and sufficiency while penalizing redundancy and conflict. It re-encodes the pool after every acquisition, so evidence already gathered changes the ranking of what remains.

Critical strengths
  1. The motivation for "relevance-based ranking has no notion of which query demands are already satisfied" is good
  2. The human-problem-solving analogy is good.
  3. QDG is well designed
Critical weaknesses
  1. The parser is built from finite task templates and lexical rules, and reaches 84.8 and 80.1 edge F1. That is decent but far from solved, and the templates are per benchmark. The paper reports oracle graphs raise task metrics by 1.81 and 1.96 points, so parser error costs about 2 points, which is larger than most of the gains reported over baselines.
  2. the acquisition score in Eq. 7 has six weighted terms, alpha through eta. That is a lot of hyperparmeters with no optimality guarantee. Table 6 varies only tau and rho. The remaining weights are fixed and unablated.
Questions for authors
  1. Can you compare against Frame-Voyager or another learned selector trained on the same supervision? All current selector comparisons are against unsupervised rules.
  2. How hard is it to write QDG templates for a new benchmark, and what is the parser F1 on VSI-Bench? The method's generality depends on this, and 84.8 versus 80.1 on the two supervised sets suggests benchmark sensitivity.
  3. Oracle graphs raise task metrics by 1.81 and 1.96 points, which is larger than the gain over your best matched control. How much of the remaining headroom is parser error versus routing error?
2 comments
AuthorsOpenReview ↗

Dear Reviewer TBak

Dear Reviewer TBak,

Thank you for your constructive and technically insightful comments. In our rebuttal, we clarified how the QDG templates can be adapted to a new benchmark and reported parsing accuracy on VSI-Bench. We also added experiments that separate the effects of graph correction and evidence selection, evaluate sensitivity to the acquisition weights, and compare EgoSCOPE with Frame-Voyager using the same training data.

With about one day remaining in the discussion period, we would appreciate hearing whether these results address your concerns. If they do, we would be grateful if you could consider updating your evaluation. We would also be happy to clarify any remaining points.

Thank you again for your time and for helping us improve the paper!

Best wishes!

I read the rebuttal. I keep the positive score, however, I must acknowledge I am not really famillar with this area.

Reviewer_GaU1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoSCOPE chooses which clips to show a video model when you can only show a few. First, hand-written rules turn the question into a small graph of what the answer needs. For "where was the cup placed last?", that's the cup, the last time it was put down, and where it ended up. Then a small model looks at 64 candidate clips and picks 8, one at a time. At each step it prefers clips that cover parts of the question that aren't covered yet, and it avoids repeats and contradictions. After that, a frozen answer model is called once. The selector is trained on Ego4D NLQ and EgoLifeQA and then used on VSI-Bench without more training. It reports 25.07 R@[email protected] on NLQ, 48.4 on EgoLifeQA. and 44.74 on VSI-Bench with LLaVA-OV-7B. That's 0.8 to 1.4 points better than the best other selector given the same inputs.

Critical strengths
  1. The main comparison is set up the right way. Every selector gets the same candidates, the same budget, the same answer model and prompt, and one call. Including a learned baseline without the graph is a good touch.

  2. The coverage and duplicate numbers help show what the selector is actually doing.

  3. It's cheap, with router overhead of about 0.1 s or less per question.

Critical weaknesses
  1. Some reported numbers can't come from the stated test sets. This is my main concern.
  • EgoLifeQA. The paper doesn't give category sizes, but every baseline category score in Table 2 (including EgoGraph's, from another paper) fits exactly with 125 / 126 / 61 / 125 / 63 questions, which add up to 500. For example, 45.9 is 28 out of 61. With these sizes, EgoSCOPE's 48.7, 48.2, 47.4 and 60.1 can't come from any number of correct answers, and Sec. 4.4 says these are single runs.
  • Ego4D NLQ. All 19 baseline numbers in Table 1 are whole counts out of 4,552. EgoSCOPE's 34.26 and 59.43 aren't (1559 gives 34.25, 1560 gives 34.27), and neither is DPP's 23.78 in Table 16.

Please explain and share the per-question predictions. App. B.2 and App. D say a script rebuilds the tables from them.

  1. Settings were tuned on the reported NLQ set. Table 13 shows the NLQ validation set was used both to pick the checkpoint and to report Table 1, and loss weights, score weights and τ were all "selected on validation data". With a margin of about 1.2 points over the best other selector, that matters.

  2. The VSI-Bench numbers need explaining. 8-frame LLaVA-OV-7B gets 37.17 here vs 32.4 in the VSI-Bench paper. In Table 22, three very different models land within 0.6 points of each other, and there are no baselines for the new models, so we can't tell if the gain carries over.

Questions for authors

see weakness

  1. Please explain the EgoLifeQA and NLQ numbers that aren't possible with 500 or 4,552 questions, and share the per-question predictions for Tables 1, 2 and 16.

  2. What do the NLQ results look like if you pick settings on a held-out part of the training data instead?

1 comment
AuthorsOpenReview ↗

Dear Reviewer GaU1

Thank you again for your constructive and detailed comments. In our rebuttal, we clarified how the original EgoLifeQA and Ego4D NLQ scores were computed and reported new results on all 500 EgoLifeQA questions and all 4,552 NLQ queries. We also reran NLQ with checkpoint and hyperparameter selection restricted to a held-out subset of the training data. Under this protocol, EgoSCOPE outperforms the matched recurrent control without QDG by 0.92 percentage points. On VSI-Bench, we clarified the difference between the local and published baseline settings and added matched comparisons across three backbones.

As the discussion period draws to a close, we would appreciate hearing whether these clarifications and results address your concerns. If they do, we would be grateful if you would consider updating your evaluation. We would be happy to clarify any remaining points.

Thank you again for your time and for helping us improve the paper!

Best wishes!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank all three reviewers for their comments. During the rebuttal, we provided point-by-point responses, conducted additional controlled experiments, and incorporated the results and clarifications into the revised manuscript and supplementary material.

Reviewer TBak recognized our motivation to track unresolved query demands, the analogy to human problem solving, and the QDG design. Reviewer TBak raised questions about QDG adaptation, parsing accuracy, the effects of parsing and routing errors, acquisition weights, and comparison with a learned selector. In response, we clarified template adaptation (Sec. 3.2 and Appendix A.4) and added a VSI parser audit and controlled experiments on graph correction and evidence selection (Appendix B.5). We evaluated the acquisition weights (main-text Table 8 and Appendix E.3) and compared EgoSCOPE with Frame-Voyager using the same training data and evaluation budget (main-text Table 7 and Appendix E.2).

During the discussion, TBak confirmed having read our rebuttal and maintained the positive score. TBak's questions were technically specific and addressed core aspects of the method, particularly the distinction between parser and router errors and the contribution of the acquisition terms. We sincerely thank TBak for reviewing our response. These comments and suggestions helped us improve the experimental analysis and the paper.

Reviewer GaU1 recognized the matched evaluation design, the learned control without QDG, and the low routing overhead. Reviewer GaU1 raised concerns about the reported scores, NLQ model selection, and the VSI baseline and backbone comparisons. We clarified the scoring protocols and provided aggregate counts for recomputing the results (Appendix B.4 and F), and added complete-list evaluations (main-text Tables 1, 2 and 6). We repeated NLQ evaluation with model selection confined to the training data (Sec. 4.1 and Appendix B.3-B.4), clarified the eight-frame VSI baseline (main-text Table 5), and added matched comparisons across backbones (Appendix C.4, Table 34). We posted a detailed response and invited further feedback. We thank GaU1 for providing these comments.

Reviewer Tnfd recognized the use of the query-side demand graph as a dynamic control state for evidence routing and the formulation of evidence selection around jointly sufficient evidence. Reviewer Tnfd requested more precise positioning of the work and stronger evidence for the core mechanisms. The reviewer's suggestions included comparisons with ordinary question decomposition, untyped demand states, and simpler coverage objectives, as well as analyses of parsing errors, query complexity, candidate quality, evidence budget, and runtime. We added the simpler learned controls (main-text Table 6 and Appendix E.1), parsing and candidate-pool diagnostics (Appendix B.5), and complexity, budget and runtime analyses (Appendix E.4 and C.4). We clarified the relation to prior work (Secs. 1-2), the evaluation protocols (Secs. 4.3-4.4 and Appendix B.4), and the method's scope (Appendix E.4).

Tnfd's comments directly examined whether the core modules provide benefits over simpler alternatives and prompted additional experiments on the method's central design choices. We sincerely thank Tnfd for these technically focused and constructive suggestions, which strengthened the experimental evidence and improved the clarity of the paper.

We thank all three reviewers and the Area Chair for the time devoted to reviewing our work.

Best wishes,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.