- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EGOPASS-BENCH to address the insufficient evaluation of changes in observer–entity interactions in existing benchmarks. The benchmark contains 95 egocentric videos and 577 distinct questions across five tasks: wearer translation, viewpoint and path turning, external-object motion, determining whether an object impedes passage, and detecting objects that impede passage. GPT-6 Astra leads on most evaluation metrics on EGOPASS-BENCH. However, its performance in detecting objects that impede passage is less than one percentage point above the baseline that always returns an empty set, showing that accurate motion recognition does not ensure reliable detection of objects that impede passage.
Critical strengths
This work evaluates existing models' ability to detect objects that impede passage in egocentric navigation scenes. It can be used to assess models' ability to assist wearers with navigation and obstacle avoidance.
Critical weaknesses
- The proposed dataset offers limited novelty. SAW-Bench, MotionBench, RiskBench, and DRAMA already cover some closely related capabilities. EGOPASS-BENCH mainly differs in comparing the attention required by objects at different time points and detecting objects that impede passage in egocentric navigation scenes. Its novelty is therefore relatively limited.
- The proposed dataset provides insufficient coverage, with too few videos and questions. Only 251 questions concern whether objects impede passage. Although the data collection considers six types of environments, this does not establish whether key passage-risk scenarios are adequately covered. The paper lacks systematic coverage statistics and analyses based on obstacle types and interaction scenarios. Moreover, is it reasonable that the baseline that always returns an empty set achieves a Discovery score of nearly 60%?
Questions for authors
- To address Weakness 1, design additional new tasks for passage scenarios.
- To address Weakness 2, collect more data while considering coverage across different obstacle types and interaction scenarios.
4 comments
W1. Limited novelty
Thank you for this comment. EgoPass directly scores the direction of change in passage constraints and completeness of the relevant-object set, exposing misjudgments and omissions that persist despite correct local recognition or answers. SAW-Bench already evaluates observer-centric spatial, motion, and action reasoning, and DRAMA includes risk-object localization and object-level questions. The table below specifies what the additional measurements require.
| Evaluation | Required output | Scoring basis |
|---|---|---|
| SAW-Bench | Selected options for questions about position, direction, routes, or action feasibility | Multiple-choice accuracy |
| DRAMA: main evaluation in the original paper | Bounding box of an important risk object and a description of its risk | Box overlap and description-quality metrics |
| EgoPass: specified-object judgment | Change in the same object's passage constraint on the wearer between two specified times | Change-category accuracy |
| EgoPass: uncued discovery | All relevant objects and their change labels, with an empty set permitted | One-to-one matching, omissions and extra reports, and exact set-and-label correctness |
In Figure 3, the pedestrian approaches but steps aside to open the path, so the reference is decreasing constraint. Correctly locating the pedestrian or describing the approach can still coexist with reversing this direction of change. Linked questions provide direct evidence: among 67 comparisons with stationary objects and increasing passage constraints, Astra recognizes stationarity in every case but misjudges the passage change in 21. The motion and passage answers are therefore scored separately as correct and incorrect.
Among 56 linked comparisons where Astra correctly judges the specified object's passage impact, it omits the same object in 8 independently answered uncued responses. These omissions reduce discovery credit and prevent exact set-and-label correctness. The scores distinguish explaining one object from identifying all relevant objects and correctly interpreting their changes, addressing which objects require attention and when attention should increase or decrease. Sections 5.2–5.3 provide the analyses.
W2. Insufficient coverage
Thank you for this suggestion. The added data cover interactions with traffic participants and static passage constraints, including complex scenes with multiple object types. Of the 284 evaluation clips, 260 assess 11 types of passage-relevant objects. The table follows the paper's categories, grouping types occurring in fewer than 6 evaluation clips as Other. A clip may contain multiple types.
| Assessed object | Evaluation clips | Assessed object | Evaluation clips |
|---|---|---|---|
| Cars and other road vehicles | 80 | Bollards, posts, and cones | 54 |
| Riders with their vehicles | 64 | Trees, branches, and hedges | 47 |
| Pedestrians | 50 | Street furniture, signs, and carts | 26 |
| Two-wheelers assessed separately | 11 | Gates and fences | 11 |
| Curbs, steps, and ground boundaries | 68 | Other (pushchairs, rocks, and similar obstacles) | 5 |
All 14 visual configurations score higher on discovery in crossing encounters than in passages constrained by fixtures or vegetation. Astra scores 60.48% on crossing encounters (62 nonempty questions) and 29.83%–30.31% on fixture- or vegetation-constrained passage (138 nonempty questions). Among the 145 evaluation clips containing a nonempty discovery reference, 23 involve crossing paths, 51 involve space constrained by fixtures or vegetation, 36 involve oncoming encounters, 32 involve ground boundaries, 12 involve same-direction travel or following, 8 involve passing stationary people or vehicles, and 10 involve other interactions. Categories can overlap within a clip (Section 5.1; Appendix D.1, Tables 19–20).
The always-empty score is determined by the proportion of empty references. This proportion decreases from 65/110 (59.09%) before expansion to 211/574 (36.76%) after expansion. Astra scores 55.69%–55.81% on these 574 questions.
-
The revision adds statistics on object types and interaction scenarios. The rationale for choosing these categories and assessing their coverage remains unclear. Were the categories predefined or derived from the collected examples? Please explain why types such as curbs, steps, and ground boundaries are grouped together, and how you checked for missing or underrepresented obstacle scenarios.
-
The authors clarify that the task evaluates changes in passage constraints and add paired analyses showing that correct motion recognition can coexist with incorrect passage judgments. However, the practical significance of the HIGHER/LOWER labels remains unclear. An object labeled LOWER may still require avoidance.
Thank you for the follow-up feedback; we clarify both points below.
1. Category design and coverage checks. Categories were defined before data collection based on common situations encountered in everyday walking. We cross-check the predefined categories against targets in specified-object questions and reference entities in uncued discovery questions, counting each object category once per evaluation clip. We then inspect the constituent types within merged categories and the distribution of oncoming, crossing, same-direction, and overtaking interactions to identify gaps and rare situations obscured by broad category totals. All 11 object categories are represented; counts and rare types are reported in Appendix D.1. Curbs, steps, and ground boundaries are grouped as terrain constraints because elevation changes and edges of walkable surfaces affect foot placement and available passage space, potentially causing trips or falls if overlooked.
2. Practical value of constraint changes. We introduced HIGHER/LOWER to reduce reliance on individual interpretations of “risk” by comparing how the same object’s passage constraints change between two time points. LOWER indicates reduced constraint; even when avoidance remains necessary, whether constraints are increasing or easing informs updates to alerts and preparations to slow down or turn. To assess whether these criteria can be understood and applied consistently, we compared the two participants’ independent responses. Across the 295 entities matched in both responses, change-label agreement was 96.95%, with Cohen’s κ = 0.946, supporting the reproducibility of judgments under these criteria.
Thank you for this suggestion. The revision retains the five original tasks and adds complex passage situations, native-video comparisons, and human evaluation under the same conditions to test the measurement value of the existing design. The tasks separately assess motion state, changes in passage constraints on the wearer, and completeness of the relevant-object set; each is answered independently and scored separately. The linked results above show that correct motion judgments can coexist with incorrect constraint-change judgments, and correct specified-object judgments can coexist with omissions in uncued discovery. These results demonstrate the design's diagnostic value: distinguishing correct local answers from complete understanding of a passage situation, and locating errors in impact judgment or object coverage.
Yes. The expansion covers both interactions with traffic participants and static passage constraints. In the full interaction statistics, 36 evaluation clips involve oncoming encounters, 23 involve crossing paths, 51 involve passage constrained by fixtures or vegetation, and 32 involve ground boundaries. Categories can overlap. These situations include the combined effects of pedestrians and vehicles with curbs, vegetation, gates, or other objects. Appendix D.1 provides the distributions.