acceptodds
Withdrawn from ICLR 2027

EgoPass-Bench: Identifying Objects That May Affect the Wearer's Passage in Egocentric Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding an egocentric environment requires reasoning about how surrounding entities affect the observer's actions. Existing evaluations of multimodal foundation models (MFMs) examine observer localization, motion trajectories, and action feasibility, but leave the discovery and interpretation of evolving observer–entity interactions insufficiently assessed. We introduce EgoPass-Bench, a benchmark for observer-relative interaction understanding in real-world egocentric video. EgoPass-Bench comprises 95 manually reviewed egocentric clips recorded using Ray-Ban Meta (Gen 2) smart glasses and 577 distinct questions across five tasks assessing motion perception, interaction judgments about specified entities, and relevant-entity discovery without target cues. Our evaluation of eight MFMs reveals a pronounced discrepancy between motion perception and interaction understanding. On EgoPass-Bench, GPT-6 Astra leads on most evaluation metrics and achieves approximately 90% accuracy in recognizing both observer and external-object motion, yet it achieves only 60.00% on relevant-entity discovery without target cues. Further analysis reveals frequent inclusion of entities that neither pose a plausible physical conflict nor constrain passage, showing that accurate motion recognition does not ensure reliable relevant-entity discovery.

Primary area:
datasets and benchmarks
Keywords:
egocentric video understanding, multimodal foundation models, video benchmarks, motion perception, safe passage

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XbYNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EGOPASS-BENCH to address the insufficient evaluation of changes in observer–entity interactions in existing benchmarks. The benchmark contains 95 egocentric videos and 577 distinct questions across five tasks: wearer translation, viewpoint and path turning, external-object motion, determining whether an object impedes passage, and detecting objects that impede passage. GPT-6 Astra leads on most evaluation metrics on EGOPASS-BENCH. However, its performance in detecting objects that impede passage is less than one percentage point above the baseline that always returns an empty set, showing that accurate motion recognition does not ensure reliable detection of objects that impede passage.

Critical strengths

This work evaluates existing models' ability to detect objects that impede passage in egocentric navigation scenes. It can be used to assess models' ability to assist wearers with navigation and obstacle avoidance.

Critical weaknesses
  1. The proposed dataset offers limited novelty. SAW-Bench, MotionBench, RiskBench, and DRAMA already cover some closely related capabilities. EGOPASS-BENCH mainly differs in comparing the attention required by objects at different time points and detecting objects that impede passage in egocentric navigation scenes. Its novelty is therefore relatively limited.
  2. The proposed dataset provides insufficient coverage, with too few videos and questions. Only 251 questions concern whether objects impede passage. Although the data collection considers six types of environments, this does not establish whether key passage-risk scenarios are adequately covered. The paper lacks systematic coverage statistics and analyses based on obstacle types and interaction scenarios. Moreover, is it reasonable that the baseline that always returns an empty set achieves a Discovery score of nearly 60%?
Questions for authors
  1. To address Weakness 1, design additional new tasks for passage scenarios.
  2. To address Weakness 2, collect more data while considering coverage across different obstacle types and interaction scenarios.
4 comments

Q1. Additional passage tasks

Thank you for this suggestion. The revision retains the five original tasks and adds complex passage situations, native-video comparisons, and human evaluation under the same conditions to test the measurement value of the existing design. The tasks separately assess motion state, changes in passage constraints on the wearer, and completeness of the relevant-object set; each is answered independently and scored separately. The linked results above show that correct motion judgments can coexist with incorrect constraint-change judgments, and correct specified-object judgments can coexist with omissions in uncued discovery. These results demonstrate the design's diagnostic value: distinguishing correct local answers from complete understanding of a passage situation, and locating errors in impact judgment or object coverage.

Q2. Obstacle and interaction coverage

Yes. The expansion covers both interactions with traffic participants and static passage constraints. In the full interaction statistics, 36 evaluation clips involve oncoming encounters, 23 involve crossing paths, 51 involve passage constrained by fixtures or vegetation, and 32 involve ground boundaries. Categories can overlap. These situations include the combined effects of pedestrians and vehicles with curbs, vegetation, gates, or other objects. Appendix D.1 provides the distributions.

W1. Limited novelty

Thank you for this comment. EgoPass directly scores the direction of change in passage constraints and completeness of the relevant-object set, exposing misjudgments and omissions that persist despite correct local recognition or answers. SAW-Bench already evaluates observer-centric spatial, motion, and action reasoning, and DRAMA includes risk-object localization and object-level questions. The table below specifies what the additional measurements require.

EvaluationRequired outputScoring basis
SAW-BenchSelected options for questions about position, direction, routes, or action feasibilityMultiple-choice accuracy
DRAMA: main evaluation in the original paperBounding box of an important risk object and a description of its riskBox overlap and description-quality metrics
EgoPass: specified-object judgmentChange in the same object's passage constraint on the wearer between two specified timesChange-category accuracy
EgoPass: uncued discoveryAll relevant objects and their change labels, with an empty set permittedOne-to-one matching, omissions and extra reports, and exact set-and-label correctness

In Figure 3, the pedestrian approaches but steps aside to open the path, so the reference is decreasing constraint. Correctly locating the pedestrian or describing the approach can still coexist with reversing this direction of change. Linked questions provide direct evidence: among 67 comparisons with stationary objects and increasing passage constraints, Astra recognizes stationarity in every case but misjudges the passage change in 21. The motion and passage answers are therefore scored separately as correct and incorrect.

Among 56 linked comparisons where Astra correctly judges the specified object's passage impact, it omits the same object in 8 independently answered uncued responses. These omissions reduce discovery credit and prevent exact set-and-label correctness. The scores distinguish explaining one object from identifying all relevant objects and correctly interpreting their changes, addressing which objects require attention and when attention should increase or decrease. Sections 5.2–5.3 provide the analyses.

W2. Insufficient coverage

Thank you for this suggestion. The added data cover interactions with traffic participants and static passage constraints, including complex scenes with multiple object types. Of the 284 evaluation clips, 260 assess 11 types of passage-relevant objects. The table follows the paper's categories, grouping types occurring in fewer than 6 evaluation clips as Other. A clip may contain multiple types.

Assessed objectEvaluation clipsAssessed objectEvaluation clips
Cars and other road vehicles80Bollards, posts, and cones54
Riders with their vehicles64Trees, branches, and hedges47
Pedestrians50Street furniture, signs, and carts26
Two-wheelers assessed separately11Gates and fences11
Curbs, steps, and ground boundaries68Other (pushchairs, rocks, and similar obstacles)5

All 14 visual configurations score higher on discovery in crossing encounters than in passages constrained by fixtures or vegetation. Astra scores 60.48% on crossing encounters (62 nonempty questions) and 29.83%–30.31% on fixture- or vegetation-constrained passage (138 nonempty questions). Among the 145 evaluation clips containing a nonempty discovery reference, 23 involve crossing paths, 51 involve space constrained by fixtures or vegetation, 36 involve oncoming encounters, 32 involve ground boundaries, 12 involve same-direction travel or following, 8 involve passing stationary people or vehicles, and 10 involve other interactions. Categories can overlap within a clip (Section 5.1; Appendix D.1, Tables 19–20).

The always-empty score is determined by the proportion of empty references. This proportion decreases from 65/110 (59.09%) before expansion to 211/574 (36.76%) after expansion. Astra scores 55.69%–55.81% on these 574 questions.

  1. The revision adds statistics on object types and interaction scenarios. The rationale for choosing these categories and assessing their coverage remains unclear. Were the categories predefined or derived from the collected examples? Please explain why types such as curbs, steps, and ground boundaries are grouped together, and how you checked for missing or underrepresented obstacle scenarios.

  2. The authors clarify that the task evaluates changes in passage constraints and add paired analyses showing that correct motion recognition can coexist with incorrect passage judgments. However, the practical significance of the HIGHER/LOWER labels remains unclear. An object labeled LOWER may still require avoidance.

Thank you for the follow-up feedback; we clarify both points below.

1. Category design and coverage checks. Categories were defined before data collection based on common situations encountered in everyday walking. We cross-check the predefined categories against targets in specified-object questions and reference entities in uncued discovery questions, counting each object category once per evaluation clip. We then inspect the constituent types within merged categories and the distribution of oncoming, crossing, same-direction, and overtaking interactions to identify gaps and rare situations obscured by broad category totals. All 11 object categories are represented; counts and rare types are reported in Appendix D.1. Curbs, steps, and ground boundaries are grouped as terrain constraints because elevation changes and edges of walkable surfaces affect foot placement and available passage space, potentially causing trips or falls if overlooked.

2. Practical value of constraint changes. We introduced HIGHER/LOWER to reduce reliance on individual interpretations of “risk” by comparing how the same object’s passage constraints change between two time points. LOWER indicates reduced constraint; even when avoidance remains necessary, whether constraints are increasing or easing informs updates to alerts and preparations to slow down or turn. To assess whether these criteria can be understood and applied consistently, we compared the two participants’ independent responses. Across the 295 entities matched in both responses, change-label agreement was 96.95%, with Cohen’s κ = 0.946, supporting the reproducibility of judgments under these criteria.

Reviewer_tdTGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces EgoPass-Bench, a benchmark for identifying entities that may affect a smart-glasses wearer’s passage and judging how their influence changes. It contains 95 egocentric clips and 577 questions across five tasks covering motion perception, specified-object passage relations, and uncued entity discovery. Eight models are evaluated, with preceding-clip context tested on a subset. The authors report frequent entity-selection and relation errors despite strong performance on motion recognition.

Critical strengths

The benchmark uses real smart-glasses footage and separates motion perception, specified-entity relation judgments, and target-free discovery on shared clips, which allows these abilities to be compared directly. The paper is also transparent about its own limitations, including the empty-set baseline and the human–model input difference.

Critical weaknesses

1. The reference judgments need stronger validation. Reference answers are provided by the video recorders. Although two participants answered the questions independently, the paper does not clarify whether they were independent of the reference annotators or how disagreements with the references were resolved. The prompts define passage relevance through physical conflict and available space, but do not sufficiently explain how competing cues determine a unique attention-change label. For example, Fig. 3 labels an approaching pedestrian as requiring less attention because they move aside, without clearly explaining how proximity, urgency, and path separation are weighed. Human agreement is lowest for turning and passage relations (κ ≈ 0.60), making independent validation particularly important. The benchmark contains 95 clips from 42 recordings and 110 free-form questions, only 45 of which have nonempty references. The insufficient-evidence analysis uses a 16-question diagnostic subset. Without uncertainty estimates that account for questions sharing source footage, the stability of model differences remains unclear.

2. The motion-vs-discovery gap is hard to interpret. The paper contrasts approximately 90% accuracy on wearer-translation and external-object-motion questions with a 60.00% entity-discovery score, but the latter is a custom penalized set score rather than ordinary accuracy. With 65 of 110 discovery questions having empty references, always returning an empty set scores 59.09%, virtually matching the best model and exceeding all others. The penalty for additional reports makes always returning an empty set an effective shortcut, while the trade-off between false reports and missed passage-relevant entities is not justified. Although the authors report nonempty-case results and precision/recall, this comparison does not cleanly isolate differences in the underlying capabilities. The paper also describes one-to-one entity matching without specifying who or what performed it, how ambiguous descriptions were resolved, or whether matching was blind to model identity. Finally, the 91.14% human discovery score comes from candidate-assisted responses, while models receive no candidate list. This misalignment further limits its interpretation as a human–model capability gap.

Questions for authors

Needs further clarification:

  1. Were the two human participants independent of the video recorders and reference annotators, and did they receive the same relevance rules as the models (App. D.5)? How were disagreements with the references resolved?
  2. How were the specified objects for the external-object-motion and passage-relation tasks selected, and by whom? Please report the answer distributions for both tasks, including how often the answer is "not relevant at either time."
  3. How was the one-to-one matching between free-form descriptions and reference entities performed, and was it blind to model identity? What motivates the metric's trade-off between false reports and missed entities, given that always returning an empty set nearly matches the best model?
  4. Beyond SAW-Bench's action-feasibility task, what do the passage-relation and discovery tasks evaluate? Please substantiate the ratings in Table 1, add dataset scale, annotation source, and validation procedures, and discuss related work on assistive navigation for blind and low-vision users.

Suggestions, not hard requests:

  1. Feature the empty- and nonempty-case discovery scores more prominently, add blind (question-only) and trivial baselines, evaluate humans without candidate lists, and report uncertainty estimates that account for questions sharing source recordings.
  2. The abstract should note the 59.09% empty-set baseline alongside the 60.00% discovery score and clarify that "attention requirement" refers to physical passage constraints. Fig. 3's caption should be more informative to help explain both panels.
4 comments

E1. Consent, privacy, and data release

Thank you for this suggestion. The example referenced in the review now appears in Figure 3, with identifiable faces obscured. All camera wearers consented to recording. Before release, we will further protect incidental bystanders' privacy by obscuring faces, legible license plates, and other identifying details in each clip, while verifying that motion and passage space remain interpretable.

Upon acceptance, we will publicly release identity-obscured evaluation clips, questions, reference answers, and scoring materials. The release documentation will specify access procedures, the data license, and permitted uses.

Q4. Related benchmarks and assistive navigation

Thank you for this suggestion. SAW-Bench already evaluates observer-centric spatial, motion, and action reasoning. EgoPass makes changes in each object's passage constraint and completeness of the relevant-object set explicit output requirements and scoring targets. Accordingly, Table 1 assigns EgoPass ✓ in the cued reasoning and uncued discovery column, while RiskBench's risk-object localization and DRAMA's object-level questions receive △. Ratings depend on whether the task explicitly requires the output; △ indicates evaluation of a related capability. Appendix D.6 discusses the task definitions, scale, and direct comparison using Figure 3. Our reference construction, independent human evaluation, and blinded scoring are described in Sections 3.6 and 4.1 and Appendices B.2–B.3 and C.2.

In a field evaluation of navigation assistance, UNav provided turn-by-turn guidance to 20 blind and low-vision participants on unfamiliar routes, assessing wayfinding through navigation errors and completion time. Pundlik et al. studied walking obstacle avoidance in 31 participants with severe visual impairments. With participants continuing to use their usual white cane or guide dog, wristband vibration warnings reduced obstacle contacts relative to a silent condition. UNav field evaluation · Wearable collision-warning trial

EgoPass evaluates the visual judgments that precede an obstacle warning: identifying passage-relevant objects and determining whether their constraints on the walker are increasing or decreasing. In Figure 3, the pedestrian approaches but yields space; 6 of 8 frame-based configurations nevertheless predict higher required attention. Treating decreasing distance as increasing passage difficulty in such situations could produce unnecessary alerts for blind and low-vision users.

S5. Baselines and uncertainty

Comparisons on the same questions show that visual input improves accuracy; humans also outperform Astra on discovery questions with both empty and nonempty references. GPT-5.6 Sol scores 45.54% with questions alone and 57.47% with frames, a paired gain of 11.9 ± 3.1 percentage points (95% CI; main-text Table 2 and Appendix C.2).

On the 211 discovery questions with empty references, Astra scores 69.67% and the two participants average 83.65%. On the 363 questions requiring objects, Astra scores 47.57%–47.75%, the participants average 77.02%, and the always-empty baseline scores 0 (Appendix C.2, Table 15).

S6. Abstract and Figure 3

The abstract defines the physical meaning of required attention and reports the human–model gap and empty-set comparison on all 574 discovery questions in the same sentence. Required attention is the need to monitor an object, prepare to respond, or adjust movement because it may constrain passage. Across all 574 discovery questions, the two-person mean exceeds even Astra's upper-bound score by 23.65 ± 7.45 percentage points (paired 95% CI). The abstract also reports the always-empty baseline of 36.76% and Astra's advantage of at least 18.9 percentage points on the same question set. The corresponding estimate from cluster resampling is 18.9 ± 8.5 percentage points (paired 95% CI); Section 5.1 and Appendix C.2 provide the calculations. Figure 3 shows the approaching but yielding pedestrian on the left. On the right, the upper and lower panels report, by model family, matched and extra entity counts (a), and discovery scores for empty and nonempty references (b). Each row displays a lower bound as the main value and a rightmost “+” increment for the maximum addition allowed by unresolved identity matches. The increment is measured in entities in (a) and percentage points in (b).

Q2. Target selection and answer distributions

Camera wearers nominate targets, and the research team finalizes them. The questions explicitly include objects that do not affect passage, so naming a target does not reveal its relevance. If the observations establish that an object does not affect passage at either time, the reference is irrelevant at both times. If visibility or trajectory evidence is inadequate, the reference is insufficient evidence. The irrelevant-at-both-times category accounts for 268/675 questions (39.70%) in the current evaluation, compared with 86/141 (60.99%) before expansion.

Object-motion categories distinguish stationary, moving, a change in motion state, and insufficient evidence. Passage-impact references are distributed as follows.

Passage-impact referenceQuestions
Irrelevant at both times268
Passage constraint increases181
Passage constraint decreases181
Insufficient evidence45
Total675

This distribution also appears in Appendix D.2, Table 22.

Q3. Blinded matching and scoring

Model names are hidden during entity matching. Human scorers identify objects question by question using appearance, position, and trajectory. When a case is ambiguous, two scorers revisit the observations, discuss the evidence, and record their judgment. A further review covers 717 distinct descriptions on 63 questions with complete references; both model names and predicted change labels are hidden during matching. The scoring records distinguish human judgments, tool-assisted matches, and unresolved identities.

When a description can still refer to multiple objects, we compute each model's lowest and highest feasible scores under one-to-one matches consistent with the visual evidence. Full-set bounds for discovery and exact set-and-label accuracy retain all 574 questions. Main-table point estimates use cases whose score is determined for that metric, with denominators reported separately. Conditional label accuracy is the pooled ratio of correctly labeled objects to matched objects. Appendix B.2 provides the calculations.

Scoring accounts for both omissions and false positives because walking assistance must identify all passage-relevant objects while avoiding unnecessary alerts.

Astra on all 574 uncued discovery questions (%)False-positive penalty 0.5Penalty 1 (default)Penalty 2
Score60.02–60.1055.69–55.8155.14–55.31

When the false-positive penalty is halved or doubled, Astra's lower-bound score remains above the always-empty baseline of 36.76% (Section 5.1).

Exact set-and-label accuracy requires the complete object set and all change labels to be correct. Conditional label accuracy measures the proportion of matched objects assigned the correct change label. Appendix C.1 reports the denominators, full-set score bounds, and 95% confidence intervals obtained by cluster resampling of evaluation clips for cases whose score is determined for the relevant metric.

W1. The reference judgments need stronger validation

Thank you for highlighting the apparently conflicting cues in Figure 3. The pedestrian moves closer but steps left, increasing the space available ahead of the wearer; the reference is therefore lower required attention. At each comparison time, we use the visible trajectories, relative arrival times, and passage width to judge whether a conflict is plausible within the next two seconds or whether the wearer needs to adjust their steps immediately. If the observations do not establish the direction of change, the reference is insufficient evidence.

  • The pilot study familiarizes participants with consistent criteria, and the team reviews and finalizes the references. Two participants who took no part in filming or reference construction first independently completed 17 practice clips and 144 pilot questions, then reviewed the criteria using the observations. The team reviewed and corrected disputed references using the same criteria.
  • The formal evaluation preserves independent responses and uses the same reference version. Both participants independently answered 2,201 benchmark questions. They could replay the permitted observation windows but did not discuss responses or receive references or feedback. The table below reports agreement on their formal responses.
TaskPreviously reported κAgreeing formal responsesAgreementCurrent Cohen's κ
Viewpoint and path turning0.61215/26082.69%0.745
Passage impact of a specified object0.60554/67582.07%0.750

Questions with an insufficient-evidence reference have increased from 16 before expansion to 72. On these 72 questions, 9 of the 14 visual configurations answer none correctly, and the best result is 8/72. To estimate sampling uncertainty, we perform 10,000 bootstrap resamples of the 284 evaluation clips, grouped by collection source and stratified by collection batch, using shared draws across all models and baselines. Human scores are the mean of the two participants on the sampled questions. The statistical method is in Appendix B.2, human results in Appendix C.2, and complete insufficient-evidence results in Appendix D.4.

W2. Motion-vs-discovery gap

Thank you for distinguishing these metrics. Linked responses show errors in both passage-impact judgment and uncued discovery; humans still outperform Astra under the same uncued conditions. We link independently answered questions about the same scene and object. Among 313 comparisons where Astra correctly answers both wearer-translation and object-motion questions, it answers the object's passage question incorrectly in 96. Among 152 passage–discovery comparisons linked by explicit reference correspondence and identical observation windows, Astra answers 56 passage questions correctly but omits the same object in 8 of the corresponding uncued responses.

After removing candidate cues, the two-person mean discovery score still exceeds Astra's highest feasible score by 23.65 ± 7.45 percentage points (paired 95% CI). The two participants average 79.46%, while Astra's full-set upper bound is 55.81%. Both receive the same questions and observation windows and are assessed using the same scoring rules. The blinded object-matching procedure is described in Question 3 below.

Q1. Independent human evaluation

Yes. Neither participant took part in filming or reference construction, and the formal evaluation used the same criteria as the model evaluation. The undergraduate and graduate participants first completed 17 practice clips and 144 pilot questions, then answered independently. The team reviewed disputed references using consistent criteria, while each participant's formal responses were retained separately. One turning question whose corrected reference had no matching answer option was excluded for every model and participant, leaving 2,201 questions for formal scoring. All 574 discovery questions provided no candidate objects; respondents had to supply their own object descriptions and change labels.

The participants' mean multiple-choice accuracy is 85.80%, and their mean uncued discovery score is 79.46%. Agreement and Cohen's κ by task are reported in Appendix C.2, Table 13.

Reviewer_FiseOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • The paper proposes a new benchmark, EgoPass-Bench, for evaluating whether multimodal models understand how surrounding objects affect the wearer's passage in egocentric video.
  • Data: 95 clips recorded with Ray-Ban Meta (Gen 2), across six scene types (intersections, sidewalks, residential areas etc). 2 FPS.
  • What is passage relevance? It is defined as a plausible collision or insufficient passage space within a 2 second horizon.
  • Evaluations: eight top models (GPT, Gemini, etc), with and without the preceding clip history. Two human raters.
  • The results show that the best models (GPT-6 Astra) achieves 90% on wearer and object motion. And only 60% on entity discovery. Humans are at 91%.
Critical strengths
  • Interesting and novel task. Egocentric multimodal evaluations are valuable as models improve performance in this space. This benchmark can open doors for more research efforts in this direction.

  • The task design is clean and simple. Separation between motion perception and relational reasoning and discovery.

  • The authors have captured real egocentric data from consumer glasses and evaluated multiple frontier models manually.

Critical weaknesses
  • Limited scale of the benchmark: EgoPassBench consists of 95 clips from 42 videos. Only 53 reference objects across 110 discovery questions. In comparison to existing video benchmarks like MVBench, EgoSchema, the scale is tiny. It would also be helpful to include statistics on the datasets like clip duration, object categories and scene diversity in the paper. With such a small scale, drawing reliable conclusions is hard.

  • Entity discovery evaluation: Please correct me if I am wrong here, 65 of 110 discovery questions have an empty reference set, so always predicting an empty set scores 59%, whereas Astra is at 60%? Fixing this skew in the evals would be helpful. A trivial baseline should not be this close to the best performing model.

  • Annotation reliability and reproducibility: The reference answers are provided by the subjects, and it appears that the inter-annotator agreement is moderate for relations (about 0.6). More details on how exactly the annotations were obtained and the quality checks would be helpful. For instance, it is not clear how the free form descriptions are matched to reference objects? Additionally, please clarify if the dataset will be made public and the license details.

  • Baselines: The models evaluated are not inherently video models. Currently API models are provided with contact sheets instead of videos. However the benchmark and the task are temporal in nature. It would be helpful to evaluate video LLMs like Qwen-VL or LLaVA-Video as additional video baselines on the dataset.

Questions for authors
  • The scale of dataset. Do the authors feel meaningful conclusions can be draw from the current eval set?
  • The skewed evaluations with a trivial baseline
  • Quality guarantee for the annotations and reproducibility of the evals (without a human rater).
  • Additional video LLMs as baselines.
2 comments

Q1. Dataset scale

Thank you for this question. The expanded evaluation covers 284 evaluation clips of walking situations, 2,201 benchmark questions, and 14 visual configurations. The analyses in Section 5 support a common conclusion: correct recognition of local states does not yet ensure a complete and reliable understanding of the wearer's passage situation in the evaluated models. Specifically:

  • Correct recognition of an object's motion can coexist with an incorrect judgment of its impact on the wearer's passage.
  • Correct judgments about discovered objects can coexist with omissions of other passage-relevant objects, leaving the situation incompletely understood.
  • When visual evidence is insufficient, the evaluated models tend to give definite answers rather than reliably recognize insufficient evidence.

Q2. Trivial baseline

Astra's advantage over the always-empty baseline comes from genuine object discovery on nonempty questions. Across all 574 questions, Astra reports 503 objects and correctly matches 278–279 of the 431 reference entities. The always-empty baseline matches 0.

On the 211 questions with empty references, the always-empty baseline scores 100%, compared with Astra's 69.67%. On the 363 questions requiring objects, the baseline scores 0%, compared with Astra's 47.57%–47.75%.

Q3. Annotation quality and reproducibility

The reported scores can be recomputed using the scoring scripts. The scripts read the recorded object correspondences and match counts. Ambiguous new descriptions are handled through the same blinded review procedure. Across 14 configurations and 2,201 questions, 30,796 of the 30,814 planned responses are valid. We retain the earliest valid output for each question. The 18 cases without a valid response receive zero on question-level metrics; conditional label accuracy is computed over successfully matched objects. Appendix B.2 describes the materials and methods.

Q4. Additional video LLMs as baselines

We added six native-video configurations (W4). For Qwen3.8 Max and Flash, native-video configurations have slightly higher overall multiple-choice accuracy, while frame-based configurations have higher discovery scores. Both conditions use the same benchmark questions.

ModelMultiple-choice accuracy: frames / video (%)Discovery: frames (%)Discovery: video (%)
Qwen3.8 Max58.39 / 58.5743.50–44.0539.66–40.53
Qwen3.8 Flash54.46 / 56.3639.40–40.6534.09–35.69

W1. Limited scale of the benchmark

Thank you for this suggestion. The expanded evaluation covers both dynamic traffic interactions and static spatial constraints. The combined number of specified-object passage questions and uncued discovery questions has increased from 251 to 1,249. Discovery questions contain 431 reference entities, counted across questions. Across the full benchmark, 80 evaluation clips assess road vehicles and 68 assess curbs, steps, or other ground boundaries. Interaction coverage includes 36 clips with oncoming encounters, 23 with crossing paths, and 51 with passage constrained by fixtures or vegetation. The median observation interval is 2.5 seconds, with a range of 2–5 seconds (Sections 3.4–3.5; Appendix D.1).

Evaluation unitBefore expansionCurrent evaluation
Evaluation clips / benchmark questions95 / 577284 / 2,201
Passage judgments for specified objects141675
Uncued object discovery110574
Discovery questions requiring at least one object45363

W2. Entity discovery evaluation

Thank you for identifying this shortcut. After expansion, Astra exceeds the always-empty baseline by 18.9 ± 8.5 percentage points (paired 95% CI). We added complex scenes with multiple object types and passages constrained by fixtures or vegetation, increasing the number of discovery questions requiring objects from 45 to 363. The always-empty score consequently falls from 59.09% to 36.76%. The following table compares performance on all 574 discovery questions in the expanded benchmark.

Discovery score (%)Empty reference · 211 questionsNonempty reference · 363 questionsAll · 574 questions
Always-empty baseline100.000.0036.76
GPT-6 Astra69.6747.57–47.7555.69–55.81

The ranges in the table reflect ambiguity in object identity; the 95% CI above reflects variability under cluster resampling.

W3. Annotation reliability and reproducibility

Thank you for this suggestion. Agreement between the two independent participants on passage judgments is Cohen's κ = 0.750. Camera wearers proposed the references, which the team reviewed and corrected against the observations permitted for each question, using consistent criteria. Two participants who took no part in filming or reference construction first completed 17 practice clips and 144 pilot questions, then independently answered all 2,201 benchmark questions. Their passage judgments agree on 554/675 questions (82.07%).

For discovery questions, both participants and models had to identify relevant objects without candidate cues and judge how each object's impact on passage changed. The first scorer matched model descriptions question by question without seeing model names. For ambiguous cases, two scorers revisited the observations, discussed the evidence, and recorded their judgment. Upon acceptance, we will release identity-obscured evaluation clips, questions, reference answers, and the available responses, scoring decisions, and scoring scripts.

W4. Baselines

Thank you for this suggestion. We have evaluated six native-video configurations on the same questions, including three Qwen configurations, as shown below. Gemma 4 31B achieves 55.00% turning accuracy, the highest among all 14 visual configurations.

Native-video configurationMultiple-choice accuracy (%)Uncued object-discovery score (%)
Qwen3.8 Max58.5739.66–40.53
Qwen3.8 Flash56.3634.09–35.69
Qwen3.5-397B-A17B55.8728.95–29.53
GLM-5.3 Flash55.1339.02–40.30
Gemma 4 31B54.7636.30–37.80
MiniMax M352.4931.18–32.46

The discovery ranges are the bounds over all 574 questions reported in Appendix C.1, Table 8; main-text Table 2 reports point estimates from cases with a determined score.

Comments on the paper

2 comments

We thank all three reviewers for their suggestions. We have added 189 evaluation clips depicting everyday walking situations. The benchmark now comprises 284 evaluation clips and 2,201 questions, excluding the pilot study. Table 2 reports results for 8 frame-based configurations, 6 native-video configurations, and two independent human participants on the same question set. Human performance is reported as the mean of the two participants.

We recruited two independent participants, an undergraduate student and a graduate student, to repeat the human baseline evaluation. They first completed a pilot study with 17 practice clips and 144 questions to familiarize themselves with the criteria, then independently answered all 2,201 benchmark questions. Humans and models received the same benchmark questions and response requirements. All 574 discovery questions provided no candidate objects; respondents had to identify the relevant objects themselves and describe how each object's impact on passage changed.

We estimate 95% confidence intervals through resampling to assess the stability of model scores and their differences from human and baseline performance. We perform 10,000 bootstrap resamples of the 284 evaluation clips, grouped by collection source and stratified by collection batch. Clips and questions within each group are sampled together, using the same draws across all models and tasks. Intervals use a normal approximation, with four additional random seeds to check numerical stability. The two participants remain fixed across resamples. Appendix B.2 provides the details.

Dear Reviewers,

Authors posted their responses to your queries. At the moment, only one reviewer has followed up on the authors' comments.

Please read the author's feedabck and post your replies so that we get a chance to clarify points which are possibly still unclear and may require further discussion.

Thank you for supporting the ICLR revision process and contributing to discussion.

Best, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.