acceptodds
Withdrawn from ICLR 2027

VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

28 views

Abstract

Blindness and low vision limit many people's ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains  1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14 representative vision-language models zero-shot on VISTA and design a lightweight multimodal model baseline that integrates all six modalities. Experimental results show that, although existing models can often describe egocentric scenes, they remain limited in generating goal-directed assistive guidance. VISTA thus provides a benchmark for evaluating multimodal models that could improve everyday accessibility.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Egocentric Video Understanding, Multimodal Benchmarking, Assistive AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZGhWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a new egocentric multimodal benchmark named VISTA, which aims to assist blind and low-vision users. The most distinctive features of this dataset are its rich multimodal signals and goal-driven action instructions. Experiments on this dataset reveal that goal-driven assistance can potentially raise safety issues. However, the involvement of blind and low-vision people is very limited. Without careful involvement from this community, this dataset could be just another general egocentric dataset.

Critical strengths

Rich multimodal signals and a focus on assistance for blind and low-vision people (although not validated).

Critical weaknesses
  1. Limited involvement of the blind and low-vision community. To build these kind of dataset, feedbacks from these specific community is most important. For example two prior work established the involvement of these communities [1, 2].
  2. Potentially inappropriate modalities, such as gaze and IMU. Gaze and IMU signals collected from sighted people might not be helpful for blind and low-vision people.

[1] Xiao, Junbin, et al. "Egoblind: Towards egocentric visual assistance for the blind." Advances in Neural Information Processing Systems 38 (2026). [2] Song, Inpyo, et al. "Video question answering for people with visual impairments using an egocentric 360-degree camera." arXiv preprint arXiv:2405.19794 (2024).

Questions for authors

Did the construction of this benchmark involve blind and low-vision people? As I understand it, the paper includes only one reviewer from this community, who reviewed 27 clips and 34 annotations (a very small proportion of the benchmark).

Reviewer_HvjQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VISTA, a multimodal egocentric benchmark for goal-oriented assistance to blind and low-vision (BLV) users. The benchmark comprises 997 recordings spanning ten assistive task categories, annotated for scene description, visual question answering, and goal-conditioned guidance. It further provides synchronized wearable sensor streams, including gaze, motion, hand tracking, and spatial trajectories. The authors evaluate 14 vision-language models and propose a multimodal baseline that integrates additional sensor information into pretrained VLMs. The experiments demonstrate that goal-conditioned guidance remains challenging for existing models, while the proposed adaptation framework improves performance across several backbones.

Critical strengths
  1. The paper addresses an important gap between visual scene understanding and actionable assistance for BLV users. The combination of scene descriptions, visual question answering, and goal-conditioned instructions offers a useful extension to conventional egocentric VQA benchmarks.

  2. VISTA incorporates synchronized wearable sensor streams that could support future research on multimodal assistive systems. The evaluation covers a diverse set of VLMs and highlights meaningful limitations in goal-conditioned guidance, providing useful insights for the community.

Critical weaknesses
  1. The experiments do not adequately isolate the contribution of multimodal sensor fusion. In Table 3, additional sensor inputs and LoRA-based task adaptation are introduced simultaneously, without a controlled RGB-only fine-tuning baseline. Furthermore, modality-specific ablations are not provided, making it difficult to determine which sensor streams contribute to the observed improvements and whether all six modalities are necessary. The manuscript also does not report repeated-run statistics or provide an accessible implementation repository. These limitations weaken the experimental evidence supporting the effectiveness and reproducibility of the proposed multimodal framework.

  2. The recordings were collected by sighted participants simulating assistive scenarios, and direct BLV community evaluation covers only 27 clips and 34 annotations. Moreover, model-generated guidance is evaluated using a single automatic judge, without reported human validation of its assessments. Given the safety-sensitive nature of navigation and hazard-detection tasks, stronger user-centered evaluation and human–judge agreement analysis would improve confidence in the benchmark's reliability and practical relevance.

Questions for authors
  1. Could the authors provide an RGB-only LoRA baseline and modality-specific ablations under identical training settings to clarify the independent contributions of additional sensor modalities?

  2. Could the authors provide human validation of the automatic judge, particularly regarding the usefulness and safety of generated guidance? How well do its assessments align with judgments from BLV users or human experts?

Reviewer_XW3bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VISTA, a new benchmark for AI systems measuring their goal-oriented guidance capabilities for blind and low vision (BLV) users. The dataset in total covers around 1000 recordings (~3h of video in total), split into train/validation/test sets. The data was recorded with Gen1 Aria glasses which record sensor information missing from prior BLV assistance benchmarks, such as IMU, GPS, barometer, magnetometer, SLAM cameras, and gaze and hand tracking. The benchmark further includes natural language annotations used for scene description, VQA, or instruction tasks. The paper provides baselines for the benchmark, relying on off-the-shelf multimodal LLMs and a LoRA-finetuned VLM consuming additional sensor streams provided by the Aria/VRS recordings. The baselines highlight a research gap in goal-oriented assistance tasks and demonstrate that sensor fusion can meaningfully improve model performance on these tasks.

Critical strengths
  1. The paper introduces a useful benchmark that goes beyond the current “offering” in terms of task and modality coverage, and it targets a meaningful human-centered problem/task.
  2. The BLV community was involved in the creation of the benchmark by reviewing a subset of clips. (though it would be helpful to know the degree of their vision impairment and whether other members of the community agree with their assessments).
Critical weaknesses
  1. The benchmark is relatively small (only about 3 hours across all data splits). This might be sufficient for pure benchmarking/evaluation but rather limited for the `"development of assistive models that use synchronized wearable sensor streams'' as the paper states, as it is unlikely the training split is useful beyond LoRA-style adaptation of already capable models.
  2. There seems to be no held-out/hidden portion (e.g. as part of a leaderboard) or canary that could avoid the test set from leaking into MLLMs training data.
  3. The central contribution is a benchmark but the paper does not describe the release plan, license information, hosting and Croissant files, the content/coverage of consent forms, geographic location of data collection (e.g. relevant as biometric eye tracking data collected in certain regions falls under legislations such as the EU AI Act, GDPR or BIPA), etc. – I highly recommend adding a proper data card (https://arxiv.org/abs/2204.01075) or follow guidelines from dataset/benchmark tracks like NeurIPS E&D (https://neurips.cc/Conferences/2026/CallForEvaluationsDatasets, Metadata and Responsible AI.)
  4. The evaluation protocol is based on a single grader model (GPT 6 Luna), and there appears no human validation for this protocol nor reported agreement with other model-based graders.
  5. Some related work is missing entirely, e.g. I recommend positioning the work with Karamolegkou et al. (https://aclanthology.org/2025.acl-long.1260/). Some is already cited but should be included in Table 1 (WalkVLM and GuideDog).
Questions for authors
  1. What recording profile was used for the Aria glasses? The different profiles vary significantly in e.g. framerate and resolution.
  2. To what extent were members of the BLV community involved in the ideation of this study, and to what extent does it serve the communities’ needs? This ties in with the stated limitation that data collectors were sighted, so their movement patterns and goal assistance scenarios might not reflect true BLV patterns.
  3. (suggestion) Basic training/test split statistics should be provided in the main paper – they are currently hard to find in the appendices.
  4. The paper's example for a goal–instruction pair seems highly flawed for someone who can neither see a paved path, nor a dirt bed: “Instruction: “Take 2 steps to the right, and follow the paved path around the bush, avoiding the dirt bed.” Yet this instruction seems to have passed several rounds of reviewing. What were the exact instructions given to annotators and reviewers?
  5. Relatedly, the paper mentions “trained annotators” – how and for what were they trained exactly?
  6. Why are Aria sensor streams represented by 8 learnable queries specifically? This decision seems somewhat ad-hoc, so I wonder if it was ablated or inspired by a published technique.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.