- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a framework for proactive visual assistance for blind and low-vision (BLV) people, where an agent watching a first-person video stream decides on its own when to speak and what to say. The authors add a proactive annotation layer to 439 EgoBlind video segments (4.28 h), in which five sighted–BLV annotator pairs label 7,482 timestamped PUSH/NO-PUSH decisions and write 1,794 assistance messages. They propose BLV-SceneAgent, an ~8B streaming VLM fine-tuned on these labels, combined with a small intervention controller (TCI) that uses the VLM's speak/silence margin, scene change, and a ledger of already-delivered notifications to avoid redundant speech. On the test split, TCI improves macro-F1 by 6.3 points over the fine-tuned VLM with a P95 latency of 479 ms, and a live study with six BLV participants reports higher usefulness and trust and lower annoyance.
Critical strengths
- The paper addresses an important and socially beneficial problem. Visual assistance for BLV people remains largely reactive, requiring users to ask about cues they may not know exist. Enabling an assistant to proactively notify users of relevant changes in their surroundings, while avoiding redundant speech, could meaningfully improve their safety and independence in daily life.
- A practical real-time system evaluated with BLV users. The agent runs under 0.5 s at P95, the controller adds under 4 ms, the multimodal cache is reused for generation, and the system is tested live with BLV participants across four everyday settings.
Critical weaknesses
W1. Novelty and positioning relative to prior BLV assistance work are unclear. The videos are reused from EgoBlind [1], which already frames BLV assistance causally in real time (timestamp-restricted); the new contribution is an annotation layer on 439 segments (4.28 h). The core idea of deciding when to notify and suppressing redundant reminders was already proposed in WalkVLM [2], which uses temporal-aware adaptive prediction to reduce redundant reminders for BLV walking assistance. WalkVLM and other BLV guidance work (e.g., VL-GUIDE [3], GuideDog [4]) are neither cited nor compared. The paper should clarify what is new beyond extending this idea to more scene types.
W2. The evaluation does not show that TCI is better than existing or simpler approaches. Table 1 compares only the paper's own variants (pretrained, SFT, SFT+TCI). Moreover, decisions are evaluated only at the annotated timestamps, each from a 3-second clip and the preceding action history, rather than over the continuous stream. Although the paper argues that the task is "a sequential agent problem rather than framewise classification followed by captioning" (Sec. 1), this evaluation setting can be addressed by conventional offline VLMs prompted with the same clip and history (e.g., Qwen3-VL, GPT-4o, Gemini). Therefore, both offline VLMs and existing proactive/streaming models (e.g., MMDuet, Dispider, Proact-VL, EgoSpeak, WalkVLM), as well as simple heuristics (e.g., thresholding the native margin, similarity-based repetition filtering), are necessary baselines. Evaluating only at annotated points also means that false alarms between them, the main source of annoyance for BLV users, are never measured. Finally, results are from a single run on a small test set (64 videos), and the annotators' expertise, vision status, guidelines, and agreement are not reported (see Question 3).
W3. Content quality and the user study are under-evaluated. Assistance messages are scored only by similarity to a single reference (token F1, ROUGE-L, BERTScore), so incorrect or unsafe guidance (e.g., a wrong direction), which is critical for BLV users, is never analyzed. The user study is small (six participants, two trials per setting), reports only means without variance or significance tests, and again compares only the paper's own variants.
References
[1] Xiao et al., EgoBlind: Towards Egocentric Visual Assistance for the Blind. NeurIPS 2025 Datasets and Benchmarks. https://arxiv.org/abs/2503.08221
[2] Yuan et al., WalkVLM: Aid Visually Impaired People Walking by Vision Language Model. ICCV 2025. https://arxiv.org/abs/2412.20903
[3] Kim et al., Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals? arXiv 2025. https://arxiv.org/abs/2510.00766
[4] GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance. ACL 2026. https://arxiv.org/abs/2503.12844
Questions for authors
-
Under the per-timestamp evaluation, how do (a) offline VLMs prompted with the same 3-second clip and action history (e.g., Qwen3-VL, GPT-4o, Gemini), (b) existing proactive/streaming models (e.g., MMDuet, Proact-VL, EgoSpeak, WalkVLM's reminder mechanism), and (c) simple heuristics perform on the same test set?
-
Can the agent be evaluated at a fixed decision rate over the full stream (e.g., every 0.5 s), reporting false PUSHes between annotated points?
-
What are the backgrounds of the annotators, and how were the labels produced? Since the labels define what counts as a useful notification and come from only five sighted–BLV pairs, please describe: (a) whether any annotators had relevant expertise (e.g., orientation and mobility instructors or trained sighted guides); (b) the BLV partners' vision status (totally blind vs. low vision), age range, and experience with assistive technology; (c) the annotation guidelines, annotator training, and how disagreements were resolved; and (d) the inter-annotator agreement on PUSH/NO-PUSH decisions across pairs.
-
How often do delivered messages contain incorrect or unsafe guidance? A targeted human or model-based error analysis would help.
-
Could the authors report variance and significance tests for the user study, and whether participants knew which system they were using?
-
The paper includes a study with BLV participants but no ethics statement. Could the authors add one?
-
Which streaming VLM is used as the backbone? The paper mentions only an "8.77B streaming backbone" with native ⟨SILENCE⟩/⟨RESPONSE⟩ tokens and cites Yao et al. (2026); please state the model explicitly.