acceptodds
Withdrawn from ICLR 2027

Beyond Reactive Visual Assistance: A Real-Time Scene-Interaction Agent for Blind and Low-Vision People

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual assistance for blind and low-vision (BLV) people remains largely query-driven: users must ask about cues they may not know exist. Real-time BLV assistance from continuous first-person video instead requires an agent that decides when evidence warrants speech, avoids redundant notifications, and delivers scene-appropriate guidance while that evidence remains actionable. We introduce a BLV-informed proactive scene-interaction dataset by densely annotating 439 multi-scene first-person video segments from BLV users' daily lives, yielding 7,482 timestamp-aligned PUSH/NO-PUSH decisions along continuous trajectories and assistance messages for 1,794 PUSH decisions. We present BLV-SceneAgent, a real-time observe–decide–act–commit agent built on a streaming VLM. Its Temporal Contextual Intervention (TCI) policy integrates the native response margin with a causal recurrent belief, signed state change, and a committed-notification ledger, and is refined under the agent's predicted action history. NO-PUSH terminates at the action boundary, whereas PUSH reuses the multimodal cache for content generation. On a segment-disjoint test set, TCI improves steady-state macro-F1 by 6.3 points over calibrated SFT, with a P95 action-boundary latency of only 478.8 ms. A live study across four settings shows that the agent surpasses baselines in usefulness, trust, and reduced annoyance. We will publicly release our dataset and model to support further research on proactive visual assistance for blind and low-vision people.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
proactive visual assistance, blind and low-vision people, streaming vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UH7gOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework for proactive visual assistance for blind and low-vision (BLV) people, where an agent watching a first-person video stream decides on its own when to speak and what to say. The authors add a proactive annotation layer to 439 EgoBlind video segments (4.28 h), in which five sighted–BLV annotator pairs label 7,482 timestamped PUSH/NO-PUSH decisions and write 1,794 assistance messages. They propose BLV-SceneAgent, an ~8B streaming VLM fine-tuned on these labels, combined with a small intervention controller (TCI) that uses the VLM's speak/silence margin, scene change, and a ledger of already-delivered notifications to avoid redundant speech. On the test split, TCI improves macro-F1 by 6.3 points over the fine-tuned VLM with a P95 latency of 479 ms, and a live study with six BLV participants reports higher usefulness and trust and lower annoyance.

Critical strengths
  1. The paper addresses an important and socially beneficial problem. Visual assistance for BLV people remains largely reactive, requiring users to ask about cues they may not know exist. Enabling an assistant to proactively notify users of relevant changes in their surroundings, while avoiding redundant speech, could meaningfully improve their safety and independence in daily life.
  2. A practical real-time system evaluated with BLV users. The agent runs under 0.5 s at P95, the controller adds under 4 ms, the multimodal cache is reused for generation, and the system is tested live with BLV participants across four everyday settings.
Critical weaknesses

W1. Novelty and positioning relative to prior BLV assistance work are unclear. The videos are reused from EgoBlind [1], which already frames BLV assistance causally in real time (timestamp-restricted); the new contribution is an annotation layer on 439 segments (4.28 h). The core idea of deciding when to notify and suppressing redundant reminders was already proposed in WalkVLM [2], which uses temporal-aware adaptive prediction to reduce redundant reminders for BLV walking assistance. WalkVLM and other BLV guidance work (e.g., VL-GUIDE [3], GuideDog [4]) are neither cited nor compared. The paper should clarify what is new beyond extending this idea to more scene types.

W2. The evaluation does not show that TCI is better than existing or simpler approaches. Table 1 compares only the paper's own variants (pretrained, SFT, SFT+TCI). Moreover, decisions are evaluated only at the annotated timestamps, each from a 3-second clip and the preceding action history, rather than over the continuous stream. Although the paper argues that the task is "a sequential agent problem rather than framewise classification followed by captioning" (Sec. 1), this evaluation setting can be addressed by conventional offline VLMs prompted with the same clip and history (e.g., Qwen3-VL, GPT-4o, Gemini). Therefore, both offline VLMs and existing proactive/streaming models (e.g., MMDuet, Dispider, Proact-VL, EgoSpeak, WalkVLM), as well as simple heuristics (e.g., thresholding the native margin, similarity-based repetition filtering), are necessary baselines. Evaluating only at annotated points also means that false alarms between them, the main source of annoyance for BLV users, are never measured. Finally, results are from a single run on a small test set (64 videos), and the annotators' expertise, vision status, guidelines, and agreement are not reported (see Question 3).

W3. Content quality and the user study are under-evaluated. Assistance messages are scored only by similarity to a single reference (token F1, ROUGE-L, BERTScore), so incorrect or unsafe guidance (e.g., a wrong direction), which is critical for BLV users, is never analyzed. The user study is small (six participants, two trials per setting), reports only means without variance or significance tests, and again compares only the paper's own variants.

References

[1] Xiao et al., EgoBlind: Towards Egocentric Visual Assistance for the Blind. NeurIPS 2025 Datasets and Benchmarks. https://arxiv.org/abs/2503.08221

[2] Yuan et al., WalkVLM: Aid Visually Impaired People Walking by Vision Language Model. ICCV 2025. https://arxiv.org/abs/2412.20903

[3] Kim et al., Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals? arXiv 2025. https://arxiv.org/abs/2510.00766

[4] GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance. ACL 2026. https://arxiv.org/abs/2503.12844

Questions for authors
  1. Under the per-timestamp evaluation, how do (a) offline VLMs prompted with the same 3-second clip and action history (e.g., Qwen3-VL, GPT-4o, Gemini), (b) existing proactive/streaming models (e.g., MMDuet, Proact-VL, EgoSpeak, WalkVLM's reminder mechanism), and (c) simple heuristics perform on the same test set?

  2. Can the agent be evaluated at a fixed decision rate over the full stream (e.g., every 0.5 s), reporting false PUSHes between annotated points?

  3. What are the backgrounds of the annotators, and how were the labels produced? Since the labels define what counts as a useful notification and come from only five sighted–BLV pairs, please describe: (a) whether any annotators had relevant expertise (e.g., orientation and mobility instructors or trained sighted guides); (b) the BLV partners' vision status (totally blind vs. low vision), age range, and experience with assistive technology; (c) the annotation guidelines, annotator training, and how disagreements were resolved; and (d) the inter-annotator agreement on PUSH/NO-PUSH decisions across pairs.

  4. How often do delivered messages contain incorrect or unsafe guidance? A targeted human or model-based error analysis would help.

  5. Could the authors report variance and significance tests for the user study, and whether participants knew which system they were using?

  6. The paper includes a study with BLV participants but no ethics statement. Could the authors add one?

  7. Which streaming VLM is used as the backbone? The paper mentions only an "8.77B streaming backbone" with native ⟨SILENCE⟩/⟨RESPONSE⟩ tokens and cites Yao et al. (2026); please state the model explicitly.

Reviewer_835oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BLV-SceneAgent, a real-time policy that decides whether a streaming vision-language model should speak to a blind or low-vision user. On 1090 non-initial test decisions from 439 EgoBlind segments, the pretrained streamer reaches PUSH precision 0.255, and calibrated adaptation of the same backbone still sits at macro-F1 0.761. The proposed policy scores the native response margin of one causal prefill, adjusts it with a recurrent belief, a signed scene-change gain, and an eight-entry ledger of notifications already delivered, and speaks only when that score passes a fitted threshold. On that segment-disjoint test set it reaches macro-F1 0.824, against 0.761 for the adapted backbone, with a P95 action latency of 478.8 ms. Overall, this study mainly contributes a per-tick rule for when to interrupt with scene guidance, rather than a new vision-language architecture.

Critical strengths
  1. The experimental evaluation is relatively comprehensive and the performance looks good. It separates backbone adaptation from temporal selectivity, checks that precision and recall rise together, and shows that a frozen decoder still yields higher delivery scores when the push times improve. On the 1,090 non-initial test decisions, macro-F1 moves from 0.761 to 0.824, while the P95 action-boundary latency on one A40 stays at 478.8 ms.

  2. The paper is organized around a distinction that matters in use: for a blind or low-vision user, the same visible event is worth saying only while it is new and actionable, so assistance is a sequence of interrupt-or-stay-silent decisions tied to what has already been spoken. The method follows that claim, from a single action boundary to a controller whose repetition term can only suppress a repeated notification, to generation that reuses the same prefill and commits memory only after speech succeeds.

Critical weaknesses
  1. The headline gain does not isolate the proposed policy. Adaptation does most of the work: the macro-F1 moves from 0.412 for the pretrained streamer to 0.761 after BLV adaptation, and only then from 0.761 to 0.824 with Temporal Contextual Intervention. The abstract calls that second step a 6.3-point gain over calibrated SFT. Section 4.2 instead defines the baseline as the supervised-fine-tuning model plus a separately trained gated recurrent unit. There is no row for the fine-tuned model’s own silence/response score, and no comparison with a published proactive policy. The large jump shows that BLV labels change the behavior. The smaller jump does not show the necessities of the ledger, the signed change term, and self-history refinement.

  2. The number used to support that jump cannot be audited. A predicted PUSH counts as correct when its time falls inside an “acceptable notification interval,” but the paper never states how long the interval is or how a second PUSH inside the same interval is charged. Section 4.2 says the comparisons use a paired bootstrap over the 64 test videos, and Sections 4.3–4.8 report no interval and no test. A wide interval or a non-significant difference would shrink or remove the 6.3-point macro-F1 gain and the 8.5-point PUSH-F1 gain.

Questions for authors
  1. Could the author provide anonymous code repository for reviewing?
  2. Sections 4.3 and 4.6 report the decision scores and the delivery scores for the same 1,090 decisions in separate subsections. Can the delivery table sit with Table 1, so the claim that better timing changes which messages count as useful can be checked without treating it as a second experiment?
  3. How is a predicted PUSH matched to a reference? Please state the length of each acceptable notification interval and whether a second PUSH inside the same interval counts as a false positive. Do the macro-F1 of 0.824 and the PUSH F1 of 0.743 still hold if a reference can be matched only once, at the nearest prediction inside a fixed window of 1 second?
  4. The abstract compares TCI with calibrated SFT, while Section 4.2 defines the Table 1 baseline as supervised fine-tuning plus a separately trained gated recurrent unit. What is the test macro-F1 and PUSH F1 if the decision is only a validated threshold on the fine-tuned model's own silence/response margin, with no gated recurrent unit and no TCI?
Reviewer_WaBSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BLV-SceneAgent, a proactive visual assistant for blind and low-vision (BLV) users that learns when to speak and when to remain silent during continuous first-person video. The authors add chronological intervention annotations to 439 EgoBlind video segments, yielding 7,482 PUSH/NO-PUSH decisions and 1,794 assistance messages through five sighted–BLV annotator dyads. An action-first fine-tuned streaming VLM provides representations and conditional message generation. A Temporal Contextual Intervention (TCI) controller combines the native response margin, a recurrent belief, state-change features, and a committed-notification ledger. The paper reports improvements in intervention prediction and delivery quality, low model-side action latency, and favorable ratings in a six-participant BLV user study.

Critical strengths

1. An important problem formulation with meaningful involvement of the intended users. The paper recognizes that useful visual assistance requires selecting when to interrupt rather than continuously describing the scene. Conditioning intervention on previously communicated information is well motivated. The paired annotation protocol is a strength, particularly because BLV annotators determine whether speech would be useful and author the assistance messages. The resulting temporal supervision is a potentially useful resource for proactive assistance research.

2. Positive empirical results and a coherent execution design. Table 1 reports an increase in macro-F1 from 0.761 to 0.824 and PUSH F1 from 0.658 to 0.743 over the BLV-adapted baseline, with improvements in both precision and recall. Table 3 provides supporting ablations for the temporal mechanisms. Reusing the same multimodal cache for conditional generation and stopping early on NO-PUSH are sensible design choices. The 476K-parameter controller adds relatively little model-side computation, and the reported user ratings provide encouraging preliminary feedback.

Critical weaknesses

1. The validity of fixed reference labels under the agent's own notification history is not established. The paper explicitly motivates history-dependent supervision by noting that an observation may warrant PUSH before it has been communicated and NO-PUSH afterward (Section 1). However, self-history refinement uses the policy's predicted actions while retaining the original reference labels (Section 3.4). After a missed notification, a later reference NO-PUSH may reasonably require a reminder; after an early notification, a later reference PUSH may be redundant. Acceptable timing intervals do not by themselves resolve this dependence on what was actually communicated.

2. The temporal evaluation and annotation protocol are insufficiently specified. Section 4.2 considers a predicted PUSH temporally correct when it falls within a reference interval, but does not specify one-to-one event matching, how multiple predictions within an interval are penalized, or how event intervals map to the reported pointwise classification metrics. The construction of decision timestamps and acceptable intervals, and the handling of annotator disagreement, also require clarification. Without these definitions, the main results are difficult to reproduce and their sensitivity to the evaluation protocol is unclear.

Questions for authors
  1. How do you establish that the original PUSH/NO-PUSH labels remain valid after the policy misses or advances a notification? Can you provide BLV judgments conditioned on predicted histories, and clarify which histories are used for the VLM inputs, cached states, and notification ledger during training and evaluation?
  2. How are decision timestamps and acceptable notification intervals annotated? Please specify event matching, duplicate-notification penalties, the conversion to classification metrics, and annotation quality control, ideally with evaluation pseudocode.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.