acceptodds
Under review as a conference paper at ICLR 2027

BLV-Gate: Benchmarking Proactive Communication in Blind and Low-Vision Assistance

Abstract

Continuous visual assistance for blind and low-vision (BLV) users requires an assistant to decide whether to communicate, when to respond, and what information is useful, while avoiding unnecessary output. Existing evaluations focus on isolated questions or predefined event recall, providing limited evidence about this sequence-level communication problem. We introduce \system, a benchmark built from 499 verified egocentric video sessions across five everyday scene categories, containing 5,863 temporally localized communication events with grounded visual facts and reference responses. \system evaluates complete causal response sequences across intervention regulation, timing and latency, grounding and response quality, and session-level information burden. We also introduce SenseBridge Agent, a companion baseline with causal visual reasoning and memory. Experiments with ten representative baselines reveal clear trade-offs among event coverage, communication burden, response quality, and latency: periodic systems achieve higher recall at the cost of frequent output, conservative streaming systems miss necessary events, proactive assistants show stronger potential, and stronger backbones improve grounding and response quality without resolving timely delivery. All experimental data and code are available at: https://anonymous.4open.science/r/BLV-Gate-7A03.

Primary area:
datasets and benchmarks
Keywords:
Blind and Low Vision, Vision Language Model, Proactive Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_vpS1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BLV-Gate introduces a new eval for visual assistance from EgoBlind and VIABench videos by annotating each with inform, warn and VQA events. The system under test observing a same video responds (or does not) and the responses are compared against the annotated events using a judge (for matching response/event, and validating quality/validity). The annotation is done by creating an agentic pipeline the outputs of which are reviewed by annotators watching full videos who can either accept, reject, or add new events. SenseBridge is introduced as a comparative baseline. Paper results show stark tradeoffs between event coverage, communication volume, latency and quality.

Critical strengths
  • BLV-Gate evaluates end to end video replay for agent, which is a more holsitic mreasurement approach than prior benchmarks which measure similar things (viabench and walkvlm)
  • Human annotation is done by 3 humans watching the full video sessions, and can remove or add events. Humans have full context while annotating
  • The experiments provide useful guidance about design choices and provide quantified tradeoffs between coverage, conversation load, quality and latency in this particular setting. Giving others a reference point on how to measure and make these tradeoffs in their setup
Critical weaknesses
  • There appears to be some inconsistency in the reported numbers between the main body and the appendix. SenseBridge GPT numbers for Outdoor Walking and Shopping for example do not appear to match
  • An age-old problem in measuring reaction time in evals is how to account for changes in infrastructure, model deployment settings and load (and model size) to get apples to apples comparison. BLV-Gate does not sufficiently address the issues or attempt to solve them but things like output latency and deadline violations are reported as comparable
  • Similar in spirit to VIABench and WalkVLM in terms of measuring events, with the key contribution is balancing converage with conversation load. However, it is not established in any way that the excess responses from the models are not helpful (i.e. the human annotations do not have a lot of useful missing cues which the models may catch). Issues can be two fold, humans may be biased to only consider events at a particular time in video that the original pipeline captured (ignoring other events at same moment that models being evaluated may raise) and humans my not add events sufficiently leading to the overall distribution of events being evaluated leaning heavily towards the pipeline that generated them. Ideally (a) a human review of false positive events from the model would prove that they are indeed conversation noise and (b) an annotation of a small sample of video sessions without pipeline generated events would yield similar distribution of events on those videos as the existing benchmark events. This is my main concern and I am happy to raise the score if sufficiently resolved.
Questions for authors
  • Of the 5863 events, how many were in the 7466 candidate events, and how many were newly created by human annotators?
1 comment

Dear Reviewer vpS1,

Thank you for the detailed review and for recognizing that end-to-end replay is a more holistic measurement than prior benchmarks, that annotators review complete sessions and can add or remove events, and that the quantified trade-offs give others a reference point for their own systems. Your main concern is whether excess model responses are truly noise and whether the annotations are anchored to the generating pipeline. All revisions will be included in the latest version of the paper.


Q1. Whether unmatched model responses are truly unhelpful. (Weakness 3a)

A1. We ran the proposed human review of false positives. Annotators reviewed 318 responses counted as false interventions, sampled across all ten configurations. For each, they saw the causal video clip, the model's earlier responses and the nearby annotated events.

  • 80.8% were confirmed as genuine noise:
    • mistimed relative to the event they describe;
    • repeated;
    • unnecessary;
    • visually wrong.
  • At most 12.3% contained useful information missing from the annotations. Most of these were only partly correct; only 2.8% were fully correct and useful.
  • The remaining 6.9% were attribution misses by the judge.

Q2. Possible anchoring of annotations to the generating pipeline. (Weakness 3b)

A2. We ran the proposed annotation without pipeline-generated events. Two annotators independently annotated 40 held-out sessions (33.7 min, stratified across the five scenes) from scratch, without seeing any candidate events.

  • The annotations do not lean towards pipeline-specific events.
    • 83–86% of the blind annotators' events match an existing annotated event.
    • Event types agree in 89–90% of matched pairs, with a median timing offset under 1 s.
    • Annotators working without candidates therefore raise few events that the benchmark lacks.
  • The difference is granularity, not content.
    • Blind annotators mark about 4 events/min versus 10.3 in the benchmark, with fewer facts per event.
    • They recover 97.6% of session-opening events but fewer incremental updates; they tend to merge consecutive updates into one event.
    • This makes recall on BLV-Gate a strict, fine-grained criterion. It does not change the over-communication finding: even at the blind annotators' coarser density, every evaluated configuration speaks more often (4.6–61.3 outputs/min). Agreement between the two blind annotators is event-level F1 = 0.607, with event-type κ = 0.71–0.75. Both studies and all human labels will be added to the paper.

Q3. Provenance of the 5,863 events. (Question 1)

A3. Of the 5,863 events in the benchmark:

  • 3,311 were retained unchanged;
  • 2,548 were retained after human editing of timing, facts, response or type.
  • 10 were newly created by reviewers.
  • 1617 events were deleted from the candidates. Reviewers added very few events because the candidate generator already covered nearly all points they considered worth communicating. Review therefore consisted mainly of correcting timing, facts and event types, and of pruning candidates. The blind re-annotation in A2 confirms this independently: annotators without candidates rarely raise events the benchmark lacks.

Q4. Inconsistency between main text and appendix. (Weakness 1)

A4. Thank you for spotting this. SenseBridge Agent was updated to a second version (v2) before submission. Table 3, Table 4, Figure 3 and Section 5.3 report v2, but the SenseBridge Agent (GPT-5.6-Luna) rows in Tables 5–9 were not updated. We have replaced them with v2 results. The appendix now matches Section 5.3 (Outdoor Walking F1 = 0.452; Shopping & Service P = 0.130, R = 0.795), and the event-weighted recall over scenes equals the Table 3 value of 0.815.


Q5. Latency comparability across infrastructure. (Weakness 2)

A5. We agree that measured latency reflects deployment as much as the model. BLV-Gate therefore separates two axes:

  • Intervention quality, scored at the decision point and independent of infrastructure. This is the main focus of the benchmark.
  • Delivery, measured as output latency and deadline violation rate under the stated deployment. We now state explicitly that latency-based metrics are system-level observations, not controlled comparisons between models. In particular, the gap between API-served and locally served models mainly reflects serving infrastructure. We also report, per system, the serving setup (local A800 with vLLM, or API) and the latency distribution (median, p95). To compare policies independently of infrastructure, we added a sensitivity analysis that assigns the same assumed latency L to every system. Up to L = 2 s, the ranking of systems by delivered recall is nearly identical to the decision-time ranking (one adjacent swap), while every system loses 28–57% of its recall. End-to-end timeliness on real wearable hardware will be validated in our planned on-device deployment study.
Reviewer_iK5ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces BLV-GATE, a benchmark for communication regulation in continuous visual assistance for blind and low-vision users. It contains 499 sessions and 5,863 annotated events, and evaluates intervention precision/recall, timing, factual coverage, response quality, and communication burden. It also introduces SenseBridge, a memory-based agent baseline. Experiments with ten configurations illustrate trade-offs between event coverage, unnecessary interventions, and generation latency.

Critical strengths

The paper addresses a consequential evaluation problem: continuous assistance requires selective, timely communication, and isolated answer accuracy does not capture unnecessary interruptions or missed opportunities.

The combination of human-verified event annotations and separate measurements of coverage, content quality, latency, and output burden provides a potentially useful diagnostic resource. In particular, separating decision timing from output latency exposes limitations that recall alone would hide.

Critical weaknesses
  • The reported results are internally inconsistent in ways that affect the main findings. Table 3 reports 81.5% recall for SenseBridge with GPT-5.6-Luna, whereas Tables 5–9 report scene recalls ranging from 62.84% to 76.91%. Under consistent aggregation, overall recall cannot exceed every constituent scene recall; weighting these values by the event counts in Table 2 gives approximately 72.12%. Moreover, the appendix reports lower recall than the same-backbone Periodic VLM in every scene, contradicting the aggregate improvement in Table 3. These discrepancies prevent a reliable assessment of the claimed advantage.

  • The validity and incremental value of the evaluation are insufficiently established. The human-agreement study validates response-quality scoring, but not event attribution or fact coverage, although these judgments determine the benchmark’s core results. Unmatched outputs may include useful information omitted from the annotations, and character counts have not been validated against users’ experienced burden. The contribution also needs sharper differentiation from EgoServe and EgoPro-Bench, which already evaluate intervention timing and precision/recall, and WalkVLM-LR, which explicitly addresses redundant assistance. These gaps matter because the benchmark itself is the primary contribution.

Questions for authors
  • Can you reconcile Table 3, Tables 5–9, and the scene-level numbers in Section 5.3? Do they use identical runs, data, and aggregation rules?

  • What human verification supports event attribution and fact coverage? Have unmatched outputs been audited to distinguish genuinely unnecessary interventions from annotation omissions?

  • How is 1 Hz decision-making implemented when average generation takes approximately 12 seconds? Can subsequent decisions access only responses already completed and delivered? How are unanswered queries retained after their one-shot injection, particularly for delayed-evidence VQA?

  • What capability or failure mode does the proposed protocol reveal beyond the closest existing evaluations, and is there evidence that these additional measurements better reflect user preferences?

2 comments

Dear Reviewer iK5R,

Thank you for the careful review and for recognizing that continuous assistance requires selective, timely communication, and that separating decision timing from output latency reveals limitations hidden by recall. The reviewer correctly identified an inconsistency in our appendix, and the questions on evaluation validity led us to run three new human studies. All tables in the revision are generated from the same runs on the verified annotations with a single aggregation script. All revisions will be included in the latest version of the paper.


Q1. Inconsistency between Table 3, Tables 5–9 and Section 5.3.

A1. This was a synchronization error in the appendix. SenseBridge Agent was updated to a second version (v2) before submission:

  • Table 3, Table 4, Figure 3 and Section 5.3 already report v2.
  • The SenseBridge Agent (GPT-5.6-Luna) rows in Tables 5–9 were not updated. We have replaced these rows with v2 results. The event-weighted recall over the five scenes is now 0.815, matching Table 3. The scene numbers also match Section 5.3: Outdoor Walking F1 = 0.452; Shopping & Service P = 0.130, R = 0.795. After the correction, SenseBridge Agent's recall exceeds Periodic VLM with the same backbone in every scene. More importantly, Periodic VLM is an always-respond reference designed for high coverage. It speaks at every decision point (61.3 outputs/min), so its recall comes at the cost of over-broadcasting. This trade-off is what BLV-Gate is designed to expose.

Q2. Validity of event attribution, fact coverage and unmatched outputs.

A2. We ran three human studies.

  1. Audit of unmatched outputs. Annotators reviewed 318 responses counted as false interventions, sampled across all ten configurations, each with its causal video clip and the model's earlier responses.
  • At most 12.3% contained useful information missing from the annotations, and most of these were only partly correct; only 2.8% were fully correct and useful.
  • 80.8% were confirmed as genuine noise: mistimed, repeated, unnecessary or visually wrong.
  • The remaining 6.9% were attribution misses by the judge.
  1. Validation of attribution and fact coverage. On 620 responses, the judge rarely misses a match that humans confirm among responses credited as correct (1.1%). Fact-level agreement with humans is κ = 0.39. The judge is stricter than humans on absolute FC (40.8 vs. 70.4; r = 0.44). We therefore use FC to compare systems, not as an absolute coverage rate.
  2. Blind re-annotation. Two annotators independently annotated 40 held-out sessions from scratch. 83–86% of their events match existing annotations, so the annotations rarely miss events a human would raise. Inter-annotator event F1 is 0.607. Characters per minute is an observable proxy. We have not validated it against BLV users' experienced burden and now state this as a limitation.

Q3. 1 Hz decisions, memory and query handling.

A3. BLV-Gate evaluates whether a model intervenes correctly and whether the response is delivered in time as two separate axes.

  • 1 Hz decisions. We invoke the model at every 1 Hz decision point on the causally available frames, and evaluate its decision at that time. This tests intervention correctness independently of serving infrastructure.
  • Delivery. Generation time is reported separately as output latency and deadline violation rate (DVR). For API-served models, this time is dominated by API network and service latency. The 99.8% DVR of the API-served configurations is exactly how the benchmark exposes that such systems cannot deliver in real time. Because replay is sequential, a model's context may include responses that would not yet have been delivered in deployment. We now state this explicitly in Section 3.3.
  • Memory and questions. These follow each method's native design, which is part of what is compared:
    • MMDuet2, VideoLLM-online and LiveStar keep their streaming history or memory;
    • Periodic VLM and SimpleStream have no memory but re-attach recent questions at every decision point;
    • SenseBridge Agent keeps a memory of its previous responses and is question-triggered, answering only in the round where a question arrives.
  • The benchmark measures the consequence of this design: SenseBridge Agent's timely recall is 55.8% on present-evidence VQA but 30.6% on delayed-evidence VQA. We describe each baseline's memory and query mechanism in Appendix C.2.
AuthorsOpenReview ↗

Follow-up on Q4


Q4. Added value over existing evaluations and user preferences. A4. BLV-Gate is designed around what helps a blind or low-vision user, rather than around generic proactive interaction. We added EgoServe, EgoPro-Bench and WalkVLM-LR to Table 1 and clarified the differences:

  • Scope. EgoServe and EgoPro-Bench evaluate general egocentric or personalised assistance, scoring intervention timing per event. WalkVLM-LR targets walking guidance only. BLV-Gate is built entirely from BLV-captured video across five everyday settings. Its event types (INFORM, WARN, VQA) reflect what a blind user needs to be told, warned about or answered.
  • Safety first. Facts are weighted by importance (critical = 5, general = 3, low-value = 1), so missing an immediate hazard costs far more than missing context. WARN events are additionally checked against delivery deadlines, because a correct warning that arrives late does not help a user who cannot see the hazard.
  • Listening burden. A blind user receives all information through audio and cannot skim it. BLV-Gate therefore scores the complete output sequence and penalises repeated or unnecessary speech, not only missed events. This reveals a failure that event-level evaluation cannot see: SenseBridge Agent (GPT-5.6-Luna) produces 26.5 redundant outputs/min, and redundant outputs outnumber correct ones in 90.2% of its sessions.
  • Help that actually arrives. Delivered assistance diverges from decision quality: rankings by decision-time F1 and by delivered F1 differ.

Comments on the paper

1 comment

Dear Reviewer VjRg,

Thank you for the constructive review and for recognizing that extending evaluation from isolated responses to complete communication sequences makes proactive visual assistance measurable, and that the coverage–burden trade-off provides reproducible evidence of current capability boundaries. We address the reviewer's questions on delivery timing, the added value of sequence-level evaluation and annotation agreement below. All revisions will be included in the latest version of the paper.


Q1. Decision-time versus completion-time intervention metrics. (Weakness 1)

A1. BLV-Gate is designed to test two things: whether a model decides to intervene proactively at the right moment, and whether what it says is useful. We therefore compute intervention P/R/F1 at the decision point, which separates the intervention policy from serving infrastructure. Output latency and the deadline violation rate (DVR) are reported separately, so readers can see both. End-to-end timeliness on real wearable hardware will be validated in our planned on-device deployment study.

As requested, we added completion-time metrics.

SystemF1 (decision)F1 (delivered)WARN R (decision)WARN R (delivered)
SimpleStream0.3730.3260.4680.212
Periodic VLM (Qwen3-VL-4B)0.3260.2950.5440.296
Periodic VLM (Qwen3-VL-8B)0.3030.2620.4940.160
VideoLLM-online0.1410.1280.0820.044
JoyAI0.2460.0760.1830.063
SenseBridge (GPT-5.6-Luna)0.3900.0010.8710.000
SenseBridge (Qwen3-VL-8B)0.3430.0000.4380.000
Periodic VLM (GPT-5.6-Luna)0.3470.0000.6950.000

The rankings change as the reviewer anticipated. The decision-time and delivered F1 rankings are uncorrelated (Kendall τ = 0.00).

  • Locally served systems with sub-second latency keep most of their F1.
  • API-served systems drop to near zero. Their latency (e.g., median 9.3 s and p95 28.5 s for SenseBridge Agent with GPT-5.6-Luna) is dominated by API network and service time rather than by the intervention policy.
  • WARN events are the most affected, because their windows are short. Even the fastest systems lose about half of their WARN recall.

Q2. Added value of sequence-level evaluation over VIABench. (Weakness 2)

A2. VIABench evaluates whether a model raises the annotated reminders, i.e., event recall. Under this protocol, a response that repeats an already-reported event, or speaks when nothing new has happened, costs nothing. However, these behaviours are precisely what burdens a blind user, who receives all information by listening and cannot skim past it. BLV-Gate scores the complete output sequence. Every utterance is credited, mistimed, redundant or false, so repeated and excessive broadcasting become visible and measurable. The two protocols lead to different conclusions:

  • Repeated broadcasting. The always-respond Periodic VLM re-announces the same event about 34–37 times per minute. In over 97% of its sessions, redundant outputs outnumber correct ones. Event-level recall cannot see this, because each event is credited only once.
  • Excessive broadcasting. Systems that speak more achieve higher recall but also more false interventions. Ranked by recall alone, Periodic VLM places 2nd–4th of ten configurations. Ranked by precision and by false interventions per minute, it falls to the bottom. Across all configurations, recall is negatively correlated with precision (Kendall τ = −0.20) and with a low false-intervention rate (τ = −0.47). An event-level evaluation would therefore favour indiscriminate narration, whereas sequence-level evaluation reveals the trade-off that a proactive assistant for BLV users must manage.

Q3. Annotation agreement, independence and BLV involvement.

A3. We conducted a supplementary study to quantify annotation agreement. Two annotators independently annotated 40 held-out sessions (33.7 min, stratified across the five scenes) from scratch. They had no access to candidate events or to each other's labels.

AspectAgreement between the two annotators
Intervention necessityevent-level F1 = 0.607
Event category (INFORM / WARN)91–92% agreement, κ = 0.71–0.75
Ideal response timemedian offset 0.7–0.8 s
Acceptable windowmean IoU 0.47

The blind annotations also agree with the benchmark annotations:

  • 83–86% of blind events match an annotated event;
  • event categories agree in 89–90% of matched pairs. No BLV users were involved in validating the usefulness of events. We state this as a limitation; a user study is part of our planned on-device deployment.

Q4. Repository access.

A4. We apologise for the inconvenience. We have re-checked and refreshed the anonymous repository (https://anonymous.4open.science/r/BLV-Gate-7A03/).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.