- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BLV-Gate introduces a new eval for visual assistance from EgoBlind and VIABench videos by annotating each with inform, warn and VQA events. The system under test observing a same video responds (or does not) and the responses are compared against the annotated events using a judge (for matching response/event, and validating quality/validity). The annotation is done by creating an agentic pipeline the outputs of which are reviewed by annotators watching full videos who can either accept, reject, or add new events. SenseBridge is introduced as a comparative baseline. Paper results show stark tradeoffs between event coverage, communication volume, latency and quality.
Critical strengths
- BLV-Gate evaluates end to end video replay for agent, which is a more holsitic mreasurement approach than prior benchmarks which measure similar things (viabench and walkvlm)
- Human annotation is done by 3 humans watching the full video sessions, and can remove or add events. Humans have full context while annotating
- The experiments provide useful guidance about design choices and provide quantified tradeoffs between coverage, conversation load, quality and latency in this particular setting. Giving others a reference point on how to measure and make these tradeoffs in their setup
Critical weaknesses
- There appears to be some inconsistency in the reported numbers between the main body and the appendix. SenseBridge GPT numbers for Outdoor Walking and Shopping for example do not appear to match
- An age-old problem in measuring reaction time in evals is how to account for changes in infrastructure, model deployment settings and load (and model size) to get apples to apples comparison. BLV-Gate does not sufficiently address the issues or attempt to solve them but things like output latency and deadline violations are reported as comparable
- Similar in spirit to VIABench and WalkVLM in terms of measuring events, with the key contribution is balancing converage with conversation load. However, it is not established in any way that the excess responses from the models are not helpful (i.e. the human annotations do not have a lot of useful missing cues which the models may catch). Issues can be two fold, humans may be biased to only consider events at a particular time in video that the original pipeline captured (ignoring other events at same moment that models being evaluated may raise) and humans my not add events sufficiently leading to the overall distribution of events being evaluated leaning heavily towards the pipeline that generated them. Ideally (a) a human review of false positive events from the model would prove that they are indeed conversation noise and (b) an annotation of a small sample of video sessions without pipeline generated events would yield similar distribution of events on those videos as the existing benchmark events. This is my main concern and I am happy to raise the score if sufficiently resolved.
Questions for authors
- Of the 5863 events, how many were in the 7466 candidate events, and how many were newly created by human annotators?
Dear Reviewer vpS1,
Thank you for the detailed review and for recognizing that end-to-end replay is a more holistic measurement than prior benchmarks, that annotators review complete sessions and can add or remove events, and that the quantified trade-offs give others a reference point for their own systems. Your main concern is whether excess model responses are truly noise and whether the annotations are anchored to the generating pipeline. All revisions will be included in the latest version of the paper.
Q1. Whether unmatched model responses are truly unhelpful. (Weakness 3a)
A1. We ran the proposed human review of false positives. Annotators reviewed 318 responses counted as false interventions, sampled across all ten configurations. For each, they saw the causal video clip, the model's earlier responses and the nearby annotated events.
Q2. Possible anchoring of annotations to the generating pipeline. (Weakness 3b)
A2. We ran the proposed annotation without pipeline-generated events. Two annotators independently annotated 40 held-out sessions (33.7 min, stratified across the five scenes) from scratch, without seeing any candidate events.
Q3. Provenance of the 5,863 events. (Question 1)
A3. Of the 5,863 events in the benchmark:
Q4. Inconsistency between main text and appendix. (Weakness 1)
A4. Thank you for spotting this. SenseBridge Agent was updated to a second version (v2) before submission. Table 3, Table 4, Figure 3 and Section 5.3 report v2, but the SenseBridge Agent (GPT-5.6-Luna) rows in Tables 5–9 were not updated. We have replaced them with v2 results. The appendix now matches Section 5.3 (Outdoor Walking F1 = 0.452; Shopping & Service P = 0.130, R = 0.795), and the event-weighted recall over scenes equals the Table 3 value of 0.815.
Q5. Latency comparability across infrastructure. (Weakness 2)
A5. We agree that measured latency reflects deployment as much as the model. BLV-Gate therefore separates two axes: