- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies proactive safety warning in streaming video, where systems generally need to issue warnings before an incident actually occurs. The authors propose ForeWarn, which uses a small VLM to predict near-term incident risk and uses the risk score to determine both whether to issue a warning immediately and when to call a verifier. When short-horizon risk is high, the system issues a warning without waiting for the verifier. For longer-horizon risks, the verifier can confirm whether a hazardous state is present. The authors report that, at a comparable false-alarm rate, ForeWarn achieves higher pre-incident recall while reducing verifier calls; however, its generalizability across scenarios remains unestablished.
Critical strengths
The problem and the proposed design are well aligned. The paper distinguishes between a verifier’s accuracy and whether its results arrive in time, arguing that online warning depends not only on correct recognition but also on delivering a warning while action is still possible. Short-horizon risk triggers an immediate warning, while longer-horizon risk is combined with verification results to issue an advisory warning, addressing both fast-onset incidents and slowly developing hazards.
Critical weaknesses
- The key risk scores are not probability-calibrated. The paper states that r3 and r5 are used as scores rather than calibrated probabilities. Thus, r3=0.9 does not mean that the true probability of an incident is 90%. More extensive experiments are needed to establish how stable the thresholds are across scenarios and to justify how they are selected.
- The problem is valuable, but the distinction from prior work needs to be clarified. Fast–slow models, selective model invocation, and inference cascades have been studied in related work. The authors should explain what their design adds beyond existing fast–slow pipelines and selective VLM routing, and why these mechanisms cannot be reduced to generic confidence-based routing.
Questions for authors
- How was the immediate-warning threshold Θu=0.9 chosen? Was it selected using only the development set? Did the authors evaluate recall, false-alarm rate, and balanced accuracy across different threshold values?
- Given that r3 and r5 are not probability-calibrated, how stable are the thresholds across datasets or cameras? Did the authors validate the method in independent scenarios beyond those used for deployment-specific calibration?
- Could the authors report confidence intervals for the main aggregate results and quantify how end-to-end device latency affects timely recall?
5 comments
Response to Reviewer h1sV (2/2)
Q3. Confidence intervals for the main results, and the effect of device latency on timely recall.
Wilson intervals for every domain are in Appendix H.6, and the device experiments are in Appendix H.9. The macro recall difference against StreamSense is +0.197, with an interval from +0.138 to +0.257. The clip-FAR intervals are wide because the normal pools hold 16 to 88 clips. In the recorded device runs accumulated lag roughly halves the timely hits on every tier, from 117 to 56 of 299 on the 64 GB tier. A new control experiment on that tier has an on-board baseline of 70 timely hits. Running the verifier off the board on a separate server raises them to 176, with 0.519 of normals alert-free against 0.803. The controlled recordings stay at 1 to 2 of 35 on the two larger tiers, and Section 4.5 reports these as decision misses and not lag.
Ethics flag: consent, privacy and review for the self-recorded trials.
The submission did not describe these three points, and we answer them in turn.
- Who was recorded, and consent. The 65 controlled recordings were staged and recorded by members of our laboratory. Everyone who appears is a researcher on this project or a student of the same laboratory. No one outside the laboratory was recruited, and no member of the public was filmed. Each person who appears agreed orally to the use of the recordings in this study.
- Privacy. The recordings are used only for the execution evaluation of Section 4.5. They are kept on our group's storage and are not part of any released dataset. No frame of them appears in the paper. The release holds only their per-step scores, decisions and timing, together with a manifest of clip codes, labels, incident times and durations. It holds no video, image or audio.
- Ethics review. The recordings were not submitted for institutional ethics review. The participants were limited to the researchers on this project and students of our laboratory, as stated above. If the ethics reviewers consider a formal determination necessary, we will request one from our institution and report its outcome.
Our Ethics Statement described the recordings as "volunteer enactments" and noted that the archive holds no consent or institutional-review documentation. That wording did not say who the volunteers were or whether a review took place, and we have rewritten the statement to give the points above directly.
Follow-up to Reviewer h1sV
Dear Reviewer h1sV,
Thank you again for your review. Our response is posted above, together with a revised PDF. The revision reports how the immediate threshold was chosen and adds a sweep over it on all five domains (Section 3.3, Appendix H.2). It positions the method against cascades and confidence routing with a direct comparison (Sections 2 and 4.3), and it adds intervals for the main results (Appendix H.6). The Ethics Statement now describes who was recorded, consent and review status.
Could you let us know whether these changes address your concerns, including the ethics point, or which points remain open? We will answer any further question before the discussion closes on November 18. If the concerns are resolved, we would be grateful if you could reflect this in your assessment.
Best regards,
The authors
Response to Reviewer h1sV (1/2)
Thank you for the review and for raising the ethics point in a constructive way. References below are to the revised paper, where changes are in blue. R / F denotes pre-incident recall and the clip false-alert rate (FAR) on held-out normal clips, and α is the nominal clip-FAR budget of site calibration.
We first note one fact about the summary. The macro recall of 0.670 against 0.473 for StreamSense comes at a macro clip FAR of 0.362 against 0.291, with similar verifier calls. With StreamSense re-thresholded to at most ForeWarn's clip FAR, the lead is 13.3 points, and it comes from the staged falls and EDF (Section 4.2, Table 10). Two StreamSense rows in the submission had used the zero-shot scout, and the revision corrects them (Table 1).
W1. The risk scores are not calibrated probabilities, and threshold stability and selection need more evidence.
The thresholds are operating points and not calibrated probabilities. The revision reports the reliability of the scores and adds a rule that sets the immediate threshold from calibration normals (Appendices H.8 and H.2).
W2. What does the design add beyond fast-slow pipelines, selective invocation and cascades, and why is it not generic confidence routing?
ForeWarn differs from a cascade in its signal and in its action, and the difference is measurable. With the same scout, verifier and calibration rule, a controller in which every alert waits for the verifier loses 7 of 31 clips on the staged falls and 19 of 42 on EDF (Section 4.3, Appendix H.4). In a cascade or a confidence router, the small model's uncertainty decides whether the large model is consulted, and the output on that step waits for the large model. ForeWarn differs in two ways. First, its signal is a forecast of time-to-incident at two horizons and not confidence in the present answer. Second, the path with the least time never waits. The short horizon issues the warning from the scout, and verification runs asynchronously where it can only add an advisory. On the other three domains the two controllers are close. At tight budgets the second model adds recall on long-runway footage. On the long-runway home pool Home-L at α = 0.3 the full policy gives 0.370 / 0.432, against 0.204 / 0.318 for a single threshold on the scout (Appendix H.3). Section 2 now positions the method against cascades and selective routing in these terms.
Q1. How was θu = 0.9 chosen?
It was fixed before the benchmark runs from a sweep on EDF recordings, and the other four domains inherit the value without tuning. Those recordings are also the EDF evaluation videos, so the EDF results carry selection bias. The revision states this in Section 3.3 and reports a new sweep of the threshold on all five domains (Appendix H.2). On the falls the new sweep gives 0.968 / 0.312 at a threshold of 0.80, 0.935 / 0.125 at 0.90 and 0.871 / 0.125 at 0.95. On the falls and EDF a higher threshold only removes recall, and a lower one adds a little recall at a much higher false-warning rate.
Q2. Stability across datasets or cameras, and validation beyond the scenarios used for calibration.
The advisory thresholds are site-specific by design, and one calibration pass on normal footage sets them. An additional set of 259 clips adds two domains outside the scout's training sources (Appendices H.8 and H.10).