acceptodds
Withdrawn from ICLR 2027

Heimdall: Hindsight Distillation for Proactive Warning in Streaming Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming vision-language models can judge whether a scene is dangerous. Proactive warning asks them to say so before the accident, within a preset cap on false alarms, and on the device beside the camera. However, training such a warner requires supervision, and the annotated accident times that anticipation methods train on are expensive to obtain and absent for a newly deployed camera's footage. We observe that a frozen large verifier is uncertain at the moment a warning is due but confident a few seconds later, once the accident is in view. This later verdict can become a training target for the earlier moment without any annotation. We propose Heimdall, which queries a frozen 8B verifier a few seconds after each window of the training stream, converts those later verdicts into a multi-horizon risk target for the window itself, and distills on that target a 2B model that runs in real time on a Jetson. At deployment the two models run as one online policy. Each alarms when its risk score crosses its threshold, and the two thresholds are chosen together so that the policy as a whole stays within the cap. On the held-out test split of PaSBench-Video, Heimdall warns before the accident on 1.5 times as many clips as a zero-shot model with the same verifier at half its false-alarm rate, and on a third more than a model distilled from the same verifier at the same moment. Published warners that find more accidents raise two to four times as many false alarms as the cap allows.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
proactive warning, streaming video, vision-language models, hindsight distillation, false-alarm calibration, on-device inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.83 over 6 reviews, above 27% of the others.
Reviewer_QFJKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses proactive accident warning in streaming video under a fixed false-alarm cap, with a small model running on an edge device. The key idea is that a frozen 8B vision-language verifier is uncertain at the moment a warning should fire but confident a few seconds later once the accident is visible. Heimdall queries the verifier at each second after a training window, takes the maximum of its three-second risk answers over horizons of 1 to 5 seconds, and converts these into a five-bucket time-to-accident target for the earlier window. A 2B host is distilled on this target with LoRA. No accident-time annotation is used. At deployment the host scores every one-second window and the verifier answers every two seconds without blocking. Either can alarm, and their two thresholds are chosen jointly on a pool of 214 external normal clips so that the whole policy stays within the budget. A miss-rate identity argues that streaming memories can only shift scores and so cannot help at a low false-alarm operating point. On a self-made test split of PaSBench-Video with 181 hazardous and 78 normal clips, Heimdall warns before the accident on 30.4 percent of clips at 5.1 percent false alarms, versus 20.4 at 10.3 for the undistilled pair and 22.7 at 6.4 for a host distilled from the same verifier at the same moment. It matches an annotation-supervised host at zero margin and leads at three and five seconds. Results are confirmed on a 482-clip multi-source corpus and the loop is run on two Jetson boards.

Critical strengths
  1. The idea is simple and well-motivated. Using the deployed verifier's own later verdict as a training target removes the need for annotated accident times, and the same-time and fixed-offset controls cleanly isolate the value of asking later from the value of distillation itself.
  2. The label-free target matches annotation supervision. The hindsight host reaches the same R0 as the host trained on human accident times and leads at three and five seconds, which is the paper's most consequential result.
  3. Calibrating the two-model policy jointly under one cap is a practical contribution. Section 3.1 shows that per-path calibration realizes roughly twice the nominal false alarms, and Eq. 2 fixes this.
  4. Real deployment is demonstrated. The closed loop runs on Jetson Orin boards with measured latency, power, and schedule drift, and the paper shows how quantization and timing remove zero-margin hits.
Critical weaknesses
  1. The central promise is not demonstrated. The abstract motivates the method by footage from "a newly deployed camera" that lacks annotations, but training uses the curated, hazard-rich PaSBench training split. On the one genuinely new site (self-collected), the method is not retrained and reaches 5.7 percent R0 at 13.3 percent false alarms after recalibration (Table 15). Learning from a new camera's own mostly-normal footage is never tested.
  2. The false-alarm evidence rests on very few clips. The test split has 78 normals, so "half the false-alarm rate" is 4 versus 8 clips, and the development set has 20 normals, so a 5 percent FAR difference is one clip. Many claims about operating points are at this granularity.
  3. The multi-horizon maximum adds little over a single later query. The fixed-offset teacher matches Heimdall exactly at R3, R5, and FAR and trails only at zero margin, so the specific target construction of Eq. 3 contributes 11 clips at the loosest margin.
  4. The theoretical analysis is thin and its assumptions are violated. Proposition 1 assumes independent window scores within a clip, which overlapping one-second windows do not satisfy, and Corollary 1 is close to tautological while assuming a memory leaves normal scores unchanged, which it does not. The analysis serves mainly as a narrative device.
  5. The target is weakly aligned with the event. Its AUROC against the annotated event is 0.66 to 0.69 and only 9 to 16 percent of top-bucket windows actually contain an accident. The paper does not explain why distilling on such a noisy target performs as well as annotations.
  1. The PaSBench-Video protocol baseline is a reimplementation without released code and warns on at most 7.2 percent of clips. Without validation against the original paper's numbers, this row may understate that protocol.
Questions for authors
  1. Can you train or fine-tune the host on footage from the self-collected site itself, or on a held-out mostly-normal stream, to show that the method works on a new camera's footage as the abstract claims?
  2. Can SPRINT be given an operating point (for example, k consecutive positive answers or the yes-token probability) so that it can be compared at matched false-alarm rates?
  3. Was the annotation-supervised host trained on the same 3,573 windows as Heimdall?
  4. How many normal clips would be needed for the false-alarm comparisons to be statistically meaningful, and can you report confidence intervals on FAR for Table 1?
Reviewer_fYUsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the "Heimdall" method, which converts the judgments made by a frozen VLM teacher at later time points into training signals for earlier temporal windows, thereby reducing the need for "time-to-accident" annotations. The method computes the maximum risk—as assessed by the teacher—across multiple future intervals, transforms the differences in these nested risks into "soft labels" mapped to specific time buckets, and uses them to train a lightweight primary model.

Critical strengths
  1. The methodology is well-designed and effective. The proposed hindsight distillation decouples future-looking supervision during training from low-latency action execution during inference, resulting in a clear system architecture. By avoiding the need for the student model to align with the teacher model’s predictions across multiple future time segments, the framework effectively translates the teacher model’s post-hoc assessment capabilities into the student model’s ability to make proactive, real-time decisions. Furthermore, it does not rely on manual annotations of the precise timing for each incident.
  2. The experimental evaluation—including both ablation studies and comparative analyses—is comprehensive, with key results demonstrating improvements over existing methods. Compared to an undistilled zero-shot system, Heimdall achieves higher recall and a lower false-alarm rate. The study also compares various configurations—such as using a teacher model with real-time access, a teacher model with a fixed delay, and ground-truth manual annotations—which helps to clarify whether "leveraging future information" provides any tangible performance gains.
Critical weaknesses
  1. The effectiveness of multi-time-horizon targets is limited, as the targets representing the highest future risk do not always align precisely with the predicted timing of an accident. By having the "teacher" model predict whether an accident will occur within a future time window—and then taking the maximum value across these predictions—the effective time range for "high-risk" classifications is expanded. This approach may also inadvertently amplify isolated, high-confidence (but potentially erroneous) predictions from the teacher model. Furthermore, the primary performance gains are concentrated at the R0R_0 horizon; for R3R_3 and R5R_5, the method offers no significant advantage over a simpler "teacher" model based on a single, fixed-offset time horizon.
  2. Cross-scenario generalization remains a challenge. The high false-positive rate encountered in new scenarios necessitates calibration using "normal" video data. This suggests that the balance between the false-positive budget and overall performance is highly sensitive to the specific scenario—a finding that undermines the paper’s core motivation
Questions for authors
  1. How should the "future-maximum" buckets be interpreted in terms of time? For which types of hazards does this target offer a clear advantage over a fixed future offset—as opposed to simply broadening the high-risk time interval?
  2. While avoiding the need for manual annotation of accident timestamps saves costs, would it be more effective to train the model using manually annotated data? Data with precisely labeled accident timestamps could also be used to train a model to predict, 3 seconds in advance, whether an accident is about to occur.
  3. What is the total offline cost associated with "hindsight supervision"? What is the accuracy of the teacher model?
Reviewer_zE7KOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a 2B streaming VLM host to warn of imminent accidents without annotated accident times. The supervision comes from the same frozen 8B verifier that runs at deployment. A few seconds after each training window the verifier is queried, and the max of its verdicts over a certain horizon becomes a multi-horizon risk target distilled into the host via LoRA. At inference, host and verifier thresholds are jointly calibrated on a pool of normal clips under one clip-level false-alarm budget.

Critical strengths
  • The paper is well written.

  • The end-to-end edge deployment on real boards is more than most papers in this area attempt.

Critical weaknesses
  • Novelty is thin. The method shares the privileged-future distillation pattern of existing works. The paper's own fixed-offset control (single query at t+4s) captures most of the gain, so the multi-horizon max only matters at tau=0, and the paired test for even that is unreported.

  • The target is biased toward false positives and the paper trains on it anyway. The max-over-horizons labels windows whose accident falls just beyond the horizon, while the top bucket's true positive rate is only 9–16%. Training windows are also selected by the teacher score. One verifier hallucination inside (t, t+5s] poisons the target at t, with no correction.

  • The headline budget guarantee fails in the motivating regime.

  • The theory is near-trivial on violated assumptions. Proposition 1 assumes independent windows within a clip, false for adjacent seconds of one unfolding precursor, and the empirical rho is estimated from 3 normal and 17 signal tail windows. The corollary is a one-line observation, and the memory baselines it dismisses are QA systems run on a warning task they were never built for.

Questions for authors

See Weaknesses

Reviewer_x8crOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Heimdall is a method for training and deploying a pre-warning model for streaming video without incident time annotations. The core hindsight distillation queries a frozen 8B verifier (Qwen3-VL-8B) for the “probability of an accident within 3 seconds” every second of [t, t+k] after each training window t. The maximum value across four horizons (1, 2, 3, and 5 seconds) is converted into a 5-bucket time-to-accident goal, and a 2B host (Qwen3-VL-2B, LoRA), which only views current footage, is distilled using the same query format. During deployment, the host scores the window every second, and the verifier runs asynchronously on a 2-second cycle, allowing either to issue an alert. Two thresholds are jointly selected from an external healthy clip pool (214 clips) to ensure that the total number of false alarms for the policy is less than or equal to the clip-unit budget B (main operating point B=10). Using the miss-rate identity under the independent window assumption and the score-shift upper bound (Prop. 1, Cor. 1), the authors establish that streaming memory cannot shift operating points beyond the budget-fixed threshold, while distillation can. It is argued that there is a result. In a test where PaSBench-Video was directly segmented by video unit (181 hazardous/78 normal), it reported R0 30.4%, R3 17.7%, and FAR 5.1% at B=10, and compared to a zero-shot pair without distillation (20.4%/13.3%, FAR 10.3%), same-time teacher (22.7%), fixed-offset teacher (24.3%), and annotated supervised host (30.4%), it claims that the gain comes from asking the teacher later. It evaluates the results of a verified corpus of 482 clips from 7 sources and a self-collected site with a fixed threshold transfer, comparisons with streaming memory, cascade, and protocol baselines, and real-time execution connecting a Jetson host and a server verifier.

Critical strengths

The system-level evaluation, including false alarms and delays, is thorough, and the results reporting is transparent. The Host and verifier are treated as a single alert policy to jointly correct two thresholds from an external normal pool (Eq. 2), and leading alerts are measured based on the actual alert commit time. The evaluation extends to the real-time loop connecting the Jetson host and the server verifier. It discloses 10 types of streaming memory, verdict write-back, voice results from fusion rules, R3 and R5 ties with fixed-offset teachers, increased false alarms during transitions and low recall in the field, as well as details regarding data exposure and duplication (Tables 8 and 9) and deduplication analysis. The fact that the operating conditions of the real-time alert system are reflected in the evaluation and that readers can independently determine the scope of improvement is rare in papers in this field, thereby enhancing the credibility of the reported figures.

The approach of learning leading alerts without incident time annotations is practical, and it provides control experiments that isolate the effects of future observations. It trains students who have only viewed up to the present based on the risk judgment of teachers who have seen future footage, and compares them with same-time and fixed-offset teachers within the same learning window. In independent tests, compared to zero-shot pairs, it showed an R0 of 20.4%→30.4% and a FAR of 10.3%→5.1% (sign test p=0.0005), tied with hosts trained with human annotations in R0 (30.4%), and outperformed them in R3 and R5 (17.7/14.9 vs 14.9/12.2). This is direct evidence for the claim that label-free goals can replace annotations and has practical significance in a deployment situation where there are no annotations on every new camera.

Critical weaknesses
  1. The additional gain of the Hindsight target is not sufficiently separated from its uncertainty, and the range of the proven effect is narrow. In Table 1, Heimdall has an R0 7.7 pp and 6.1 pp higher than the same-time and fixed-offset supervising, respectively. Since this represents a net increase of 14 and 11 detections based on 181 hazardous clips, the results are not negligible. However, the reported paired sign test is only compared to zero-shot pairs, and there are no confidence intervals for the difference compared to the direct paired comparison test with the two supervised controls that justify the proposed target. While there is one end-to-end iteration in Appendix C, variances with more than three seedings were not reported. Notably, the fixed-offset supervising and Heimdall are identical at R3=17.7%, R5=14.9%, and FAR=5.1%. As the authors acknowledge, at this operating point, the additional gain of the max-over-horizons aggregation appears only in the increase in warnings immediately before an accident, not in a longer lead time. Absolute performance is also an issue. At FAR 5%, R3 17.7% and R5 14.9% are far from deployable levels, and in self-collected sites, R0 remains at 5.7% even after field recalibration. Since the memory·cadence·write-back results selected from the development set of 37 hazardous/20 normal clips differ by 1 to 4 clips, they should be read only as exploratory evidence. Therefore, the proven contribution is narrowed to "increases the warning just before an accident (τ=0) by 11 to 14 clips," and the core contribution must be rephrased to fit this range, and direct pairwise comparisons and seed variances must be reported. This is the main reason for placing my rating on the borderline.

  2. The conclusions regarding the limitations of streaming memory are broader than the scope of theoretical and experimental application. Corollary 1 limits detection increment when the normal score distribution and threshold are fixed and the signal score receives only non-negative bounded increments. Cases where general memory changes the separation of normal and dangerous scores or shifts the threshold through recalibration are not addressed, and since there are arms in Table 2 that lose existing detection, it is unclear whether the experiment satisfies that assumption. There are also more direct counterexamples. In the development set, OASIS shows a higher R3 at a FAR of 10%, the same as Heimdall (16.2% vs. 10.8%, Table 11), and in Table 13, HERMES outperforms Heimdall at all lead times τ≥1 second, although the operating point is different (FAR 20%). In other words, based solely on the results disclosed by the authors, the memory baseline is dominant as the lead time increases. This is incompatible with the expression in contribution (3) “a streaming memory cannot move the operating point at a low false-alarm rate while supervising the host can.” The conclusion should be limited to the mechanism actually evaluated (score-shifting memory) and operating range (τ≈0, low FAR), and compared with the memory baseline using recall–FAR curves by lead time in independent tests. In its current form, it is difficult to accept the conclusion of the analysis presented as the third contribution.

Questions for authors
  1. Can you provide direct paired detection differences, confidence intervals, and tests between Heimdall and same-time/fixed-offset teachers? What about variances with seeding 3 or more times? How do you define the additional contribution of the max-over-horizons objective when fixed-offset and R3, R5, and FAR are identical?

  2. Since all hosts share a training window selected based on the hindsight criterion (R_k4 > 0.2), the same-time control is not a "sample when not looking into the future." Can you provide an ablation that separates the effect of the timing of the teacher query from the effect of the window selection (e.g., same-time hosts trained in a window selected with same-time scores)?

  3. What is the proportion of times at the teacher query time point s ∈ (t, t+k) where an annotation thought frame is already included? What is the verifier's post-accident detection accuracy when annotations are known? Depending on this answer, it is determined whether the hindsight target is "learning without annotations" or "learning where annotations are replaced by detector pseudo-labels," and the difference from temporally shifted distillation (Patera et al., 2026) also depends on this.

  4. How is Corollary 1 applied to memory arms where normal score distributions or thresholds change, or where existing detections are lost? Are you willing to limit the conclusions regarding memory general to the actual evaluated mechanism (score shift type)?

  5. Can the R0 comparison in Figure 4 be extended to R3 and R5, and can OASIS and HERMES be compared in independent tests by setting thresholds in the calibration pool? What is Heimdall's R0/R3 for FARs such as SPRINT (FAR 21.8%) and HERMES (20%)? Realization Please report the uncertainty of FAR and server call costs together.

Reviewer_Gp2BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents Heimdall, an annotation‑free proactive warning system for streaming safety‑monitoring video that leverages hindsight distillation: it retroactively obtains risk labels from a frozen 8B multimodal verifier’s later confident post‑accident judgements to train a lightweight 2B LoRA‑adapted on‑device host model, and deploys both models as a unified online policy with jointly calibrated thresholds to respect a global false‑alarm budget; theoretical analysis shows pure streaming‑memory approaches hit a performance ceiling by merely shifting scores, while extensive experiments on PaSBench‑Video and cross‑domain confirmation datasets, alongside Jetson edge‑hardware validation, demonstrate Heimdall yields better pre‑accident recall at lower false‑alarm rates compared to zero‑shot, same‑time distillation, memory‑based and annotation‑supervised baselines, with thorough ablation and failure‑mode analysis confirming gains stem from hindsight supervision rather than other implementation choices.

Critical strengths
  1. Novel annotation‑free hindsight‑distillation supervision for proactive video hazard warning. Instead of relying on expensive human‑annotated accident timestamps required by prior accident‑anticipation work, Heimdall generates training targets using later confident outputs from the already‑deployed large verifier model. This is practically consequential because it enables adapting safety warning systems to newly‑installed cameras where accident annotations do not exist, removing a major real‑world deployment bottleneck for streaming safety vision‑language models.

  2. Principled joint threshold calibration plus formal theoretical analysis under a hard false‑alarm budget. Real‑world safety warning systems must obey strict false‑alarm constraints, yet most existing streaming video‑LLM baselines lack such explicit bounded‑budget guarantees. The paper jointly calibrates thresholds for its two‑model policy and provides a miss‑rate identity proving that streaming‑memory techniques only shift scores and hit a hard performance ceiling under low false‑alarm budgets. This theoretical grounding strengthens empirical results and offers actionable reasoning for why the proposed distillation yields tangible improvements, rather than only reporting benchmark numbers.

Critical weaknesses
  1. Your hindsight distillation leverages future outputs from the verifier to generate supervision for past windows. Prior work such as temporally‑shifted distillation and RiskProp[1] also uses future video information for supervision. Could you explicitly articulate the core novelty gap between your approach and these existing future‑aware distillation methods? Temporally‑shifted distillation also uses teacher outputs from future timestamps; why is your approach more than just a variant of fixed‑offset temporal distillation? Additionally, R_k(t) is not a true probability and brings label noise. Have you quantified how label noise introduced by this hindsight pseudo‑label hurts model performance, and under what conditions this pseudo‑label would fail?

  2. The joint‑calibration of two‑model policy under a unified false‑alarm budget is an interesting contribution. However your domain‑transfer experiments clearly show thresholds calibrated on the source normal pool cannot transfer to unseen target domains and will violate the false‑alarm budget B. Real‑world cold‑start deployments for new cameras often lack sufficient in‑domain normal clips for calibration. To what extent is this joint‑calibration innovation practical for real‑world cold‑start scenarios? Is this calibration scheme largely unusable without sufficient target‑domain normal data?

  3. In your ablation, you compare same‑time teacher, fixed‑offset single‑step teacher and full multi‑horizon hindsight distillation with max‑pooling over future horizons. The performance gain comes from two factors: accessing future timestamps, and taking maximum across multiple future horizons. Could you disentangle and report the individual contribution of these two components? How much improvement originates purely from “looking into the future”, and how much comes from multi‑horizon max aggregation?

  4. For your core innovation hindsight distillation: is it a general‑purpose learning paradigm, or more of a task‑specific heuristic tailored for this proactive warning setting? What are the key prerequisites that must hold for this hindsight pseudo‑label trick to work?

[1] RiskProp: Collision-Anchored Self-Supervised Risk Propagation For Early Accident Anticipation. CVPR 2026.

Questions for authors

See Weaknesses

Reviewer_VToWOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Heimdall, a warning system for streaming video, which aims to predict accidents without requiring manually annotated accident times. The key idea is: training a 2B model with predictions from a frozen 8B VL model on later windows. This model is designed to recognize risks earlier. At inference time, the two models work together with jointly calibrated thresholds. Experiments on PaSBench-Video show improved recall and reduced FP compared to baselines. The paper also evaluates the method on additional datasets and Jetson devices.

Critical strengths

This paper addresses a practical problem: training proactive warning models without manually annotated accident timestamps. Using a verifier’s later predictions to supervise earlier video windows is well motivated. On the held-out test set, Heimdall improves pre-accident recall from 20.4% to 30.4% over the zero-shot pair while reducing FAR from 10.3% to 5.1%. Joint alarm calibration, wall-clock evaluation, and Jetson experiments further strengthen the paper’s practical relevance.

Critical weaknesses
  1. The motivation behind the adaptation to the deployment of newly installed cameras is not completely supported by the evaluation in the paper. While the student is trained on PaSBench-Video split, it fails to show the ability to distill the hindsight on new cameras' unlabeled data. In the case of the confirmation corpus, transferring of the thresholds raises FAR to 20.8%, but recalibration lowers it to 5.5% at the expense of reducing R0 from 30.8% to 18.4% (Table 15). On the self-collected site, recalibration reaches just 5.7% R0. These results leave the effectiveness of actual site-specific adaptation unclear.

  2. The added value of the multi-horizon target over a simpler fixed-offset teacher is not fully established. In the held-out test set, both achieve identical R3 (17.7%), R5 (14.9%), and FAR (5.1%), with Heimdall improving only R0 (30.4% vs. 24.3%). Appendix C provides similar results when per-stream thresholds are used. Since the reported significance tests compare against the zero-shot baseline rather than directly against the fixed-offset teacher, it remains unclear how consistent the additional benefit of the multi-horizon design is.

Questions for authors
  1. Can the authors evaluate hindsight distillation on unlabeled footage from a new camera? Given the limited results on the self-collected site (Table 15), what evidence supports its effectiveness for new-camera adaptation?
  2. What is the benefit of the multi-horizon target over fixed-offset distillation? Please provide a direct paired comparison and clarify the role of joint calibration, since both methods achieve identical R3 and R5 (Table 1).
  3. Does Corollary 1 apply to memory methods that change both normal and hazard score distributions? How do the authors explain OASIS achieving higher R3 than Heimdall at the same FAR (Table 13)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.