- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a training-free framework that turns a frozen streaming VLM into a proactive and asynchronous system, namely FORESIGHT. Two copies of the same model share one KV cache. An Ingest copy continuously appends admitted frames, and a Think copy wakes at scheduled checks, reads a snapshot of the cache and writes a structured plan. Each plan sets when to check next, which question to revisit, the sampling rate, which object classes to keep or ignore, whether to compact the cache, and whether the evidence suffices to respond. A live executor applies the plan, and a per-task calibrated gate decides when to emit answers. FORESIGHT reports joint F1 on OmniPro Online against for the strongest baseline, and gains of and points over its backbone on StreamingBench and OVO-Bench.
Critical strengths
-
Proactive responding improves substantially with a frozen backbone in using the proposed method. The gain comes mainly from timing: Time F1 is 43.7 against 24.4, while content accuracy is similar at 54.6 against 54.4 (Table 5). This matters to my assessment because it shows that an inference-time controller alone can improve when a streaming VLM chooses to respond.
-
The system design is simple and backbone-agnostic. A single writer to the shared KV cache, with read-only snapshots, lets planning run without pausing ingestion, and schema-guided decoding with diff-based updates keeps each plan short. Because no weights are trained, the same controller can in principle be applied to other backbones, which makes the design generally useful for the community.
Critical weaknesses
- My main concern will be the evaluation protocol of the paper seems to favors FORESIGHT, and several reported numbers do not match the stated protocol.
- The response gate is fitted per task on % of each evaluation set. As far as the paper states, the baselines receive no such calibration.
- FORESIGHT is evaluated on the 80% held-out split while baselines use the full sets. The backbone rows are copied from Lu et al. (2026). FORESIGHT’s StreamingBench averages exclude Proactive Output, whereas the baseline averages include it.
- Several OVO-Bench accuracies cannot arise from the stated split (Question 2).
- The experiments do not isolate the central claim that the model plans its future perception.
- Removing FPS, Keep Classes, Ignore Classes, or Compact Now changes joint F1 by at most 0.4 points (Table 4). The perception-planning fields named in the title therefore do not measurably affect accuracy on OmniPro.
- No experiment compares the anticipatory schedule with a fixed-interval schedule at the same check rate.
- Efficiency is modeled, instead of measured. The efficiency claims rest on a cost model whose factors, in the paper’s words, “come from the model, not from measurement” (Appendix G). FORESIGHT’s real-time factor appears only as a point in Figure 1, without a measurement protocol. The roughly 150 GH200 GPU-hours for one OmniPro evaluation (Appendix B.1), against 143 hours of footage (Appendix G.1), suggest a real-time factor at or above one, which is hard to reconcile with Figure 1.
Questions for authors
(W1)
-
How does FORESIGHT perform with one gate setting shared across all tasks? What do the strongest baselines achieve when their own response thresholds receive the same per-task calibration, where such thresholds exist (e.g., QueryStream’s similarity threshold)? Please also report the gate settings for Cumulative-Count, Event-Narration, and Deduplicated-Count, and the variance across random calibration splits.
-
Could the authors report StreamingBench under one averaging convention for all methods? With Proactive Output included, FORESIGHT’s Contextual score is 44.50. The group weights that reproduce the overall scores of Qwen3-VL-8B and MiniCPM-o 4.5 (2,500, 1,000, and 1,000 questions) then give an overall of about 64.8 rather than 66.00. Could the authors also evaluate the frozen backbone under the same protocol, frame budget, and held-out split as FORESIGHT on StreamingBench and OVO-Bench?
(W2)
-
Please compare FORESIGHT with a fixed-interval schedule at the same average check rate, using the same frozen model, prompts, and gate.
-
Could the authors report the measured check rate, admitted frame rate, visual token count, latency, and real-time factor of FORESIGHT on each benchmark, with and without the perception-planning fields? How many GPUs serve one stream, and do the Ingest and Think twins share a GPU?
-
How are visual tokens assigned to Keep and Ignore Classes inside the frozen encoder? If this step relies on an additional detector or segmentation model, what does it cost? How does Compact Now compact the KV cache, and what does each snapshot of the cache cost in memory and time?
2 comments
[W2 / Q1] Planning future perception and matched fixed scheduling
The perception-planning fields allocate computation to make real-time operation possible. At a similar check rate (about 0.30 vs. 0.25 checks per second), a fixed check schedule and frame rate require 1.70 s of computation per second of video, versus 0.67 s with FORESIGHT's model-chosen ones. Adaptive FPS reduces vision encoding from 273 to 59 ms and Ingest prefill from 396 to 98 ms per second of video; Think time approximately halves. Section CR3 of our common response above provides the measured comparison.
The interval choices also track upcoming events: a 1-second interval is chosen 79% of the time when an event is less than 1 s away, while an 8-second interval is chosen 58–64% of the time when an event is more than 30 s away or there is none. These annotations are never supplied to the planner. The full distribution is in Section CR3 of our common response above.
[W2 / Q2] Measured work, latency, RTF, hardware, and Figure 1
We now report measured costs on OmniPro instead of relying on the cost model. One GH200 serves each stream and is shared by both twins; peak memory is 65 GB. FORESIGHT checks about once every 4 s on average and runs in real time: each second of video takes 1.0 s of wall time, and no frame is dropped. Section CR3 of our common response above gives the measured module costs and fixed-scheduling comparison.
In compute, ingestion still dominates, as in Appendix G, because prefill processes far more tokens per second than the Think twin decodes; in wall time, token-by-token decoding takes the larger share, yet the total still fits within real time. Figure 1's real-time factor follows this protocol, and we will state it in the caption. Scored timing reflects actual answer availability to the user.
[W2 / Q3] Class pruning, compaction, and snapshot costs
Class pruning. An additional frozen CLIP ViT-L/14 model scores image patches against the Keep and Ignore class names. Its score map is resampled to Qwen3-VL's token grid, and the top 70% of visual tokens are retained before they enter the KV cache. CLIP has 0.43B parameters and costs about 17 ms per frame on one GH200. Section CR4 of our common response above gives the exact scoring procedure.
Snapshot. In our implementation, a snapshot copies the shared KV cache, which is the simplest way to keep the Think twin's reads separate from Ingest's writes. It costs 11 ms per check and one extra copy of the cache in memory (see Section CR3 of our common response above). The copy can be removed entirely: Ingest writes into a fixed, pre-allocated buffer and publishes the new length, so a snapshot becomes just that length ("the first L tokens") and nothing is copied. The Think twin's own tokens then go into a small private buffer and attend to the shared prefix plus this buffer without locking.
Compact Now. Compaction rewrites old tokens in the cache, at most once every 30 s. Because it modifies tokens that the Think twin reads, it runs between Think steps.
Thank you for the careful numerical checks and for recognizing that a frozen-backbone controller can improve proactive response timing. We address W1, W2, and each question below and refer to the numbered sections of our common response for the full tables and shared explanations.
[W1 / Q1] Shared gate, calibrated baselines, missing settings, and split variability
Following your feedback, we removed the in-context examples that motivated the original threshold fitting. The revised FORESIGHT uses one gate setting shared across all nine tasks, with no setting fitted to data, and obtains 27.14 macro Joint F1, 46.85 macro Time F1, and 59.24 macro content accuracy on the full 932-sample OmniPro set. On those same samples without calibration, the original prompt reaches 16.67 Joint F1. Per-task calibration raises QueryStream only from 6.7 to 7.36. Section CR1 of our common response above gives the comparison, prompt diagnostic, and uncertainty.
For completeness, the settings omitted from Table 10 in the submitted calibrated version were: Cumulative-Count 0.925, edge, 7 s; Event-Narration 0.10, level, 7 s; Deduplicated-Count 0.992, edge, 5 s. Since the revised FORESIGHT has no calibration split, there is no variance across splits; across three end-to-end runs, the standard deviation of macro Joint F1 is 1.06 points.
[W1 / Q2] StreamingBench aggregation and matched frozen-backbone evaluation
You are right about the aggregation. FORESIGHT fits no thresholds on StreamingBench and uses the full evaluation set; the 20%/80% split applied only to OmniPro. Our submitted overall excluded Proactive Output for FORESIGHT while including it for baselines. Including it gives 44.50 Contextual and 64.77 overall, matching your 64.8. The gain over the published backbone changes from +6.7 to +5.5, while FORESIGHT remains first, 2.07 points above the published MiniCPM-o 4.5 score. We will correct Tables 2 and 7 and the headline statements.
We also ran the frozen backbone under FORESIGHT's protocol: on identical samples, Qwen3-VL-8B scores 58.41, MiniCPM-o 4.5 61.45, and FORESIGHT 64.77. FORESIGHT therefore improves its own backbone by 6.36 points and leads MiniCPM-o 4.5 by 3.32 points. Section CR2 of our common response above gives the formula and matched-run results.
[W1] OVO-Bench accuracies and stated split
Thank you; your check exposed errors in our description and several reported per-task scores. FORESIGHT fits nothing on OVO-Bench and uses the full evaluation set, so the stated 20%/80% split does not apply. Section CR2 of our common response above provides the recalculated per-task accuracies and the corrected 62.29 overall; we will correct the text and affected tables.
We replaced the Qwen3-VL-8B and MiniCPM-o 4.5 rows copied from Lu et al. (2026) with our own runs under the same OVO-Bench v2 protocol, video-only input, frame budget, and full evaluation set. The frozen backbone scores 45.86, MiniCPM-o 4.5 59.10, and FORESIGHT 62.29. Thus FORESIGHT improves its own backbone by 16.43 points and leads MiniCPM-o 4.5 by 3.19 points. The matched comparison is in Section CR2 of our common response above.