acceptodds
Under review as a conference paper at ICLR 2027

Foresight: Planning Future Perception in Streaming VLMs without Retraining

Abstract

Existing streaming vision-language models (VLMs) continuously perceive and reason over visual streams, but their computational pathways remain fixed throughout inference. Consequently, they cannot adapt computation to evolving scene dynamics, where different future events demand different levels and forms of perception. We show that streaming VLMs inherently possess the ability to anticipate the immediate future, and leverage this capability to dynamically configure future computation in a training-free manner. Realizing such anticipatory computation, however, is very challenging: future anticipation must be sufficiently reliable to guide computation, planning must run concurrently with streaming inference, and online reconfiguration must incur negligible overhead. To address these challenges, we introduce a dual-stream architecture comprising two Siamese LLMs with shared weights and input encoders. The first LLM continuously processes incoming tokens, while the second runs ahead of the stream to anticipate future context, plan future computation, and generate task responses without interrupting streaming inference. The resulting computation plan is executed online through an efficient reconfiguration protocol with lightweight diff-based updates, enabling dynamic adaptation with negligible overhead. With a frozen Qwen3-VL-8B backbone, FORESIGHT achieves 23.0 mean joint F1 on OmniPro Online evaluation beating strongest trained baseline by 9.5%, while improving the backbone by 6.7 on StreamingBench and 15.4 on OVO-Bench, with the largest gain of 18.7 when evidence arrives later in the video stream. Our source code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, streaming, training free, proactive

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tTqBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a training-free framework that turns a frozen streaming VLM into a proactive and asynchronous system, namely FORESIGHT. Two copies of the same model share one KV cache. An Ingest copy continuously appends admitted frames, and a Think copy wakes at scheduled checks, reads a snapshot of the cache and writes a structured plan. Each plan sets when to check next, which question to revisit, the sampling rate, which object classes to keep or ignore, whether to compact the cache, and whether the evidence suffices to respond. A live executor applies the plan, and a per-task calibrated gate decides when to emit answers. FORESIGHT reports 23.023.0 joint F1 on OmniPro Online against 13.513.5 for the strongest baseline, and gains of 6.76.7 and 15.415.4 points over its backbone on StreamingBench and OVO-Bench.

Critical strengths
  1. Proactive responding improves substantially with a frozen backbone in using the proposed method. The gain comes mainly from timing: Time F1 is 43.7 against 24.4, while content accuracy is similar at 54.6 against 54.4 (Table 5). This matters to my assessment because it shows that an inference-time controller alone can improve when a streaming VLM chooses to respond.

  2. The system design is simple and backbone-agnostic. A single writer to the shared KV cache, with read-only snapshots, lets planning run without pausing ingestion, and schema-guided decoding with diff-based updates keeps each plan short. Because no weights are trained, the same controller can in principle be applied to other backbones, which makes the design generally useful for the community.

Critical weaknesses
  1. My main concern will be the evaluation protocol of the paper seems to favors FORESIGHT, and several reported numbers do not match the stated protocol.
  • The response gate is fitted per task on 2020% of each evaluation set. As far as the paper states, the baselines receive no such calibration.
  • FORESIGHT is evaluated on the 80% held-out split while baselines use the full sets. The backbone rows are copied from Lu et al. (2026). FORESIGHT’s StreamingBench averages exclude Proactive Output, whereas the baseline averages include it.
  • Several OVO-Bench accuracies cannot arise from the stated split (Question 2).
  1. The experiments do not isolate the central claim that the model plans its future perception.
  • Removing FPS, Keep Classes, Ignore Classes, or Compact Now changes joint F1 by at most 0.4 points (Table 4). The perception-planning fields named in the title therefore do not measurably affect accuracy on OmniPro.
  • No experiment compares the anticipatory schedule with a fixed-interval schedule at the same check rate.
  • Efficiency is modeled, instead of measured. The efficiency claims rest on a cost model whose factors, in the paper’s words, “come from the model, not from measurement” (Appendix G). FORESIGHT’s real-time factor appears only as a point in Figure 1, without a measurement protocol. The roughly 150 GH200 GPU-hours for one OmniPro evaluation (Appendix B.1), against 143 hours of footage (Appendix G.1), suggest a real-time factor at or above one, which is hard to reconcile with Figure 1.
Questions for authors

(W1)

  • How does FORESIGHT perform with one gate setting shared across all tasks? What do the strongest baselines achieve when their own response thresholds receive the same per-task calibration, where such thresholds exist (e.g., QueryStream’s similarity threshold)? Please also report the gate settings for Cumulative-Count, Event-Narration, and Deduplicated-Count, and the variance across random calibration splits.

  • Could the authors report StreamingBench under one averaging convention for all methods? With Proactive Output included, FORESIGHT’s Contextual score is 44.50. The group weights that reproduce the overall scores of Qwen3-VL-8B and MiniCPM-o 4.5 (2,500, 1,000, and 1,000 questions) then give an overall of about 64.8 rather than 66.00. Could the authors also evaluate the frozen backbone under the same protocol, frame budget, and held-out split as FORESIGHT on StreamingBench and OVO-Bench?

(W2)

  • Please compare FORESIGHT with a fixed-interval schedule at the same average check rate, using the same frozen model, prompts, and gate.

  • Could the authors report the measured check rate, admitted frame rate, visual token count, latency, and real-time factor of FORESIGHT on each benchmark, with and without the perception-planning fields? How many GPUs serve one stream, and do the Ingest and Think twins share a GPU?

  • How are visual tokens assigned to Keep and Ignore Classes inside the frozen encoder? If this step relies on an additional detector or segmentation model, what does it cost? How does Compact Now compact the KV cache, and what does each snapshot of the cache cost in memory and time?

2 comments

Thank you for the careful numerical checks and for recognizing that a frozen-backbone controller can improve proactive response timing. We address W1, W2, and each question below and refer to the numbered sections of our common response for the full tables and shared explanations.

[W1 / Q1] Shared gate, calibrated baselines, missing settings, and split variability

Following your feedback, we removed the in-context examples that motivated the original threshold fitting. The revised FORESIGHT uses one gate setting shared across all nine tasks, with no setting fitted to data, and obtains 27.14 macro Joint F1, 46.85 macro Time F1, and 59.24 macro content accuracy on the full 932-sample OmniPro set. On those same samples without calibration, the original prompt reaches 16.67 Joint F1. Per-task calibration raises QueryStream only from 6.7 to 7.36. Section CR1 of our common response above gives the comparison, prompt diagnostic, and uncertainty.

For completeness, the settings omitted from Table 10 in the submitted calibrated version were: Cumulative-Count 0.925, edge, 7 s; Event-Narration 0.10, level, 7 s; Deduplicated-Count 0.992, edge, 5 s. Since the revised FORESIGHT has no calibration split, there is no variance across splits; across three end-to-end runs, the standard deviation of macro Joint F1 is 1.06 points.

[W1 / Q2] StreamingBench aggregation and matched frozen-backbone evaluation

You are right about the aggregation. FORESIGHT fits no thresholds on StreamingBench and uses the full evaluation set; the 20%/80% split applied only to OmniPro. Our submitted overall excluded Proactive Output for FORESIGHT while including it for baselines. Including it gives 44.50 Contextual and 64.77 overall, matching your 64.8. The gain over the published backbone changes from +6.7 to +5.5, while FORESIGHT remains first, 2.07 points above the published MiniCPM-o 4.5 score. We will correct Tables 2 and 7 and the headline statements.

We also ran the frozen backbone under FORESIGHT's protocol: on identical samples, Qwen3-VL-8B scores 58.41, MiniCPM-o 4.5 61.45, and FORESIGHT 64.77. FORESIGHT therefore improves its own backbone by 6.36 points and leads MiniCPM-o 4.5 by 3.32 points. Section CR2 of our common response above gives the formula and matched-run results.

[W1] OVO-Bench accuracies and stated split

Thank you; your check exposed errors in our description and several reported per-task scores. FORESIGHT fits nothing on OVO-Bench and uses the full evaluation set, so the stated 20%/80% split does not apply. Section CR2 of our common response above provides the recalculated per-task accuracies and the corrected 62.29 overall; we will correct the text and affected tables.

We replaced the Qwen3-VL-8B and MiniCPM-o 4.5 rows copied from Lu et al. (2026) with our own runs under the same OVO-Bench v2 protocol, video-only input, frame budget, and full evaluation set. The frozen backbone scores 45.86, MiniCPM-o 4.5 59.10, and FORESIGHT 62.29. Thus FORESIGHT improves its own backbone by 16.43 points and leads MiniCPM-o 4.5 by 3.19 points. The matched comparison is in Section CR2 of our common response above.

[W2 / Q1] Planning future perception and matched fixed scheduling

The perception-planning fields allocate computation to make real-time operation possible. At a similar check rate (about 0.30 vs. 0.25 checks per second), a fixed check schedule and frame rate require 1.70 s of computation per second of video, versus 0.67 s with FORESIGHT's model-chosen ones. Adaptive FPS reduces vision encoding from 273 to 59 ms and Ingest prefill from 396 to 98 ms per second of video; Think time approximately halves. Section CR3 of our common response above provides the measured comparison.

The interval choices also track upcoming events: a 1-second interval is chosen 79% of the time when an event is less than 1 s away, while an 8-second interval is chosen 58–64% of the time when an event is more than 30 s away or there is none. These annotations are never supplied to the planner. The full distribution is in Section CR3 of our common response above.

[W2 / Q2] Measured work, latency, RTF, hardware, and Figure 1

We now report measured costs on OmniPro instead of relying on the cost model. One GH200 serves each stream and is shared by both twins; peak memory is 65 GB. FORESIGHT checks about once every 4 s on average and runs in real time: each second of video takes 1.0 s of wall time, and no frame is dropped. Section CR3 of our common response above gives the measured module costs and fixed-scheduling comparison.

In compute, ingestion still dominates, as in Appendix G, because prefill processes far more tokens per second than the Think twin decodes; in wall time, token-by-token decoding takes the larger share, yet the total still fits within real time. Figure 1's real-time factor follows this protocol, and we will state it in the caption. Scored timing reflects actual answer availability to the user.

[W2 / Q3] Class pruning, compaction, and snapshot costs

Class pruning. An additional frozen CLIP ViT-L/14 model scores image patches against the Keep and Ignore class names. Its score map is resampled to Qwen3-VL's token grid, and the top 70% of visual tokens are retained before they enter the KV cache. CLIP has 0.43B parameters and costs about 17 ms per frame on one GH200. Section CR4 of our common response above gives the exact scoring procedure.

Snapshot. In our implementation, a snapshot copies the shared KV cache, which is the simplest way to keep the Think twin's reads separate from Ingest's writes. It costs 11 ms per check and one extra copy of the cache in memory (see Section CR3 of our common response above). The copy can be removed entirely: Ingest writes into a fixed, pre-allocated buffer and publishes the new length, so a snapshot becomes just that length ("the first L tokens") and nothing is copied. The Think twin's own tokens then go into a small private buffer and attend to the shared prefix plus this buffer without locking.

Compact Now. Compaction rewrites old tokens in the cache, at most once every 30 s. Because it modifies tokens that the Think twin reads, it runs between Think steps.

Reviewer_24b6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Foresight proposes a training-free framework for proactive streaming video understanding with a frozen vision-language model. Its central claim is that the model can anticipate the arrival of task-relevant evidence and use this anticipation to allocate future computation. The controller decides when to reason again, which unresolved question to revisit, and how densely to sample upcoming observations. The architecture separates perception and planning into two execution roles sharing model weights, a visual encoder, and a causal KV cache. Ingest continuously processes admitted observations and is the sole writer to the visual history. Think reads a cache snapshot and generates plans asynchronously, allowing perception to continue during planning. Generated plans remain outside the visual history to separate observations from predictions. Each plan distinguishes transient evidence from persistent control. The evidence state determines whether sufficient information is available to respond, together with an answer and event timestamp when appropriate. Control updates specify the next checking time, a deferred question, sampling density, class-based token pruning, and cache compaction. A live executor applies these updates, while temporal gates suppress premature or duplicate responses. The claimed contributions are a formulation of anticipatory computation for streaming VLMs, a shared-state architecture for asynchronous perception and planning, and a lightweight reconfiguration protocol using schema-guided decoding and partial updates. The paper argues that this combination improves proactive understanding while limiting planning overhead and avoiding backbone retraining. Its training-free characterization refers to unchanged model weights; response thresholds and gate configurations are nevertheless fitted on benchmark calibration data.

Critical strengths
  1. A practical streaming architecture. Separating ingestion from asynchronous planning addresses the difficulty of processing new observations while reasoning. A single writer maintains causal state, and the planner reads a snapshot. This provides a concrete mechanism for supporting standing instructions over evolving streams, with an explicit interface between planning and execution.

  2. Training-free integration. Shared weights and cached observations allow the controller to reuse an existing VLM. Schema-guided decoding and partial updates are sensible measures for reducing repeated generation. Although their runtime benefits require stronger measurement, the integrated design offers practical value by avoiding a separate streaming fine-tuning pipeline.

  3. Promising evidence on relevant tasks. Reported gains on OmniPro and OVO-Bench Forward Active Responding align with maintaining an instruction until useful evidence arrives. Evaluation across three benchmarks suggests benefits beyond isolated demonstrations. These results provide a substantive acceptance argument, provided their comparability and attribution can be established convincingly.

Critical weaknesses
  1. Incremental novelty remains unclear. StreamAgent already anticipates future evidence and adjusts perception in streaming video; long-video agents also perform task-driven evidence acquisition. FORESIGHT should establish what its particular scheduling, shared-state execution, and control updates contribute beyond these precedents. The current comparisons do not sufficiently isolate those differences.

  2. Anticipation and efficiency are insufficiently substantiated. Removing adaptive FPS leaves Joint F1 unchanged, and scheduling ablations barely affect Time F1. Disabling the entire plan also disrupts the response interface. Fixed or reactive controls at matched compute are needed. Efficiency curves using assumed check-rate distributions do not establish measured runtime savings.

  3. Evaluation protocols are inconsistent. FORESIGHT's held-out 80% results are compared with full-set baselines, some OmniPro modality subsets differ, and StreamingBench aggregation excludes Proactive Output for FORESIGHT while including it for baselines. These differences weaken the headline comparisons. Identical evaluation samples, aggregation rules, and uncertainty estimates are necessary.

  4. Commercial comparisons are outdated. Historical GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet results do not establish competitiveness against current real-time visual systems. Gemini 3.8 Live merits comparison; image-enabled OpenAI Realtime requires an explicit adapter. Comparisons should document causal delivery, sampling limits, response-triggering policies, and actual latency, using versions available before submission.

Questions for authors
  1. If unexpected evidence arrives while Think is waiting for its scheduled check, can Ingest trigger an earlier recheck, or must the system wait? What bounds checking intervals and sampling rates when anticipation is wrong, and how do these safeguards affect missed events and response delay?

  2. Can all methods be evaluated on identical held-out examples, modality subsets, benchmark versions, and aggregation rules? How does consistently including Proactive Output affect StreamingBench? Please clarify whether calibration and evaluation are separated by source video, and provide per-task sample counts and uncertainty estimates for the main comparisons.

  3. What replaces removed scheduling fields, and are ablation gates recalibrated? Why does removing FPS leave Joint F1 unchanged, and what measured savings does adaptation provide? Please report actual check rates, per-stream hardware, peak memory, and latency tails, clarifying whether timing reflects actual answer availability.

5 comments

Thank you for the detailed review and for recognizing the shared-state architecture and its relevance to standing instructions over evolving streams. We address the weaknesses and follow-up questions in order and refer to the numbered sections of our common response for the full tables and shared explanations.

[W1] Incremental novelty relative to StreamAgent and long-video agents

StreamAgent. StreamAgent also uses future anticipation, but its planning runs on a fixed schedule and is computationally heavy: every clip triggers a caption, multiple candidate plans (reactive, proactive, and speculative), an LLM judge that selects one, and a trigger check. In FORESIGHT, the model itself decides, from the task and the video, when the next check happens. Each plan is a single structured output, written as a diff of the fields that change, and perception and planning share one KV cache, so the stream is encoded only once and no captions are needed. FORESIGHT also does not invoke tools such as zoom-in, detection, or tracking. On a single GH200, StreamAgent needs 2.1–2.7 seconds of compute per second of video (Appendix G.1), whereas FORESIGHT runs in real time (see Section CR3 of our common response above). StreamAgent reaches 5.0 Joint F1 on OmniPro, against 27.14 for FORESIGHT (see Section CR1 of our common response above).

Long-video agents. VideoAgent starts from frames sampled from a complete, pre-recorded video, identifies missing evidence, retrieves frames from selected segments with CLIP, and captions them. FORESIGHT instead works on a stream whose future frames do not exist yet: it schedules future checks and perception while frames arrive causally, and ingestion continues during planning and answering. Moreover, in proactive streaming the system must respond to every event that satisfies the query (OmniPro has more than three events per video on average), rather than search an available video for the evidence to answer one question.

Isolating the differences. The model-chosen Next Check and FPS are key to making FORESIGHT efficient: the model checks and samples densely only when it expects an event, and sparsely otherwise. Section CR3 of our common response above reports the comparison with a fixed check schedule and frame rate, measured module costs, and the planner's interval choices relative to upcoming events.

[W2] Anticipation, adaptive FPS, and fair control ablations

Adaptive FPS and scheduling allocate computation while retaining useful responses, so unchanged Joint F1 is consistent with the intended quality–efficiency trade-off. At a similar check rate (about 0.30 vs. 0.25 checks per second), the fixed check schedule and frame rate require 1.70 s of computation per second of video, compared with 0.67 s for FORESIGHT. Adaptive FPS cuts vision encoding from 273 to 59 ms and Ingest prefill from 396 to 98 ms per second of video; Think time approximately halves. Section CR3 of our common response above reports the measured breakdown.

The planner also checks more densely before events: when the next annotated event is less than 1 s away, it chooses a 1-second interval 79% of the time; when it is more than 30 s away or there is none, it chooses an 8-second interval 58–64% of the time. The planner never sees the annotations. Section CR3 of our common response above provides the full interval distribution.

[W3 / Q2] Identical samples, modality conditions, aggregation, and uncertainty

The revised calibration-free FORESIGHT reaches 27.14 macro Joint F1, 46.85 macro Time F1, and 59.24 macro content accuracy on the full OmniPro set. The 100% rows use the same 932 samples with audio labels "none" or "helpful," from 699 source videos; Section CR1 of our common response above gives per-task counts and the comparison. Across three full-system runs, Joint F1 has a standard deviation of 1.06 points, compared with a 13.6-point margin over the strongest baseline. No evaluation sample is now used for fitting; the submitted calibration and evaluation videos were disjoint within each task.

On StreamingBench and OVO-Bench, no thresholds were fitted and results use the full evaluation sets; we will correct the split description. Our runs use identical examples, video-only input, the same frame budget, OVO-Bench v2, and one aggregation rule per benchmark. Including Proactive Output changes FORESIGHT's StreamingBench overall from 66.00 to 64.77; it remains first against the published MiniCPM-o 4.5 score of 62.70. Proactive Output is also reported separately (46.3, 95% CI 39.1–53.7). Against our own matched MiniCPM-o 4.5 runs, FORESIGHT leads by 3.32 points on StreamingBench and 3.19 points on OVO-Bench. Section CR2 of our common response above gives the aggregation and matched comparisons.

[W4] Current commercial systems

We agree that historical GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet scores do not establish competitiveness against current real-time products. Since running evaluation on the current frontier models is costly, we leave it for future work.

[Q1] Unexpected evidence during a scheduled wait

In FORESIGHT, Ingest cannot trigger a check. Hence, the next-check options for the Think LLM should be small enough not to let events be missed. In our experiments, the allowed next-check intervals are 1 s, 2 s, 4 s, and 8 s. Only 0.8% of events are missed due to a late next check.

[Q3] Replacements, recalibration, and real response efficiency

What replaces a removed field, and are the gates recalibrated? A removed field keeps its default value from the initial controller state U0, because the merge keeps any field the plan leaves out (Section 4.1). Without Next Check, the Think LLM wakes at the fixed default interval Δ0; without Next Question, each check uses only the original task instruction. The response gate was kept identical to the full system in every ablation. In the revised FORESIGHT, calibration is removed entirely.

Why removing FPS leaves Joint F1 unchanged. Its main function is to make the system efficient enough to run in real time. Without the FPS field, the stream is sampled at the full rate, so the model sees all of the visual information. With adaptive FPS, it samples sparsely when nothing relevant is happening and densely around the moments it expects to matter. Those moments still carry enough information to match full sampling at a lower cost.

Measured savings, hardware, and latency. Model-chosen Next Check and FPS reduce computation from 1.70 s to 0.67 s per second of video; Ingest prefill alone drops from 396 to 98 ms. Each stream uses one GH200 shared by both twins, with peak memory of 65 GB. Checks occur about once every 4 s on average, reaching 0.4 checks per second on event-dense streams. Scored timing reflects when the user receives the answer. Section CR3 of our common response above gives the complete measurements and answer-availability protocol.

Thank you for the rebuttal. The measured compute savings and the clarified ablation defaults address most of my concerns.

A few clarifications, none requiring new experiments:

  1. Please report the Joint/Time F1 of the fixed-schedule baseline already run in CR3, alongside its compute.
  2. Please define how the 0.8% missed events are counted. If available from existing logs, please also report the response delay caused by scheduled waits.

I will take these into account in my final assessment.

Reviewer_626xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Foresight, a training-free framework for proactive streaming video understanding built on a frozen VLM. It separates continuous video ingestion from asynchronous planning through a shared KV cache. The planner decides when to reason next, what question to revisit, and how densely to sample incoming frames. An executor applies these updates and releases responses through an evidence-based gate. Experiments on OmniPro, StreamingBench, and OVO-Bench report improvements in proactive responding and streaming video understanding.

Critical strengths
  1. The paper addresses an important practical problem. Streaming video assistants must balance continuous perception, limited computation, and timely responses. Allowing the model to plan when and what to check next is a meaningful direction.

  2. The framework has a coherent overall design. Asynchronous perception and planning, shared KV caching, and incremental plan updates form a practical control loop. Keeping planning outputs separate from visual history also helps avoid treating predictions as observed evidence.

  3. Foresight improves over its frozen backbone on StreamingBench and OVO-Bench. The ablations also show that scheduling the next check and maintaining a follow-up question contribute to performance.

Critical weaknesses
  1. The calibration and evaluation conditions are not aligned. Foresight uses 20% of each official evaluation set to calibrate its gating parameters and is evaluated on the remaining 80%, while baseline results are reported on the full set. The methods should be compared on the same subset under comparable calibration conditions. The authors should also test whether the calibrated parameters transfer to new settings without requiring recalibration on each target benchmark.

  2. The implementation of category-based pruning is unclear. The paper does not adequately explain how Keep/Ignore Classes are mapped to specific visual tokens. Please clarify how the relevant tokens are identified, where pruning occurs in the pipeline, and whether additional models are required.

  3. The gains from the backbone and the proposed strategy are not sufficiently separated. Some training-free baselines use different backbones, making it difficult to determine whether the improvements come from Qwen3-VL itself or the Foresight strategy. Comparisons using the same backbone would help isolate the contribution of the proposed method.

  4. The empirical efficiency analysis is insufficient. The overall real-time factor and theoretical cost analysis do not fully characterize response efficiency. Please report the time spent in each module and the end-to-end response latency.

Questions for authors

The questions are the same as the weakness section.

3 comments

Thank you for the careful review and for recognizing the practical importance of planning when and what to check in a visual stream. Your comments helped us clarify both the experimental protocol and the implementation. We address each concern below and refer to the numbered sections of our common response for the full tables and shared explanations.

[W1] Calibration, identical evaluation subsets, and transfer

We agree that the original 80%-versus-full-set comparison was not matched. Removing only the task-specific in-context examples enabled FORESIGHT to operate without response-gate calibration. On the full 932-sample OmniPro set, the revised configuration reaches 27.14 macro Joint F1, versus 13.5 for MiniCPM-o 4.5. On those same samples and without calibration, the original prompt reaches 16.67, so the improvement comes from removing the examples rather than changing the evaluation set.

There are now no calibrated parameters that need to transfer: FORESIGHT neither updates weights nor fits thresholds on benchmark data. Per-task calibration raises QueryStream only from 6.7 to 7.36 Joint F1. Section CR1 of our common response above provides the full comparison, explanation, sample counts, and uncertainty; Section CR2 of our common response above clarifies that StreamingBench and OVO-Bench already used full evaluation sets without threshold fitting. We will correct the manuscript accordingly.

[W2] How Keep/Ignore Classes map to visual tokens

We use an additional frozen CLIP ViT-L/14 model to score image patches against the plan's Keep and Ignore class names. Its score map is resampled to Qwen3-VL's token grid, and the top 70% of visual tokens are retained before they enter the KV cache. The additional model has 0.43B parameters and costs about 17 ms per frame on one GH200. Section CR4 of our common response above explains the exact scoring rule and where pruning occurs.

[W3] Backbone versus FORESIGHT's controller

Our reproduced training-free baselines all use the same frozen Qwen3-VL-8B backbone as FORESIGHT. On OmniPro, QueryStream reaches 6.7 Joint F1 (7.36 with per-task calibration) and StreamAgent 5.0, against 27.14 for FORESIGHT (see Section CR1 of our common response above). We also ran the frozen backbone directly under the same protocol and samples: FORESIGHT improves it by 6.36 points on StreamingBench and 16.43 points on OVO-Bench. Section CR2 of our common response above contains these matched comparisons.

[W4] Module costs and end-to-end latency

We now report measured execution costs on OmniPro with one GH200 per stream shared by both twins. The fixed check schedule and frame rate require 1.70 s of computation per second of video, versus 0.67 s with model-chosen Next Check and FPS, a 2.5× reduction. FORESIGHT runs in real time: each second of video takes 1.0 s of wall time, and no frame is dropped. A cache snapshot costs 11 ms per check, and each check decodes about 22 tokens. Section CR3 of our common response above gives the module breakdown, check rates, and memory. Scoring uses actual answer availability to the user, following the official OmniPro protocol.

Thank you for the additional experiments and clarifications. The new results alleviate my concerns about calibration and backbone differences. I still find the contribution somewhat incremental, with much of the value coming from system integration. Several implementation details also need clearer and more complete descriptions.

I hope the authors will address these remaining concerns in the final paper, and I will reconsider my rating accordingly.

Thank you for your feedback and your willingness to reconsider your rating. We believe system-level contributions are valuable in their own right, particularly when they turn existing capabilities into functionality that is difficult to achieve otherwise. FORESIGHT aims to do precisely this by exploiting the anticipatory capabilities of streaming VLMs to actively control future perception and computation.

This requires addressing several non-trivial challenges: making anticipation actionable through decisions about when to reason and what evidence to seek; adapting these decisions as the stream evolves; and executing planning concurrently with perception without incurring prohibitive overhead. We develop a carefully designed algorithm to address these challenges and validate its key design choices through systematic ablations. Unlike QueryStream, which relies on query-stream similarity to trigger responses, and StreamAgent, which plans at fixed intervals with substantial computational overhead, FORESIGHT lets the VLM dynamically determine when and how to inspect the stream. Its shared-state asynchronous execution and lightweight plan reconfiguration make this practical in real time, with a frozen backbone and no additional training.

We have uploaded a revised version of the PDF, with all changes highlighted in blue, and expanded the implementation details to support reproducibility. We would be happy to make further clarifications or additions if you feel anything still remains unclear or incomplete.

Thank you again for your constructive feedback, and for helping us improve the paper.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common answer to all reviewers

Thank you for the careful reviews. Your comments helped us clarify the evaluation protocol, isolate the controller's contribution, and report measured execution costs. We address the shared questions below, followed by individual replies.

[CR1] Calibration-free OmniPro evaluation, matched samples, and uncertainty

We sincerely thank the reviewers for this feedback, which enabled us to improve both the manuscript and the method's results significantly. It prompted us to revisit the need for calibration and led to a simpler, calibration-free configuration with stronger performance.

Our original threshold fitting was motivated by the observation that the event-sufficiency score, p_hit, had different probability levels across tasks, suggesting that different tasks required different response thresholds. Following the reviewers' feedback, we investigated this behavior further and found that it was induced by the task-specific in-context examples included in our prompt to explain the task and response behavior. This finding is consistent with the known tendency of in-context examples to bias a model's output probabilities (Zhao et al., "Calibrate Before Use", ICML 2021). Removing only these examples, with the remaining instructions unchanged, resolved the behavior that motivated per-task fitting: FORESIGHT now works well with one shared gate setting and no response-gate calibration.

We also agree that comparing our original 80% held-out results with full-set baseline results did not establish a matched evaluation. With calibration no longer needed, we now evaluate FORESIGHT on the full set used by the baselines.

The revised calibration-free FORESIGHT obtains 27.14% macro Joint F1, 46.85% macro Time F1, and 59.24% macro content accuracy, using the official OmniPro Online scoring procedure. We also completed a calibration diagnostic for QueryStream using Qwen3-VL-8B: thresholds were selected on 182 examples and frozen before evaluation on 750 held-out examples. The table below reproduces the submitted paper's Table 1 and adds these two configurations. Scores are Joint F1 (%).

Model / configurationCalibration doneScored subsetEATGSMSCCACCENDCSIMacro mean
LiveStarNo100%9.70.80.00.014.70.01.60.06.03.6
MMDuet2No100%12.55.314.911.221.45.33.712.714.711.3
MiniCPM-o 4.5No100%50.112.69.88.023.89.03.03.81.413.5
Dispider†No100%2.30.00.00.08.50.34.50.19.72.8
StreamAgentNo100%6.81.44.01.911.30.93.78.17.15.0
QueryStream, published thresholdsNo100%8.21.43.11.017.66.67.27.18.36.7
FORESIGHT, with calibration (previous)Yes, 20%80% held-out44.717.916.419.133.816.311.529.218.523.0
FORESIGHT, without calibration (new)No100%48.7321.5320.5324.1037.9220.8814.4232.9423.2127.14
QueryStream, fitted thresholds (new)Yes, 20%80% held-out8.960.006.972.6517.917.164.269.159.157.36

[CR1 continued] Results, conclusions, and uncertainty

Why the results improve. With the task-specific in-context examples, p_hit was high almost everywhere in the stream, so the model tended to trigger everywhere. More importantly, p_hit remained high after the first event, so the second, third, and later events were hidden behind it. Per-task calibration (Appendix H) could only partly compensate: a threshold can suppress spurious triggers, but it cannot recover events masked by a confidence that does not fall between them. After removing the examples, p_hit is high mainly at the correct events and falls between them, so later events are no longer hidden. All nine tasks improve (+2.9 to +5.0 Joint F1 relative to the submitted configuration, which was scored on its 80% split), and both Time F1 (43.7 → 46.85) and content accuracy (54.6 → 59.24) increase. On the same 932 samples and without calibration, the original prompt reaches 16.67 Joint F1 and the revised prompt 27.14, showing that the improvement comes from removing the examples, not from the change of evaluation set.

The conclusions are unchanged and reinforced. FORESIGHT remains the best method on OmniPro Online, now on the same full set as the baselines, and its margin over the strongest trained baseline, MiniCPM-o 4.5, grows from 9.5 to 13.6 Joint F1 points. It now neither updates weights nor fits any threshold on benchmark data, so the training-free characterization covers the response gate as well as the backbone. Per-task calibration raises QueryStream only from 6.7 to 7.36. The architecture, plan, and controller are unchanged; we only removed the in-context examples from the prompt, which made calibration unnecessary. We will update the manuscript accordingly.

Sample counts and uncertainty. The OmniPro rows marked "100%" use the same 932 samples: all samples which do not require audio, scored with the official OmniPro Online protocol. The calibrated diagnostic rows use their held-out subsets, as indicated in the table.

EATGSMSCCACCENDCSITotal
Samples3812142733810296283148932

Across three end-to-end runs of the full system, the standard deviation of macro Joint F1 is 1.06 points, whereas FORESIGHT's margin over the strongest baseline is 13.6 points, more than ten times larger. For the calibrated QueryStream run, the 95% bootstrap confidence interval is 5.92–8.82 Joint F1. Because FORESIGHT no longer fits any threshold, no evaluation sample is used for fitting. In the submitted version, the 20%/80% split was random within each task, so calibration and evaluation videos were disjoint within each task.

[CR2] StreamingBench and OVO-Bench aggregation and matched-backbone comparisons

Fitting and evaluation sets. Section 4.1 and Appendix B.1 describe a 20%/80% split for every benchmark, but thresholds were fitted only on OmniPro; this is why Appendix H lists gate settings for OmniPro alone. On StreamingBench and OVO-Bench, FORESIGHT fits no thresholds, and its results are computed on the full evaluation sets; the revised configuration fits none on OmniPro either. We will correct Section 4.1 and Appendix B.1 accordingly.

All methods we run are scored on identical examples, with the same benchmark version (OVO-Bench v2), video-only input, the same frame budget, and one aggregation rule per benchmark.

StreamingBench. Our submitted overall excluded Proactive Output for FORESIGHT, whereas the baseline overalls include it. Under the baselines' convention, with Proactive Output included in Contextual, FORESIGHT scores 79.30 on Real-Time, 48.70 on Omni-Source, and 44.50 on Contextual. Weighting these groups by their question counts (2,500, 1,000, and 1,000) gives

Overall=79.30×2500+48.70×1000+44.50×10004500=64.77.\text{Overall} = \frac{79.30 \times 2500 + 48.70 \times 1000 + 44.50 \times 1000}{4500} = 64.77.

The gain over the published backbone score (59.30) therefore changes from +6.7 to +5.5. FORESIGHT still ranks first, 2.07 points above the published MiniCPM-o 4.5 score (62.70): the correction changes the margin, not the ranking. We still report Proactive Output separately (46.3, 95% CI 39.1–53.7), since it scores timing rather than answer choice. We will correct Tables 2 and 7, the abstract, the introduction, and the conclusion.

Model (our runs, same protocol and samples)Overall
Qwen3-VL-8B (frozen backbone)58.41
StreamAgent56.45
MiniCPM-o 4.561.45
FORESIGHT64.77

On identical samples, FORESIGHT improves its own frozen backbone by 6.36 points and leads MiniCPM-o 4.5 by 3.32 points.

OVO-Bench. The numerical check exposed errors in our description and several reported per-task scores. The stated 20%/80% split does not apply to OVO-Bench: FORESIGHT fits nothing on this benchmark, and its numbers are computed on the full evaluation set. The recalculated accuracies below match the full-set question counts. We will correct the text and affected tables.

TaskScore
ACR71.56
ATR76.72
STU65.17
FPD79.21
OJR73.37
EPM62.3
ASI69.6
OCR85.91

ACR: Action Recognition; ATR: Attribute Recognition; STU: Spatial Understanding; FPD: Future Prediction; OJR: Object Recognition; EPM: Episodic Memory; ASI: Action Sequence Identification; OCR: Optical Character Recognition. Scores are accuracy (%).

With these corrections, the Real-Time average is 75.32 and the Backward average is 55.17. Averaging the three task-group means, with Forward at 56.37, gives an overall of 62.29.

Model (our runs, same protocol and samples)Overall
Qwen3-VL-8B (frozen backbone)45.86
QueryStream (Qwen3-VL-8B)46.80
StreamAgent48.41
MiniCPM-o 4.559.10
FORESIGHT62.29

On identical samples, FORESIGHT improves its own frozen backbone by 16.43 points and leads MiniCPM-o 4.5 by 3.19 points.

[CR3] Anticipatory scheduling, adaptive FPS, and measured execution costs

The purpose of adaptive FPS and scheduling is to allocate computation while retaining useful responses; therefore unchanged Joint F1 or a small Time F1 change is consistent with the intended quality–efficiency trade-off. We profiled FORESIGHT end to end on OmniPro, with one GH200 per stream shared by both twins. At a similar check rate (about 0.30 vs. 0.25 checks per second), we compare a fixed check schedule and frame rate with FORESIGHT's model-chosen ones:

ModuleFixed check schedule and frame rateFORESIGHT (model-chosen Next Check and FPS)
Vision encoder, per second of video (per frame)273 ms (262 ms)59 ms (118 ms)
Ingest prefill, per second of video396 ms98 ms
Cache snapshot, per check11 ms11 ms
Think, per check (of which decoding)3.45 s (3.42 s)2.07 s (2.06 s)
Decoded tokens per check (ms per token)20.2 (169 ms)21.7 (95 ms)
Think, per second of video1,024 ms508 ms
Gate and bookkeeping after generation, per check1.48 s0.84 s
Computation per second of video1.70 s0.67 s

With a fixed check schedule and frame rate, the same system needs 1.70 s of computation for every second of video, so it cannot keep up with a live stream. With the model-chosen Next Check and FPS, it needs 0.67 s, 2.5× less, and runs in real time: each second of video takes 1.0 s of wall time, and no frame is dropped. Adaptive FPS cuts the vision encoder from 273 to 59 ms and the Ingest prefill from 396 to 98 ms per second of video, and the Think time halves, from 1,024 to 508 ms. Planning stays cheap because the Think twin reads the shared KV cache instead of re-encoding the stream: a snapshot costs 11 ms, and each check decodes only about 22 tokens.

The planner anticipates events. The table below groups all planning steps by the time remaining until the next annotated event and reports which next-check interval the planner chose. The planner never sees these annotations. When an event is less than 1 s away, it chooses the shortest interval (1 s) 79% of the time; when the next event is more than 30 s away or there is none, it chooses the longest interval (8 s) 58–64% of the time. The choice shifts steadily with the distance to the next event: the planner checks densely just before events and backs off when nothing is coming, instead of checking on a fixed clock.

Next annotated event in1-second interval2-second interval4-second interval8-second interval
>0–1 s79%13%7%1%
>1–3 s64%26%8%2%
>3–10 s29%36%25%10%
>10–30 s9%16%34%41%
>30 s6%9%27%58%
No future annotated event2%7%27%64%

Hardware, memory, and check rates. Each stream runs on one GH200, shared by both twins, with a peak memory of 65 GB including the full KV cache. FORESIGHT checks about once every 4 s on average; the check rate has a long tail that reaches 0.4 checks per second on event-dense streams.

Latency and answer availability. The scored time reflects when the user actually receives the answer, which is streamed as soon as the trigger fires, following the official OmniPro protocol.

[CR4] How Keep/Ignore Classes map to visual tokens

For each frame, CLIP ViT-L/14 produces a 16×16 grid of patch features, projected into CLIP's joint image–text space. The Keep and Ignore class names in the current plan are encoded with CLIP's text encoder and cached until the plan changes. For each patch, we compute its cosine similarity to every class embedding and score it as its highest similarity to a Keep class minus 0.5 times its highest similarity to an Ignore class. The 16×16 score map is bilinearly resampled to Qwen3-VL's visual-token grid, and the top 70% of each frame's visual tokens are kept; the rest are dropped before they enter the KV cache.

This step uses one additional frozen model, CLIP ViT-L/14 (0.43B parameters). Its measured cost is about 17 ms per frame on one GH200. In exchange, 30% of the visual tokens never reach the Ingest LLM, which reduces prefill computation and KV-cache memory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.