acceptodds
Under review as a conference paper at ICLR 2027

Self-Focus: Native Active Perception for Streaming Interaction Models

Abstract

Deciding when to respond depends on the evidence a model has acquired. We introduce Self-Focus, a streaming interaction model that learns active perception natively within its interaction policy; thereby jointly deciding what evidence to acquire and when it is sufficient to respond. As the video stream progresses, Self-Focus constructs its own working context through active sensing, continuously, before and after a query arrives. We initialize the policy through supervised fine-tuning on our streaming episodes, then optimize it with GRPO over the policy's self-generated context. We introduce a delayed sensing credit to estimate each sensing action's advantage in subsequent interaction return. Self-Focus achieves state-of-the-art performance on both proactive and retrospective tasks across four benchmarks. The resulting query-agnostic context also improves offline long-video reasoning and provides a stronger initialization for iterative video agents, enabling better performance at fewer search iterations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
interaction models, streaming video, MLLMs, VLMs

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ESYuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-Focus, a streaming interaction model that jointly learns active perception and response decisions. The model can selectively retain, discard, retrieve, or examine video frames at higher resolution. The authors introduce delayed sensing credit to optimize sensing decisions using future interaction outcomes. Experiments on proactive and retrospective video understanding benchmarks show improvements over existing streaming and interaction models.

Critical strengths
  1. The paper addresses an important and well-motivated problem. Existing streaming interaction models mainly focus on when to respond, while evidence acquisition is often handled separately. Learning sensing and response decisions within a single policy is a reasonable approach. The method is also clearly presented.
  2. The experimental evaluation is extensive. The authors consider both proactive and retrospective tasks across several benchmarks. Tables 3 and 4 provide useful evidence for the benefits of joint sensing, delayed sensing credit, and different sensing actions. The offline long-video experiments further demonstrate the value of the learned working context.
Critical weaknesses
  1. The methodological novelty of delayed sensing credit could be better established. Eq. (2) is closely related to existing reward-to-go estimators with a leave-one-out baseline. Table 3 shows improvements over trajectory-level GRPO, which is encouraging. However, the comparison does not fully isolate the effect of temporal credit assignment from differences in advantage normalization. Appendix D.4 establishes an unbiased policy gradient under certain assumptions, while acknowledging that the estimator is not a same-state action advantage. Further analysis would help clarify the specific contribution of the proposed estimator.
  2. The robustness of long-horizon streaming interaction needs further analysis. Appendix D.3 limits RL training segments to 256 seconds, with earlier context largely reconstructed from reference actions. The authors also use partially self-generated prefixes and evaluate on longer videos. However, the results do not separately examine how performance changes over long multi-turn interactions. An analysis across stream lengths, query counts, and delays between evidence and query arrival would help establish the method's long-horizon robustness.
Questions for authors
  1. Could the authors further clarify the contribution of delayed sensing credit beyond standard reward-to-go estimation? How sensitive are the results to the choice of advantage normalization and baseline?
  2. How does performance vary with stream duration, the number of previous interactions, and the delay between evidence and query arrival? In particular, does performance degrade for histories much longer than those used during RL training?
  3. Figure 4 provides useful runtime results, but could the authors also compare total computation and memory costs with the baselines? In particular, how much overhead comes from low-resolution previews, high-resolution sensing, and the CPU archive?
  4. Could the authors provide confidence intervals or results across multiple random seeds, particularly for benchmarks with small performance differences? It would also be helpful to clarify how StreamingBench-PO accuracy is aggregated.
5 comments
AuthorsOpenReview ↗

Rebuttal - 1/3

We sincerely thank the reviewer for their positive assessment and encouraging feedback. We are pleased that the reviewer finds our problem important and recognizes the importance of jointly learning sensing and response decisions, the extensive evaluation, and the value of our learned working context for long-video understanding. Below, we address the remaining concerns.

W1 & Q1.

W1 “Eq. (2) is closely related to existing reward-to-go estimators with a leave-one-out baseline.”

Q1 “Could the authors further clarify the contribution of delayed sensing credit beyond standard reward-to-go estimation?”

We agree and clarify that our contribution is not a new return estimator, but learning evidence acquisition from subsequent interaction outcomes within a unified streaming interaction policy. The challenge is learning what evidence to acquire before future queries are known. Our streaming grammar (Sec. 3.2), constructed episodes (Sec. 3.3), and SFT initialization (Sec. 3.4) enable causal RL rollouts that share future frames and queries but construct different working contexts. This allows us to apply such reward-to-go estimators with a leave-one-out baseline to credit earlier sensing decisions from later interaction outcomes, without a separate sensing reward or learned critic. We will clarify this distinction in the revision.

W1 “However, the comparison does not fully isolate the effect of temporal credit assignment from differences in advantage normalization”

Q1 “How sensitive are the results to the choice of advantage normalization and baseline?”

The original comparison in Tab. 3 (rows 5 and 7) already controls for advantage normalization: both use the same leave-one-out baseline, without standard-deviation (SD) normalization, and share the same initialization, data, and training setup. Only the return horizon differs. We will relabel row 5 as “GRPO with trajectory-level credit” to make this explicit.

Normalization. Following your suggestion, we additionally trained both horizons with standard-deviation (SD) normalization (low budget):

Credit AssignmentSD norm.SPOT PNROVO RTOVO BWD
Trajectory-levelYes18.582.049.6
Trajectory-level (Tab. 3, row 5)No18.882.250.5
Delayed sensing creditYes19.782.353.7
Delayed sensing credit(Tab. 3, row 7)No20.482.655.6

Delayed sensing credit consistently outperforms trajectory-level credit with and without SD normalization, confirming that the improvement is not due to a normalization mismatch. SD normalization reduces performance for both methods, supporting our choice to train without it.

Baseline. For a group of (G) rollouts, leave-one-out and group-mean advantages differ only by a constant factor G/(G-1). They therefore preserve the sign and relative ordering of action credits. The main difference in our comparison is the return horizon.

AuthorsOpenReview ↗

Rebuttal - 2/3

W2 and Q2.

W2 “However, the results do not separately examine how performance changes over long multi-turn interactions.”

Q2 “In particular, does performance degrade for histories much longer than those used during RL training?”

We agree that an explicit long-horizon analysis would strengthen the paper. Although RL segments are limited to 256s, each starts from a reconstructed streaming history, not an empty context (Fig. 3b): earlier observations in the working context, previous interactions in dialogue history, and absolute timestamps. At inference, the working context remains bounded regardless of stream length.

“An analysis across stream lengths, query counts, and delays between evidence and query arrival...”

Q2 “...stream duration, the number of previous interactions, and the delay between evidence and query arrival?”

Thank you for suggesting these three axes; we report each below. We will include these tables and associated discussions in the appendix.

Stream duration. Our offline Long VideoQA evaluation (Tab. 5) already tests query-agnostic sensing on videos extending well beyond the 256s RL segments, with improvements over the base model at every matched context budget. On average, these videos run up to 9.6× the RL segment length. To examine online interactions, we additionally split queries by their arrival time and run Self-Focus at low budget.

Ask timeSPOT PNROVO RTOVO BWD
<256s20.582.755.8
>=256s20.182.455.2
of which >=600s19.782.054.6

Performance remains similar even when queries arrive well beyond the RL segment length, with only minor decreases across the tasks.

Number of previous Interactions. We run this analysis on SPOT PNR, which includes queries referring to previous interactions. We retain the five most recent interactions in dialogue history and report performance by query turn positions.

Turn in Session1st2nd-3rd4th-6th7th+
PNR T-F120.520.520.420.2

Performance remains stable across turn positions, indicating little degradation as interactions accumulate in the evaluated sessions.

Evidence-to-query delay. Since the online benchmarks do not provide explicit evidence intervals for this analysis, we evaluate on 200 held-out retrospective episodes from HT-Step, COIN, and Charades-STA validation splits, with no training-video overlap. We compare trajectory-level credit (Tab. 3, row 5) with delayed sensing credit (row 7), using 50 episodes per delay range at the low budget. Evidence in context measures the percentage of episodes with at least one frame from the annotated evidence interval available before query arrival (retained) or at response time (retained or retrieved).

Evidence-to-query delayTrajectory-level acc. (%)Delayed credit acc. (%)Evidence in context before query (traj. / delayed)Evidence in context at response (traj. / delayed)
Recent (0-3s)848494/96%96/98%
Near (8-30s)687844/58%56/72%
Far (60-256s)566210/16%18/28%
Longer (256-600s)56606/10%12/20%

When evidence is recent, both variants retain it at similar rates. As the delay increases, delayed sensing credit retains more relevant evidence before query arrival and achieves equal or higher accuracy, even beyond the 256s RL segment length. Keeping more of the evidence helps in two ways: the model can answer from its working context directly, and when it needs to retrieve, the kept entries give retrieve() timestamped anchors near the evidence. This is precisely the behavior delayed sensing credit aims to learn: preserving evidence before its relevance to future queries is known.

AuthorsOpenReview ↗

Rebuttal - 3/3

Q3.

“how much overhead comes from low-resolution previews, high-resolution sensing, and the CPU archive”

Thank you for this suggestion. We provide a component-wise latency breakdown on a single RTX A5000 GPU in Tab. A below.

Table A. Component latency (ms). Values marked * are measured at the full working-context budget (2K/4K/8K for low/medium/high).

Frame decoding (1fps)13.9
Visual encoding of one low-res preview16.2
Visual encoding of one high-res focus27.5
Prefilling of one low-res preview*45.0/54.3/74.1
Prefilling of one high-res focus*105.2/122.2/159.0
Decoding a sensing action*61.5/66.0/74.6
Eviction + re-indexing*9.0/17.9/36.1
one retrieve() from the CPU archive25.0
avg. TTFT for OVO-Bench315/335/352

High-resolution focus() incurs additional encoding and prefill costs, but is invoked selectively, while low-resolution previews are processed at every frame. Retrieval from the CPU archive adds only 25 ms per call. Overall, Self-Focus achieves average OVO-Bench time-to-first-token (TTFT) of 315-352 ms, comparable to the lightweight DSCache (326 ms). We exclude response generation latency, which varies with answer length. We will include this breakdown in the revised appendix. With peak GPU memory of 18.3/19.6/22.2 GB (low/medium/high) on a single 24 GB RTX A5000, Self-Focus runs at the evaluated 1-2 fps input rates in real-time for all benchmarks without dropping frames.

Q4.

“Could the authors provide confidence intervals or results across multiple random seeds…”

Following prior baselines, we use greedy decoding for all main results (Tabs. 1, 2, 5, and 6), including reproduced baselines, and all ablations (Tabs. 3 and 4), ensuring deterministic evaluation without sampling variability.

“particularly for benchmarks with small performance differences?”

Following your suggestion, we additionally evaluate Self-Focus (low) over five decoding seeds at temperature 1.0 on the two real-time benchmarks, where performance differences are relatively small.

TaskGreedy (Tab. 2)5 seeds mean ±SD5 seeds 95% CI
OVO-Bench Real-time82.682.3±0.6[81.6, 83.0]
StreamingBench Real-time82.282.0±0.4[81.5, 82.5]

The stochastic results remain close to greedy decoding, with modest variation across decoding seeds. We will include these results and clarify the evaluation protocol in the revision.

“It would also be helpful to clarify how StreamingBench-PO accuracy is aggregated.”

We use the official StreamingBench scorer on the Proactive Output (PO) task. Each query is issued once at its annotated timestamp, and the model decides when to respond. The score averages two accuracies: timing (the response falls within 2 seconds of the ground-truth time) and answer (the response is on time and its output is correct). We report this score over all 250 questions, using the same protocol for our reproduced JoyAI-VL-Interaction baseline. We would also like to point out that App. C incorrectly states 50 questions, referring to the subset evaluated in the original StreamingBench paper. We thank you for prompting us to identify this discrepancy and will correct it to 250 in the revision.

Reviewer_JHzZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Self-Focus to jointly learn evidence acquisition and response timing in streaming video, supporting frame retention, high-resolution observation, and historical retrieval. Experiments on multiple streaming benchmarks demonstrate its effectiveness. The paper also includes targeted evaluations of specific components.

Critical strengths
  • The experiments are well designed and validate the effectiveness of each sensing action.
Critical weaknesses
  • The differences from existing streaming agents and VLMs should be clarified, particularly regarding the interaction protocol, action space, and training objective. The current discussion does not sufficiently distinguish the proposed interaction design from prior work.
  • Eq. (2) uses a return-to-go estimator with a leave-one-out baseline, so its methodological novelty remains unclear. Please clarify what is new beyond this standard formulation and whether the GRPO comparison uses matched advantage normalization and scaling.
  • The latency impact of retrieve() in streaming interaction is insufficiently characterized.
Questions for authors

See weaknesses above

5 comments
AuthorsOpenReview ↗

Rebuttal - 1/2

We sincerely thank the reviewer for their positive assessment and constructive feedback. Below, we address the three concerns with clarifications and additional experimental evidence.

W1.

“The differences from existing streaming agents and VLMs should be clarified”

Thank you for highlighting the need to clarify these distinctions. We summarize the differences between existing approaches and Self-Focus in Tab. A below.

Table A. Comparison of evidence acquisition and interaction across video models.

CategoryRepresentative modelsOnlineEvidence Acquisition (Sensing)Response Timing (Interaction)Jointly Learned?
Video AgentsVideoZoomer, OmniAgentXLearned; query-conditioned iterative retrievalN/A (offline)N/A
Streaming VLMsTimeChat-online, HERMES✓Not learned; query-agnostic token pruningNo; External triggers computed post-hocX
Streaming VLMsQueryStream, OASIS✓Not learned; query-conditioned pruning/retrievalNo; External triggers computed post-hocX
Interaction ModelsMMDuet2, JoyAI-VL-Interaction✓Not learned; External, fixed protocolLearnedX
Active Perception for Interaction ModelsSelf-Focus✓LearnedLearned✓

As shown in Tab. A, video agents learn evidence acquisition over offline videos where queries are known beforehand. Interaction models learn when to respond, but not how to acquire the evidence supporting those responses. Some streaming VLMs do explore evidence acquisition, but as a form of separately designed module to manage memory. We argue that this separation is a fundamental limitation: deciding when sufficient evidence is available to respond is inherently dependent on what evidence it has acquired. This motivates us to jointly learn these two i.e. evidence acquisition and streaming interaction, and the first to do so, to our knowledge.

“regarding the interaction protocol, action space, and training objective”

As discussed in L61-L63, our central motivation is to explore the reciprocal dependency between sensing and interaction: when to respond (interaction) depends on what evidence has been acquired (sensing), while what to acquire depends on what the model may need to answer. To our knowledge, Self-Focus is the first model to do so. Accordingly, our protocol integrates sensing (drop(), keep(), focus(), retrieve()) and response (<silent>, <response>) actions (Sec. 3.2). However, we find that there can be multiple sensing trajectories leading to a successful interaction. As such, no single ground truth exists for such sensing supervision. Here, following our reciprocal dependency argument, we learn sensing from its contribution to future interaction outcomes (Sec. 3.5). Tab. 3 supports this choice: jointly optimizing sensing and response through delayed sensing credit (row 7) improves proactive, Real-Time, and Backward performance over learning interaction alone (row 4) or using a separate sensing reward (row 6). This supports our argument that sensing should be optimized for the interactions it serves, rather than rewarded independently. We will include Tab. A in the appendix to clarify these distinctions.

AuthorsOpenReview ↗

Rebuttal - 2/2

W2.

“Eq. (2) uses a return-to-go estimator with a leave-one-out baseline, so its methodological novelty remains unclear. Please clarify what is new beyond this standard formulation”

We agree with your observation that Eq. (2) uses a conventional return-to-go estimator with a leave-one-out baseline. We do not claim novelty in the estimator itself; our contribution is using future interaction outcomes to train sensing decisions made before the query is known.

Our design choices in Secs. 3.2-3.5 make this possible: the streaming grammar defines sensing and response decisions within one policy; constructed episodes cover different temporal relationships between queries and evidence; SFT initializes both decisions; and causal RL rollouts construct different working contexts while observing the same future frames and queries. Together, these choices enable conventional return-to-go estimator with a leave-one-out baseline to train earlier sensing decisions from later interaction outcomes, without a separate sensing reward or learned critic.

“whether the GRPO comparison uses matched advantage normalization and scaling.”

Yes, both comparisons use matched normalization and scaling. Trajectory-level GRPO (Tab. 3, row 5) and delayed sensing credit (Tab. 3, row 7) use the same leave-one-out baseline, without group-standard-deviation normalization. They share the SFT initialization, training data, and optimization settings. The only difference is the return horizon: trajectory-level GRPO assigns a fixed total-return advantage to every action, whereas Eq. (2) assigns a time-varying advantage from the return not yet realized. Labeling row 5 simply as “GRPO” may have caused confusion, since standard GRPO applies group-standard-deviation normalization. We will rename it “GRPO with trajectory-level credit” and explicitly clarify the matched normalization and scaling.

Following your suggestion, we further isolate the effect of normalization by training both return horizons with and without group-standard-deviation normalization at the low context budget (Tab. B).

Table B.

Credit AssignmentSD norm.SPOT PNROVO RTOVO BWD
Trajectory-levelYes18.582.049.6
Trajectory-level (Tab. 3, row 5)No18.882.250.5
Delayed sensing creditYes19.782.353.7
Delayed sensing credit (Tab. 3, row 7)No20.482.655.6

Delayed sensing credit consistently outperforms trajectory-level credit with and without SD normalization. In particular, Backward accuracy improves by 4.1 points with normalization and 5.1 points without. Thus, the improvement is not explained by differences in advantage normalization. SD normalization reduces performance under both return horizons, supporting our choice to train without it. We will include this controlled comparison in the revision.

W3.

“The latency impact of retrieve() in streaming interaction is insufficiently characterized.”

Thank you for this suggestion. We provide additional runtime measurements on a single 24GB RTX A5000 in Tab. C below. Following ReKV, Self-Focus maintains a CPU-offloaded archive of historical frame KVs, enabled in all retrieval-based results in our paper. A single retrieve() call takes approximately 25 ms with the archive, compared with 300 ms without it. Since retrieval is invoked for approximately 30% of OVO-Bench queries, the archive reduces average time-to-first-token (TTFT) by 80-93ms across context budgets, with negligible changes in accuracy (≤0.2 points). The archive uses 2.1 GB of CPU memory on average (14.9 GB peak for a 30min OVO video), without increasing GPU memory. Importantly, all three budgets run on a single 24GB GPU, with average TTFT below 500 ms even without the archive. We will include Tab. C and these measurements in the appendix.

Table C. Computed on one 24GB A5000. CPU memory refers to archive-specific storage

Self-Focus budgetOVO RTOVO BWDTTFT (ms) avg. on OVOPeak GPU Memory (GB)Avg./Peak CPU Memory (GB)
low w/o CPU archive82.755.840818.30
low w/ CPU archive82.655.631518.32.1/14.9
medium w/o CPU archive82.264.341719.60
medium w/ CPU archive82.264.433519.62.1/14.9
high w/o CPU archive82.366.143222.20
high w/ CPU archive82.366.035222.22.1/14.9

Dear Reviewer JHzZ,

As the discussion period is nearing its end, we wanted to check whether our rebuttal and additional experiments have addressed your concerns. We would be happy to provide any further clarification.

Best regards,

Authors

Reviewer_uoy1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Focus, a streaming interaction model that jointly learns active perception and response decisions. A unified policy controls sensing actions (drop(), keep(), focus(), and retrieve()) and decides when to respond or remain silent. The authors construct 100K streaming episodes and adopt a two-stage SFT and RL training procedure with delayed sensing credit. Experiments report improvements across proactive and retrospective video understanding benchmarks, further supporting the importance of active perception in streaming interactions.

Critical strengths
  1. The proposed RL training procedure improves performance over the authors' SFT-only and trajectory-level GRPO variants. The evaluation covers several relevant streaming benchmarks, with ablations examining the contributions of different sensing actions.

  2. The paper constructs 100K streaming interaction episodes covering proactive, retrospective, and mixed interactions, providing training data for jointly learning sensing and response behaviors.

Critical weaknesses

W1. Limited methodological novelty and insufficient differentiation from prior work. The paper largely integrates established sensing operations into a unified interaction policy. Closely related studies, such as QueryStream (ICLR 2026) and Thinking-QwenVL (ICLR 2026), have already explored adaptive visual information selection, evidence-aligned response timing, and memory-constrained online video understanding. Although Self-Focus extends these settings to learned query-agnostic sensing, its fundamental methodological advances over prior work are not clearly established. Furthermore, the proposed delayed sensing credit closely resembles conventional reward-to-go optimization with a leave-one-out baseline.

Several directly relevant streaming interaction models over the past year are also missing from the experimental comparisons, they raise questions about the paper's broad state-of-the-art claims. Moreover, OVO-Bench Forward Active Responding subtask and a separate evaluation of mixed interactions are absent, despite their direct relevance to the proposed method. These omissions further limit the evidence for the method's advantages over existing approaches.

W2. The experiments do not convincingly demonstrate the paper's central motivation. A central motivation is to learn which visual evidence should be preserved before future queries are known. However, Self-Focus maintains a separate CPU-offloaded archive that allows previously discarded historical frames to be retrieved after query arrival. In Table 4, enabling retrieve() increases OVO-Bench Backward accuracy from 44.2% to 55.6% when keep() and focus() are available. This suggests that a substantial portion of the improvement may come from query-conditioned historical retrieval rather than learned query-agnostic evidence retention.

W3. The performance improvements are not sufficiently compelling relative to the added complexity. In Table 2, the parameter-matched SimpleStream-8B, using only four recent frames without additional training, achieves 81.4%/54.0% on OVO-Bench Real-Time/Backward, compared with 82.6%/55.6% for Self-Focus. These modest improvements come with substantially greater complexity, including 100K constructed training episodes, SFT, RL, adaptive sensing, and historical retrieval. Although higher context budgets yield more substantial backward-memory gains, the corresponding increases in memory usage and inference cost are not adequately examined.

Questions for authors
  1. After applying drop(), what are the average effective retained-frame rate and the proportions of drop(), keep(), and focus() actions? How sensitive is accuracy and response timing to changes in the retained-frame rate and sensing decisions?
  2. What specific capabilities or methodological advances distinguish Self-Focus from closely related approaches such as QueryStream and Thinking-QwenVL? A clearer discussion and targeted comparison would help establish the contribution beyond integrating existing sensing and interaction mechanisms.
  3. Given the competitive performance of SimpleStream-8B using only four frames, could the authors provide comparisons under matched visual-token budgets, inference costs, and memory constraints?
7 comments
AuthorsOpenReview ↗

Rebuttal - 1/5

We sincerely thank the reviewer for their detailed and constructive feedback, and for recognizing the value of our training procedure and the breadth of our evaluation. We address each concern below with clarifications, further explanations, and additional experimental evidence.

W1 & Q2.

W1. “...its fundamental methodological advances over prior work are not clearly established”

Q2. “What specific capabilities or methodological advances distinguish Self-Focus from closely related approaches”

Thank you for highlighting the need to clarify our methodological contributions. Our central contribution is learning active perception within the streaming interaction policy, rather than adaptive visual selection or response timing in isolation. Existing interaction models learn when to respond from separately constructed visual contexts, while active perception methods typically select evidence given a known query. Self-Focus jointly learns what evidence to acquire and when it is sufficient to respond, including sensing decisions made before a future query is known.

This introduces a distinct learning problem: sensing decisions determine the evidence available for future interactions, yet their usefulness may only become apparent after the query arrives. Since there is no unique ground-truth sensing trajectory, we train a unified policy through downstream interaction outcomes. The same policy supports proactive, retrospective, and mixed interactions, without separate task-specific prompts or inference procedures.

“...the proposed delayed sensing credit closely resembles conventional reward-to-go optimization with a leave-one-out baseline”

We agree with your observation that Eq. (2) uses a conventional reward-to-go formulation with a leave-one-out baseline. We do not claim novelty in the estimator itself; our contribution is using future interaction outcomes to train sensing decisions made before the query is known.

To make this possible, we introduce a streaming interaction grammar (Sec. 3.2), construct 100K streaming episodes (Sec. 3.3), initialize sensing and response actions through SFT (Sec. 3.4), and jointly optimize them through causal RL rollouts (Sec. 3.5). Each rollout group shares the same prefix, future frames, and query arrivals, but constructs different working contexts through its own sensing actions. Together, these choices make it possible to use conventional reward-to-go optimization with a leave-one-out baseline to train earlier sensing decisions from later interaction outcomes, without a separate sensing reward or learned critic.

Tab. 3 empirically supports this formulation: learning sensing within the interaction policy improves proactive and real-time performance over response-only training, while delayed sensing credit further improves backward accuracy by 5.1 over trajectory-level credit. Appendix D.4 provides additional policy-gradient justification. We will make this distinction more explicit in the revision.

AuthorsOpenReview ↗

Rebuttal - 2/5

W1 “Closely related studies, such as QueryStream (ICLR 2026) and Thinking-QwenVL (ICLR 2026)...”

Q2 “...distinguish Self-Focus from closely related approaches such as QueryStream and Thinking-QwenVL?”

Thank you for highlighting these relevant works. Both address important aspects of streaming video understanding, but differ from Self-Focus in how evidence acquisition and response decisions are controlled and learned:

  • QueryStream uses training-free, query-conditioned token pruning (QDP) and externally specified response-triggering rules (RTAR). Its visual selection depends on an available query, rather than learning which observations to retain before future queries arrive.

  • Thinking-QwenVL learns query-guided evidence integration and response readiness through progressive reasoning. Its interaction is organized around a single known query, rather than learning query-agnostic sensing actions jointly with response decisions across multiple queries/turns of interaction.

In contrast, Self-Focus learns to acquire and preserve evidence before a query arrives, then adapts its sensing and response decisions once the query is known, in a single policy.

The performance gap is substantial (Tab. A below). On OVO-Bench, even at the low budget, Self-Focus outperforms QueryStream by 19.5 on Real-Time and 10.7 on Backward accuracy, and Thinking-QwenVL by 17.9 and 11.3, respectively.

Proactive Evaluation (Response Timing). Neither work provides a publicly available, runnable implementation of its response-timing mechanism. Thinking-QwenVL has not released its inference code or model weights. QueryStream's RTAR implementation (for proactive trigger) is also unavailable. Their RTAR triggering thresholds were grid-searched on OVO-Bench Forward validation (QueryStream; App. A.6, Tab. 8). More importantly, its RTAR evaluation selects trigger points post-hoc using the full video (App. A.4). Thus, its reported evaluation does not establish online response timing over a streaming video. This reflects a common practice when response timing was not learned as part of the model’s policy - QueryStream's backbone, TimeChat-Online, similarly lacks released proactive triggering code (GitHub issue #28). We therefore evaluate QueryStream's released QDP component using SPOT-Bench's Qwen2.5-VL harness. It achieves 6.4 Timeliness-F1 on PNR, compared with 20.4 for Self-Focus (low).

W1 “Several directly relevant streaming interaction models over the past year are also missing from the experimental comparisons, they raise questions about the paper's broad state-of-the-art claims.”

Thank you for this suggestion. We provide additional comparisons with published models like QueryStream, Thinking-QwenVL, Streamo, OASIS, ThinkStream, and VST in Tab. A below alongside the baselines already evaluated in our submission. We will include these additional comparisons in the appendix.

Table A. Additional streaming model comparisons. *Reproduced by us with SPOT-Bench’s Qwen2.5-VL harness

ModelVenueSPOT PNROVO RTOVO BWDOVO FAR
QueryStreamICLR 20266.4*63.144.940.2
Thinking-QwenVLICLR 2026-64.744.336.1
StreamoCVPR 2026-67.449.257.0
OASISCVPR 2026-78.157.2-
DSCacheICML 2026-71.744.849.7
ThinkStreamECCV 2026-67.052.3-
VSTECCV 2026-67.256.754.0
SimpleStreamarXiv 2026-81.454.0-
StreamTTTarXiv 2026-78.159.9-
Self-Focus-8B (low)-20.482.655.661.7
Self-Focus-8B (medium)-20.982.264.463.7
Self-Focus-8B (high)-20.382.366.065.0
AuthorsOpenReview ↗

Rebuttal - 3/5

Selection of our Baselines:

  • For retrospective evaluation, our original comparisons (Tab. 2) were selected to examine the perception-memory trade-off highlighted by SimpleStream: improving recall of past events (Backward) often comes at the expense of perceiving recent events (Real-Time). We therefore included SimpleStream, DSCache, and StreamTTT, which represent strong recent baselines addressing this tradeoff. Self-Focus advances this perception-memory frontier. The additional comparisons in Tab. A further support this finding.

  • For proactive evaluation we compare against interaction models with native response-timing policies, which like Self-Focus, learn when to remain silent and when to respond without external triggering based on post-hoc selection. Our comparisons (Tab. 1) therefore include MMDuet2, the baseline for ProactiveVideoQA, and JoyAI-VL-Interaction, our closest interaction-model baseline, post-trained from the same Qwen3-VL-8B backbone.

“Moreover, OVO-Bench Forward Active Responding subtask and a separate evaluation of mixed interactions are absent, despite their direct relevance to the proposed method.”

Thank you for raising this point. We would like to clarify that OVO-Bench Forward Active Responding (FAR) does not evaluate autonomous response timing under its released protocol on github. Instead, it evaluates accuracy at predefined timestamps and evaluates retrospectively via answer correctness (most yes/no) without a timing-aware metric. Following SimpleStream, our original OVO-Bench evaluation (Tab. 2) therefore focuses on Real-Time and Backward tasks to examine the perception-memory trade-off, while proactive response timing is evaluated directly on SPOT-Bench, ProactiveVideoQA, and StreamingBench-PO (Tab. 1).

Nevertheless, following your suggestion, we additionally evaluate Self-Focus on FAR using its released protocol (Tab. A). Self-Focus achieves 61.7%/63.7%/65.0% at low/medium/high budgets, outperforming existing models who report performance on FAR, all numbers are taken from their respective papers. QueryStream and Thinking-QwenVL numbers cannot be compared with ours under a matched setup. The lack of a matched evaluation protocol was also one reason we did not include FAR in our original submission, where we instead prioritized benchmarks that directly evaluate online response timing. However, FAR results demonstrate Self-Focus's strong ability to understand and recognize evolving context, even when autonomous response timing is not evaluated. We will include Tab A in the appendix and clarify our choice of OVO-Bench tasks in Appendix C.1.

W1 “separate evaluation of mixed interactions are absent”

Thank you for this important point. We agree that our gains on queries whose evidence lies before (OVO Backward), during (OVO Real-time) and after (SPOT-Bench) the query do not by themselves show the gain on mixed interactions, where the response is due later but depends on what happened earlier. To our knowledge no benchmark evaluates this setting, and we could not find one at submission time.

We therefore constructed 150 mixed queries from held-out HT-Step and COIN validation videos, following our episode construction (Sec. 3.3), without overlap with training data. Each query requires recalling an earlier action and responding when a later action finishes. For example, while a cook fries a second batch of fish, the model is asked to report what happened before frying began, once the current batch is finished. We evaluate answer correctness and response timing separately, as well as their conjunction.

Table B.

ModelCorrectOn timeCorrect & On time
Qwen3-VL-8B28.00.00.0
JoyAI-VL-Interaction38.052.022.0
Self-Focus (low)47.372.735.3

Qwen3-VL-8B answers as soon as the query arrives, so its accuracy stays near chance (25%) and is never on time. JoyAI-VL-Interaction-8B keeps only recent 16 frames but also long-term text summaries of the earlier stream. The text summaries are generic, so it is unable to properly recall previous events required to answer the query. Self-Focus is correct most often and answers both correctly and on time on 35.3 % of the queries, against 22.0 % for JoyAI. We will include these results, the evaluation protocol, and additional examples in the appendix.

Reviewer_fkNvOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose Self-Focus, a method for joint evidence acquisition and response timing for streaming video interaction tasks. For incoming streaming frames, their model decides to add to context or focus on a higher resolution view of the frame, while for each query (human interaction) the model retrieves an interval from the observed past. They use temporal grounding datasets to construct ground-truth targets in this format and first perform supervised fine-tuning. Then an RL tuning stage uses a GRPO setup with interaction rewards aimed at sensing decisions made before a future query is revealed and query-conditioned responses. Results across several proactive interaction benchmarks, streaming video benchmarks, and offline long-video QA benchmarks establish performance of their method.

Critical strengths
  1. Clear demonstration that learning what evidence to acquire improves learning when to respond for interactive tasks.
  2. Extensive evaluation across multiple related video tasks.
  3. GRPO reward objectives are interesting and useful for broader interactive and proactive video model training.
Critical weaknesses
  1. Memory eviction is not learned, simply drops oldest as heuristic. Is there a reason for not attempting to learn this? The method cannot explicitly protect important old evidence or replace a newer, less useful frame. More reasoning on this design decision would be helpful.
  2. Pre-query sensing depends on predictable future questions. This raises the question on whether the pre-query sensing is overfit to the domains of these benchmarks, especially given the GRPO training on dataset containing queries of similar domains (e.g. QAEgo4D for training vs multiple Ego4D derived benchmarks for evaluation).
  3. “Active Perception” claim could be somewhat of an oversell. The retrieval cannot iteratively correct itself. A query permits one contiguous-interval retrieval. If the model retrieves the wrong interval or needs evidence scattered across the video, it cannot search again based on what it discovers.
Questions for authors

See weaknesses.

Also could you provide detailed discussion on improvements from query-agnostic sensing (before a query arrives) versus query-conditioned sensing and retrieval (after a query arrives)?

5 comments
AuthorsOpenReview ↗

Rebuttal - 1/3

We thank the reviewer for their positive assessment and thoughtful, constructive feedback. We particularly appreciate their recognition of our central finding, extensive evaluation, and the value of our GRPO reward objectives for broader interactive and proactive video model training. Their insightful observations motivated additional analyses and helped us clarify important aspects of our method. We address each point below.

W1.

"Memory eviction is not learned…cannot explicitly protect important old evidence or replace a newer, less useful frame…”

Thank you for raising this point! We agree that FIFO eviction cannot explicitly protect important older evidence. However, Self-Focus already learns to avoid retaining less useful incoming frames through drop().

“Is there a reason for not attempting to learn this?”

We considered learning eviction as an additional policy action initially, but chose FIFO for two reasons.

  1. Scope of learning. Our primary goal is to learn which evidence to acquire for future interactions, particularly before queries are known. Unlike drop(), keep(), and focus(), which act on an incoming frame, learned eviction would require selecting among all retained frames, introducing an action space that grows with the working context. It would also require assigning credit for replacing previously acquired evidence. Jointly learning these decisions introduces a separate problem beyond our scope, which we leave for future work where it can be studied comprehensively.

  2. Empirical impact. We measured how frequently FIFO eviction occurs and its effect on answer correctness across the three streaming tasks.

Table A. Eviction (%) / incorrect→correct (%) / correct→incorrect (%) i.e. changes in answer correctness between runs w/ and w/o FIFO eviction

Self-FocusSPOT PNR (future)OVO RT (present)OVO Backward (past)
low12.8/0.2/0.013.3/0.4/0.312.2/0.4/0.2
medium8.8/0.1/0.08.5/0.3/0.26.0/0.3/0.1
high4.5/0.1/0.02.8/0.1/0.10.8/0.0/0.0

Despite nonzero eviction rates, changes in answer correctness remain small (≤0.4) across all tasks and budgets. This suggests that FIFO rarely evicts evidence required for future queries in these online benchmarks.

Alternative eviction rule. FIFO is a replaceable component of our implementation. Here, we replace it with another query-agnostic importance-based eviction rule following HERMES, without retraining Self-Focus. We evaluate at the low budget, where eviction is most frequent.

Table B. Performance with FIFO versus HERMES (at a frame-level) eviction.

Self-Focus (low)SPOT PNR (future)OVO RT (present)OVO Backward (past)
FIFO20.482.655.6
HERMES20.482.655.5

Performance remains essentially unchanged, suggesting that FIFO is not a major limitation on these online benchmarks. Importantly, alternative eviction rules can be incorporated without retraining the learned sensing policy. We will clarify this design choice and include this analysis in the revision.

AuthorsOpenReview ↗

Rebuttal - 2/3

W2.

“This raises the question on whether the pre-query sensing is overfit to the domains of these benchmarks”

Thank you for raising this concern about whether pre-query sensing overfits to predictable future questions or similar domains.

Cross-domain generalization. Our training draws from multiple video sources beyond Ego4D (App. D.1). To examine whether pre-query sensing overfits to similar domains, we evaluate its query-agnostic working context on offline Long VideoQA (Sec. 4.4), where the context is constructed before the question is known. In particular, VideoMME and LongVideoBench contain independently collected web videos spanning education, artistic performance, knowledge and other domains beyond our training sources. Despite these differences, Self-Focus outperforms the base Qwen3-VL-8B and JoyAI-VL-Interaction on both benchmarks and Streamo on LongVideoBench (Tab. 5). These results show that learned pre-query sensing remains effective beyond the training video sources.

Training Sources. We use largely the same training data sources as existing streaming models (e.g., Streamo, MMDuet2, TimeChat-Online) for comparable training domains, but construct our streaming episodes from the original temporal grounding annotations (Sec. 3.3). We ensure no video overlap between training and evaluation. Self-Focus uses only 100K streaming episodes, compared with Streamo's 465K instruction-tuning examples and JoyAI-VL-Interaction's roughly 4M reported training samples. Taken together, these comparisons suggest that the gains are unlikely to be explained solely by greater training-data scale or exposure to the evaluation domains, and are consistent with the benefits of learning better evidence acquisition.

W3.

““Active Perception” claim could be somewhat of an oversell.”

We do not define active perception solely through retrieve(). While recent offline video methods often formulate active perception as iterative retrieval, its broader principle is to actively decide what evidence to acquire (Bajcsy, 1988; Aloimonos et al., 1988). In our streaming setting, this takes the form of drop(), keep(), and focus() decisions as frames arrive. Our sensing is iterative over time and feedback-dependent: retaining a frame may make subsequent frames redundant, while dropping one may lead the policy to retain similar evidence later. Unlike corrective search over an offline video, these decisions operate causally as new observations arrive. This is our instantiation of active perception for streaming video. We will clarify this distinction in the introduction.

“The retrieval cannot iteratively correct itself. A query permits one contiguous-interval retrieval.”

We agree. We limit retrieve() to one contiguous interval per query to bound retrieval overhead during streaming interaction. This is a design choice rather than a requirement of our active perception formulation.

“If the model retrieves the wrong interval or needs evidence scattered across the video, it cannot search again based on what it discovers.”

We agree that our current retrieve() cannot correct an unsuccessful retrieval or gather evidence from multiple disjoint intervals. However, our learned query-agnostic working context can also support iterative retrieval by downstream video agents. In Tab. 6, we initialize frozen agents with Self-Focus's retained frames, improving accuracy while reducing subsequent search iterations. This demonstrates how our sensing policy complements iterative retrieval, even though our online interaction policy currently permits only one retrieval.

AuthorsOpenReview ↗

Rebuttal - 3/3

Questions:

“detailed discussion on improvements from query-agnostic sensing (before a query arrives) versus query-conditioned sensing and retrieval (after a query arrives)?”

Thank you so much for pointing this out! We acknowledge an omission in Tab. 4: while it ablates individual sensing actions, it does not isolate pre-query sensing from post-query sensing and retrieval, potentially giving the impression that retrieve() alone drives backward performance. We address this with a new ablation below that keeps the learned post-query policy fixed while varying the pre-query working context.

Table C. Effect of pre-query sensing with a fixed learned post-query policy. Retrieval (%) denotes valid retrieve(s,e) calls. “Kept all at 1fps” retains frames without the context-budget constraint.

Pre-querySPOT PNR (future)OVO RT (present)OVO Backward (past)Pre-query frames (mean/max)Valid retrieve calls
Nothing Kept19.274.344.40/00.0%
Kept all at 1fps19.076.551.6232.0/31663.2%
Self-Focus (low)20.482.655.614.4/3233.9%
Self-Focus (med)20.982.264.424.5/6430.0%
Self-Focus (high)20.382.366.032.3/12829.2%

Discussion:

  • Pre-query sensing grounds subsequent retrieval. Self-Focus does not observe or search the archive directly; it predicts retrieve(s,e) from its working context, using retained timestamped frames to identify missing evidence. Without pre-query frames, the policy instead emits null retrieve() calls (0% valid retrieval), and backward accuracy drops to 44.4%.

  • Learned sensing matters beyond retaining more frames. Keeping all frames at 1fps improves backward accuracy (44.4 to 51.6), but remains below learned pre-query sensing even at the low budget (55.6), which retains only 14.4 frames on average.

  • Pre-query sensing and post-query retrieval are complementary. Increasing the working-context budget improves backward accuracy, while valid retrieval calls become slightly less frequent. This is consistent with more relevant evidence already being available when queries arrive.

Thus, the gains attributed to retrieve() in Tab. 4 depend on the context acquired through earlier sensing decisions. Sec. 4.4 (Tabs. 5-6) further shows that our learned drop()/keep()/focus() policy constructs a query-agnostic context that outperforms uniform sampling and offline keyframe selection at matched budgets, while improving downstream video agents. We will include this ablation in the main paper to clarify how pre-query evidence acquisition supports subsequent sensing and interaction.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author update following the discussion period

We thank the reviewers and Area Chair for their time, thoughtful feedback, and positive assessments of our paper's strengths. As the discussion period draws to a close, we would like to summarize how we have addressed the concerns raised through clarifications, additional experiments and revisions to the manuscript.

Our central contribution is making active perception a native, learned capability of streaming interaction models. We demonstrate why this is necessary: learning when to respond without learning what evidence to acquire is insufficient, revealing a reciprocal dependency between sensing and interaction. To our knowledge, Self-Focus is the first streaming interaction model to jointly optimize these decisions through downstream interaction outcomes, including learning to acquire evidence before its relevance to future interactions is known. Our main results, controlled ablations, and additional experiments conducted during the discussion period provide direct empirical evidence for this contribution.

We have updated the manuscript accordingly, with all revisions highlighted in blue:

  • Clarifying our contribution (Secs. 1-2). Following Reviewer fkNv's suggestion, we clarify the sequential, feedback-dependent nature of streaming active perception. We also revise the Related Work (particularly L128-L142) to better distinguish Self-Focus from prior methods and highlight our novelty and contributions, addressing concerns raised by Reviewers uoy1 and JHzZ.
  • Separating sensing and retrieval (new Tab. A; Sec. 4.3, L431–L461). Following suggestions from Reviewers fkNv and uoy1, we add a controlled ablation isolating query-agnostic sensing while keeping the learned query-conditioned policy fixed. While Sec. 4.4 already demonstrated the utility of the self-generated query-agnostic context on offline Long VideoQA, we recognized that the ablations in Tab. 4 alone did not sufficiently disentangle sensing from retrieval on streaming benchmarks. The new experiment addresses this gap and provides a clearer transition to Sec. 4.4, strengthening the evidence for learning to acquire useful context before a query is known.
  • Expanding baselines and strengthening the perception-memory comparison (Tab. 2; Fig. 5). We initially prioritized the strongest baselines targeting the perception-memory trade-off in retrospective benchmarks. Following Reviewer uoy1's suggestion, we expand Tab. 2 with additional published streaming methods and add a visualization of the perception-memory frontier in Appendix E (Fig. 5). The figure shows that Self-Focus advances this frontier: improvements in backward accuracy do not come at the cost of real-time accuracy, which remains essentially unchanged across context budgets. We further validate this advantage against SimpleStream under approximately matched visual-token budgets.
  • Additional interaction and robustness evaluations (Appendix E). Following suggestions from the reviewers, we add evaluations on mixed interactions (uoy1), OVO-Bench Forward Active Responding (uoy1), long-horizon streaming robustness (BaWq), and FIFO eviction sensitivity (fkNv). We thank the reviewers for highlighting these complementary evaluation axes, which further strengthen the empirical evidence for Self-Focus across diverse streaming interaction settings.
  • Runtime profile (Sec. 4.5). We clarify our rolling KV cache and CPU-offloaded archive. For all the evaluated benchmarks, Self-Focus sustains real-time streaming without buffering or dropping frames. Beyond meeting this real-time requirement, Self-Focus achieves inference latency comparable to efficiency-focused methods such as StreamingVLM and DSCache, while substantially outperforming them across all evaluated tasks.

We thank the reviewers and AC again for their thoughtful feedback and encouraging assessment of our work. We believe these discussions have further strengthened the paper and its empirical foundations. In streaming systems, future information needs are often unknown; we hope Self-Focus represents a step toward always-on models that can acquire and preserve useful evidence for future interactions.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.