- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Self-Focus, a streaming interaction model that jointly learns active perception and response decisions. The model can selectively retain, discard, retrieve, or examine video frames at higher resolution. The authors introduce delayed sensing credit to optimize sensing decisions using future interaction outcomes. Experiments on proactive and retrospective video understanding benchmarks show improvements over existing streaming and interaction models.
Critical strengths
- The paper addresses an important and well-motivated problem. Existing streaming interaction models mainly focus on when to respond, while evidence acquisition is often handled separately. Learning sensing and response decisions within a single policy is a reasonable approach. The method is also clearly presented.
- The experimental evaluation is extensive. The authors consider both proactive and retrospective tasks across several benchmarks. Tables 3 and 4 provide useful evidence for the benefits of joint sensing, delayed sensing credit, and different sensing actions. The offline long-video experiments further demonstrate the value of the learned working context.
Critical weaknesses
- The methodological novelty of delayed sensing credit could be better established. Eq. (2) is closely related to existing reward-to-go estimators with a leave-one-out baseline. Table 3 shows improvements over trajectory-level GRPO, which is encouraging. However, the comparison does not fully isolate the effect of temporal credit assignment from differences in advantage normalization. Appendix D.4 establishes an unbiased policy gradient under certain assumptions, while acknowledging that the estimator is not a same-state action advantage. Further analysis would help clarify the specific contribution of the proposed estimator.
- The robustness of long-horizon streaming interaction needs further analysis. Appendix D.3 limits RL training segments to 256 seconds, with earlier context largely reconstructed from reference actions. The authors also use partially self-generated prefixes and evaluate on longer videos. However, the results do not separately examine how performance changes over long multi-turn interactions. An analysis across stream lengths, query counts, and delays between evidence and query arrival would help establish the method's long-horizon robustness.
Questions for authors
- Could the authors further clarify the contribution of delayed sensing credit beyond standard reward-to-go estimation? How sensitive are the results to the choice of advantage normalization and baseline?
- How does performance vary with stream duration, the number of previous interactions, and the delay between evidence and query arrival? In particular, does performance degrade for histories much longer than those used during RL training?
- Figure 4 provides useful runtime results, but could the authors also compare total computation and memory costs with the baselines? In particular, how much overhead comes from low-resolution previews, high-resolution sensing, and the CPU archive?
- Could the authors provide confidence intervals or results across multiple random seeds, particularly for benchmarks with small performance differences? It would also be helpful to clarify how StreamingBench-PO accuracy is aggregated.
5 comments
Rebuttal - 2/3
W2 and Q2.
W2 “However, the results do not separately examine how performance changes over long multi-turn interactions.”
Q2 “In particular, does performance degrade for histories much longer than those used during RL training?”
We agree that an explicit long-horizon analysis would strengthen the paper. Although RL segments are limited to 256s, each starts from a reconstructed streaming history, not an empty context (Fig. 3b): earlier observations in the working context, previous interactions in dialogue history, and absolute timestamps. At inference, the working context remains bounded regardless of stream length.
“An analysis across stream lengths, query counts, and delays between evidence and query arrival...”
Q2 “...stream duration, the number of previous interactions, and the delay between evidence and query arrival?”
Thank you for suggesting these three axes; we report each below. We will include these tables and associated discussions in the appendix.
Stream duration. Our offline Long VideoQA evaluation (Tab. 5) already tests query-agnostic sensing on videos extending well beyond the 256s RL segments, with improvements over the base model at every matched context budget. On average, these videos run up to 9.6× the RL segment length. To examine online interactions, we additionally split queries by their arrival time and run Self-Focus at low budget.
| Ask time | SPOT PNR | OVO RT | OVO BWD |
|---|---|---|---|
| <256s | 20.5 | 82.7 | 55.8 |
| >=256s | 20.1 | 82.4 | 55.2 |
| of which >=600s | 19.7 | 82.0 | 54.6 |
Performance remains similar even when queries arrive well beyond the RL segment length, with only minor decreases across the tasks.
Number of previous Interactions. We run this analysis on SPOT PNR, which includes queries referring to previous interactions. We retain the five most recent interactions in dialogue history and report performance by query turn positions.
| Turn in Session | 1st | 2nd-3rd | 4th-6th | 7th+ |
|---|---|---|---|---|
| PNR T-F1 | 20.5 | 20.5 | 20.4 | 20.2 |
Performance remains stable across turn positions, indicating little degradation as interactions accumulate in the evaluated sessions.
Evidence-to-query delay. Since the online benchmarks do not provide explicit evidence intervals for this analysis, we evaluate on 200 held-out retrospective episodes from HT-Step, COIN, and Charades-STA validation splits, with no training-video overlap. We compare trajectory-level credit (Tab. 3, row 5) with delayed sensing credit (row 7), using 50 episodes per delay range at the low budget. Evidence in context measures the percentage of episodes with at least one frame from the annotated evidence interval available before query arrival (retained) or at response time (retained or retrieved).
| Evidence-to-query delay | Trajectory-level acc. (%) | Delayed credit acc. (%) | Evidence in context before query (traj. / delayed) | Evidence in context at response (traj. / delayed) |
|---|---|---|---|---|
| Recent (0-3s) | 84 | 84 | 94/96% | 96/98% |
| Near (8-30s) | 68 | 78 | 44/58% | 56/72% |
| Far (60-256s) | 56 | 62 | 10/16% | 18/28% |
| Longer (256-600s) | 56 | 60 | 6/10% | 12/20% |
When evidence is recent, both variants retain it at similar rates. As the delay increases, delayed sensing credit retains more relevant evidence before query arrival and achieves equal or higher accuracy, even beyond the 256s RL segment length. Keeping more of the evidence helps in two ways: the model can answer from its working context directly, and when it needs to retrieve, the kept entries give retrieve() timestamped anchors near the evidence. This is precisely the behavior delayed sensing credit aims to learn: preserving evidence before its relevance to future queries is known.
Rebuttal - 3/3
Q3.
“how much overhead comes from low-resolution previews, high-resolution sensing, and the CPU archive”
Thank you for this suggestion. We provide a component-wise latency breakdown on a single RTX A5000 GPU in Tab. A below.
Table A. Component latency (ms). Values marked * are measured at the full working-context budget (2K/4K/8K for low/medium/high).
| Frame decoding (1fps) | 13.9 |
| Visual encoding of one low-res preview | 16.2 |
| Visual encoding of one high-res focus | 27.5 |
| Prefilling of one low-res preview* | 45.0/54.3/74.1 |
| Prefilling of one high-res focus* | 105.2/122.2/159.0 |
| Decoding a sensing action* | 61.5/66.0/74.6 |
| Eviction + re-indexing* | 9.0/17.9/36.1 |
| one retrieve() from the CPU archive | 25.0 |
| avg. TTFT for OVO-Bench | 315/335/352 |
High-resolution focus() incurs additional encoding and prefill costs, but is invoked selectively, while low-resolution previews are processed at every frame. Retrieval from the CPU archive adds only 25 ms per call. Overall, Self-Focus achieves average OVO-Bench time-to-first-token (TTFT) of 315-352 ms, comparable to the lightweight DSCache (326 ms). We exclude response generation latency, which varies with answer length. We will include this breakdown in the revised appendix. With peak GPU memory of 18.3/19.6/22.2 GB (low/medium/high) on a single 24 GB RTX A5000, Self-Focus runs at the evaluated 1-2 fps input rates in real-time for all benchmarks without dropping frames.
Q4.
“Could the authors provide confidence intervals or results across multiple random seeds…”
Following prior baselines, we use greedy decoding for all main results (Tabs. 1, 2, 5, and 6), including reproduced baselines, and all ablations (Tabs. 3 and 4), ensuring deterministic evaluation without sampling variability.
“particularly for benchmarks with small performance differences?”
Following your suggestion, we additionally evaluate Self-Focus (low) over five decoding seeds at temperature 1.0 on the two real-time benchmarks, where performance differences are relatively small.
| Task | Greedy (Tab. 2) | 5 seeds mean ±SD | 5 seeds 95% CI |
|---|---|---|---|
| OVO-Bench Real-time | 82.6 | 82.3±0.6 | [81.6, 83.0] |
| StreamingBench Real-time | 82.2 | 82.0±0.4 | [81.5, 82.5] |
The stochastic results remain close to greedy decoding, with modest variation across decoding seeds. We will include these results and clarify the evaluation protocol in the revision.
“It would also be helpful to clarify how StreamingBench-PO accuracy is aggregated.”
We use the official StreamingBench scorer on the Proactive Output (PO) task. Each query is issued once at its annotated timestamp, and the model decides when to respond. The score averages two accuracies: timing (the response falls within 2 seconds of the ground-truth time) and answer (the response is on time and its output is correct). We report this score over all 250 questions, using the same protocol for our reproduced JoyAI-VL-Interaction baseline. We would also like to point out that App. C incorrectly states 50 questions, referring to the subset evaluated in the original StreamingBench paper. We thank you for prompting us to identify this discrepancy and will correct it to 250 in the revision.
Rebuttal - 1/3
We sincerely thank the reviewer for their positive assessment and encouraging feedback. We are pleased that the reviewer finds our problem important and recognizes the importance of jointly learning sensing and response decisions, the extensive evaluation, and the value of our learned working context for long-video understanding. Below, we address the remaining concerns.
W1 & Q1.
We agree and clarify that our contribution is not a new return estimator, but learning evidence acquisition from subsequent interaction outcomes within a unified streaming interaction policy. The challenge is learning what evidence to acquire before future queries are known. Our streaming grammar (Sec. 3.2), constructed episodes (Sec. 3.3), and SFT initialization (Sec. 3.4) enable causal RL rollouts that share future frames and queries but construct different working contexts. This allows us to apply such reward-to-go estimators with a leave-one-out baseline to credit earlier sensing decisions from later interaction outcomes, without a separate sensing reward or learned critic. We will clarify this distinction in the revision.
The original comparison in Tab. 3 (rows 5 and 7) already controls for advantage normalization: both use the same leave-one-out baseline, without standard-deviation (SD) normalization, and share the same initialization, data, and training setup. Only the return horizon differs. We will relabel row 5 as “GRPO with trajectory-level credit” to make this explicit.
Normalization. Following your suggestion, we additionally trained both horizons with standard-deviation (SD) normalization (low budget):
Delayed sensing credit consistently outperforms trajectory-level credit with and without SD normalization, confirming that the improvement is not due to a normalization mismatch. SD normalization reduces performance for both methods, supporting our choice to train without it.
Baseline. For a group of (G) rollouts, leave-one-out and group-mean advantages differ only by a constant factor G/(G-1). They therefore preserve the sign and relative ordering of action credits. The main difference in our comparison is the return horizon.