- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how previously self-generated text affects the responsiveness of closed-loop streaming video LLMs to new visual information. The authors introduce an evaluation protocol grounded in temporal annotations, using in-loop current-step recognition probes and narration judging to quantify this effect. They also propose Retained-Output Contrastive Decoding (ROCD), a training-free method that compares prediction distributions with and without direct access to retained generated text and suppresses some of the token-probability increases associated with that access. Experiments on COIN show that retained narration reduces current-step recognition performance. ROCD improves Top-1 accuracy by 10.2, 8.1, and 5.0 percentage points for StreamingVLM, LiveCC, and Proact-VL, respectively, and also shows a trend toward improved scores in generated-narration evaluation.
Critical strengths
- The method is simple and training-free. Without updating model parameters, it compares token prediction distributions with and without direct access to retained self-generated text and suppresses positive probability increases associated with that text.
Critical weaknesses
-
There is insufficient evidence for the prevalence and importance of the problem. Through current-step recognition experiments on COIN, the authors find that retained self-generated text reduces models' responsiveness to new visual information. However, COIN primarily contains instructional videos with clearly delineated procedural steps, and the paper does not validate this phenomenon on other types of video data. The experiments establish a negative effect of retained text on COIN step recognition, but do not adequately show that the effect is similarly prevalent and important in general streaming video narration.
-
Improvements in actual video narration quality from ROCD have not been sufficiently validated. The authors identify the negative effect of retained text through step recognition on COIN, then propose ROCD and validate it primarily through step recognition metrics on the same dataset. Although the experiments support improvements in these metrics, step recognition does not directly represent the quality of the narration actually generated. Moreover, ROCD is applied during both generation and probe readout, so the main experiments do not adequately distinguish improvements in the generation process from reweighting of probe-answer probabilities. Table 3 additionally evaluates complete narrations, but primarily relies on descriptive scores from a fixed LLM judge, without sufficient analysis of statistical reliability or validation on independent video scenarios. The current evidence therefore does not adequately support the conclusion that ROCD reliably improves general streaming video narration quality.
Questions for authors
-
The authors evaluate Text Clearing in Narration Judging but do not report its probe performance during complete closed-loop runs. Could they compare direct removal of retained generated text with ROCD in terms of both probe accuracy and Narration Judging under the same experimental conditions? Also, would retaining no previously generated text at all produce different results from Text Clearing?
-
Current-step recognition measures the model's ability to identify the active step rather than the quality of its actual generated narration. Is there evidence of a reliable relationship between higher probe accuracy and improved final narration quality? In particular, when different interventions on retained text may affect the two metrics differently, how should improvements in probe accuracy be interpreted in terms of actual narration quality?
-
The LLM judge used for Narration Judging does not see the original video and evaluates only the annotated step timeline and generated text. How do the authors establish that these scores adequately reflect agreement between the generated text and the actual visual content? In particular, can the protocol detect narration that is semantically consistent with the current step but contains incorrect visual details? If it cannot, how do the authors assess the extent to which ROCD improves actual video narration quality?
-
All experiments are conducted on COIN, which mainly contains instructional videos with explicit procedural steps and temporal boundaries. Could the authors provide further evidence that the negative effect of retained self-generated text is not specific to this dataset or its step recognition task, and that ROCD remains practically useful in other streaming video narration settings?