acceptodds
Withdrawn from ICLR 2027

Annotation-Grounded Evaluation of Self-Generated Text Effects in Closed-Loop Streaming Video LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Streaming video LLMs can retain earlier generated text in the running context, allowing previous outputs to condition later predictions. We study this retained-output dependence as a distinct property of closed-loop streaming generation. Existing evaluations do not isolate it: supplied-history settings omit the model's own generated context, while end-to-end streaming metrics conflate retained-output effects with newly arriving visual evidence. We introduce an annotation-grounded protocol for evaluating retained-output effects during closed-loop generation. In-loop probing reads current-step predictions at annotated times, while narration judging evaluates generated output against the same timestamped reference timeline. Fixed-run reconstructions on COIN show that retained narration can reduce responsiveness to changing visual evidence, particularly across annotated step transitions. We further introduce Retained-Output Contrastive Decoding (ROCD), a training-free method that contrasts prefix-matched views with and without direct access to retained generated text. Across multiple streaming video LLMs, ROCD increases responsiveness to newly arriving visual evidence, with corresponding changes in both probe readouts and generated narration. Together, the protocol characterizes how retained output affects adaptation to changing visual evidence, while ROCD provides a training-free method for reducing this dependence during streaming inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, Streaming visual-language models, Inference-Time Prior

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_anrqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how previously self-generated text affects the responsiveness of closed-loop streaming video LLMs to new visual information. The authors introduce an evaluation protocol grounded in temporal annotations, using in-loop current-step recognition probes and narration judging to quantify this effect. They also propose Retained-Output Contrastive Decoding (ROCD), a training-free method that compares prediction distributions with and without direct access to retained generated text and suppresses some of the token-probability increases associated with that access. Experiments on COIN show that retained narration reduces current-step recognition performance. ROCD improves Top-1 accuracy by 10.2, 8.1, and 5.0 percentage points for StreamingVLM, LiveCC, and Proact-VL, respectively, and also shows a trend toward improved scores in generated-narration evaluation.

Critical strengths
  1. The method is simple and training-free. Without updating model parameters, it compares token prediction distributions with and without direct access to retained self-generated text and suppresses positive probability increases associated with that text.
Critical weaknesses
  1. There is insufficient evidence for the prevalence and importance of the problem. Through current-step recognition experiments on COIN, the authors find that retained self-generated text reduces models' responsiveness to new visual information. However, COIN primarily contains instructional videos with clearly delineated procedural steps, and the paper does not validate this phenomenon on other types of video data. The experiments establish a negative effect of retained text on COIN step recognition, but do not adequately show that the effect is similarly prevalent and important in general streaming video narration.

  2. Improvements in actual video narration quality from ROCD have not been sufficiently validated. The authors identify the negative effect of retained text through step recognition on COIN, then propose ROCD and validate it primarily through step recognition metrics on the same dataset. Although the experiments support improvements in these metrics, step recognition does not directly represent the quality of the narration actually generated. Moreover, ROCD is applied during both generation and probe readout, so the main experiments do not adequately distinguish improvements in the generation process from reweighting of probe-answer probabilities. Table 3 additionally evaluates complete narrations, but primarily relies on descriptive scores from a fixed LLM judge, without sufficient analysis of statistical reliability or validation on independent video scenarios. The current evidence therefore does not adequately support the conclusion that ROCD reliably improves general streaming video narration quality.

Questions for authors
  1. The authors evaluate Text Clearing in Narration Judging but do not report its probe performance during complete closed-loop runs. Could they compare direct removal of retained generated text with ROCD in terms of both probe accuracy and Narration Judging under the same experimental conditions? Also, would retaining no previously generated text at all produce different results from Text Clearing?

  2. Current-step recognition measures the model's ability to identify the active step rather than the quality of its actual generated narration. Is there evidence of a reliable relationship between higher probe accuracy and improved final narration quality? In particular, when different interventions on retained text may affect the two metrics differently, how should improvements in probe accuracy be interpreted in terms of actual narration quality?

  3. The LLM judge used for Narration Judging does not see the original video and evaluates only the annotated step timeline and generated text. How do the authors establish that these scores adequately reflect agreement between the generated text and the actual visual content? In particular, can the protocol detect narration that is semantically consistent with the current step but contains incorrect visual details? If it cannot, how do the authors assess the extent to which ROCD improves actual video narration quality?

  4. All experiments are conducted on COIN, which mainly contains instructional videos with explicit procedural steps and temporal boundaries. Could the authors provide further evidence that the negative effect of retained self-generated text is not specific to this dataset or its step recognition task, and that ROCD remains practically useful in other streaming video narration settings?

Reviewer_C19pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of retained self-generated text on future predictions in closed-loop streaming video LLMs. The authors introduce an annotation-grounded evaluation framework using in-loop probing and timeline-based narration judging on the COIN dataset. They demonstrate that retaining prior model outputs reduces responsiveness to visual changes, especially during step transitions. To mitigate this issue, they present Retained-Output Contrastive Decoding (ROCD), a training-free technique using cache selection to contrast closed-loop views against views without direct access to generated context. Across three streaming video LLMs, ROCD improves current-step recognition, post-transition visual updates, and narration quality.

Critical strengths

The paper isolates the confounding effect of self-generated text context from visual evidence in streaming generation through an evaluation methodology. Additionally, ROCD provides an efficient, training-free inference mechanism via cache selection that improves visual responsiveness without full stream replay.

Critical weaknesses
  1. Methodological novelty.
    ROCD's core operation, w(y) = min(p_CL(y), p_ROE(y)), is a straightforward contrastive reweighting rule. The paper needs a more compelling demonstration of what is fundamentally new relative to context-aware decoding and prior methods for mitigating the propagation of generated-text errors.

  2. Limited external validation.
    All experiments use COIN, the main probe evaluates recognition within a fixed annotation-derived answer space, and narration quality is judged by one LLM. Consequently, the evidence does not yet establish broad improvements in real-world streaming understanding.

Questions for authors
  1. Assessment-critical clarification.
    How does ROCD differ substantively from existing context-aware contrastive decoding methods, particularly in its theoretical motivation and behavior beyond applying the contrast to retained self-generated text? A stronger comparison with relevant alternatives could materially change the novelty assessment.

  2. Future-work suggestion.
    Can the proposed evaluation and ROCD be validated on a second streaming-video benchmark? Such evidence would establish whether the observed gains generalize beyond COIN and the selected judge configuration.

Reviewer_PpmwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates a problem in closed-loop streaming video large language models: previously generated narrations remain in the context, coupled with video evidence, and together affect subsequent predictions. The authors propose an evaluation protocol based on COIN step annotations, comprising two parts: first, inserting a multiple-choice question during execution to determine the current step and retrieving the model's in-loop probing; second, having LLM judges score the narrations against the annotated timeline. Experiments show that retained narrations reduce the model's recognition of the current step, especially after step switching. Based on this, the authors propose a training-free Retained-Output Contrast Decoding (ROCD): comparing the two prediction distributions when the model "sees" and "does not see" retained outputs, only suppressing the token probability that increases due to seeing old outputs. ROCD improved the accuracy of step identification across all three models, and the narration-judging scores also move in the desired direction.

Critical strengths
  1. The research question is clear, and the evaluation design has clear diagnostic value. The paper studies how streaming video LLMs are affected by their own retained outputs, which bears directly on the reliability of continuous video understanding. By fixing the video time, visual input, and probe question while varying only the retained text, the authors isolate the effect of retained outputs and show that it weakens updates after step transitions.

  2. The method is simple, training-free, and yields consistent gains across systems. ROCD contrasts predictions with and without access to retained outputs while preserving the speak-or-silence probability at each decoding step. It improves step-recognition accuracy by 10.2, 8.1, and 5.0 percentage points on the three models, the narration-judging scores move in the desired direction on all five criteria.

Critical weaknesses
  1. Current evidence is insufficient to attribute the gains to improved closed-loop generation. ROCD is applied both during narration generation and at probe readout. Section 4 already shows that merely replacing the retained narration with a placeholder at probe time improves accuracy by 11.0–16.3 percentage points. Therefore, the current results cannot distinguish whether the gains in Table 2 and Fig. 3 stem from improved narration content or from the changed readout rule. Two additional experiments are recommended: one applying ROCD only during generation, with standard probe readout; the other using base generation and applying ROCD only at probe readout.

  2. Insufficient baselines. The probe experiments compare ROCD only against VCD, which perturbs the visual input and does not target the retained text that the paper identifies as the source of the problem. Baselines that directly target the retained text are missing from the probe experiments, such as text clearing (which appears only in the narration-judging results in Table 3) and decoding directly from the retained-output-excluded distribution pROEp_{\mathrm{ROE}}. The paper also does not compare against CAD and RVD, the most closely related methods, both of which are already discussed in the related work. It is therefore difficult to assess whether ROCD is necessary.

Questions for authors
  1. Since ROCD is also applied at probe readout, and Section 4 shows that replacing the retained narration with a placeholder at probe time already yields 11.0–16.3 points higher accuracy, the gains in Table 2 and Fig. 3 may partly reflect the readout rule rather than improved narration. In addition, the probe experiments compare only against VCD, which targets the visual input rather than the retained text. Could the authors (i) report results with ROCD applied only during generation (with standard readout) and only at readout (with base generation), and (ii) compare on the probe metrics with baselines that directly target the retained text, such as text clearing, decoding from pROEp_{\mathrm{ROE}} alone, and streaming adaptations of CAD and RVD?

  2. ROCD suppresses the influence of retained outputs regardless of their correctness (Section 7). This suits COIN, where the current step is largely visible in the current frames, but retained narration can also carry information that is no longer visible, e.g., in StreamingVLM, whose text window is longer than its visual window. How does ROCD perform when such earlier context is needed, for example on probes about the previous step or how many times an action has occurred?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.