acceptodds
Withdrawn from ICLR 2027

PEARL: Personalized Streaming Video Understanding with Runtime-Defined Concepts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human cognition of new concepts is inherently a streaming process: we continuously recognize new objects or identities and update our memories over time. However, current methods are largely limited to static images or offline videos. This disconnects continuous visual input from instant real-world feedback, limiting their ability to provide the real-time, interactive personalized responses essential for future AI assistants. To bridge this gap, we first propose and formally define the novel task of Personalized Streaming Video Understanding (PSVU). To facilitate research in this new direction, we introduce PEARL-Bench, the first comprehensive benchmark designed specifically to evaluate this challenging setting. It evaluates a model's ability to respond to runtime-defined personalized concepts across Concept-Definition QA, Real-Time QA, and Past-Time QA, each further organized into fine-grained reasoning categories. Concept-Definition QA introduces user-defined concepts into the stream, Real-Time QA queries their current states and includes a new Action category for clip-grounded personalized motion concepts in addition to frame-grounded entity concepts, and Past-Time QA tests long-range retrieval of historical personalized evidence. PEARL-Bench comprises 132 unique videos and 2,173 fine-grained annotations with precise timestamps. To tackle this challenging new setting, we further propose PEARL, a plug-and-play strategy that serves as a strong baseline. Evaluations across 8 offline and online models demonstrate that it brings consistent PSVU improvements when applied to 3 distinct architectures, proving to be highly effective and robust. We hope this work advances vision-language model (VLM) personalization and inspires further research into streaming personalized AI assistants.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Multimodal Personalization, Vision-Language Models, Memory-Augmented Models, Video Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uY2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the Personalized Streaming Video Understanding (PSVU) task, where the user defines named entities or actions during a video stream and later ask questions about such entities or actions. The model must understand these definitions and properly retrieve the relevant frames from the video to answer the question. The paper also contributes PEARL-Bench, a benchmark for the PSVU task containing 132 videos, and PEARL, a training-free method for this task.

Experiments apply PEARL to three main backbones and show that it can robustly improve question answering accuracies, although the absolute performance is still low compared to human levels.

Critical strengths

The paper proposes a novel task setting and corresponding benchmark. Remembering user-defined objects is highly relevant in applications of visual personal assistants.

Critical weaknesses
  • The proposed task setting makes some strong simplifying assumptions about real-time interactions with humans. In the proposed benchmark, the target is always visible, there is never ambiguity on the target, and the instructions are always well aligned with the frames that the target is shown. Real-time interactions with humans might involve ambiguous references, delayed arrival of instructions, or clarifications.
  • The paper does not provide sufficient information about the annotation process, including the models, prompts, and decision rules for ablation-based filtering, as well as whether any multi-annotator agreement are measured and how disagreement resolution was done.
Questions for authors

How are entity frames and action-clip boundaries selected?

Reviewer_ha1YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for streaming video understanding in which users define concepts during a video and later ask about their current or past states. It also proposes PEARL, a memory and retrieval framework that improves QA accuracy under the paper’s evaluation protocol.

Critical strengths
  • The benchmark brings runtime concept definition and current and historical QA into one setting.
  • PEARL shows clear accuracy gains across three backbone models.
Critical weaknesses
  • The baselines use different frame rates and amounts of visual evidence. The paper also lacks a simple reference-frame-plus-retrieval baseline under matched budgets. As a result, the comparisons show a gain for the complete PEARL system but do not isolate the value of its specific design.
  • The experiments do not directly measure continuous real-time operation or long-range retrieval quality. Reported latency excludes video ingestion and concept registration, while Past-Time QA accuracy does not reveal whether the correct historical evidence was retrieved. These gaps leave two central motivations only partially validated.
Questions for authors
  • How does PEARL compare with ReKV and simple retrieval under matched backbone, frame-rate, and visual-token budgets? This would help isolate the contribution of PEARL’s memory and retrieval design.
  • What are retrieval Recall@K and QA accuracy at different query-to-evidence time gaps? This would show whether the method remains effective when relevant evidence is genuinely distant.
  • What are throughput and P95 latency when video ingestion, concept registration, and queries run concurrently? Query-only latency does not establish whether the system can keep up with a live stream.
  • How many questions were removed during benchmark filtering, and do results differ between real and synthetic videos? These breakdowns would clarify whether the reported gains depend on benchmark construction or video domain.
Reviewer_sQeQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PEARL studies video understanding with concepts introduced during an interaction.

A query can name a person or demonstrate an action, then ask questions about that concept as the video continues. The method stores concept examples and video history, retrieves relevant evidence, and passes it to a frozen VLM. The insight makes sense and the model reports improvement over baselines. My main concern is whether the evaluation really tests online understanding and whether the baselines receive the same information as it can test whether the memory architecture is especially novel and necessary.

Critical strengths

The paper propose new concepts with current and historical video questions is useful and design useful method based on it. The action category is also a worthwhile extension beyond remembering a named person or object. The figures and tables presentation are clear.

PEARL improves several backbones with detailed prompts, segmentation settings, latency measurements, and several supplementary ablations.

Critical weaknesses

W1. Main comparison concerns

Most of the real-time improvement in Table 3 appears when concept memory is added: the score rises from 15.84 to 51.41, while the full system reaches 53.83. This shows that remembering the definition matters. It does not yet show that the proposed memory organization is needed.

How does PEARL compare with the same backbone given the original definition, visual example, and current clip directly in context? Does the advantage remain when methods receive the same concept examples, sampling rate, available history, and visual-token budget?

W2. Online evaluation.

Since the method uses scene-based clips, questions can arrive before a clip has ended. If a question arrives at second five of an eight-second clip, which frames are available to the system? How are the current clip and retrieved neighbors truncated at the query time?

W3. The reported score mixes answer accuracy with option-order robustness

Appendix F.2 counts a question as correct only if all four answer placements are answered correctly. This is a stricter measure than ordinary accuracy. Figure 2 also shows examples with two or three choices.

Are those shortened illustrations, or does the dataset contain different option counts? What scoring rule applies to each case?

W4. The contribution

The system combines familiar components: stored examples, descriptions, query rewriting, and video retrieval. Their combination is reasonable, but the paper sometimes treats the broader ideas of editable personalization and historical memory as new.

What capability requires PEARL’s particular design rather than an existing memory-based video system supplied with the same definitions?

Although a benchmark contribution with a useful baseline would be sufficient in principle, the paper would be easier to assess if it stated that contribution directly.

W5. Related work.

The following references are recommended to be cited. [1] Streaming Video Question-Answering with In-context Video KV-Cache Retrieval [2] Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

Questions for authors

Q1. The 2,173 interactions include 498 concept definitions and 1,675 scored questions. Can these counts be distinguished wherever the benchmark size is reported?

Q2. Several questions come from the same video. What are the confidence intervals when videos are used as the resampling units?

Q3. The timing results appear to measure answering after memory is available. Do they include ingestion, embedding, registration, and indexing? What is the continuous processing throughput?

Q4. Figure 1 contains “Definiton,” Figure 4 contains “Retrival” and “+PSVU,” and some Yo’LLaVA citations have incomplete fields. Can these be corrected and the method name used consistently?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.