- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes the Personalized Streaming Video Understanding (PSVU) task, where the user defines named entities or actions during a video stream and later ask questions about such entities or actions. The model must understand these definitions and properly retrieve the relevant frames from the video to answer the question. The paper also contributes PEARL-Bench, a benchmark for the PSVU task containing 132 videos, and PEARL, a training-free method for this task.
Experiments apply PEARL to three main backbones and show that it can robustly improve question answering accuracies, although the absolute performance is still low compared to human levels.
Critical strengths
The paper proposes a novel task setting and corresponding benchmark. Remembering user-defined objects is highly relevant in applications of visual personal assistants.
Critical weaknesses
- The proposed task setting makes some strong simplifying assumptions about real-time interactions with humans. In the proposed benchmark, the target is always visible, there is never ambiguity on the target, and the instructions are always well aligned with the frames that the target is shown. Real-time interactions with humans might involve ambiguous references, delayed arrival of instructions, or clarifications.
- The paper does not provide sufficient information about the annotation process, including the models, prompts, and decision rules for ablation-based filtering, as well as whether any multi-annotator agreement are measured and how disagreement resolution was done.
Questions for authors
How are entity frames and action-clip boundaries selected?