- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CEDVS-Bench, a benchmark for continuous detection of language-defined events in video streams. Before a video starts, the model receives a fixed set of event descriptions. At every second it must decide, using only past frames, whether to emit a START or END for any candidate or stay silent. Candidates include positive events with annotated intervals and absent events at three difficulty levels. The benchmark has 2,840 videos drawn from Charades, QVHighlights, and CG-Bench, organized into Short, Medium, and Long subsets, with 15,983 positive instances and 66,158 absent candidates. Annotations are produced by an automatic pipeline. Qwen-VL-Max captions each 2-second clip, Gemini-2.5-Pro extracts and merges macro events of at least 6 seconds, Qwen-VL-Max verifies positives from sampled frames, Gemini generates Medium and Hard negatives, and Qwen-VL-Max checks them against 16 to 20 sampled frames. A human audit covers 110 videos. Metrics include onset localization, decision timeliness, a simulated response timeliness that replays measured service time through a FIFO queue, false alarm rate by difficulty, and real-time factor. Eight window-based VLMs with five recent frames and three native streaming models are evaluated. Qwen3-VL-8B performs best but response F1 collapses under processing cost, false alarm rates rise with semantic similarity and observation horizon, and adding more frames or splitting candidates across calls often hurts.
Critical strengths
- The task framing is practically important and distinct from existing streaming QA. Maintaining a persistent candidate set, triggering at onset, and staying silent on absent events is what a deployed video assistant actually needs, and no prior benchmark evaluates all three jointly at this scale.
- Explicit absent candidates with construction-defined difficulty levels are a valuable design. Pairing Hard negatives that share scene and actor with the positive directly probes whether a model grounds the specific action rather than the context.
- The processing-aware response timeliness metric is a genuine contribution. Replaying measured service time through a FIFO queue (Eq. 7) exposes that a correct and timely decision can still arrive too late, which standard timeliness metrics miss.
- The authors provide extensive analysis. Horizon truncation on the same videos (Table 7), duration and occupancy breakdowns, frame budget and candidate grouping ablations with paired bootstrap intervals, and verified qualitative cases go well beyond a leaderboard.
Critical weaknesses
- The positive events are not the kind of events the task is motivated by. The example annotation in the supplement shows events such as "Video Intro Animation", "Montage of various short scenes", and "Person sitting on the floor" tiling 98 percent of a 20-minute video back to back. These are dense-caption segment titles, not triggerable events like "someone opens the cabinet".
- Boundaries are never independently annotated. The human boundary review is accept-or-correct with 90 percent acceptance (Table B.1), which the paper itself says is not agreement between independent annotations. Even the offline full-video condition reaches only 59 onset F1 on the easiest subset (Table 5), suggesting substantial label noise.
- Positive validity on CEDVS-Short is only 83.33 percent. One in six positives on the subset where models perform best is judged invalid by humans, yet all retained positives remain in the benchmark.
- The annotation pipeline and the best model share a family. Qwen-VL-Max writes the captions and verifies positives and negatives, and Qwen3-VL-8B outperforms every other model by a wide margin (Table 4). This confound is not discussed or controlled.
- Absence is not verified for long videos. Medium and Hard negatives are checked with 16 frames across videos up to 20 minutes, Easy negatives are not checked at all, and the paper does not say how human auditors verified absence over full recordings.
- The evaluation interface likely dominates the results. Scores swing by 25 F1 points with frame count and by 15 points with candidate grouping (Table G.4), near-zero scores for native streaming models are attributed to interface differences, and no human or oracle reference is provided to anchor what a good score looks like.
Questions for authors
- What fraction of positive events in each subset are dense-caption style segments (for example intros, montages, "person sitting") versus discrete triggerable actions? Could you report the distribution of event coverage ratio per video?
- How were human auditors able to verify full-video absence of negatives for CEDVS-Long videos averaging 15 minutes?
- Can you evaluate a non-Qwen captioner or verifier on a subset to test whether the Qwen3-VL advantage persists when the annotation pipeline uses a different model family?
- Why are the 17 percent of CEDVS-Short positives rejected by human audit retained?
7 comments
Response to Reviewer SECA (2/4)
Weakness 2, Weakness 3 and Question 4:ndependent Annotation and Correction of Known Errors
Response2: Thank you for asking why the rejected records were retained and for requesting independent boundary annotations. We re-scored the models after applying the annotation corrections and added independent annotations to assess label consistency.
The submitted scoring set applied automatic filtering, while human rejection decisions were stored separately and had not been applied to it. The 83.33% validity rate refers to the reviewed Short positives. Removing rejected records and applying available boundary corrections, with predictions fixed, lowers Short Onset F1 by 0.46–3.48 percentage points across the eight models. Qwen remains first on the corrected manually reviewed subsets (Table R3-2b).
Table R3-2b. Two matched annotation comparisons across all eight models
| Subset | Model | Old annotation: original | Old annotation: corrected | Independent annotation: original | Independent annotation: A | Independent annotation: B |
|---|---|---|---|---|---|---|
| Short | Qwen3-VL-8B | 41.94 | 40.38 | 38.96 | 38.36 | 35.14 |
| Short | InternVL3.5-8B | 27.51 | 25.61 | 26.09 | 25.23 | 26.79 |
| Short | MiniCPM-V-2.6 | 26.67 | 26.21 | 25.49 | 20.41 | 24.24 |
| Short | LLaVA-OneVision-1.5-8B | 26.43 | 24.12 | 23.81 | 22.95 | 22.76 |
| Short | LLaVA-NeXT-Video-7B | 13.77 | 11.59 | 15.47 | 14.69 | 16.85 |
| Short | VideoLLaMA3-7B | 24.64 | 21.16 | 25.00 | 24.19 | 27.20 |
| Short | LongVA-7B | 16.52 | 15.14 | 22.06 | 21.21 | 21.05 |
| Short | LongVU-7B | 38.03 | 35.77 | 29.33 | 25.35 | 27.78 |
| Medium | Qwen3-VL-8B | 23.99 | 23.55 | 34.12 | 33.72 | 35.15 |
| Medium | InternVL3.5-8B | 10.55 | 10.16 | 14.94 | 13.99 | 12.71 |
| Medium | MiniCPM-V-2.6 | 7.33 | 6.94 | 12.87 | 13.37 | 11.87 |
| Medium | LLaVA-OneVision-1.5-8B | 11.32 | 11.09 | 19.83 | 19.66 | 18.50 |
| Medium | LLaVA-NeXT-Video-7B | 4.49 | 4.59 | 8.66 | 8.60 | 9.56 |
| Medium | VideoLLaMA3-7B | 6.01 | 5.77 | 9.39 | 9.32 | 8.82 |
| Medium | LongVA-7B | 6.01 | 5.73 | 9.60 | 9.52 | 9.80 |
| Medium | LongVU-7B | 12.89 | 12.31 | 18.28 | 19.15 | 17.68 |
| Long | Qwen3-VL-8B | 10.04 | 10.27 | 10.59 | 11.63 | 10.78 |
| Long | InternVL3.5-8B | 2.19 | 2.28 | 6.74 | 7.78 | 6.86 |
| Long | MiniCPM-V-2.6 | 2.04 | 1.82 | 2.01 | 1.99 | 2.03 |
| Long | LLaVA-OneVision-1.5-8B | 3.33 | 3.24 | 5.17 | 5.98 | 5.24 |
| Long | LLaVA-NeXT-Video-7B | 0.86 | 0.89 | 6.64 | 7.51 | 6.73 |
| Long | VideoLLaMA3-7B | 3.72 | 3.71 | 4.07 | 4.03 | 4.12 |
| Long | LongVA-7B | 3.15 | 2.75 | 11.63 | 11.36 | 12.05 |
| Long | LongVU-7B | 1.75 | 1.85 | 9.40 | 10.60 | 9.59 |
We assessed ranking consistency separately within each review set, comparing each revised reference with its own original baseline(Table R3-2c). Across the eight models, correcting known errors gives Spearman correlations of 0.976 / 1.000 / 0.976 on Short / Medium / Long. Replacing the original annotations with either independent annotator’s reference gives correlations between 0.857 and 1.000. Qwen remains first after known-error correction on all three subsets and under both independent references on Short and Medium. On Long, Qwen and LongVA remain the top two models under all three independent references, although their order changes under A. These comparisons show that the annotation changes largely preserve the relative model ordering on the reviewed sets.
Table R3-2c. Ranking consistency within the two review sets
| Review set and comparison | Short: Spearman ρ | Medium: Spearman ρ | Long: Spearman ρ |
|---|---|---|---|
| Known-error review: original vs. corrected | 0.976 | 1.000 | 0.976 |
| Independent review: original vs. A | 0.857 | 1.000 | 0.976 |
| Independent review: original vs. B | 0.929 | 0.976 | 1.000 |
Two annotators independently watched 60 complete videos without seeing the original annotations or each other’s labels. On the candidates used for model evaluation, their event-level Onset F1@3s agreement is 61.02 [42.31, 78.69] on Short, 89.05 [83.21, 94.49] on Medium, and 92.04 [83.76, 98.25] on Long (95% CIs). These scores measure agreement between full-video annotations.
For a matched model comparison, Qwen’s offline Onset F1 on the Short annotation candidates is 40.00 with the original labels, 50.98 with A, and 38.46 with B (Table R3-2). The paper’s 59.11 was computed on the full Short evaluation set. The independent references change scores in both directions, and we will report this sensitivity alongside the Short results.
Response to Reviewer SECA (2/4)-Weakness 2, Weakness 3 and Question 4
Table R3-2. Full-video offline and causal online Onset F1 on the common manually reviewed subset subsets
| Subset | Model | Offline: original / A / B | Online: original / A / B |
|---|---|---|---|
| Short | Qwen3-VL-8B | 40.00 / 50.98 / 38.46 | 38.96 / 38.36 / 35.14 |
| Short | InternVL3.5-8B | 22.73 / 20.00 / 29.27 | 26.09 / 25.23 / 26.79 |
| Short | MiniCPM-V-2.6 | 4.76 / 5.26 / 10.26 | 25.49 / 20.41 / 24.24 |
| Medium | Qwen3-VL-8B | 47.33 / 46.62 / 49.21 | 34.12 / 33.72 / 35.15 |
| Medium | InternVL3.5-8B | 15.25 / 15.00 / 15.93 | 14.94 / 13.99 / 12.71 |
| Medium | MiniCPM-V-2.6 | 0.00 / 0.00 / 0.00 | 12.87 / 13.37 / 11.87 |
| Medium | VideoLLaMA3-7B | 0.00 / 0.00 / 0.00 | 9.39 / 9.32 / 8.82 |
We will correct or remove all errors identified through human review before release and retain the original scoring reference separately for reproducibility.Manually reviewing the entire benchmark would require substantial annotation effort, so we will provide a human-reviewed subset. Every candidate in this subset will be checked: invalid positives and negatives will be removed, and unacceptable boundaries will be corrected or the affected records excluded from evaluation.
Response to Reviewer SECA (1/4)
Weakness 1 and Question 1: Event Types, Coverage, and Evaluation by Track
Response1: Thank you for pointing out that our event labels include discrete actions, sustained states, and video segments. We now separate these targets into three tracks and report their composition and detection performance.
Table R3-1 reports the composition of positive instances based on automatic classification of their descriptions. Activities and states form the largest group in each subset. The 99.5% median coverage on Long shows that the dense coverage in example is common.
Table R3-1. Positive-event composition and per-video coverage
We assign actions/transitions to the Core Track, activities/states to the Extended Track, and editing/topic segments to the Auxiliary Track. Table R3-1b reports Onset F1 and Decision F1 at 3 seconds using the original predictions, with only positive candidates in each track included. Qwen’s Core Onset F1 is 56.70 / 39.10 / 11.50 on Short / Medium / Long, so the low Long score persists when discrete actions are evaluated separately.
Table R3-1b. Results by track: Onset F1 / Decision F1 (%) at 3 s
We will also clarify what onset means for each type. For “a person sitting on the floor,” an Extended Track state, onset is the first time that state is visible in the recording, or zero if it is already present at video start. This does not require observing the act of sitting down. The revised paper will discuss event triggering through the Core Track and present the other two tracks as broader monitoring tasks.