acceptodds
Desk rejected at ICLR 2027

CEDVS-Bench: Speak or Stay Silent? A Benchmark for Continuous Event Detection in Video Streams

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Video assistants must recognize events while they unfold, decide when evidence justifies a report, and remain silent when a requested event never occurs. Retrospective localization and answers at selected query times only partially capture this requirement. We introduce CEDVS-Bench, a unified benchmark for continuous detection of language-defined events in video streams. Candidate descriptions are specified before observation; models make causal decisions every second, without access to future frames or event boundaries. The benchmark contains 2,840 videos, 15,983 positive event instances, and 66,158 absent candidates across short, medium, and long video settings. Our evaluation jointly measures onset localization, decision timeliness, processing-aware response availability, and rejection of absent events. Experiments on both window-based VLMs and native streaming models show that accurate localization does not necessarily lead to timely reporting, while semantically similar absent events remain difficult to reject, especially over longer observation horizons. CEDVS-Bench provides a unified testbed for accurate, timely, and reliable continuous event monitoring in video streams.

Primary area:
datasets and benchmarks
Keywords:
Event Detection, Streaming Video, Real-Time Video Understanding, Decision Timeliness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SECAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CEDVS-Bench, a benchmark for continuous detection of language-defined events in video streams. Before a video starts, the model receives a fixed set of event descriptions. At every second it must decide, using only past frames, whether to emit a START or END for any candidate or stay silent. Candidates include positive events with annotated intervals and absent events at three difficulty levels. The benchmark has 2,840 videos drawn from Charades, QVHighlights, and CG-Bench, organized into Short, Medium, and Long subsets, with 15,983 positive instances and 66,158 absent candidates. Annotations are produced by an automatic pipeline. Qwen-VL-Max captions each 2-second clip, Gemini-2.5-Pro extracts and merges macro events of at least 6 seconds, Qwen-VL-Max verifies positives from sampled frames, Gemini generates Medium and Hard negatives, and Qwen-VL-Max checks them against 16 to 20 sampled frames. A human audit covers 110 videos. Metrics include onset localization, decision timeliness, a simulated response timeliness that replays measured service time through a FIFO queue, false alarm rate by difficulty, and real-time factor. Eight window-based VLMs with five recent frames and three native streaming models are evaluated. Qwen3-VL-8B performs best but response F1 collapses under processing cost, false alarm rates rise with semantic similarity and observation horizon, and adding more frames or splitting candidates across calls often hurts.

Critical strengths
  1. The task framing is practically important and distinct from existing streaming QA. Maintaining a persistent candidate set, triggering at onset, and staying silent on absent events is what a deployed video assistant actually needs, and no prior benchmark evaluates all three jointly at this scale.
  2. Explicit absent candidates with construction-defined difficulty levels are a valuable design. Pairing Hard negatives that share scene and actor with the positive directly probes whether a model grounds the specific action rather than the context.
  3. The processing-aware response timeliness metric is a genuine contribution. Replaying measured service time through a FIFO queue (Eq. 7) exposes that a correct and timely decision can still arrive too late, which standard timeliness metrics miss.
  4. The authors provide extensive analysis. Horizon truncation on the same videos (Table 7), duration and occupancy breakdowns, frame budget and candidate grouping ablations with paired bootstrap intervals, and verified qualitative cases go well beyond a leaderboard.
Critical weaknesses
  1. The positive events are not the kind of events the task is motivated by. The example annotation in the supplement shows events such as "Video Intro Animation", "Montage of various short scenes", and "Person sitting on the floor" tiling 98 percent of a 20-minute video back to back. These are dense-caption segment titles, not triggerable events like "someone opens the cabinet".
  2. Boundaries are never independently annotated. The human boundary review is accept-or-correct with 90 percent acceptance (Table B.1), which the paper itself says is not agreement between independent annotations. Even the offline full-video condition reaches only 59 onset F1 on the easiest subset (Table 5), suggesting substantial label noise.
  3. Positive validity on CEDVS-Short is only 83.33 percent. One in six positives on the subset where models perform best is judged invalid by humans, yet all retained positives remain in the benchmark.
  4. The annotation pipeline and the best model share a family. Qwen-VL-Max writes the captions and verifies positives and negatives, and Qwen3-VL-8B outperforms every other model by a wide margin (Table 4). This confound is not discussed or controlled.
  5. Absence is not verified for long videos. Medium and Hard negatives are checked with 16 frames across videos up to 20 minutes, Easy negatives are not checked at all, and the paper does not say how human auditors verified absence over full recordings.
  6. The evaluation interface likely dominates the results. Scores swing by 25 F1 points with frame count and by 15 points with candidate grouping (Table G.4), near-zero scores for native streaming models are attributed to interface differences, and no human or oracle reference is provided to anchor what a good score looks like.
Questions for authors
  1. What fraction of positive events in each subset are dense-caption style segments (for example intros, montages, "person sitting") versus discrete triggerable actions? Could you report the distribution of event coverage ratio per video?
  2. How were human auditors able to verify full-video absence of negatives for CEDVS-Long videos averaging 15 minutes?
  3. Can you evaluate a non-Qwen captioner or verifier on a subset to test whether the Qwen3-VL advantage persists when the annotation pipeline uses a different model family?
  4. Why are the 17 percent of CEDVS-Short positives rejected by human audit retained?
7 comments
AuthorsOpenReview ↗

Response to Reviewer SECA (1/4)

Weakness 1 and Question 1: Event Types, Coverage, and Evaluation by Track

Response1: Thank you for pointing out that our event labels include discrete actions, sustained states, and video segments. We now separate these targets into three tracks and report their composition and detection performance.

Table R3-1 reports the composition of positive instances based on automatic classification of their descriptions. Activities and states form the largest group in each subset. The 99.5% median coverage on Long shows that the dense coverage in example is common.

Table R3-1. Positive-event composition and per-video coverage

SubsetAction/transition (%)Activity/state (%)Editing/topic (%)Coverage median [IQR] (%)
Short39.260.60.291.4 [81.8, 96.6]
Medium27.162.110.896.0 [90.5, 100.0]
Long40.348.810.899.5 [98.1, 100.0]

We assign actions/transitions to the Core Track, activities/states to the Extended Track, and editing/topic segments to the Auxiliary Track. Table R3-1b reports Onset F1 and Decision F1 at 3 seconds using the original predictions, with only positive candidates in each track included. Qwen’s Core Onset F1 is 56.70 / 39.10 / 11.50 on Short / Medium / Long, so the low Long score persists when discrete actions are evaluated separately.

Table R3-1b. Results by track: Onset F1 / Decision F1 (%) at 3 s

SubsetModelCore: Event TriggeringExtended: Activity & State MonitoringAuxiliary: Video Segment Monitoring
ShortQwen3-VL-8B56.70 / 50.0556.02 / 54.2522.22 / 44.44
ShortInternVL3.5-8B46.71 / 40.0449.39 / 47.660.00 / 0.00
ShortMiniCPM-V-2.641.73 / 31.2144.02 / 36.9322.22 / 0.00
ShortLLaVA-OneVision-1.5-8B43.03 / 39.1150.16 / 46.6918.18 / 0.00
ShortLLaVA-NeXT-Video-7B28.00 / 23.4030.47 / 28.240.00 / 0.00
ShortVideoLLaMA3-7B44.43 / 35.8946.17 / 40.850.00 / 0.00
ShortLongVA-7B31.37 / 23.0638.51 / 31.910.00 / 0.00
ShortLongVU-7B50.68 / 41.3156.82 / 49.1540.00 / 40.00
MediumQwen3-VL-8B39.10 / 37.8438.20 / 42.3238.45 / 38.83
MediumInternVL3.5-8B24.03 / 22.6729.79 / 29.1427.51 / 27.39
MediumMiniCPM-V-2.622.77 / 17.2824.28 / 18.5221.95 / 16.24
MediumLLaVA-OneVision-1.5-8B29.39 / 27.7233.21 / 33.6730.12 / 27.70
MediumLLaVA-NeXT-Video-7B13.87 / 13.2917.97 / 16.9114.83 / 14.27
MediumVideoLLaMA3-7B17.65 / 14.0622.19 / 18.7820.21 / 16.78
MediumLongVA-7B19.97 / 15.5724.06 / 20.6323.49 / 18.72
MediumLongVU-7B24.47 / 17.9328.35 / 22.6329.03 / 22.75
LongQwen3-VL-8B11.50 / 10.4813.72 / 12.3323.35 / 20.60
LongInternVL3.5-8B5.70 / 4.767.42 / 6.0418.27 / 14.14
LongMiniCPM-V-2.64.03 / 2.355.40 / 3.939.09 / 8.06
LongLLaVA-OneVision-1.5-8B4.74 / 4.617.92 / 6.9819.06 / 16.34
LongLLaVA-NeXT-Video-7B3.79 / 3.464.93 / 5.0015.14 / 14.34
LongVideoLLaMA3-7B5.64 / 3.876.10 / 5.1215.19 / 14.54
LongLongVA-7B4.26 / 3.716.28 / 5.4314.33 / 13.26
LongLongVU-7B4.66 / 3.946.98 / 5.457.98 / 6.71

We will also clarify what onset means for each type. For “a person sitting on the floor,” an Extended Track state, onset is the first time that state is visible in the recording, or zero if it is already present at video start. This does not require observing the act of sitting down. The revised paper will discuss event triggering through the Core Track and present the other two tracks as broader monitoring tasks.

AuthorsOpenReview ↗

Response to Reviewer SECA (2/4)

Weakness 2, Weakness 3 and Question 4:ndependent Annotation and Correction of Known Errors

Response2: Thank you for asking why the rejected records were retained and for requesting independent boundary annotations. We re-scored the models after applying the annotation corrections and added independent annotations to assess label consistency.

The submitted scoring set applied automatic filtering, while human rejection decisions were stored separately and had not been applied to it. The 83.33% validity rate refers to the reviewed Short positives. Removing rejected records and applying available boundary corrections, with predictions fixed, lowers Short Onset F1 by 0.46–3.48 percentage points across the eight models. Qwen remains first on the corrected manually reviewed subsets (Table R3-2b).

Table R3-2b. Two matched annotation comparisons across all eight models

SubsetModelOld annotation: originalOld annotation: correctedIndependent annotation: originalIndependent annotation: AIndependent annotation: B
ShortQwen3-VL-8B41.9440.3838.9638.3635.14
ShortInternVL3.5-8B27.5125.6126.0925.2326.79
ShortMiniCPM-V-2.626.6726.2125.4920.4124.24
ShortLLaVA-OneVision-1.5-8B26.4324.1223.8122.9522.76
ShortLLaVA-NeXT-Video-7B13.7711.5915.4714.6916.85
ShortVideoLLaMA3-7B24.6421.1625.0024.1927.20
ShortLongVA-7B16.5215.1422.0621.2121.05
ShortLongVU-7B38.0335.7729.3325.3527.78
MediumQwen3-VL-8B23.9923.5534.1233.7235.15
MediumInternVL3.5-8B10.5510.1614.9413.9912.71
MediumMiniCPM-V-2.67.336.9412.8713.3711.87
MediumLLaVA-OneVision-1.5-8B11.3211.0919.8319.6618.50
MediumLLaVA-NeXT-Video-7B4.494.598.668.609.56
MediumVideoLLaMA3-7B6.015.779.399.328.82
MediumLongVA-7B6.015.739.609.529.80
MediumLongVU-7B12.8912.3118.2819.1517.68
LongQwen3-VL-8B10.0410.2710.5911.6310.78
LongInternVL3.5-8B2.192.286.747.786.86
LongMiniCPM-V-2.62.041.822.011.992.03
LongLLaVA-OneVision-1.5-8B3.333.245.175.985.24
LongLLaVA-NeXT-Video-7B0.860.896.647.516.73
LongVideoLLaMA3-7B3.723.714.074.034.12
LongLongVA-7B3.152.7511.6311.3612.05
LongLongVU-7B1.751.859.4010.609.59

We assessed ranking consistency separately within each review set, comparing each revised reference with its own original baseline(Table R3-2c). Across the eight models, correcting known errors gives Spearman correlations of 0.976 / 1.000 / 0.976 on Short / Medium / Long. Replacing the original annotations with either independent annotator’s reference gives correlations between 0.857 and 1.000. Qwen remains first after known-error correction on all three subsets and under both independent references on Short and Medium. On Long, Qwen and LongVA remain the top two models under all three independent references, although their order changes under A. These comparisons show that the annotation changes largely preserve the relative model ordering on the reviewed sets.

Table R3-2c. Ranking consistency within the two review sets

Review set and comparisonShort: Spearman ρMedium: Spearman ρLong: Spearman ρ
Known-error review: original vs. corrected0.9761.0000.976
Independent review: original vs. A0.8571.0000.976
Independent review: original vs. B0.9290.9761.000

Two annotators independently watched 60 complete videos without seeing the original annotations or each other’s labels. On the candidates used for model evaluation, their event-level Onset F1@3s agreement is 61.02 [42.31, 78.69] on Short, 89.05 [83.21, 94.49] on Medium, and 92.04 [83.76, 98.25] on Long (95% CIs). These scores measure agreement between full-video annotations.

For a matched model comparison, Qwen’s offline Onset F1 on the Short annotation candidates is 40.00 with the original labels, 50.98 with A, and 38.46 with B (Table R3-2). The paper’s 59.11 was computed on the full Short evaluation set. The independent references change scores in both directions, and we will report this sensitivity alongside the Short results.

AuthorsOpenReview ↗

Response to Reviewer SECA (2/4)-Weakness 2, Weakness 3 and Question 4

Table R3-2. Full-video offline and causal online Onset F1 on the common manually reviewed subset subsets

SubsetModelOffline: original / A / BOnline: original / A / B
ShortQwen3-VL-8B40.00 / 50.98 / 38.4638.96 / 38.36 / 35.14
ShortInternVL3.5-8B22.73 / 20.00 / 29.2726.09 / 25.23 / 26.79
ShortMiniCPM-V-2.64.76 / 5.26 / 10.2625.49 / 20.41 / 24.24
MediumQwen3-VL-8B47.33 / 46.62 / 49.2134.12 / 33.72 / 35.15
MediumInternVL3.5-8B15.25 / 15.00 / 15.9314.94 / 13.99 / 12.71
MediumMiniCPM-V-2.60.00 / 0.00 / 0.0012.87 / 13.37 / 11.87
MediumVideoLLaMA3-7B0.00 / 0.00 / 0.009.39 / 9.32 / 8.82

We will correct or remove all errors identified through human review before release and retain the original scoring reference separately for reproducibility.Manually reviewing the entire benchmark would require substantial annotation effort, so we will provide a human-reviewed subset. Every candidate in this subset will be checked: invalid positives and negatives will be removed, and unacceptable boundaries will be corrected or the affected records excluded from evaluation.

Reviewer_4MRUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CEDVS-Bench, a benchmark for streaming event detection. In this task, a model monitors a set of language-specified events and decides every second whether each event has started, ended, or neither. Beyond onset accuracy, the evaluation also covers decision timeliness, response availability, and false alarms on absent events. The authors compare recent-window VLMs with native streaming systems and analyze how performance varies with video length, event duration, negative difficulty, and several other factors.

Critical strengths
  1. The problem formulation is well motivated. The paper measures onset localization, timely decisions, response availability, and absent-event rejection separately, rather than using one offline localization score as a proxy for continuous monitoring.

  2. The paper is careful and transparent about how the benchmark was built and evaluated. The paper provides detailed descriptions of positive-event extraction, negative-event generation, visual verification, human spot checks, parsing rules, service-time simulation, and differences between native-model interfaces. It also clearly states that these native interfaces should not be used to produce a single ranking of response efficiency.

Critical weaknesses
  1. The benchmark labels still carry measurable construction uncertainty. Positive events start from caption timelines and Gemini event extraction before visual verification, and some negative candidates are also generated by Gemini. In the retained audit sample, positive validity is 93.08% and negative validity is 97.70% (Table 3). The 90.06% figure in Table B.1 is an accept-or-correct rate for determinate boundary judgments, not agreement between independently produced annotations. Since the audit covers 110 videos rather than the full 2,840-video benchmark, these figures do not provide a direct full-dataset error estimate;

  2. The paper also does not report inter-annotator agreement. This matters because both the positive/negative labels and the onset tolerances affect the reported model rankings.

  3. Cross-subset comparisons mix duration with several other changes. CEDVS-Short, Medium, and Long draw on different source collections and have different candidate counts, event distributions, and duration ranges. The fixed-video horizon analysis on CEDVS-Long helps isolate exposure time, but it does not remove the confounding in comparisons between the three benchmark subsets. In particular, the high FAR and near-zero response F1 reported for CEDVS-Long shared-window runs may reflect source, candidate load, event density, duration, and service-time effects together. Claims about a general “long-video” degradation should therefore be stated cautiously.

Questions for authors
  1. Can you provide full benchmark label-quality statistics or stratified human-review results, together with independent annotator agreement for temporal boundaries? Please also explain why 1-, 2-, and 3-second tolerances are appropriate given the boundary uncertainty.

  2. Easy negative candidates are sampled from other videos and do not receive the same full-video visual verification as Medium and Hard candidates. How are occurred, absent, and unresolved candidates represented in the final export, and can you release verification status and provenance for every candidate?

  3. Do the main-table model differences have video-paired confidence intervals or significance tests, and can you provide true end-to-end online measurements for response F1 despite the different streaming interfaces? If not, how should the comparative response-efficiency claims be restricted?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4MRU (1/3)

Weakness 1, Weakness 2 and Question 1: Independent Human Annotation, Annotation Sensitivity, and Temporal Tolerances

Response1: Thank you for asking for independent human validation and a justification of the time tolerances. We conducted a stratified annotation of 60 videos and examined how different reference annotations and onset tolerances affect the model results.

We sampled 20 videos from each subset, stratifying Short and Long by duration and candidate count, and Medium by candidate-count quartile. We selected up to four positive and six negative candidates per video. Two annotators watched each video in full and independently labelled the sampled candidates without seeing the original annotations or each other’s labels. Overall, they agreed on event occurrence for 95.25% of candidates (κ = 0.893), and 95.27% of their paired onset annotations differed by no more than 3 seconds (Table R2-1).

Table R2-1. Overall results of the independent 60-video annotation

CheckResult [95% CI, where applicable]
Occurrence agreement (%)95.25 [93.76, 96.85]
Cohen κ0.893 [0.859, 0.929]
Positive occurrence validity — A (%)93.78 [90.82, 96.43]
Positive occurrence validity — B (%)92.23 [89.05, 95.31]
Negative occurrence validity — A (%)99.44 [98.59, 100.00]
Negative occurrence validity — B (%)98.59 [97.46, 99.44]
Paired onsets within 3 s (%)95.27 [92.70, 97.63]
Onset discrepancy: median / P95 (s)0.00 / 2.60

The 90.06% previously reported in Table B.1 was the acceptance rate of existing boundaries. The new annotation measures agreement between independently produced annotations. This agreement varies by subset: paired onsets agree within 3 seconds in 86.21% of cases on Short, 95.31% on Medium, and 98.68% on Long (R2-1a).

Table R2-1a: Inter-Annotator Agreement on Event Occurrence and Onset Timing

SubsetOccurrence agreement (%) [95% CI]Cohen κPaired onsets within 3 s (%) [95% CI]
Short89.44 [85.44, 93.29]0.71586.21 [77.78, 95.45]
Medium96.30 [93.65, 98.93]0.91995.31 [90.62, 98.55]
Long98.98 [97.50, 100.00]0.97998.68 [96.05, 100.00]

Keeping model predictions fixed, we then recomputed Onset F1 at 3 seconds on the manually reviewed candidates using the original annotations and each annotator’s reference (A and B; Table R2-1b). Qwen remains first on Short and Medium under all three references. On Long, Qwen and LongVA exchange first place, with their score difference remaining below 1.3 percentage points. Spearman correlations with the original rankings range from 0.857 to 1.000. The largest individual score change is 5.1 percentage points, for MiniCPM-V on Short under reference A.

Table R2-1b. Fixed predictions under original, A and B references

ModelShort: original / A / BMedium: original / A / BLong: original / A / B
Qwen3-VL-8B38.96 / 38.36 / 35.1434.12 / 33.72 / 35.1510.59 / 11.63 / 10.78
InternVL3.5-8B26.09 / 25.23 / 26.7914.94 / 13.99 / 12.716.74 / 7.78 / 6.86
MiniCPM-V-2.625.49 / 20.41 / 24.2412.87 / 13.37 / 11.872.01 / 1.99 / 2.03
LLaVA-OneVision-1.5-8B23.81 / 22.95 / 22.7619.83 / 19.66 / 18.505.17 / 5.98 / 5.24
LLaVA-NeXT-Video-7B15.47 / 14.69 / 16.858.66 / 8.60 / 9.566.64 / 7.51 / 6.73
VideoLLaMA3-7B25.00 / 24.19 / 27.209.39 / 9.32 / 8.824.07 / 4.03 / 4.12
LongVA-7B22.06 / 21.21 / 21.059.60 / 9.52 / 9.8011.63 / 11.36 / 12.05
LongVU-7B29.33 / 25.35 / 27.7818.28 / 19.15 / 17.689.40 / 10.60 / 9.59

Table R2-1c. Ranking consistency under independent annotations

Comparison across eight modelsShortMediumLong
Spearman ρ: original vs. A0.8571.0000.976
Spearman ρ: original vs. B0.9290.9761.000
Qwen rank: original / A / B1 / 1 / 11 / 1 / 12 / 1 / 2

For the choice of tolerance, the annotators’ paired onsets agree within 1, 2, 3, and 5 seconds in 90.53%, 94.67%, 95.27%, and 95.86% of cases, respectively. We retain 3 seconds as the main tolerance: widening it to 5 seconds increases pooled agreement by only 0.59 percentage points. The 1- and 2-second results show performance under stricter timing requirements, while 5 seconds provides an additional sensitivity check. On the original main-table evaluation records, Qwen has the highest Onset F1 at every tested tolerance on all three subsets (Table R2-2).

AuthorsOpenReview ↗

Response to Reviewer 4MRU (1/3)-Weakness 1, Weakness 2 and Question 1

Table R2-2. Onset tolerance sensitivity on the original main-table evaluation records

SubsetModel1 s2 s3 s (Table 4)5 s
ShortQwen3-VL-8B30.0137.0541.7547.80
ShortInternVL3.5-8B18.8521.1723.1927.32
ShortMiniCPM-V-2.617.7221.2424.2127.16
ShortLLaVA-OneVision-1.5-8B17.0220.5223.9628.73
ShortLLaVA-NeXT-Video-7B9.7611.0612.1413.79
ShortVideoLLaMA3-7B16.1019.3722.0926.20
ShortLongVA-7B11.5513.7715.7318.35
ShortLongVU-7B23.9928.4932.7538.12
MediumQwen3-VL-8B10.8514.9219.8826.18
MediumInternVL3.5-8B5.236.697.969.65
MediumMiniCPM-V-2.64.616.016.767.83
MediumLLaVA-OneVision-1.5-8B5.958.0910.0712.55
MediumLLaVA-NeXT-Video-7B3.814.094.224.59
MediumVideoLLaMA3-7B4.204.965.296.01
MediumLongVA-7B4.806.117.078.90
MediumLongVU-7B6.889.0610.2412.07
LongQwen3-VL-8B4.886.417.789.40
LongInternVL3.5-8B2.753.363.924.53
LongMiniCPM-V-2.61.682.202.513.14
LongLLaVA-OneVision-1.5-8B2.513.153.724.59
LongLLaVA-NeXT-Video-7B2.432.522.592.78
LongVideoLLaMA3-7B2.402.863.103.57
LongLongVA-7B3.804.224.555.41
LongLongVU-7B2.572.953.334.03

A full manual review of the benchmark would require substantial annotation effort. We will therefore release a human-reviewed subset with identified errors corrected or removed, together with the independent annotations, candidate verification status, and construction pipeline. The reported quality statistics will be identified as results from the manually reviewed sample.

AuthorsOpenReview ↗

Response to Reviewer 4MRU (2/3)

Weakness 3:Confounding across Subsets and Observation Time within the Same Videos

Response 2: Thank you for pointing out the confounding in the subset comparisons. We will distinguish the accumulation of false reports during longer observation of the same videos from performance differences across the three subsets.

We compared the same CEDVS-Long videos and event candidates over the first 30 seconds, the first 150 seconds, and the full video (Table R2-3). Qwen’s cumulative FAR increases from 14.65% to 68.16%, while its false STARTs per query-minute decrease from 0.314 to 0.051. InternVL shows the same pattern. More absent candidates therefore receive at least one false report as observation continues, even though the average number of false reports per query-minute decreases.

Table R2-3. Exposure within the same Long recordings

HorizonEligible onsetsDecision F1: Qwen / InternVLFAR: Qwen / InternVLFalse STARTs/query-min: Qwen / InternVL
First 30 s58814.04 / 7.5014.65 / 29.330.314 / 0.587
First 150 s14459.62 / 4.1934.73 / 68.000.153 / 0.272
Full video56967.00 / 3.1868.16 / 85.920.051 / 0.056

At each horizon, reports are cut off by decision time, and later reference onsets are excluded from scoring rather than relabelled as absent. The eligible onset counts therefore increase from 588 to 1,445 and 5,696, so the Decision F1 values are computed over different event sets. We will make this scoring rule explicit in the table caption.

Short, Medium, and Long also differ in data source, candidate count, event density, and model processing time. We will revise the main discussion and conclusion to describe CEDVS-Long’s high FAR and low Response F1 under these combined conditions, without attributing the differences to video length alone.

Question 2: Candidate Verification Status, Provenance, and Release Format

Response3: Thank you for asking us to clarify how candidates are verified and represented in the release. We will provide verification status and provenance for every candidate, alongside its occurrence label and whether it is included in scoring.

Easy negatives are sampled from other videos without automatic visual verification. Medium and Hard negatives receive checks using sampled frames. Across all three subsets, only negatives marked kept entered evaluation; removed, unsure, and records missing required verification results were excluded. For Easy candidates, kept indicates retention without necessarily implying a visual check.

In the independent annotation of 109 Easy negatives, annotator A judged 108 absent and one present, while annotator B judged all 109 absent. The disputed candidate came from CEDVS-Medium. Excluding it changes Onset F1 by at most 0.20 percentage points and leaves all eight model rankings unchanged on the matched Medium manually reviewed subset (Table R2-4).

Table R2-4. Sensitivity to independently flagged Easy negatives

ModelOriginal OnsetFilter by AFilter by BΔ A (pp)Rank: orig. / A / BFAR: orig. → A
Qwen3-VL-8B34.1234.3234.120.201 / 1 / 130.83 → 30.25
InternVL3.5-8B14.9415.0014.940.064 / 4 / 485.00 → 84.87
MiniCPM-V-2.612.8712.9012.870.045 / 5 / 590.83 → 90.76
LLaVA-OneVision-1.5-8B19.8319.9119.830.092 / 2 / 275.00 → 74.79
LLaVA-NeXT-Video-7B8.668.708.660.038 / 8 / 8100.00 → 100.00
VideoLLaMA3-7B9.399.429.390.037 / 7 / 798.33 → 98.32
LongVA-7B9.609.649.600.046 / 6 / 677.50 → 77.31
LongVU-7B18.2818.3818.280.103 / 3 / 354.17 → 53.78

The release will distinguish occurrence judgments (occurred, absent, or unresolved) from verification status and scoring eligibility. Metadata will include original labels, available automatic and human judgments, and provenance, with unchecked Easy candidates marked not_checked. We will retain both annotators’ judgments and the original scoring reference, and provide a corrected reference excluding records identified as invalid or unresolved. The manually reviewed scoring subset will contain only candidates whose occurrence labels are agreed upon or resolved through adjudication.

Reviewer_29cuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CEDVS-Bench, a benchmark for continuous detection of language-defined events in causal video streams. Event candidates are specified before observation, and a model must emit START/END transitions or remain silent at every second. The benchmark contains 2,840 videos, 15,983 positive event instances, and 66,158 absent candidates over short, medium, and long settings. It evaluates onset localization, decision timeliness, simulated processing-aware response timeliness, and false alarms on absent events. Experiments cover eight window-based VLMs and three native streaming models and reveal substantial gaps between recognition, timely reporting, and sustained rejection.

Critical strengths
  1. The task is important and meaningfully distinct from retrospective localization or occasional streaming QA. Persistent pre-specified event interests, explicit absent candidates, and per-second causal decisions jointly capture the practical "speak or stay silent" requirement.

  2. The evaluation is unusually careful about systems behavior. It separates predicted onset, decision time, and FIFO-simulated completion time; includes input preparation, inference, parsing, and state updates in service time; reports false-alarm rates with recall; and documents important interface and comparability limitations. The benchmark scale and the short-to-long exposure analysis make the failure modes consequential.

Critical weaknesses
  1. Positive events and boundaries are generated from two-frame-per-two-second captions rather than independently annotated video, and the retained human audit still finds 34 of 506 positive records invalid, 42 of 1,830 determinate negatives invalid, and 48 of 483 determinate boundaries unacceptable. The paper explicitly notes that human-rejected records are not necessarily removed and that accepted boundaries are not independently re-annotated. This noise is material when the primary tolerances are only 1--3 seconds and the source construction itself uses a two-second grid. A benchmark intended to distinguish localization from timeliness should correct all known errors and more extensively validate onset boundaries.

  2. Onset and decision metrics do not use a common matching rule. OED uses greedy nearest-onset matching, whereas DT/SRT use a lexicographic dynamic program, so DT can be numerically higher than OED (as already occurs in the main table). This prevents the intended localization-to-decision gap from being interpreted as the cost of causal delay alone. In addition, native streaming models use different candidate presentation, cadence, state, and timing instrumentation, so the headline multi-family table is descriptive rather than a controlled benchmark ranking

  3. The omission of LiveStar is notable here: it directly models the response-versus-silence decision through semantic verification and evaluates both response content and timing in continuous online video. While CEDVS-Bench imposes persistent language-defined event candidates and explicit START/END transitions, evaluating or at least discussing it would more clearly establish what the benchmark adds beyond existing proactive response-timing settings. The core metric decomposition should use one shared matching objective, with model-family results clearly separated until interfaces are standardized.

Questions for authors
  1. Will all human-identified invalid positives, invalid negatives, and unacceptable boundaries be corrected or removed before release? What is the resulting score sensitivity when audited corrections are applied?

  2. What do all main conclusions look like when OED, DT, and SRT use the same optimal one-to-one matching objective and differ only in their eligibility constraints?

  3. How stable are model rankings under bootstrap confidence intervals over videos, particularly for the very small response-F1 values and the native-model results?

  4. Since source videos come from Charades, QVHighlights, and CG-Bench, what video-ID/perceptual-hash audit addresses contamination in the evaluated VLMs' pretraining or instruction tuning?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 29cu (1/4)

Weakness 1 and Question 1:Independent Annotation and the Impact of Annotation Corrections

Response1: Thank you for highlighting the need to assess how annotation errors affect the evaluation. Our new analysis shows that correcting known errors has a limited effect on Onset F1 and largely preserves the model rankings on the manually reviewed subsets.We re-scored all eight models with their predictions held fixed, after removing invalid positive and negative records and applying the available boundary corrections. Onset F1 decreases by 0.46–3.48 percentage points on Short and changes by less than 0.6 points on Medium and Long(Table R1-1). Qwen3-VL-8B remains first on all three subsets. The top two models in each subset are unchanged; the remaining rank changes occur between models with close scores.

Table R1-1. Known-error correction across all eight models

SubsetModelOriginal Onset F1 (%)Corrected Onset F1 (%)Δ corrected−original (pp) [95% CI]
ShortQwen3-VL-8B41.9440.38−1.55 [−7.10, 3.07]
ShortInternVL3.5-8B27.5125.61−1.90 [−6.61, 1.86]
ShortMiniCPM-V-2.626.6726.21−0.46 [−3.96, 2.49]
ShortLLaVA-OneVision-1.5-8B26.4324.12−2.31 [−6.03, 0.62]
ShortLLaVA-NeXT-Video-7B13.7711.59−2.19 [−4.83, −0.23]
ShortVideoLLaMA3-7B24.6421.16−3.48 [−7.68, 0.04]
ShortLongVA-7B16.5215.14−1.38 [−4.08, 0.58]
ShortLongVU-7B38.0335.77−2.26 [−7.36, 1.94]
MediumQwen3-VL-8B23.9923.55−0.44 [−1.39, 0.36]
MediumInternVL3.5-8B10.5510.16−0.39 [−1.09, 0.14]
MediumMiniCPM-V-2.67.336.94−0.38 [−0.98, 0.19]
MediumLLaVA-OneVision-1.5-8B11.3211.09−0.23 [−1.06, 0.50]
MediumLLaVA-NeXT-Video-7B4.494.59+0.10 [−0.50, 0.65]
MediumVideoLLaMA3-7B6.015.77−0.24 [−0.85, 0.28]
MediumLongVA-7B6.015.73−0.27 [−0.78, 0.10]
MediumLongVU-7B12.8912.31−0.59 [−1.55, 0.18]
LongQwen3-VL-8B10.0410.27+0.22†
LongInternVL3.5-8B2.192.28+0.09†
LongMiniCPM-V-2.62.041.82−0.22†
LongLLaVA-OneVision-1.5-8B3.333.24−0.09†
LongLLaVA-NeXT-Video-7B0.860.89+0.03†
LongVideoLLaMA3-7B3.723.71−0.01†
LongLongVA-7B3.152.75−0.39†
LongLongVU-7B1.751.85+0.09†

† Long is a descriptive sensitivity check.

The corrected Onset F1@3 s rankings have Spearman correlations of 0.976, 1.000, and 0.976 with the original rankings on Short, Medium, and Long, respectively(Table R1-1a). Qwen3-VL-8B remains first, and the top two models retain their positions in every subset.

Table R1-1a. Ranking consistency before and after known-error correction

Comparison across eight modelsShortMediumLong
Spearman ρ: original vs. corrected0.9761.0000.976
Qwen rank: original → corrected1 → 11 → 11 → 1
Top-two ordering preservedYesYesYes

To check the onset boundaries independently, two annotators watched 60 videos in full, 20 from each subset, and labelled the sampled candidates without seeing the original annotations or each other’s labels. They agreed on whether an event occurred for 95.25% of candidates (Cohen’s κ = 0.893), and 95.27% of their paired onset annotations differed by no more than 3 seconds (Table R1-2).

Table R1-2. Independent Annotation: agreement and remaining sensitivity

CheckResult
Occurrence agreement (%)95.25 [93.76, 96.85]
Cohen κ0.893 [0.859, 0.929]
Paired onsets within 3 s (%)95.27 [92.70, 97.63]

Before release, we will correct or remove all errors already identified by the human reviews. Manually reviewing the entire benchmark would require substantial annotation effort, so we will provide a human-reviewed subset. Every candidate in this subset will be checked: invalid positives and negatives will be removed, and unacceptable boundaries will be corrected or the affected records excluded from evaluation.

AuthorsOpenReview ↗

Response to Reviewer 29cu (2/4)

Weakness 2 and Question 2: Unified Matching and Interpretation of Metric Differences

Resopnse2: Thank you for pointing out the inconsistent matching rules and the need to clarify the metric differences. Re-evaluation with a shared matching objective leaves all 72 F1 scores and the model rankings unchanged for the eight shared-window models.

We now use the same optimal one-to-one matching objective for all three metrics: first maximize the number of eligible matches, then minimize total absolute onset error. The metrics differ only in their matching conditions. Onset allows the reported onset to fall within 3 seconds of the reference onset. Decision requires the decision time to fall within 0–3 seconds after the reference onset. Response requires a decision at or after the reference onset and simulated completion within the same 0–3-second window. The new rule changes 656 matched pairs but leaves TP, FP, and FN unchanged for every candidate. Consequently, all 72 F1 scores remain unchanged to two decimal places (Table R1-3).

Table R1-3. Submitted Table 4 scores under original and unified matching

SubsetModelOnset: original → unifiedDecision: original → unifiedResponse: original → unified
ShortQwen3-VL-8B41.75 → 41.7539.03 → 39.0314.62 → 14.62
ShortInternVL3.5-8B23.19 → 23.1921.42 → 21.425.38 → 5.38
ShortMiniCPM-V-2.624.21 → 24.2119.40 → 19.403.91 → 3.91
ShortLLaVA-OneVision-1.5-8B23.96 → 23.9622.09 → 22.091.65 → 1.65
ShortLLaVA-NeXT-Video-7B12.14 → 12.1410.86 → 10.861.82 → 1.82
ShortVideoLLaMA3-7B22.09 → 22.0918.88 → 18.882.81 → 2.81
ShortLongVA-7B15.73 → 15.7312.49 → 12.491.31 → 1.31
ShortLongVU-7B32.75 → 32.7527.73 → 27.734.26 → 4.26
MediumQwen3-VL-8B19.88 → 19.8821.05 → 21.059.75 → 9.75
MediumInternVL3.5-8B7.96 → 7.967.74 → 7.740.67 → 0.67
MediumMiniCPM-V-2.66.76 → 6.765.13 → 5.130.04 → 0.04
MediumLLaVA-OneVision-1.5-8B10.07 → 10.079.94 → 9.940.03 → 0.03
MediumLLaVA-NeXT-Video-7B4.22 → 4.224.00 → 4.00< 0.01 → < 0.01
MediumVideoLLaMA3-7B5.29 → 5.294.41 → 4.410.00 → 0.00
MediumLongVA-7B7.07 → 7.075.89 → 5.89< 0.01 → < 0.01
MediumLongVU-7B10.24 → 10.248.00 → 8.000.01 → 0.01
LongQwen3-VL-8B7.78 → 7.787.00 → 7.000.06 → 0.06
LongInternVL3.5-8B3.92 → 3.923.18 → 3.180.04 → 0.04
LongMiniCPM-V-2.62.51 → 2.511.82 → 1.820.03 → 0.03
LongLLaVA-OneVision-1.5-8B3.72 → 3.723.35 → 3.350.00 → 0.00
LongLLaVA-NeXT-Video-7B2.59 → 2.592.50 → 2.500.00 → 0.00
LongVideoLLaMA3-7B3.10 → 3.102.51 → 2.510.00 → 0.00
LongLongVA-7B4.55 → 4.554.02 → 4.020.00 → 0.00
LongLongVU-7B3.33 → 3.332.70 → 2.700.00 → 0.00

The interpretation of the metric differences still needs clarification. A model can make a timely detection while reporting an inaccurate onset timestamp. Such a prediction can count towards Decision F1 without counting towards Onset F1. This explains why Decision F1 can exceed Onset F1 even under unified matching, as it does for Qwen3-VL-8B on Medium (21.05 vs. 19.88). We will revise the text to describe onset accuracy, decision timeliness, and simulated response timeliness separately, and remove the interpretation of the Onset–Decision gap as the cost of causal delay alone.

We will also present shared-window and native streaming models in separate tables, since their input interfaces, update frequencies, and timing protocols differ. Comparisons across these two groups will be described as observations under their respective settings.

AuthorsOpenReview ↗

Response to Reviewer 29cu (3/4)

Weakness 3: Relation to LiveStar and Our Task Contribution

Response3: Thank you for highlighting LiveStar. We will add it to the related work and clarify the differences in task setup and evaluation between OmniStar and CEDVS-Bench.

LiveStar studies when to respond or remain silent in a video stream. OmniStar includes online temporal grounding with start/end boundaries and repeated occurrences of the same query. In its reported OTG evaluation, intervals of specified lengths are ranked by average query perplexity, and the selected interval is evaluated using TimDiff (Appendix D.2).

CEDVS-Bench asks models to monitor a fixed set of event candidates throughout the video, including candidates that never occur, and issue START/END reports as the stream progresses. We score missed and extra detections, distinguish onset accuracy from decision and simulated response timeliness, and separately measure false reports on absent candidates. This evaluates whether a model can detect the requested events promptly while avoiding reports for events that do not occur. We will make this comparison explicit in the revised paper.We also use the shared matching objective described above and will present shared-window and native streaming models in separate tables.

Question 3:Ranking Stability and Confidence Intervals for Native Streaming Models

Response4: Thank you for asking how stable the model rankings are. The paired bootstrap results support Qwen3-VL-8B’s lead in Onset and Decision F1, while some differences in Long Response F1 and the native models’ Decision F1.

We computed 95% confidence intervals for the submitted Table 4 scores using 2,000 bootstrap resamples over videos (Table R1-4). Qwen3-VL-8B has the highest Onset and Decision F1 on all three subsets. On Long, all eight Response F1 scores are at most 0.06%, and five round to 0.00%. Below, F1 scores are reported as percentages and paired differences as percentage points.

Table R1-4. Submitted Table 4 scores with video-bootstrap confidence intervals

SubsetModelOnset F1 [95% CI]Decision F1 [95% CI]Response F1 [95% CI]
ShortQwen3-VL-8B41.75 [40.48, 43.01]39.03 [37.70, 40.35]14.62 [13.39, 15.91]
ShortInternVL3.5-8B23.19 [22.46, 23.97]21.42 [20.63, 22.24]5.38 [4.77, 6.00]
ShortMiniCPM-V-2.624.21 [23.28, 25.23]19.40 [18.57, 20.30]3.91 [3.39, 4.47]
ShortLLaVA-OneVision-1.5-8B23.96 [23.19, 24.78]22.09 [21.29, 22.95]1.65 [1.34, 2.00]
ShortLLaVA-NeXT-Video-7B12.14 [11.84, 12.46]10.86 [10.54, 11.19]1.82 [1.57, 2.07]
ShortVideoLLaMA3-7B22.09 [21.12, 23.16]18.88 [17.91, 19.84]2.81 [2.26, 3.36]
ShortLongVA-7B15.73 [15.23, 16.22]12.49 [12.02, 13.01]1.31 [1.08, 1.56]
ShortLongVU-7B32.75 [31.53, 33.94]27.73 [26.53, 28.96]4.26 [3.64, 4.86]
MediumQwen3-VL-8B19.88 [19.25, 20.54]21.05 [20.38, 21.76]9.75 [9.23, 10.28]
MediumInternVL3.5-8B7.96 [7.71, 8.22]7.74 [7.46, 8.03]0.67 [0.56, 0.78]
MediumMiniCPM-V-2.66.76 [6.52, 7.00]5.13 [4.93, 5.35]0.04 [0.02, 0.06]
MediumLLaVA-OneVision-1.5-8B10.07 [9.74, 10.40]9.94 [9.60, 10.31]0.03 [0.01, 0.06]
MediumLLaVA-NeXT-Video-7B4.22 [4.10, 4.34]4.00 [3.88, 4.12]< 0.01 [0.000, 0.009]
MediumVideoLLaMA3-7B5.29 [5.11, 5.46]4.41 [4.24, 4.56]0.00 [0.00, 0.00]
MediumLongVA-7B7.07 [6.78, 7.36]5.89 [5.65, 6.14]< 0.01 [0.000, 0.012]
MediumLongVU-7B10.24 [9.79, 10.73]8.00 [7.60, 8.40]0.01 [0.00, 0.02]
LongQwen3-VL-8B7.78 [7.06, 8.61]7.00 [6.26, 7.79]0.06 [0.016, 0.113]
LongInternVL3.5-8B3.92 [3.61, 4.29]3.18 [2.90, 3.49]0.04 [0.008, 0.082]
LongMiniCPM-V-2.62.51 [2.21, 2.86]1.82 [1.58, 2.08]0.03 [0.000, 0.071]
LongLLaVA-OneVision-1.5-8B3.72 [3.29, 4.13]3.35 [2.94, 3.75]0.00 [0.000, 0.000]
LongLLaVA-NeXT-Video-7B2.59 [2.39, 2.80]2.50 [2.31, 2.72]0.00 [0.000, 0.000]
LongVideoLLaMA3-7B3.10 [2.73, 3.49]2.51 [2.25, 2.79]0.00 [0.000, 0.000]
LongLongVA-7B4.55 [4.04, 5.07]4.02 [3.55, 4.50]0.00 [0.000, 0.000]
LongLongVU-7B3.33 [2.98, 3.69]2.70 [2.39, 3.03]0.00 [0.000, 0.000]

Comments on the paper

1 comment

Dear PCs, SACs, and ACs,

We thank the reviewers for their constructive feedback. We summarize the paper’s contributions and key findings, followed by the additional evidence and revisions addressing their main concerns.

Strengths and key findings:

  1. A practical streaming task. Models monitor a fixed set of language-defined events, report START/END transitions using only past frames, and remain silent when events are absent. All three reviewers recognize the importance of evaluating these capabilities jointly. (All reviewers)

  2. Explicit evaluation of absent-event rejection. CEDVS-Bench covers 2,840 videos, with 15,983 positive instances and 66,158 absent candidates. Hard negatives test whether models distinguish the specified action from a similar scene or actor. (29cu, SECA)

  3. Separate measures of localization and timeliness. The evaluation distinguishes reported event onsets, decision times, and FIFO-simulated response availability after processing. SECA describes the processing-aware metric as a “genuine contribution.” (All reviewers)

  4. Gaps between recognition and reliable reporting. Better recognition does not always yield better localization, and timely decisions can still arrive too late once processing costs are included. (Sections 4.2–4.4)

  5. Findings beyond model rankings. Within the same videos, cumulative false-alarm risk rises with exposure even as alarms per query-minute decrease. More frames or separate candidate calls can also hurt detection for some models. (Section 4.5; Appendix G)

Main concerns and our responses:

  1. Annotation quality and boundary reliability. (All reviewers)
    We corrected known annotation errors and tested score sensitivity on the audited subsets, and will release the corrected annotations. Two annotators independently reviewed 60 videos and agreed on event occurrence in 95.25% of cases; 95.27% of their paired onset times differed by no more than 3 seconds. (Appendix I.1)

  2. Event semantics and caption-like segments. (SECA)
    We clarify onset semantics and report separate results for actions/transitions, activities/states, and editing/topic segments. Detection remains challenging when restricted to discrete actions and transitions. (Appendix I.3)

  3. Absence verification and provenance. (4MRU, SECA)
    Both annotators judged all 119 sampled Long negatives absent through full-video review. We clarify verification and export rules and will release candidate-level provenance and human judgments. (Appendix I.1; response to SECA Q2)

  4. Matching consistency and statistical reliability. (29cu, 4MRU)
    Unified optimal matching preserves main-table F1 scores. We clarify metric interpretations and report paired video-bootstrap confidence intervals to assess the reliability of model differences. (Section 3.2; Appendix I.2)

  5. Live response timing and interface comparability. (All reviewers)
    FIFO replay reproduces live TP/FP/FN counts on 30 videos under reduced candidate load. Native-model results are reported separately, with interface differences and comparison limits documented. (Section 4.7; Appendices D–E)

  6. Checks on annotation bias and training-data overlap. (29cu, SECA)
    We tested annotation-source bias by rebuilding 60 videos without Qwen and reevaluating six models; Qwen’s lead persists. A separate audit of public training manifests identifies source-video overlap on Short, which we disclose in our response. (Appendix I.4; response to 29cu Q4)

  7. Interpreting differences across subsets. (4MRU)
    We clarify that the subsets differ in video sources and candidate loads as well as duration. We examine false-alarm accumulation by comparing different observation lengths within the same videos. (Section 4.5, Table 7)

  8. Relationship to LiveStar. (29cu)
    We added LiveStar to related work and clarified our complementary focus on persistent candidates, explicit event transitions, and sustained absent-event rejection. (Section 2)

Our responses address the main concerns about annotation quality, event definitions, metric consistency, and construction bias, and clarify the limits of timing and model comparisons. Independent human annotation, additional experiments, and sensitivity analyses provide stronger support for the benchmark’s findings on event detection, timely reporting, and sustained rejection of absent events. The additional validation strengthens the evidence for the benchmark’s reliability, while the revised claims clarify what the reported comparisons establish. We respectfully ask the committee to consider this evidence alongside the task and evaluation contributions recognized across the reviews.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.