acceptodds
Under review as a conference paper at ICLR 2027

CountRoute: Pattern Readout and Evidence-Grounded Counting for Streaming Video Counting

Abstract

Counting in a streaming video is usually treated as a single numerical inference over an increasingly long visual context. We argue that the first decision is how the count can be established. For some events a temporal pattern — the recurrence of a visual state, or an abrupt change of shot — is enough to read the count off directly. For others the count has to be grounded in semantic evidence: occurrences must be found, told apart, and added up. Video language models treat both as one direct prediction, and do poorly where a pattern would have sufficed. Evidence- grounded counting brings a second choice. Inspecting short visual blocks keeps local evidence, but each additional empty block is another opportunity for a false positive, and an event spanning blocks may be counted twice; reading a continuous textual record avoids this block-wise accumulation, but cannot recover events it never wrote down. We introduce CountRoute, a training-free streaming agent built on a query-agnostic memory of the stream. It reads a count directly off a temporal pattern only after verifying that the corresponding signal is present; otherwise it uses the record to select relevant cached blocks, and moves from inspecting their frames to reading the record as the number of selected blocks — and hence the exposure to accumulation error — grows. On SVCBench, CountRoute achieves the best overall score, ahead of existing offline and online methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Counting, Multimodal Large Language Models, Agentic Counting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZcsqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CountRoute, a training-free agent for streaming video counting that chooses between pattern-based readout and evidence-grounded counting. It maintains query-agnostic textual and visual memory, selectively inspects relevant blocks, and switches to record-level counting when sufficient evidence accumulates. On SVCBench, CountRoute achieves 38.4 overall GPA, substantially outperforming existing streaming systems and most offline baselines.

Critical strengths
  1. The paper identifies that different counting questions require different computation strategies, motivating pattern-based versus evidence-grounded readout.

  2. The component study shows consistent gains from block selection, record reading, transition counting, and periodic counting, with overall GPA increasing from 20.2 to 38.4.

  3. CountRoute achieves 38.4 overall GPA, compared with 32.0 for the strongest offline baseline and 18.7 for the strongest listed streaming baseline.

Critical weaknesses
  1. Most components are based on existing techniques; the main contribution is their routing and composition. The paper should better justify why this constitutes a substantial methodological advance.

  2. Key decisions in the routing rely on fixed thresholds. Although sensitivity studies are provided, it remains unclear how well this routing principle generalizes beyond the benchmark.

  3. Although query-time inspection is reduced, preprocessing is substantial: about 3.55s per 4s window, with roughly 2.3× the image inputs of direct 64-frame answering.

Questions for authors
  1. How much of the final improvement comes specifically from the routing policy, compared with simply combining the available readers?

  2. Can you report full end-to-end latency, throughput, memory, and model-call costs?

  3. How does the method perform on natural repetition videos rather than highly structured periodic data?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Zcsq (Part 1 / 2)

Thank you for the positive assessment of the problem decomposition and empirical gains. We address the routing contribution, threshold transfer, computational accounting, and natural-repetition evaluation below.

Q1/W1. Contribution and comparison with simple reader combinations

Counting a target event in a video stream depends on finding evidence of its occurrences. A common baseline uses the same procedure across queries, asking an MLLM to infer the count from frames sampled from the observed video prefix. The evidence needed to establish an occurrence, however, is event-specific. It may require extensive context or be apparent in a brief observation. A salient visual pattern can even give the count directly when its instances correspond one-to-one with event occurrences.

CountRoute uses these differences to decide how to count (Figure 1 of the revised paper). It first checks whether a dedicated pattern readout applies to the queried event. When counting requires context, CountRoute selects relevant blocks and chooses how to integrate the evidence based on how many blocks it selects. When few blocks are selected, it inspects their frames to preserve visual detail. For larger selections, it reads the continuous textual record to reduce double counting across block boundaries.

The controlled ablations in our response to Reviewer 3eQY, Q1/W1 separately quantify the contributions of specialized readouts and adaptive evidence reading. We compare two ways of averaging the same branch predictions: taking the mean of Cuts, Periodic, Inspect, and Read at every query point, or keeping the original pattern readouts and averaging Read and Inspect only on Evidence queries.

PolicyOverall GPA
Mean of all four branches11.55
Keep pattern readouts; mean of Read and Inspect33.43
CountRoute38.39

W2. Switching-threshold sensitivity and transfer

To assess switching-threshold sensitivity and transfer to new data, we built an event-counting evaluation set from EgoLife, a dataset of long-form egocentric recordings of daily life. Our set covers long events and short actions and contains 36 videos, 48 counting questions, and 384 query points. We used the official EgoLife captions to draft the event annotations, then manually verified them. All 48 questions route to Evidence under the submitted router. We vary only N on both datasets and evaluate the submitted N=3 on EgoLife without retuning.

Switching threshold NSVCBench GPAEgoLife-based set GPA
0: textual reading35.5161.73
237.3662.77
3: submitted default38.3967.98
437.6764.85
536.7262.77
∞: selected-block visual accumulation33.9543.24

Q2/W3. Latency and resource costs

Latency. Memory construction and query answering can run concurrently; we evaluate their latency separately. For memory construction, we use two H100 GPUs per stream: one for frame embeddings and captioning, and one for transition verification. In the full 28.0-hour SVCBench replay, memory construction keeps pace with 1× playback. From window availability to saving the memory and transition index, processing averages 2.58 s and takes at most 3.79 s per four-second window. Query answering averages 53.80 s per query point.

Model calls. Memory construction averages 2.24 visual calls and 0 text-only calls per four-second window. Query answering averages 0.84 visual calls and 1.49 text-only calls per query point.

Runtime memory and retained storage. During memory construction over the full 28.0-hour SVCBench replay, the measured peaks were 6.08 GiB of host memory (combined process PSS, including vLLM) and 64.98 GiB of GPU memory. The saved stream memory occupies approximately 2.96 MB per minute of video on disk.

Comparison with baselines using more visual input. We extend the Direct-64 baseline in Table 2 by sampling the available video prefix more densely, with frame caps of 192 and 320. All three settings use Qwen3-VL-8B and the original Table 2 prompts. We report estimated mean VLM visual input tokens per query point, counting memory construction once per video and amortizing it over the evaluated query points, with standalone frame embedding excluded.

MethodVisual tokensGPA
Direct-647,18625.34
Direct-19216,20525.27
Direct-32021,19924.67
CountRoute18,31738.39
AuthorsOpenReview ↗

Response to Reviewer Zcsq (Part 2 / 2)

Q3. Natural repetition beyond structured periodic data

We evaluate CountRoute on all 227 public PushupBench test clips. The periodicity gate routes 212 clips to recurrence counting, 14 to visual inspection, and one to caption-based reading. As in Table 7 of the submitted paper, MAE excludes clips with an absolute error greater than 50.

MethodExact accuracyOff-by-one accuracyMAE
Direct-6410.13%23.79%6.84
CountRoute (Periodic only)29.07%52.86%4.03
CountRoute28.19%51.10%4.66

Routing some clips away from recurrence slightly lowers overall accuracy relative to forced recurrence, while the full system still outperforms Direct-64.

I thank the authors for a detailed rebuttal. Most of my concerns are resolved, except the contribution is primarily a heuristic combination of existing components. I therefore keep my rating at 3.

Reviewer_jcUqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CountRoute, a training-free framework for streaming video counting that routes queries among periodic counting, shot-transition counting, and text-evidence-based counting. By combining video records, visual pattern detection, and block-wise counting, the method achieves an overall GPA of 38.4 on SVCBench. The authors also provide ablations to examine the contributions of individual components.

Critical strengths
  1. Reasonable problem decomposition. The method distinguishes periodic events, shot transitions, and events requiring semantic understanding, addressing different sources of counting errors.

  2. Useful empirical analysis. The paper includes component ablations, parameter studies, computational analysis, and discussion of routing failures.

Critical weaknesses
  1. Limited algorithmic novelty. The method primarily combines existing techniques for periodic detection, shot-transition detection, text-based retrieval, and block-wise counting. It remains unclear whether this combination constitutes a sufficiently distinctive algorithmic contribution.

  2. Insufficient evidence for real-world generalization and real-time operation. SVCBench's periodic subset is constructed by repeating the same short clip, which does not adequately represent variations in real actions. Although PushupBench provides real-video evaluation, it primarily evaluates the periodic-counting component rather than the complete routing system. Furthermore, the real-time analysis assumes that transition confirmations can be processed in parallel. Whether the system can sustain streaming operation under frequent shot changes remains unclear.

Questions for authors

1.What is the core algorithmic novelty of CountRoute compared with existing periodic-counting, shot-transition detection, and block-wise counting methods?

2.Can the complete routing system be evaluated on real periodic-action videos, rather than evaluating the periodic-counting component in isolation?

3 .How does the router ensure that a detected visual pattern semantically corresponds to the event specified by the query? How significantly do routing errors affect the final results?

  1. How does the system handle backlogs when transition confirmations cannot keep up with frequent shot changes? Can the authors report end-to-end results without relying on idealized parallel processing?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jcUq (Part 1 / 2)

Thank you for the questions about methodological distinctiveness, natural repetition, semantic applicability, and serial streaming operation. We address each point below.

Q1/W1. Methodological contribution

Counting a target event in a video stream depends on finding evidence of its occurrences. A common baseline uses the same procedure across queries, asking an MLLM to infer the count from frames sampled from the observed video prefix. The evidence needed to establish an occurrence, however, is event-specific. It may require extensive context or be apparent in a brief observation. A salient visual pattern can even give the count directly when its instances correspond one-to-one with event occurrences.

CountRoute uses these differences to decide how to count (Figure 1 of the revised paper). It first checks whether a dedicated pattern readout applies to the queried event. When counting requires context, CountRoute selects relevant blocks and chooses how to integrate the evidence based on how many blocks it selects. When few blocks are selected, it inspects their frames to preserve visual detail. For larger selections, it reads the continuous textual record to reduce double counting across block boundaries.

Dedicated readouts. On the SVCBench periodic subset, our readout achieves 67.22 GPA without training a dedicated counting model, compared with 66.17 for RepNet, which is pretrained on repetition-counting data. On queries originally routed to Cuts, our readout achieves 68.57 GPA versus 53.66 when its outputs are replaced by PySceneDetect, with routing fixed.

Q2/W2. Complete routing system on natural repetition videos

We evaluate CountRoute on all 227 public PushupBench test clips. The periodicity gate routes 212 clips to recurrence counting, 14 to visual inspection, and one to caption-based reading. As in Table 7 of the submitted paper, MAE excludes clips with an absolute error greater than 50.

MethodExact accuracyOff-by-one accuracyMAE
Direct-6410.13%23.79%6.84
CountRoute (Periodic only)29.07%52.86%4.03
CountRoute28.19%51.10%4.66

Routing some clips away from recurrence slightly lowers overall accuracy relative to forced recurrence, while the full system still outperforms Direct-64.

Q3. Semantic correspondence and impact of routing errors

Semantic alignment with the queried event. For shot-transition counting, the router checks that the question asks for shot changes, and a VLM checks whether candidate changes are shot transitions before they are counted. For periodic counting, before entering this branch, the VLM checks whether the action specified by the question repeats. Query-conditioned key-state selection further identifies a reference state representing that action for recurrence counting. These checks reduce the risk of counting an unrelated pattern.

Routing headroom. SVCBench provides ground-truth counts but no ground-truth routes. We assess the impact of routing choices by comparing CountRoute with two oracle upper bounds, keeping memory and branch predictions fixed. The outer oracle selects the best route per question with the original Evidence policy; the reader oracle keeps outer routes fixed and selects the better reader at each Evidence query point.

SettingGPAGain over CountRoute
CountRoute38.39—
Oracle outer routing43.38+4.99
Oracle evidence-reader selection46.04+7.65

With branch predictions and the other routing decision fixed, the corresponding oracle bounds the score of any admissible correction. The GPA loss due to routing errors is the GPA after correction minus CountRoute GPA, so

GPA_after_correction − CountRoute_GPA ≤ Oracle_GPA − CountRoute_GPA.

AuthorsOpenReview ↗

Response to Reviewer jcUq (Part 2 / 2)

Q4/W2. Single-GPU serial replay and verification backlogs

Serial memory construction. With the deadline policy below, the full memory construction pipeline, from frame decoding to saving each memory update, keeps pace with 1× playback on 28.0 hours of SVCBench video. Each stream uses one H100 GPU, with embedding extraction, captioning, and transition verification executed serially on shared decoded frames. Processing takes 2.90 s on average and at most 3.89 s per four-second window. Once the memory for a requested prefix is ready, query answering can run concurrently with memory construction for later windows. The replay timings measure memory construction.

Handling verification backlogs. Under fully serial execution, frequent shot changes can produce candidates faster than they can be verified, delaying subsequent memory updates. For single-GPU deployment, we add a per-window verification deadline to the full-verification pipeline, reserving 0.5 seconds for finalizing memory updates. At the deadline, we cancel any active verification, retain all completed positive and negative decisions, and count only the interrupted and unprocessed candidates using a higher candidate-score threshold of 3, without further VLM calls. Verification resumes in the next window, so unfinished candidates do not accumulate across windows.

Fallback frequency and accuracy. In this run, 28.48% of candidates use the fallback. We compare its Overall GPA on SVCBench with full verification and candidate-only counting.

Transition-counting configurationOverall GPA
Full-verification reference38.39
Verification with deadline-triggered fallback37.83
Candidate-only fallback36.96

Parallel verification. As reported in our responses to Reviewer 3eQY (Q3/W2) and Reviewer Zcsq (Q2/W3), parallel verification on two H100 GPUs kept every window within the 4 s budget across the full SVCBench replay. Every candidate was verified without fallback, reproducing the full-verification results.

Thank you for the detailed responses and additional experiments. I appreciate the efforts to address my concerns, particularly the single-GPU streaming evaluation and the experiments on natural repetition videos. However, the methodological contribution still appears largely based on combining existing components through heuristic routing (i have the same concern with Reviewer Zcsq), and the PushupBench evaluation mainly validates the periodic counting branch rather than the full routing framework under diverse real-world streaming conditions. While the rebuttal has improved my confidence in the system's practical feasibility, some concerns regarding its methodological distinctiveness and generalizability remain. Therefore, I would maintain my original rating of 2.

Reviewer_3eQYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CountRoute, a training-free framework for streaming video counting.CountRoute distinguishes between pattern-readable counting and evidence-grounded counting.For pattern-readable questions, the system employs dedicated readouts based on periodic visual-state recurrence or scene transitions.For evidence-grounded questions, the system maintains a query-agnostic textual record and a cache of video blocks. It uses the record to select relevant blocks and switches between visual block-wise counting and text-based counting according to the number of selected blocks. Experiments on SVCBench show improvements over the evaluated baselines, and additional analyses investigate periodic counting and the effect of selecting blocks in videos with sparse events.

Critical strengths

(1) A clear and intuitive analysis of different counting mechanisms. The distinction between pattern-readable and evidence-grounded counting is intuitive and provides a useful perspective on streaming video understanding. (2) A useful analysis of error accumulation in block-wise counting. The paper identifies two important failure modes of dividing a video into blocks and summing local counts.These observations are well motivated and supported by the experimental analysis. (3) Strong performance on the evaluated benchmark. CountRoute achieves an overall GPA of 38.4 on SVCBench, compared with 32.0 for the strongest reported baseline in the main comparison.The component-wise ablation studies also demonstrate that the proposed system improves as different counting mechanisms are introduced.

Critical weaknesses

(1) The contribution is primarily a heuristic combination of existing components. This is my main concern. The framework combines several established mechanisms: periodicity-based counting, scene transition detection, language-model-based evidence selection, visual block-wise counting, counting from textual descriptions and threshold-based routing. Although their integration is reasonable, the contribution appears closer to system engineering than to a general machine learning advance. (2) The computational cost and streaming efficiency are insufficiently characterized. Although CountRoute is training-free, it requires continuous video processing, caption generation, frame caching, transition verification, and additional language-model calls. The paper does not sufficiently establish whether these costs are justified by the accuracy improvements. In particular, comparisons against baselines with similar computational budgets would be more informative than comparisons based primarily on counting accuracy.

Questions for authors

(1) How much of the improvement comes from using specialized counting tools rather than from the proposed routing strategy itself? (2) How sensitive is the visual/textual switching rule to its threshold, and does the optimal threshold vary across datasets? (3) What is the total computational cost of maintaining the query-agnostic memory, including video encoding and caption generation?

1 comment

Thank you for recognizing the counting-mechanism decomposition and the analysis of block-wise counting errors. We address the contribution, attribution, threshold transfer, and computational-cost concerns below.

Q1/W1. Contribution and attribution of gains

Counting a target event in a video stream depends on finding evidence of its occurrences. A common baseline uses the same procedure across queries, asking an MLLM to infer the count from frames sampled from the observed video prefix. The evidence needed to establish an occurrence, however, is event-specific. It may require extensive context or be apparent in a brief observation. A salient visual pattern can even give the count directly when its instances correspond one-to-one with event occurrences.

CountRoute uses these differences to decide how to count (Figure 1 of the revised paper). It first checks whether a dedicated pattern readout applies to the queried event. When counting requires context, CountRoute selects relevant blocks and chooses how to integrate the evidence based on how many blocks it selects. When few blocks are selected, it inspects their frames to preserve visual detail. For larger selections, it reads the continuous textual record to reduce double counting across block boundaries.

Dedicated readouts. On the SVCBench periodic subset, our readout achieves 67.22 GPA without training a dedicated counting model, compared with 66.17 for RepNet, which is pretrained on repetition-counting data. On queries originally routed to Cuts, our readout achieves 68.57 GPA versus 53.66 when its outputs are replaced by PySceneDetect, with routing fixed.

Contribution to the full system. To measure the benefit of direct readout, we replace Periodic or Cuts with Evidence on the queries originally assigned to that readout, keeping other routing decisions fixed. To isolate the benefit of adaptive evidence integration, we retain both pattern branches and fix the Evidence reader to Read or Inspect.

SettingOverall GPADrop
CountRoute38.39—
w/o Periodic31.926.47
w/o Cuts35.183.20
w/o Inspect35.512.88
w/o Read33.954.43

Q2. Switching-threshold sensitivity and transfer

To assess switching-threshold sensitivity and transfer to new data, we built an event-counting evaluation set from EgoLife, a dataset of long-form egocentric recordings of daily life. Our set covers long events and short actions and contains 36 videos, 48 counting questions, and 384 query points. We used the official EgoLife captions to draft the event annotations, then manually verified them. All 48 questions route to Evidence under the submitted router. We vary only N on both datasets and evaluate the submitted N=3 on EgoLife without retuning.

Switching threshold NSVCBench GPAEgoLife-based set GPA
0: textual reading35.5161.73
237.3662.77
3: submitted default38.3967.98
437.6764.85
536.7262.77
∞: selected-block visual accumulation33.9543.24

Q3/W2. Memory-construction cost and accuracy versus resources

Streaming memory construction. Each stream uses two H100 GPUs: one for frame embeddings and captioning, and one for transition verification. In the full 28.0-hour SVCBench replay, memory construction keeps pace with 1× playback. The time from each four-second window becoming available to saving both memory and the transition index averages 2.58 s, with a maximum of 3.79 s. In the component timing runs, frame embedding preparation and extraction (0.64 s) and captioning requests (1.78 s) run serially on the same H100, averaging 2.42 s in total per four-second window.

Comparison at similar visual token budgets. As reported in our response to Reviewer Zcsq (Q2/W3), Direct-192 and Direct-320 use 16,205 and 21,199 visual tokens per query point, respectively, compared with 18,317 for CountRoute. At these comparable or higher visual token budgets, they achieve 25.27 and 24.67 GPA, respectively, versus 38.39 for CountRoute.

Comparison with WorldMM. We compare with WorldMM, a general-purpose agentic video understanding method, on the full SVCBench dataset. Both systems use the MiniMax M3 API for text-only calls and Qwen3-VL-8B for visual calls, with two H100 GPUs allocated per system. CountRoute uses one GPU for frame embeddings and captioning and the other for transition verification. WorldMM uses one GPU for visual generation and the other for embeddings and retrieval. We report memory construction time per minute of video and query answering time per query point.

MethodMemory construction time (s/min of video)Query answering time (s/query point)GPA
WorldMM230.6106.910.84
CountRoute38.753.838.39

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.