acceptodds
Desk rejected at ICLR 2027

When Should the Count Change? Learning State Maintenance for Causal Video Counting

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Continuous video counting requires distinguishing new observations from new objects or completed events. We introduce StaMina (State Maintenance), which learns to maintain counting state through state-conditioned updates. Recurrent visual context supports recognition; learned transitions maintain visibility, persistent identities, and completed-event records. A differentiable recurrence trains event transitions over legal paths constrained by count endpoints; visibility and association objectives train the object branch. A multi-source pipeline organizes 39.8K spatial queries and complementary event annotations into counting trajectories. On SVCBench, we evaluate counting adaptation with partial video overlap and held-out groups of linked annotations. Under prefix replay (Full) and persistent streaming (Stream), 4B and 8B models reach 41.9/36.4 and 44.9/38.2 Gaussian Precision Accuracy, respectively. The 8B model gains 10.9/3.2 points over Counting-SFT on the same queries. Matched-graph comparisons isolate phase conditioning and trajectory supervision, assessing training objectives alongside hard decisions. Online video benchmarks and count-conditioned decisions assess online understanding and task eligibility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Causal Video Counting, State Maintenance, Trajectory Supervision, Streaming Video Understanding, Test-Time Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ybyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StaMina, a state-maintenance framework for causal video counting. The method combines a Qwen3-VL backbone with gated delta memory, an explicit object-identity and visibility registry, and operator-specific finite-state event transitions. A count-constrained forward recurrence marginalizes over legal event-state trajectories compatible with observed count endpoints, while auxiliary identity and boundary losses exploit stronger supervision where available. The authors construct a multi-source training pool containing 53.6K query–answer pairs, including 39.8K spatial queries and complementary event annotations.

On SVCBench, StaMina-4B/8B obtains 41.9/44.9 GPA under Full access and 36.4/38.2 under Stream access. The 8B model improves over Counting-SFT by 10.9/3.2 GPA points on shared counting queries. The paper further presents controlled transition-learning ablations, object-identity diagnostics, transfer evaluations on OVO-Bench and OVO-S-Bench, and count-conditioned action-predicate experiments.

Critical strengths
  1. A useful and well-motivated problem formulation. The paper explicitly distinguishes visibility changes, persistent object identities, and completed-event updates. This decomposition is appropriate for causal counting, where repeated observations should not automatically trigger new counts. The proposed typed state and separate object/event branches provide a coherent way to represent these distinctions.

  2. Substantial effort toward controlled evaluation. The paper goes beyond final-answer accuracy by examining hard-prefix correctness, duplicate and missed event updates, acceptance delay, and identity-association errors. The matched-graph transition comparisons help isolate the contributions of phase conditioning and trajectory supervision. The data-provenance and supervision-family exclusion procedures are also more careful than a simple query-level split.

Critical weaknesses
  1. The problem is meaningful, and the state representation is well motivated.

The paper explicitly distinguishes object visibility, persistent object identity, and event completion. This distinction is important for continuous video counting, where repeated observations of the same object should not trigger duplicate counts, while event counting requires determining whether an action has actually completed. Explicit state representation and state-conditioned updates provide a clear and potentially useful formulation of these challenges.

  1. The evaluation includes useful mechanism-level diagnostics.

Beyond final-answer accuracy, the authors examine state-transition decisions, duplicate and missed event updates, identity-association errors, and acceptance delay. Controlled ablations investigate the contributions of phase conditioning and trajectory supervision. The paper also provides a relatively detailed account of training-data provenance and the exclusion of linked supervision. These analyses help characterize the system's behavior and failure modes.

Questions for authors
  1. What is the core algorithmic novelty of this work?

Compared with existing finite-state models, sequence-path marginalization methods, and streaming state-maintenance approaches, what is fundamentally new about StaMina's learning mechanism? The experiments show that individual components are useful, but how do the authors establish that their combination constitutes a genuinely new method rather than an integration of existing techniques?

  1. How do the authors rule out the influence of visual exposure to overlapping videos?

Given that some training parent groups overlap with SVCBench, can the authors report results separately for overlapping and non-overlapping videos, as well as an evaluation with strict source-video separation? What forms of information leakage are prevented by supervision-family exclusion, and what information may still be exploited through exposure to the same visual content?

  1. How do the structural reachability limits of periodic events affect the conclusions?

The paper reports upper coverage bounds of 44.3% and 58.2% for periodic-event endpoints under Full and Stream access, respectively. Can the authors report performance on endpoints reachable under the current observation budget and transition rules, and distinguish failures caused by architectural constraints from those caused by perception or transition-prediction errors?

  1. Why is each observation restricted to at most one state transition?

How does the method handle multiple events occurring between consecutive observations? In scenarios with sparse sampling, rapid events, or overlapping events, could the single-transition assumption systematically underestimate event counts? Under what conditions is this assumption appropriate?

  1. Are the comparisons with existing models sufficiently fair?

The external reference models retain their original evaluation protocols, whereas StaMina receives task-specific counting adaptation. Can the authors provide a unified comparison of key baselines under matched training data, task-adaptation conditions, and input budgets? How much of the observed improvement can be attributed to state maintenance itself rather than additional task-specific training?

  1. Is the automatic annotation pipeline sufficiently reliable to support the proposed training method?

The paper reports an increment-label accuracy of 85.4% for natural events. Can the authors provide more details on the annotation models, prompts, filtering criteria, and manual verification procedures, and analyze how annotation errors affect trajectory supervision, transition learning, and final counting performance? In particular, is the available supervision sufficiently reliable to support the claimed learning of fine-grained event-completion states?

Reviewer_dznYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents StaMina, a framework for maintaining counting states in continuous video. It combines visual memory, object identity tracking, and event-state transitions, and uses trajectory-level supervision to train the model. Experiments on SVCBench and other video benchmarks show improvements over several baselines, although some aspects of the evaluation remain unclear.

Critical strengths

The paper addresses an important problem: a video counting system needs to remember previously observed objects and distinguish ongoing events from completed ones. The explicit state representation is a reasonable way to handle this. The ablation results also suggest that trajectory supervision and phase information contribute to better counting performance.

Critical weaknesses

1,The proposed framework contains several interacting components, including memory, identity association, and event transitions. The experiments do not fully clarify how much each component contributes to the overall improvement over standard fine-tuning.

2.The evaluation protocol makes some results difficult to interpret. In particular, the limited observation rate restricts the number of periodic events the model can detect. The paper reports that only a subset of periodic-event endpoints is reachable under the streaming setting, but still includes all endpoints in the overall score. It would be helpful to separate errors caused by insufficient observations from those caused by incorrect model predictions.

Questions for authors
  1. Implement experiments about how much each component contributes to the overall improvement over standard fine-tuning.

  2. It would be helpful to separate errors caused by insufficient observations from those caused by incorrect model predictions.

Reviewer_dfGkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study causal video counting, where the model must tell whether a new observation is a reappearing object, a new object, or a completed event. They propose StaMina, which adds a gated delta memory and an explicit counting state to Qwen3-VL, and trains hand-designed state machines with a CTC-like loss that only needs count values at a few time points.

Critical strengths
  • The authors address a very important problem that current frontier VLMs cannot solve very well.
  • Fig. 1 is a good visualization to understand the motivation of this work.
  • The core idea, maintaining an explicit counting state and learning when the count should change (e.g., observe, occlude, see again), is straightforward and intuitive.
Critical weaknesses

1. Would the method still be effective on Qwen3.5?

Qwen3.5 already adopts a hybrid architecture with linear-attention (Gated DeltaNet) layers. Would the proposed delta memory and counting modules still provide improvements on Qwen3.5? I suggest repeating the main experiments on Qwen3.5 to verify whether these additional modules remain effective.

2. The evaluation setup and method generalizability.

There is an overlap between the training data and SVCBench (76 out of 301 videos, involving 9,216 training questions), while all reference models in Table 1 are evaluated in a zero-shot setting. This makes the direct comparison unfair. I suggest that the authors remove these overlapping videos from the training set, or report results only on the SVCBench videos that do not appear in the training set. In addition, each counting type has a manually designed state machine and several "magic numbers". The method therefore appears to be tailored to SVCBench.

I suggest evaluating it on other datasets without overlap to test whether it generalizes to counting tasks, such as image counting, counting questions in Video-MME[1] and MLVU[2], and repetitive action counting (e.g., RepCount[3]). I think repetitive action counting seems particularly well aligned with the method's intended design. However, the one-transition-per-observation design may limit the maximum number of countable cycles, which may be a fundamental limitation for fast repetitive actions. The authors should also clarify the source of the annotated periodic seeds to rule out overlap with RAC benchmarks.

3. Limited improvements in the streaming setting.

For the 8B model, the improvement over Counting-SFT in the streaming setting is not very significant (38.2 vs. 35.0). I would like to see a statistical significance analysis, along with a breakdown of which categories improve and which deteriorate. Failure cases would also help clarify where the proposed method works and where it fails.

[1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

[2] MLVU: Benchmarking Multi-task Long Video Understanding

[3] TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting

Questions for authors

Please refer to the Weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.