- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RepProbe, a benchmark of approximately 42K real and synthetic video clips for evaluating repetition counting in video language models (VLMs). Experiments on seven generalist VLMs and two specialist counters reveal substantial errors at high counts. Linear probes can decode repetition counts more accurately than native model outputs, suggesting a gap between representation and generation.
Critical strengths
- Repetition counting provides a focused diagnostic task for video understanding.
- Real and synthetic clips enable complementary analyses across counts.
- Probing reveals count information not reliably expressed in generated answers.
Critical weaknesses
-
The mechanism claim is a bit overstated, and there's no causal test backing it. Being able to linearly decode the count doesn't mean the model actually has a robust counting procedure it can use, and it certainly doesn't prove the failure sits in the output pathway. If that's really where the problem is, then probe-as-head or logit-steering on the number tokens should bring accuracy back up — but no such test is shown. As it stands the strongest claim you can defend is correlational.
-
The two readouts aren't really comparable. You're pitting a supervised probe against native generation, and those have completely different objectives, so the comparison doesn't land. Sharing the readout space — logit decoding in the same space, or constrained generation — would make it a much fairer fight.
-
Scope and ecological validity are limited, and a few of the headline signals rest on the easy cases. The preferred-number bias is just correlational; most of the real clips are exercise videos; and the synthetic clips are pretty simplistic. That last one matters, because the dissociation is strongest exactly on the geometric synthetic clips (around 92% vs 15–17%), where "periodicity is linearly there" is a much weaker claim than "the model perceives real repetitions." The frame-budget differences don't help either — they make it hard to compare models on a level playing field.
-
Scale limited. All the probing is on two 8B models, and the generalists are all 7–8B, so it's hard to tell whether the dissociation holds up at larger scale or just washes out.
Questions for authors
-
Can native outputs be evaluated using a shared 20-class decoding protocol?
-
Are the findings robust to prompts, decoding strategies, and numerical formats?
2 comments
I thank the authors for a detailed rebuttal. The shared 20-class decoding table and the final-state steering experiment are exactly the right response to my main concern.
The remaining concerns include the causal evidence is still partial. The steering dose is selected in-sample and the token sets use the full pool, so the split-half result is not a held-out validation. Besides that, the two readouts are still not feature-matched. Calibrated logits decoding and the hidden-state probe differ in input features, so the residual gap cannot be attributed to readout alone.
The issues are related to the strength of the mechanistic story and the breadth of the evidence, not the correctness of the reported results. I am therefore raising my score from 2 to 3.
Response to Reviewer 6KSt
Thank you for the specific recommendations. We added a shared answer-space decoding comparison and a final-state steering experiment, and narrowed our claims to what those tests support.
W1: Mechanism claim and causal test
Our revised interpretation is that count-related information is linearly decodable from the internal representation but is not reliably reflected in the model's own number scores or its answer. We do not claim a complete counting procedure or a predominantly output-stage failure.
We tested a final-state intervention subtracting a model-specific number-token unembedding contrast (construction and model-specific token selection in our reply to Reviewer rZK3). On high-count clips this reduces MAE for Cosmos-Reason2 (-1.63, 95% CI [-2.08, -1.20]), Video-R1 (-0.80), InternVL3.5 (-0.56), and Qwen3-VL (-0.49), with the same sign at a shared strength (α=0.05). Across 20 source-grouped split-half resamples, selecting the dose on one half reduces MAE on the other for three families (mean changes: Qwen3-VL -0.49, Video-R1 -0.80, InternVL3.5 -0.64); token sets still use the full pool, so this is not fully held-out validation (limits in our reply to rZK3). Injecting the probe's count direction produces no change (-0.01, 95% CI [-0.07, +0.05]). Numerical output preferences therefore contribute to the errors, while decodability alone does not make count information usable by the output pathway.
W2 and Q1: Shared 20-class decoding and readout calibration
Thank you for the suggestion. We evaluated all 20 counts in the model's own output space, first without supervision and then with a linear calibration trained on the same clips and labels as the hidden-state probe. All rows for a model use the same test clips.
Summed scores favor shorter tokenizations, so length normalization changes constrained decoding materially; even so, constrained decoding does not remove the counting errors. Calibration improves the model's number scores but stays below the hidden-state probe under matched supervision, with or without length normalization (3.27 and 4.24 in both cases). The gap therefore persists under matched label access and constrained answer formats, although the readouts differ in their input features.
W3: Scope, ecological validity, and frame budgets
The two components of RepProbe support complementary inferences. RepProbe-Real consists mostly of exercise videos and supports within-source variation in count, duration, and position. RepProbe-Synthetic provides controlled periodic patterns but cannot by itself establish event-level counting in natural video. We limit the synthetic conclusion to decodable periodic structure and base the high-count argument on the new 64-frame experiments, not the original 16-frame result.
With all 64 synthetic frames the answer-probe gap persists (Qwen3-VL: 23% against 93% OBO; Cosmos-Reason2: 29% against 91%), including for k≤16 (28% against 94%; 36% against 93%). On frame budgets, our matched frames-per-cycle analysis compares models at equal temporal coverage, and the new synthetic tests give both probed models the same 64 frames. The steering experiment shows output sensitivity to a number-token contrast, while its in-sample selection limits the mechanistic inference.
W4: Model scale
Probing currently covers two 8B models, and the evaluated generalists are 7B to 8B. We will state this explicitly and restrict our claims to the evaluated model sizes. Whether the dissociation holds for larger models is an open question that the current results do not answer.
Q2: Prompt, decoding, and numerical-format robustness
We tested three prompt and number-format conditions on 200 Cosmos-Reason2 clips balanced across k. Answer MAE / OBO was 3.59 / 38.5% for the default prompt, 5.37 / 31.0% for a plain prompt, and 3.19 / 38.5% for zero-padded answers. Answers are sensitive to prompt wording, but substantial error persists in every condition: the best variant reduces Many-band MAE only from 6.07 to 4.44. Constrained 20-way decoding does not remove the errors either (table above). All model answers use greedy decoding; we will add sampling-based decoding and the same variants for Qwen3-VL in the revision.