acceptodds
Under review as a conference paper at ICLR 2027

RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark

Abstract

Repetition counting requires identifying and enumerating action cycles over time. While state-of-the-art video language models describe actions fluently, they fail to reliably count how many times an action repeats. To investigate this limitation, we introduce RepProbe, a benchmark of 42K real and synthetic video clips containing exactly k in [1, 20] complete repetitions, with controls for clip position, temporal coverage, and duration. These controls allow us to study the counting failure directly and locate where in the model it arises. Counting accuracy collapses at high counts across the video language models we evaluate, and controlled experiments show this collapse is not explained by frame sampling alone. We compare model outputs with linear probes trained on the final-layer representations used to generate those outputs. We find a consistent representation-to-output dissociation: substantial count information remains linearly decodable even when models fail to report the correct count. Further analysis shows this failure arises primarily at the output stage, where answers concentrate on a small set of preferred numbers rather than the available count information. The dissociation holds on both real and synthetic videos and is not explained by clip duration, position, or scene content. These results indicate that models encode repetition structure but do not reliably report it in their outputs.

Primary area:
datasets and benchmarks
Keywords:
video language models, repetition counting, temporal reasoning, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6KStOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RepProbe, a benchmark of approximately 42K real and synthetic video clips for evaluating repetition counting in video language models (VLMs). Experiments on seven generalist VLMs and two specialist counters reveal substantial errors at high counts. Linear probes can decode repetition counts more accurately than native model outputs, suggesting a gap between representation and generation.

Critical strengths
  1. Repetition counting provides a focused diagnostic task for video understanding.
  2. Real and synthetic clips enable complementary analyses across counts.
  3. Probing reveals count information not reliably expressed in generated answers.
Critical weaknesses
  1. The mechanism claim is a bit overstated, and there's no causal test backing it. Being able to linearly decode the count doesn't mean the model actually has a robust counting procedure it can use, and it certainly doesn't prove the failure sits in the output pathway. If that's really where the problem is, then probe-as-head or logit-steering on the number tokens should bring accuracy back up — but no such test is shown. As it stands the strongest claim you can defend is correlational.

  2. The two readouts aren't really comparable. You're pitting a supervised probe against native generation, and those have completely different objectives, so the comparison doesn't land. Sharing the readout space — logit decoding in the same space, or constrained generation — would make it a much fairer fight.

  3. Scope and ecological validity are limited, and a few of the headline signals rest on the easy cases. The preferred-number bias is just correlational; most of the real clips are exercise videos; and the synthetic clips are pretty simplistic. That last one matters, because the dissociation is strongest exactly on the geometric synthetic clips (around 92% vs 15–17%), where "periodicity is linearly there" is a much weaker claim than "the model perceives real repetitions." The frame-budget differences don't help either — they make it hard to compare models on a level playing field.

  4. Scale limited. All the probing is on two 8B models, and the generalists are all 7–8B, so it's hard to tell whether the dissociation holds up at larger scale or just washes out.

Questions for authors
  1. Can native outputs be evaluated using a shared 20-class decoding protocol?

  2. Are the findings robust to prompts, decoding strategies, and numerical formats?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6KSt

Thank you for the specific recommendations. We added a shared answer-space decoding comparison and a final-state steering experiment, and narrowed our claims to what those tests support.

W1: Mechanism claim and causal test

Our revised interpretation is that count-related information is linearly decodable from the internal representation but is not reliably reflected in the model's own number scores or its answer. We do not claim a complete counting procedure or a predominantly output-stage failure.

We tested a final-state intervention subtracting a model-specific number-token unembedding contrast (construction and model-specific token selection in our reply to Reviewer rZK3). On high-count clips this reduces MAE for Cosmos-Reason2 (-1.63, 95% CI [-2.08, -1.20]), Video-R1 (-0.80), InternVL3.5 (-0.56), and Qwen3-VL (-0.49), with the same sign at a shared strength (α=0.05\alpha = 0.05). Across 20 source-grouped split-half resamples, selecting the dose on one half reduces MAE on the other for three families (mean changes: Qwen3-VL -0.49, Video-R1 -0.80, InternVL3.5 -0.64); token sets still use the full pool, so this is not fully held-out validation (limits in our reply to rZK3). Injecting the probe's count direction produces no change (-0.01, 95% CI [-0.07, +0.05]). Numerical output preferences therefore contribute to the errors, while decodability alone does not make count information usable by the output pathway.

W2 and Q1: Shared 20-class decoding and readout calibration

Thank you for the suggestion. We evaluated all 20 counts in the model's own output space, first without supervision and then with a linear calibration trained on the same clips and labels as the hidden-state probe. All rows for a model use the same test clips.

Readout (MAE / OBO)Cosmos-Reason2Qwen3-VL
Model's answer (free-form)3.54 / 39.5%6.63 / 26.3%
Constrained 20-way, zero-shot (summed log-prob)7.57 / 25.6%7.56 / 21.2%
Constrained 20-way, zero-shot (length-normalized)3.86 / 38.4%5.92 / 24.2%
Constrained 20-way, calibrated (supervised)3.27 / 46.7%4.24 / 36.5%
Hidden-state probe (supervised)2.56 / 51.4%3.01 / 47.7%

Summed scores favor shorter tokenizations, so length normalization changes constrained decoding materially; even so, constrained decoding does not remove the counting errors. Calibration improves the model's number scores but stays below the hidden-state probe under matched supervision, with or without length normalization (3.27 and 4.24 in both cases). The gap therefore persists under matched label access and constrained answer formats, although the readouts differ in their input features.

W3: Scope, ecological validity, and frame budgets

The two components of RepProbe support complementary inferences. RepProbe-Real consists mostly of exercise videos and supports within-source variation in count, duration, and position. RepProbe-Synthetic provides controlled periodic patterns but cannot by itself establish event-level counting in natural video. We limit the synthetic conclusion to decodable periodic structure and base the high-count argument on the new 64-frame experiments, not the original 16-frame result.

With all 64 synthetic frames the answer-probe gap persists (Qwen3-VL: 23% against 93% OBO; Cosmos-Reason2: 29% against 91%), including for k≤16k \le 16 (28% against 94%; 36% against 93%). On frame budgets, our matched frames-per-cycle analysis compares models at equal temporal coverage, and the new synthetic tests give both probed models the same 64 frames. The steering experiment shows output sensitivity to a number-token contrast, while its in-sample selection limits the mechanistic inference.

W4: Model scale

Probing currently covers two 8B models, and the evaluated generalists are 7B to 8B. We will state this explicitly and restrict our claims to the evaluated model sizes. Whether the dissociation holds for larger models is an open question that the current results do not answer.

Q2: Prompt, decoding, and numerical-format robustness

We tested three prompt and number-format conditions on 200 Cosmos-Reason2 clips balanced across kk. Answer MAE / OBO was 3.59 / 38.5% for the default prompt, 5.37 / 31.0% for a plain prompt, and 3.19 / 38.5% for zero-padded answers. Answers are sensitive to prompt wording, but substantial error persists in every condition: the best variant reduces Many-band MAE only from 6.07 to 4.44. Constrained 20-way decoding does not remove the errors either (table above). All model answers use greedy decoding; we will add sampling-based decoding and the same variants for Qwen3-VL in the revision.

I thank the authors for a detailed rebuttal. The shared 20-class decoding table and the final-state steering experiment are exactly the right response to my main concern.

The remaining concerns include the causal evidence is still partial. The steering dose is selected in-sample and the token sets use the full pool, so the split-half result is not a held-out validation. Besides that, the two readouts are still not feature-matched. Calibrated logits decoding and the hidden-state probe differ in input features, so the residual gap cannot be attributed to readout alone.

The issues are related to the strength of the mechanistic story and the breadth of the evidence, not the correctness of the reported results. I am therefore raising my score from 2 to 3.

Reviewer_p7JaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces RepProbe, 42K clips with exactly k = 1-20 repetitions: 32K real clips re-cut from RepCount videos (start/middle/end windows, source-level splits) and 10K synthetic 4-second clips in which one attribute changes periodically. Seven open VLMs degrade sharply at high k while the specialist ESCounts stays accurate. For Cosmos-Reason2 and Qwen3-VL, a linear probe on the final-layer residual at the last prompt token predicts k better than the model’s own answer. The authors conclude that the count is represented but not reported, with answers following a prior over a few preferred numbers.

Critical strengths

S1. The question is well posed: “not represented” and “represented but not reported” call for different fixes, and repetition counting gives an exact target.

S2. Cutting clips with different k from the same source is a sensible control, and the paper includes useful checks (FPC × k grid, shuffled-label probe, layer-wise probes, length-fair number-word readout, per-position results).

Critical weaknesses

W1. The main claim, that the count is represented but lost at readout, is not supported by the current evidence.

(a) Synthetic set. Clips last 4 s, so under the default setting in Sec. 3.3 the models see about 16 frames (16 uniform frames; Cosmos at 4 fps). With uniform sampling, exactly periodic signals with k and k+N periods (N ≈ 16) give nearly identical frames, and above k ≈ 8 there are fewer than two frames per period. The 0% Many-band OBO of all generalists (Table 1) is therefore expected from the input alone. Yet the probe reaches 91.3% (Cosmos) and 93.5% (Qwen) Many-band OBO (Table 2), so it likely uses something other than the number of periods. With duration fixed, count equals frequency, which could leak through per-frame cues or sampling details the paper does not report. Sec. 4.3 relies on this set to argue the probe reads repetition structure, so this is central. On real clips, Qwen3-VL with 16 frames is also below two frames per cycle for k ≥ 9, yet its probe keeps 39.4% Many-band OBO.

(b) Probe vs. answer. The probe is trained with labels on 1,000 clips from the same benchmark, while the models answer zero-shot. The argument that the LM head is also linear (Sec. 4.2) shows equal capacity, not that the head’s number directions align with the probed direction. The probe at block 0 already gives MAE 2.91 for Cosmos vs. 2.49 at the final block (App. A.2), which is consistent with reading visual statistics rather than a count computed by the language model. The duration control (App. A.5) uses duration only, not action type. No intervention tests whether the decodable count can actually be read out.

W2. The need for a new benchmark is not shown. The motivation is that RepCount ties count to duration and source, but the paper does not show that the same analysis on RepCount gives a different or misleading answer. In RepProbe-Real, count and duration still correlate (r = 0.62 full set, 0.55 balanced subset), and the paper relies on post-hoc statistical control, which could be applied to RepCount directly. The 32K clips are overlapping windows from about 1K videos, and the analyses use 1K-10K subsets. The full set and the FPC grid serve only the behavioral evaluation, and the high-k collapse was already reported by prior work the paper cites (Yao et al., 2025; Li et al., 2026). The benchmark and the probing analysis are therefore only loosely connected, and neither part is complete on its own.

Questions for authors

Q1. For the synthetic set: render frame rate, sampled frame indices, and whether timing parameters (e.g., flash on-time) are fixed in seconds or as a fraction of the period. How is the synthetic probe split into train and test?

Q2. How does the synthetic probe perform with shuffled frame order or a single repeated frame, and how often does it confuse k with k+16?

Q3. Do Cosmos-Reason2 and the other R1-style models produce a reasoning trace before the answer under your prompt? If so, how is comparing the answer with the last-prompt-token state justified?

Q4. Which weights are used for ESCounts and RepNet, and does Table 1 exclude clips from any videos they were trained on?

Q5. The same probe and duration controls on original RepCount videos. Does the conclusion differ?

3 comments
AuthorsOpenReview ↗

Response to Reviewer p7Ja (Part 1/2)

Thank you for the detailed review. The sampling critique identifies a real limitation in the original synthetic experiment. We test it directly with all 64 rendered frames, a restricted-count analysis, and leakage controls, and we revise the interpretation accordingly.

W1(a): Synthetic sampling and the meaning of the probe signal

The original 16-frame setting cannot resolve every period at high counts. Synthetic clips are rendered at 16 fps over four seconds (64 frames), but the default evaluation supplied 16 sampled frames to both models (indices in Q1). For k>8k > 8 this is fewer than two frames per period. A cross-budget transfer test reinforces the concern: a probe trained at 16 frames and tested at 64 keeps rank correlation with kk in the Few band (Spearman 0.69) but not in the Mid or Many bands (−0.03-0.03 and 0.050.05). We no longer treat the 16-frame high-count synthetic result as evidence.

New 64-frame experiment. We repeated the analysis with all 64 frames on a balanced 2,000-clip subset (50 training and 50 test clips per count). At k=20k = 20 the input contains 3.2 frames per period. The encoder's temporal patch size of two leaves 32 temporal positions, which motivates the restricted analysis below.

64 frames, all kkOBOExactMAE (Few / Mid / Many)
Qwen3-VL, model's answer23%11%0.99 / 5.71 / 12.68
Qwen3-VL, linear probe93%87%0.11 / 0.44 / 0.45
Cosmos-Reason2, model's answer29%12%1.82 / 3.08 / 10.39
Cosmos-Reason2, linear probe91%83%0.19 / 0.50 / 0.46

Providing all frames reduces answer MAE only partly (Qwen3-VL: 7.79 to 6.74; Cosmos-Reason2: 7.78 to 5.26), and Many-band MAE remains above 10 for both models. The low answer accuracy cannot be attributed solely to the 16-frame budget.

64 frames, k≤16k \le 16 (800 test clips)OBOExactMAE (Few / Mid / kk = 14 to 16)
Qwen3-VL, model's answer28%13%0.99 / 5.71 / 10.74
Qwen3-VL, linear probe94%89%0.14 / 0.33 / 0.29
Cosmos-Reason2, model's answer36%15%1.82 / 3.08 / 8.24
Cosmos-Reason2, linear probe93%86%0.16 / 0.41 / 0.16

The gap persists where the encoder has at least two temporal positions per period. These results support decodability of periodic structure with denser temporal input; they do not establish event-by-event counting.

Real clips at 16 frames. The same concern applies to Qwen3-VL on real clips: for k≥9k \ge 9 it has fewer than two frames per cycle, so its Many-band probe result (39.4% OBO) should not be read as resolved cycles; duration and partial cues may contribute. Two analyses are less affected. First, Cosmos-Reason2 samples at 4 fps, so real clips with k≥14k \ge 14 have about 4 to 5 frames per cycle, and its probe-answer gap holds with duration controlled: averaged over 2-second duration bins, probe MAE is 2.87 against 4.59 for the model's answers. Second, with frames per cycle matched at 8, answer MAE still rises with kk (kk = 5 to 20: 2.47 to 4.57 for Cosmos-Reason2; 2.93 to 7.86 for Qwen3-VL).

W1(b): Probe supervision, readout, and early-layer information

The review correctly distinguishes linear readout capacity from alignment between the language-model head and a supervised count probe. We added two direct comparisons.

First, we compare the hidden-state probe with a supervised calibration of the model's own 20 candidate-answer scores, using the same training clips and labels. The probe keeps a lower MAE for both models (Cosmos-Reason2: 2.56 against 3.27; Qwen3-VL: 3.01 against 4.24; full table in our reply to Reviewer rZK3). The gap persists when label access is matched, although the two readouts use different feature spaces.

Second, subtracting a model-specific number-token unembedding contrast from the final state reduces high-count MAE (Cosmos-Reason2: 7.48 to 5.85). In three families, the reduction persists on dose-selection holdouts across 20 source-grouped split-half resamples; token sets still use the full pool (protocol in our reply to rZK3). Injecting the probe direction does not help. These results implicate number-associated directions, not a precomputed count.

On the early-layer result: Cosmos-Reason2 probe MAE is 2.91 at block 0 against 2.49 at the final block, and Qwen3-VL is 4.09 against 3.05. Substantial count-predictive signal is available at the earliest probed layer, so the final-layer probe alone does not demonstrate a counting computation by the language model.

We also added the suggested action-type control, on the same real clips. Duration alone gives MAE 3.26; action type alone 4.76; duration plus action type 3.38 (change against duration alone +0.12+0.12, 95% CI [+0.01,+0.24][+0.01, +0.24]). The hidden-state probes remain better than these controls (MAE 2.56 and 3.01 against 3.26; OBO 51.4% and 47.7% against 30.1%).

AuthorsOpenReview ↗

Response to Reviewer p7Ja (Part 2/2)

W2 and Q5: Why RepProbe is needed; original RepCount audit

We ran the same probe and duration audit on original videos using their full annotated RepCount spans (n=568n = 568). The same ordering holds: probe MAE 3.04, duration-only 3.47, model's answers 3.65. We will include this comparison and will not claim that RepProbe is necessary for probing itself.

RepProbe provides a control that the original RepCount annotations cannot. In RepCount each video carries a single count, so count varies together with source, action, and scene. RepProbe-Real cuts multiple counts from 1 to 20 out of the same source video, so the same scene and action appear at many different counts. This lets us compare the model's answers and the probe across counts with scene and action held fixed. RepProbe-Real also adds start, middle, and end position controls, source-level splits, and balanced coverage of every count, and RepProbe-Synthetic fixes duration entirely. These controls do not remove the count-duration correlation in real video (r=0.62r = 0.62 overall; 0.55 on the balanced subset; 0.51 on the RepCount sources), which we audit separately and will state plainly in the motivation. Prior work documented that accuracy drops at high counts; RepProbe's contribution is a controlled setting for asking why, by linking behavior and internal representations on the same clips.

Q1: Synthetic generation and train/test split

Clips are rendered at 16 fps (64 frames in four seconds). The default 16-frame indices are [0, 4, 8, 13, 17, 21, 25, 29, 34, 38, 42, 46, 50, 55, 59, 63], from rounding 16 evenly spaced positions between 0 and 63; both probed models use this selection on synthetic clips. The 64-frame evaluation uses every frame. Pulse on-time is a fraction of each period, not a fixed duration in seconds, so duty cycle alone does not reveal kk. The new 2,000-clip analysis uses 50 training and 50 test clips per kk; the paper's synthetic results used five-fold stratified cross-validation over all 10,000 clips.

Q2: Shuffled-frame, single-frame, and k+16k + 16 checks

We trained and evaluated probes on clips with shuffled frame order and on clips formed by repeating a single frame:

Probe MAEOriginalShuffledSingle frame
Cosmos-Reason20.645.515.48
Qwen3-VL0.515.885.81

Both controls approach the MAE of a constant predictor (about 5), so low-error decoding depends on ordered temporal information. Single-frame exact match remains above the 5% chance level (24% for Cosmos-Reason2, 21% for Qwen3-VL). It is highest for orbit and pendulum (46% and 30%; 43% and 30%), and excluding both families leaves 20.5% and 16.6%. We therefore do not claim static leakage is eliminated; the control result is the deterioration in MAE. Predictions of k+16k + 16 occur in 3 of 10,000 cases; the sampled indices are not perfectly uniform, so the corresponding inputs are not identical.

Because synthetic duration is fixed, count and frequency covary. Our synthetic claim is that periodic structure is decodable when temporally visible, yet is not reliably reported. The analysis of counting beyond fixed-frequency signals rests on the variable-duration real-video experiments.

Q3: Reasoning traces and the probed token position

Under our counting prompt, Cosmos-Reason2, Qwen3-VL, Video-R1, and VideoChat-R1.5 output bare integers with no preceding reasoning trace (0 of 10,000 outputs for Cosmos-Reason2). The last-prompt-token representation therefore immediately precedes the answer. LongVILA-R1 does generate a reasoning trace, but it appears only in the behavioral evaluation and is not probed. We will state this explicitly.

Q4: Specialist checkpoints and train/test overlap

RepNet uses the Countix-trained checkpoint (TensorFlow weights in a PyTorch port). ESCounts uses the RepCount-trained checkpoint, and the original Table 1 did not exclude RepCount training-split sources for it. We recomputed its results on clips derived only from RepCount test-split videos (2,011 clips from 142 source videos): MAE 1.46, OBO 69%, exact match 32%, and Few / Mid / Many MAE 0.76 / 1.60 / 1.94. These are close to the full-evaluation results (MAE 1.32; OBO 72%), so the specialist remains substantially more accurate at high counts on held-out sources. We will report this split-safe row in Table 1.

Dear Reviewer p7Ja,

Thank you again for your detailed and constructive review. We wanted to kindly follow up on our response posted on November 14. The suggested sampling technique led us to run a full 64-frame experiment and to revise our interpretation of the synthetic results. We also added the calibration, early-layer, and action-type comparisons suggested. We will gladly incorporate this in our revised manuscript.

As the discussion period closes on today, we would be grateful to know whether our response addresses your concerns, or whether any point would benefit from further clarification. We are happy to run additional analyses if helpful.

Reviewer_rZK3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RepProbe, a diagnostic benchmark for evaluating repetition counting in video language models. The benchmark contains approximately 42K real and synthetic video clips with controlled repetition counts from 1 to 20. The authors evaluate seven general-purpose video language models and show that their counting accuracy deteriorates substantially as the number of repetitions increases, even when frame coverage is controlled.

The main contribution is an analysis of whether these failures arise from missing information in the model's representations or from difficulties in producing the correct output. By training linear probes on final-layer hidden representations, the authors show that repetition counts can be recovered substantially more accurately than through the models' native outputs. Additional experiments on synthetic videos and controls for duration and clip position support the finding that count-related information is encoded but not reliably reflected in generated answers.

Critical strengths
  1. Well-controlled benchmark design. RepProbe provides a systematic way to investigate repetition counting while controlling factors such as clip position, temporal coverage, and duration. The combination of real and synthetic videos makes the analysis more convincing and helps isolate counting failures from potential confounding factors.

  2. Interesting representation-level analysis. The substantial gap between linear probe performance and native model outputs is an interesting finding, particularly on synthetic videos where probes achieve over 90% off-by-one accuracy while native outputs remain below 20%. This provides useful evidence that video language models encode temporal information that is not reliably reflected in their predictions.

Critical weaknesses
  1. The conclusion about output-stage failure is stronger than the evidence supports. The linear probes are trained with supervised count labels, whereas the original models are evaluated zero-shot. Therefore, better probe performance demonstrates that count information is decodable from the representations, but does not necessarily establish that the models have already computed the correct counts or that the primary failure occurs at the output stage. The distinction between information availability and information utilization deserves more careful treatment.

  2. Limited evidence for the proposed failure mechanism. The authors attribute the counting failures primarily to the output pathway and preferred-number biases. However, the observed output distributions are correlational, and no intervention demonstrates that correcting the proposed readout problem improves counting accuracy. In addition, the synthetic experiments may allow the probes to exploit periodic frequency information rather than explicit event-level counting. These limitations weaken the mechanistic interpretation of the results.

Questions for authors

Given that the linear probes are trained with supervised labels while the native models are evaluated zero-shot, how can we distinguish a genuine output-stage failure from information that is present in the representations but requires additional task-specific supervision to extract?

Have the authors considered interventions on the output pathway, such as lightweight readout calibration or task-specific adaptation, to test whether the proposed output-stage explanation leads to improved counting accuracy?

In the synthetic benchmark, is it possible that the probes recover repetition counts primarily through periodic frequency information rather than representations of individual repetitions? Could the authors clarify what evidence distinguishes these possibilities?

1 comment
AuthorsOpenReview ↗

Response to Reviewer rZK3

Thank you for the clear review. The distinction drawn between information that is decodable and information the model actually uses is the right one, and it changed how we state our main result. We ran the two suggested tests: a readout calibrated with the same supervision as the probe, and an intervention on the output pathway.

Q1 and Q2: Matched supervision and calibrated readout of the model's scores

We score the 20 candidate answers ("1" through "20") in each model's own output format and compare the model's free-form answer, zero-shot constrained decoding, and a linear calibration of the 20 scores. The calibration and the hidden-state probe use the same training clips and labels, and each model's rows use the same evaluation clips (nn = 1,975 and 1,976).

Readout (MAE / OBO)Cosmos-Reason2Qwen3-VL
Model's answer (free-form)3.54 / 39.5%6.63 / 26.3%
Constrained 20-way, zero-shot (summed log-prob)7.57 / 25.6%7.56 / 21.2%
Constrained 20-way, zero-shot (length-normalized)3.86 / 38.4%5.92 / 24.2%
Constrained 20-way, calibrated (supervised)3.27 / 46.7%4.24 / 36.5%
Hidden-state linear probe (supervised)2.56 / 51.4%3.01 / 47.7%
Duration only3.26 / 30.1%3.26 / 30.1%

First, calibration of the model's own number scores improves over its free-form answer on both models (MAE change -0.27, 95% CI [-0.51, -0.04] for Cosmos-Reason2; -2.39 [-2.75, -2.04] for Qwen3-VL). Second, with the same labeled data, the calibrated readout remains less accurate than the hidden-state probe. Matching the supervision does not remove the probe's advantage: under linear readouts, the hidden state carries more accessible count-predictive signal than the 20 answer scores. The two feature spaces differ, so this does not isolate the cause, and we do not infer that the model has computed the count. Constraining answers to the 20 valid counts does not remove the gap either; summed log-probabilities penalize two-token answers, which length normalization largely corrects.

Intervention on the output pathway

We intervene on the final residual as h′=h−α∥h∥w^h' = h - \alpha \lVert h \rVert \hat{w}, where w^\hat{w} is the normalized difference between the mean LM-unembedding rows of a token set PP and a token set NN. For Qwen3-VL, InternVL3.5, and Video-R1, PP contains the four most frequently generated integers on evaluation clips with k≥14k \ge 14; NN contains rarely generated integers from 14 to 20 excluding PP. Number tokens use single-token Arabic digits for 1 to 9 where available and leading-space English words otherwise. Cosmos-Reason2 instead uses hand-selected PP = {eight, ten, twelve, fifteen, 5, 8} and NN = {one, two, three, 1, 2, 3}; these were not derived from emission counts.

ModelChange in high-count MAE95% CI
Cosmos-Reason2-1.63[-2.08, -1.20]
Video-R1-0.80[-1.15, -0.48]
InternVL3.5-0.56[-0.76, -0.38]
Qwen3-VL-0.49[-0.73, -0.28]

The dose is the largest producing at least 95% valid-integer outputs (α=0.10\alpha = 0.10 for Cosmos-Reason2 and Qwen3-VL; 0.05 otherwise); at a common α=0.05\alpha = 0.05 the improvement keeps the same sign in all four families. Adding the probe direction has negligible effect (MAE change -0.01, 95% CI [-0.07, +0.05]).

Selection and holdout limits. The main-table token sets and doses were chosen using evaluation data; the CIs are therefore descriptive after selection. For a partial check, we made 20 random source-grouped 50/50 splits of 700 high-count clips, selected α\alpha on half A, and evaluated on half B with token sets fixed from the full pool. MAE decreased on half B in every split for all three tested families (mean change: Qwen3-VL -0.49, range -0.78 to -0.09; Video-R1 -0.80, range -1.27 to -0.41; InternVL3.5 -0.64, range -0.83 to -0.39). Half A alone recovered the same token set in 9, 15, and 10 of 20 splits, respectively. Cosmos-Reason2 was not tested in this check. These overlapping resamples test dose transfer, not a fully held-out token-and-dose procedure.

These interventions show that the model's preference for certain numbers is one cause of its high-count errors, but they do not explain the errors fully.

Q3: Frequency versus event-level counting in synthetic videos

In fixed-duration periodic videos, frequency and count covary, and we now interpret the synthetic probe as decoding periodic structure, not necessarily a sequence of counted events. The new controls weaken two alternative accounts: shuffling or repeating frames raises probe MAE to a constant-predictor baseline, and the answer-probe gap remains with all 64 frames, including for k≤16k \le 16. Single-frame exact match stays above chance (details in our reply to Reviewer p7Ja), so we do not claim static leakage is absent. On real videos, where duration varies, the hidden-state probe outperforms duration-only and duration-plus-action-type predictors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank all three reviewers for their careful and constructive feedback. Their comments sharpen two questions central to RepProbe: (i) whether linear probes recover temporal information rather than sampling artifacts or static cues, and (ii) what explains the difference between count information accessible to a supervised probe and the counts in the model's own answers. We address both with temporal-coverage controls, matched-supervision readouts, and interventions on the final representation.

The evidence supports a specific empirical conclusion: count-predictive information is accessible in internal representations but is not reliably reflected in the model's own number scores or its answers. This does not mean that a model has explicitly computed the count, and we no longer claim that the failure arises primarily at the output stage. The new results support four conclusions:

  1. The synthetic gap persists when all rendered frames are provided. With all 64 frames, answer off-by-one (OBO) accuracy is 23% for Qwen3-VL and 29% for Cosmos-Reason2, whereas the probes reach 93% and 91%. Restricting to k≤16k \le 16, where the temporally patched encoder has at least two positions per period, gives answer OBO of 28% and 36% against probe OBO of 94% and 93%. Probes trained on shuffled-frame or single-frame clips fall to the MAE of a constant predictor, although residual static cues remain.

  2. The probe advantage persists under matched supervision. We score all 20 candidate counts in each model's own output format and calibrate these scores with the same training clips and labels as the hidden-state probe. Calibration improves on the model's free-form answer, but its MAE remains higher than the probe's: 3.27 against 2.56 for Cosmos-Reason2, and 4.24 against 3.01 for Qwen3-VL.

  3. Number-token steering changes high-count answers. Subtracting a model-specific contrast of number-token unembedding rows reduces high-count MAE in four model families; injecting the probe direction does not help. In 20 source-grouped split-half resamples for three families, doses chosen on one half also reduce MAE on the other half. Because token sets were selected using the full evaluation data, this checks dose transfer only. These results implicate output-associated number directions without establishing a complete mechanism.

  4. The real-video finding is not confined to re-cut clips. On full-span RepCount videos, the hidden-state probe again achieves lower MAE than the duration-only predictor and the model's answers. Adding action type to the duration baseline does not improve it.

In the revision, we will separate three levels of evidence: information decodable from internal states, information accessible through the model's own number scores, and information expressed in its answers. RepProbe's contribution is a controlled behavioral and representation-level analysis of these distinctions, not the previously documented decline in high-count accuracy.

Dear Reviewers,

A friendly reminder to check the authors’ rebuttals and engage in the discussion, as the deadline is approaching. Authors still have time to address any further questions or concerns you may have, so your timely feedback would be greatly appreciated.

Thank you for your time and effort!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.