- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether increasing temporal sampling density helps video grounding equally across event durations. Using the 4B version of TimeLens2, the authors find that increasing the sampling rate from 4 to 8 fps improves localization of short events but hurts events of intermediate duration. This interaction remains after averaging over sampling phases, while the overall change is not statistically significant. The authors investigate the effect through frame duplication, timestamp perturbations, changes to predicted interval geometry, and training interventions. They then introduce a policy that runs a second grounding pass on a local crop only when the initial prediction is short. The policy improves short event accuracy and transfers across datasets with fixed settings.
Critical strengths
-
A useful empirical finding supported by a meaningful sampling control. The paper goes beyond showing that short events are difficult. It shows that increasing density can have opposite effects on different duration groups within the same model. The phase analysis makes this result more credible.
-
The experiments examine several plausible explanations and expose the performance tradeoff. Frame duplication, timestamp perturbations, crop budget controls, and routing comparisons at matched activation rates are useful attempts to understand where the gains come from. The paper also reports the costs to other duration groups. Improving the weakest duration group can be worthwhile even when average accuracy barely changes.
Critical weaknesses
-
The proposed mechanism does not yet explain the full interaction. Duplicating frames shows that the short event gain does not require new raw visual content. It still changes the visual representation. When two frames are merged into a temporal patch, duplication changes which frames are fused, the number of representations, and the input sequence. Part of the gain could come from this change in visual encoding. Permuting timestamps shows that the model depends on readable time information. Since permutation severely damages localization, the smaller duplication gain could also reflect that general loss of performance. The quantization and offset tests partly address this issue, but do not separate token count from marker granularity. Replacing both the center and length of a prediction simply gives back the complete 8 fps prediction. That result describes the outputs without identifying the process inside the model. The unexplained negative effect is my main concern. Section 4.2 states that interval contraction does not explain the full response after phase averaging. The loss on events of 3 to 10 seconds is an important part of the finding. Do duplication and timestamp changes also reproduce this loss? If not, the positive and negative effects may have different causes. All positive model replications also use TimeLens supervision. The result on original ActivityNet annotations helps address the role of test labels, but leaves the training source unchanged. I am convinced more by the observed interaction than by the claim of a shared mechanism across generative grounding models.
-
The policy gains need the same phase check used for the main finding. In Section 4.1, the default 4 fps phase is the weakest tested phase for short events. Its mIoU is 0.3622, compared with a phase average of 0.3946. Table 2 still measures the main policy gain of 0.172 against that default prediction. The released configuration is a fair baseline. What is missing is evidence that the gain holds when the phase changes. A different first pass phase can change the predicted interval, the routing decision, and the second crop. The whole policy needs to be run again for each phase. Subtracting the phase averaged baseline from the existing policy score would not test this. Cropping changes context, the time origin, and the sampling grid as well as density. The effect of raising the full video rate from 4 to 8 fps need not match the effect of local refinement. The crop controls on other datasets are useful, but the connection is not yet clear for the main Charades result.
Questions for authors
-
Does the policy gain hold across sampling phases? Are there results from running the whole policy with different first pass phases and all selected settings fixed? Please give the paired gains by duration group and the routing activation rates. On Charades, is there a crop control that keeps the original density, or one that preserves the absolute sampling grid?
-
Do the controls also reproduce the negative effect? After phase averaging, does frame duplication hurt events of 3 to 10 seconds? Please give the factorial interaction estimates and their uncertainty. Separate significance tests for each condition do not test the difference between conditions.
-
Does the effect occur without TimeLens supervision? Do the authors have results from a model that can output short intervals but has not used TimeLens training data? If no such results are available, would they limit the claim to the tested conditions?
-
Why do the text and response curve give different values? Line 164 gives a gain of 0.059 at 2.1 seconds. The curve in Figure 1a appears close to zero at that point. Do these refer to different estimates, or does one need correction? Please explain how the curve and its zero crossing were estimated.