- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on the reason of why VLM can recognize an action yet miss details such as a brief pause or a change in speed. It builds WARP PAIRS by changing the timing and order of video frames while largely preserving appearance. Small edits are hard to detect after sampling and encoding. For larger edits, a trained classifier can often generate the answer from the model’s visual representations, even when the model is close to the random-guessing. The paper then adds TD tokens, which summarize changes between consecutive visual representations. These reduce the training steps needed to learn selected edits by up to sixfold, but bring limited gains in final accuracy and transfer to natural motion.
Critical strengths
The controlled edits make the failures easier to interpret. By largely keeping appearance fixed and changing temporal properties, the benchmark helps separate sensitivity to motion details from recognition of the action or the scene.
Critical weaknesses
The experiments show when small temporal edits become detectable, but do not clearly separate information missed during frame sampling from information weakened during encoding. Using more frames changes the available evidence, and comparing models that merge different numbers of frames also changes the encoder. This limits how precisely the paper locates the source of the failure.
TD adds both temporal difference statistics and a trainable module with extra tokens. Yet statistics without TD already perform similarly in the probe tests for peak hold and freeze (Table 19). Adding the TD statistics back into visual tokens does not help, but that branch stays weakly weighted. The results therefore do not fully establish whether the speedup comes from differencing, the added module, or where the information enters the model.
Questions for authors
Can you test the effect of temporal merging while keeping the sampled frames fixed? This would help distinguish losses caused by sampling from those caused by encoding several frames together.
Would the same TD module and token budget still speed up training with statistics that do not use temporal differences? Also, does adding TD statistics into visual tokens remain ineffective when the fusion strength is increased?