- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study investigated whether Visual Language Models (VLMs) can replicate the human visual ability to recognize content in videos based solely on visual motion. To this end, the authors proposed an algorithm that generates "appearance-free" stimuli (videos in which the textures have been replaced with i.i.d. Gaussian noise) from arbitrary videos, and used this to create a benchmark called AppearFreeVQA. Using this benchmark, the authors tested multiple VLMs and confirmed that these models cannot perceive appearance-free stimuli in the same way humans do, and that this bottleneck lies in the extraction of motion information. Furthermore, they demonstrated that using AppearFree stimuli as training data improves performance on camouflaged object detection tasks.
Critical strengths
The most significant contribution is the proposal of AppearFreeVQA. Compared to SpookyBench, it offers a wider variety of stimuli and includes more natural scene motion. The fact that the task was expanded from simply "What" (whether objects can be recognized) to "How" (motion direction or action) is also appreciated.
Critical weaknesses
Given what was already known from the SpookyBench paper, the additional differences seem incremental. It was already known that the perception of appearance-free motion is catastrophically impaired and that the bottleneck lies in extracting boundaries from motion. The gains obtained from the expanded tasks and stimuli in this new benchmark do not seem particularly significant. While the extension of the task from "What" to "How" is appreciated, it remains unclear whether this extension actually led to a deeper understanding. Perhaps the most noteworthy aspect is the "Application" (fine-tuning for camouflaged animal detection), but since these results are based on a single model, they seem somewhat preliminary to serve as a main contribution.
Another critical point concerns the generation of appearance-free stimuli, in that the necessity of MotionOptim is not sufficiently justified. While this explanation is provided in Appendix 1.8, the description in A 1.8 seems incorrect. If both foreground and background noise are independently sampled from the same i.i.d. Gaussian distribution, then even if the holes are filled with background noise, a single-frame still image would remain strictly i.i.d. Gaussian noise, and there would be no reason for discrepancies in boundary lines or local variance to occur. The authors should clarify why appearance leakage occurs in still images.
Additionally, the authors should cite the related work below and discuss how their method differs from it. While this paper highlights the ability to handle arbitrary video motions beyond hand-crafted rule-based motion, the method in the cited study already shares this characteristic.
Filip Ilic, Thomas Pock, and Richard P Wildes. Is appearance free action recognition possible? In European Conference on Computer Vision, pages 156–173. Springer, 2022.
Questions for authors
In the "Motion Misaligned" experiment, the model achieves relatively high accuracy even though the optical flow is replaced with that of a different video. Could the authors provide further explanation or intuition for why this occurs?
In the "Appearance Level Control" experiment, the stimuli are generated by solving a least-squares optimization problem. How does this formulation fundamentally differ from a simple alpha blend between the appearance-free image and the original RGB image? The current optimization approach seems unnecessarily complicated.
In Section 4.4, why was only NVILA-Lite-2B used? Would other models also show improvements? Also, wouldn't it be more effective to fine-tune the model so that optical flow information can be calculated separately and used as an additional input?