LookThere! Sparse Vision by Reinforced Selection

Essential visual information can be sparse, yet most visual computation remains dense. Though only a fraction matters, vision transformers process images as uniform sets of tokens, scaling cost with their number. As these models grow stronger, they grow larger, and the cost of computing everything climbs higher. Adaptive computation has shown that we can get by with less, but existing methods struggle at extreme sparsity and are difficult to _specialize_ to a task. We address this limitation by introducing _LookThere_, training an efficient selector of locations and an expressive extractor of representations to derive meaning from images _without ever processing the full high-resolution input_. We do so by learning the selector and extractor end-to-end with actor-critic reinforcement learning. The selector learns _where_ to look, the extractor _what_ to see, together saving computation by selecting only what is worth processing for a given task. We show that _LookThere_ selects the task-specific input, excelling at sparse recognition in high-resolution settings (traffic signs, billiards), and maintaining accuracy with as little as 0.2% of the input. It also generalizes across tasks and models, including global recognition (ImageNet classification), local recognition (ADE20K segmentation), zero-shot classification (by distillation), and regression (class agnostic enumeration). Across all settings, _LookThere_ surpasses state-of-the-art selection, providing a general and scalable framework for specialized and efficient adaptive computation.