- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies sparse video understanding through a lightweight token selector and a deeper feature extractor. Building on prior selection–extraction architectures and reinforcement-learning formulations, it introduces hierarchical temporal and spatial selection for video. Distillation pretraining prepares the models for sparse inputs, followed by joint optimization of task prediction and token selection using task supervision and group-relative policy gradients. Text conditioning allows selection to depend on a question.
Critical strengths
- Building on prior selection–extraction frameworks, the method uses a lightweight selector to inspect a low-resolution video before a deeper extractor processes selected high-resolution patches. Reinforcement learning links discrete selection directly to task performance and sparsity, while joint training allows the extractor to adapt to the selected inputs. Distillation initialization and success-gated sparsity rewards provide a reasonable training strategy for avoiding premature token removal. The resulting design aligns computational allocation with the prediction objective.
- Temporal grouping, progressive refinement to individual frames, and hierarchical spatial selection organize the large video-selection space into structured decisions. This is a well-motivated extension of image-based selection to video, where relevant evidence may be concentrated in brief moments and small regions. Question conditioning further allows the same video to yield different selections for different queries. The frame-head and question-blind ablations in Figure 6(a) provide supporting evidence for these temporal and contextual decisions.
Critical weaknesses
- Most accuracy and mAP comparisons are presented as curves or bars in Figures 2 and 7–10, without complete numerical results in the appendix. Although computational costs and selected accuracy values are reported, exact scores for every method, dataset, backbone, and sparsity level are unavailable in the manuscript. This limits verification of the reported gains, particularly small differences between methods, and makes precise comparison by subsequent work more difficult. The computational summaries also require clarification. At 95% sparsity, Figure 2 reports 30.9G, 37.1G, and 148.9G FLOPs for the proposed method, STTS, and vid-TLDR, whereas simple averages of the corresponding values in Figures 7–9 are approximately 32.87G, 39.43G, and 178.80G. Any different aggregation or configuration should be specified. Throughput measurements also lack sufficient documentation of hardware, inference batch size, numerical precision, and measurement scope. The identical efficiency entries for the proposed method and LookWhen should be identified as independent measurements or shared architecture-based estimates.
- The frame-head ablation in Figure 6(a) is conducted on the constructed A2D task combining four clips. It supports frame selection but does not separately isolate the coarse-to-fine temporal curriculum or GRPO relative to a simpler policy-gradient baseline. Since the framework builds on prior image-selection work, matched-budget ablations on standard action-recognition datasets would better establish the additional value of these mechanisms. Implementation details also require clarification, including the mask-group size (M), temporal curriculum schedule, and entropy regularization. Section 4.1 describes 40 total training epochs including 10 warmup epochs, whereas Appendix A.2.2 describes 40 RL epochs excluding 10 warmup epochs. Any distinction between the main and tuning protocols should be stated explicitly.
- Figure and table layout should be improved. In Figure 2 and Figures 7–9, the plots and accompanying tables use different widths and outer boundaries; the tables in Figures 7–9 also have uneven column spacing, particularly between LookWhen and Ours. Layout, spacing, and caption styles should be made more consistent. Figure 7 omits the percentage unit for sparsity, and accuracy differences in Figure 10 and Appendix A.2.1 should be expressed in percentage points. Figure 1’s claim that the extractor answers from selected tokens “alone” should also be corrected because selector context tokens are supplied to the extractor.
Questions for authors
- Please provide the numerical accuracy and mAP results underlying Figures 2 and 7–10. How were the “Mean over grid” values in Table 1 calculated? If all groups summarize the same complete 81-run grid, why do they imply overall averages of approximately 82.5% and 81.7%?
- How are the FLOPs in Figure 2 aggregated from the backbone-specific results in Figures 7–9? Please explain the numerical differences and specify the hardware, inference batch size, precision, and operations included in throughput measurements. Were the proposed method and LookWhen independently profiled?
- Do the main results use stochastic sampling, probability thresholds, or top-k selection at inference? How is the constraint (|P|≤K) enforced, and how does the selection rule interact with the temporal and spatial hierarchy? Please specify actual token-count statistics and whether selector register and frame tokens are included in the reported budgets and FLOPs.
- Can matched-budget ablations be provided on standard action-recognition datasets to isolate the coarse-to-fine temporal curriculum and the group-relative policy-gradient update? The existing frame-head ablation does not separately establish their contributions.
3 comments
Response to Reviewer 7DCD (2/2)
Computational Summary (Q2)
We thank the reviewer for identifying the discrepancy between Figure 2 and Figures 7-9. The measurements in Figures 7-9 are correct. Figure 2 used earlier measurements that omitted the FLOPs of operations performed by PyTorch's default fused attention kernel, which was corrected in Figures 7-9. We have corrected Figure 2 to use the same measurements as Figures 7-9.
Implementation Detail Clarification (W2)
We agree that further detail will help reproduction of our work and have expanded "Implementation Details" in Section 3.5 accordingly. We use a mask-group size of M = 32 throughout and a grid-resolution curriculum of [4, 8, 14], with each stage occupying one third of training. We have also corrected Appendix A.2.2 to match Section 4.1, specifying 40 total training epochs, including 10 warmup epochs.
Computation Measurements (Q2, W1)
We appreciate the reviewer's request for more complete measurement details. We measure FLOPs, throughput, and peak memory using each method's reference implementation on a single NVIDIA RTX A6000 with 48 GB of memory. Throughput is measured in clips per second, in evaluation mode with bf16 autocast, using the largest power-of-two batch size that fits in memory for each configuration. We have added these details to Appendix A.1 under "Measuring Computation."
LookWhen and our method share the same inference architecture and differ in how selection is trained, explaining their equivalent computational requirements at matched token budgets. We have clarified this architectural equivalence in Appendix A.1.
Figure and Table Layout (W3)
We appreciate the reviewer's care for clarity, and have adopted their suggestions to improve the presentation of our work.
Layout. The plots and tables in Figures 2 and 7-9 now span the full text width and share the same outer boundaries. The tables in Figures 7-9 use evenly spaced columns, including between LookWhen and Ours.
Captions. Figures 2, 4, and 7-10 now follow a consistent caption style, with a bold title followed by a description.
Units. Figure 7 now labels sparsity in %. Accuracy differences in Figures 4 and 10, Appendix A.2.1, and the main text are expressed in percentage points.
Figure 1. We have corrected the caption and method overview to state that the extractor uses selected patches together with the selector's register and frame tokens, which summarize a downsampled copy of the full video.
[R1] Sreehari Rammohan et al. LookThere! Sparse Vision by Reinforced Selection. NeurIPS 2026.
Dear Reviewer 7DCD,
Thank you again for your detailed feedback. We have provided responses to your concerns, including additional experimental results. As the discussion period is approaching its end, we would appreciate your thoughts on whether our responses and additional results have addressed your concerns. We would be happy to clarify any remaining questions.
Thank you for your time and consideration.
Response to Reviewer 7DCD (1/2)
We thank the reviewer for their thoughtful feedback and for recognizing the contributions of our work. We address their questions and suggestions below.
Additional Ablations (Q4, W2)
We appreciate the reviewer's suggestion to better isolate the contributions of individual components. During the rebuttal period, we ran additional ablations of the spatial hierarchy, frame head, RL algorithm, selector adaptation, reward gating, and warmup. We refer the reviewer to our response to Reviewer PpHL for the full results and discuss the requested comparisons below.
Group-relative policy-gradient update. Following the reviewer's suggestion, we compared GRPO with REINFORCE using a running reward baseline. Both use DINOv3 under matched training schedules of 10 warmup epochs followed by 30 RL epochs. At a 10% token budget, GRPO outperforms REINFORCE on every evaluated task.
Table R1. Comparison of RL algorithms at a 10% token budget
Selector adaptation. To isolate the benefit of adapting the selector during post-training, we compared our method with freezing the pretrained selector (Sec. 3.3) and fine-tuning only the extractor. We train and evaluate at 10%, 5%, and 1% token budgets, reporting final-epoch accuracy below. RL adaptation of the selector improves accuracy at every evaluated budget, with gains of 6.04–10.31 percentage points on Diving48 and up to 3.76 points on SSv2. We have included both these results in Sec. 4.3 for ablations, as well as Appendices A.4 and A.5.
Table R2. Frozen vs. jointly adapted selector at varying token budgets
Hierarchical selection. We adopted LookThere's [R1] spatial hierarchy to reduce sampling variance in the multivariate Bernoulli policy. To assess its contribution separately, we compared hierarchical and flat policies on SSv2, Jester, and EK100 (20K subsets), and the full Diving48 dataset. At a 10% token budget, DINOv3 achieves 58.75%/95.71%/33.02%/85.92% accuracy with the hierarchy, compared with 59.06%/95.76%/28.70%/85.87% without it.
Although the hierarchy produces cleaner selection masks, flat policies perform similarly, suggesting that spatial cascading is not necessary for accuracy in these settings. We hypothesize that the hierarchy is more useful at the higher resolutions used by LookThere, where the larger token count increases the action space and sampling variance. Its benefit appears reduced at the lower resolutions we use for video. We discuss these findings in Sec. 4.3 for ablations, and provide additional comparative visualizations in Appendix A.7.
Selection Rule (Q3)
During training, we sample selections from a stochastic policy. At inference, we compute each patch's selection probability under the hierarchical policy and retain the top K patches, directly enforcing the token budget. We have clarified this rule under "Implementation Details" in Section 3.5 and added token-count statistics alongside the numerical results in Table 1 in the appendix. Selector register and frame tokens are excluded from the patch-token budget but included in the reported FLOPs.
Numerical Results (Q1)
We agree that exact scores support verification and future comparisons. We have added the numerical accuracy and mAP results underlying Figures 2 and 7–10 to Table 1 in the appendix. All groups in Table 1 summarize the same complete 81-run grid. We have corrected the "Mean over grid" entries for the sparsity weights so that the grouped results consistently recover the overall mean of 82.5%.