acceptodds
Under review as a conference paper at ICLR 2027

Sparse Video Understanding Without the Heuristics

Abstract

Video recognition often requires only a handful of tokens, yet which tokens matter depends on the task. Existing approaches for sparse video computation rely on heuristics such as motion, diversity, or reconstruction to guide selection. These signals capture redundancy rather than task relevance, often retaining more tokens than necessary. We instead optimize selection directly for the task, jointly learning a lightweight selector that glimpses across space and time and a deeper extractor that processes only the selected tokens. Both are trained end-to-end through reinforcement learning, directly coupling the learning of selection with the performance of recognition. Across six action-recognition benchmarks (Kinetics-400, SSv2, Epic-Kitchens, Diving48, Jester, and Charades) and two text-conditioned tasks (A2D and CLEVRER), our approach pushes the Pareto frontier for accuracy-efficiency, outperforming state-of-the-art efficient video methods while generalizing across architectures (DINOv3, VideoMAEv2, and InternVideo2).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Vision Transformer, VideoQA, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7DCDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies sparse video understanding through a lightweight token selector and a deeper feature extractor. Building on prior selection–extraction architectures and reinforcement-learning formulations, it introduces hierarchical temporal and spatial selection for video. Distillation pretraining prepares the models for sparse inputs, followed by joint optimization of task prediction and token selection using task supervision and group-relative policy gradients. Text conditioning allows selection to depend on a question.

Critical strengths
  1. Building on prior selection–extraction frameworks, the method uses a lightweight selector to inspect a low-resolution video before a deeper extractor processes selected high-resolution patches. Reinforcement learning links discrete selection directly to task performance and sparsity, while joint training allows the extractor to adapt to the selected inputs. Distillation initialization and success-gated sparsity rewards provide a reasonable training strategy for avoiding premature token removal. The resulting design aligns computational allocation with the prediction objective.
  2. Temporal grouping, progressive refinement to individual frames, and hierarchical spatial selection organize the large video-selection space into structured decisions. This is a well-motivated extension of image-based selection to video, where relevant evidence may be concentrated in brief moments and small regions. Question conditioning further allows the same video to yield different selections for different queries. The frame-head and question-blind ablations in Figure 6(a) provide supporting evidence for these temporal and contextual decisions.
Critical weaknesses
  1. Most accuracy and mAP comparisons are presented as curves or bars in Figures 2 and 7–10, without complete numerical results in the appendix. Although computational costs and selected accuracy values are reported, exact scores for every method, dataset, backbone, and sparsity level are unavailable in the manuscript. This limits verification of the reported gains, particularly small differences between methods, and makes precise comparison by subsequent work more difficult. The computational summaries also require clarification. At 95% sparsity, Figure 2 reports 30.9G, 37.1G, and 148.9G FLOPs for the proposed method, STTS, and vid-TLDR, whereas simple averages of the corresponding values in Figures 7–9 are approximately 32.87G, 39.43G, and 178.80G. Any different aggregation or configuration should be specified. Throughput measurements also lack sufficient documentation of hardware, inference batch size, numerical precision, and measurement scope. The identical efficiency entries for the proposed method and LookWhen should be identified as independent measurements or shared architecture-based estimates.
  2. The frame-head ablation in Figure 6(a) is conducted on the constructed A2D task combining four clips. It supports frame selection but does not separately isolate the coarse-to-fine temporal curriculum or GRPO relative to a simpler policy-gradient baseline. Since the framework builds on prior image-selection work, matched-budget ablations on standard action-recognition datasets would better establish the additional value of these mechanisms. Implementation details also require clarification, including the mask-group size (M), temporal curriculum schedule, and entropy regularization. Section 4.1 describes 40 total training epochs including 10 warmup epochs, whereas Appendix A.2.2 describes 40 RL epochs excluding 10 warmup epochs. Any distinction between the main and tuning protocols should be stated explicitly.
  3. Figure and table layout should be improved. In Figure 2 and Figures 7–9, the plots and accompanying tables use different widths and outer boundaries; the tables in Figures 7–9 also have uneven column spacing, particularly between LookWhen and Ours. Layout, spacing, and caption styles should be made more consistent. Figure 7 omits the percentage unit for sparsity, and accuracy differences in Figure 10 and Appendix A.2.1 should be expressed in percentage points. Figure 1’s claim that the extractor answers from selected tokens “alone” should also be corrected because selector context tokens are supplied to the extractor.
Questions for authors
  1. Please provide the numerical accuracy and mAP results underlying Figures 2 and 7–10. How were the “Mean over grid” values in Table 1 calculated? If all groups summarize the same complete 81-run grid, why do they imply overall averages of approximately 82.5% and 81.7%?
  2. How are the FLOPs in Figure 2 aggregated from the backbone-specific results in Figures 7–9? Please explain the numerical differences and specify the hardware, inference batch size, precision, and operations included in throughput measurements. Were the proposed method and LookWhen independently profiled?
  3. Do the main results use stochastic sampling, probability thresholds, or top-k selection at inference? How is the constraint (|P|≤K) enforced, and how does the selection rule interact with the temporal and spatial hierarchy? Please specify actual token-count statistics and whether selector register and frame tokens are included in the reported budgets and FLOPs.
  4. Can matched-budget ablations be provided on standard action-recognition datasets to isolate the coarse-to-fine temporal curriculum and the group-relative policy-gradient update? The existing frame-head ablation does not separately establish their contributions.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7DCD (1/2)

We thank the reviewer for their thoughtful feedback and for recognizing the contributions of our work. We address their questions and suggestions below.

Additional Ablations (Q4, W2)
We appreciate the reviewer's suggestion to better isolate the contributions of individual components. During the rebuttal period, we ran additional ablations of the spatial hierarchy, frame head, RL algorithm, selector adaptation, reward gating, and warmup. We refer the reviewer to our response to Reviewer PpHL for the full results and discuss the requested comparisons below.

Group-relative policy-gradient update. Following the reviewer's suggestion, we compared GRPO with REINFORCE using a running reward baseline. Both use DINOv3 under matched training schedules of 10 warmup epochs followed by 30 RL epochs. At a 10% token budget, GRPO outperforms REINFORCE on every evaluated task.

Table R1. Comparison of RL algorithms at a 10% token budget

MethodDiving48SSv2 (20K)EK100 (20K)
REINFORCE81.6257.0524.67
GRPO86.4558.7929.70
Gain (%)+4.83+1.74+5.03

Selector adaptation. To isolate the benefit of adapting the selector during post-training, we compared our method with freezing the pretrained selector (Sec. 3.3) and fine-tuning only the extractor. We train and evaluate at 10%, 5%, and 1% token budgets, reporting final-epoch accuracy below. RL adaptation of the selector improves accuracy at every evaluated budget, with gains of 6.04–10.31 percentage points on Diving48 and up to 3.76 points on SSv2. We have included both these results in Sec. 4.3 for ablations, as well as Appendices A.4 and A.5.

Table R2. Frozen vs. jointly adapted selector at varying token budgets

MethodDiving48 10%Diving48 5%Diving48 1%SSv2 10%SSv2 5%SSv2 1%Jester 10%Jester 5%Jester 1%
Frozen selector79.1972.2846.3558.5153.5733.0695.4093.9181.89
GRPO86.4582.5952.3958.7956.0836.8295.7995.3582.22
Gain (%)+7.26+10.31+6.04+0.28+2.51+3.76+0.39+1.44+0.33

Hierarchical selection. We adopted LookThere's [R1] spatial hierarchy to reduce sampling variance in the multivariate Bernoulli policy. To assess its contribution separately, we compared hierarchical and flat policies on SSv2, Jester, and EK100 (20K subsets), and the full Diving48 dataset. At a 10% token budget, DINOv3 achieves 58.75%/95.71%/33.02%/85.92% accuracy with the hierarchy, compared with 59.06%/95.76%/28.70%/85.87% without it.

Although the hierarchy produces cleaner selection masks, flat policies perform similarly, suggesting that spatial cascading is not necessary for accuracy in these settings. We hypothesize that the hierarchy is more useful at the higher resolutions used by LookThere, where the larger token count increases the action space and sampling variance. Its benefit appears reduced at the lower resolutions we use for video. We discuss these findings in Sec. 4.3 for ablations, and provide additional comparative visualizations in Appendix A.7.

Selection Rule (Q3)
During training, we sample selections from a stochastic policy. At inference, we compute each patch's selection probability under the hierarchical policy and retain the top K patches, directly enforcing the token budget. We have clarified this rule under "Implementation Details" in Section 3.5 and added token-count statistics alongside the numerical results in Table 1 in the appendix. Selector register and frame tokens are excluded from the patch-token budget but included in the reported FLOPs.

Numerical Results (Q1)
We agree that exact scores support verification and future comparisons. We have added the numerical accuracy and mAP results underlying Figures 2 and 7–10 to Table 1 in the appendix. All groups in Table 1 summarize the same complete 81-run grid. We have corrected the "Mean over grid" entries for the sparsity weights so that the grouped results consistently recover the overall mean of 82.5%.

AuthorsOpenReview ↗

Response to Reviewer 7DCD (2/2)

Computational Summary (Q2)
We thank the reviewer for identifying the discrepancy between Figure 2 and Figures 7-9. The measurements in Figures 7-9 are correct. Figure 2 used earlier measurements that omitted the FLOPs of operations performed by PyTorch's default fused attention kernel, which was corrected in Figures 7-9. We have corrected Figure 2 to use the same measurements as Figures 7-9.

Implementation Detail Clarification (W2)
We agree that further detail will help reproduction of our work and have expanded "Implementation Details" in Section 3.5 accordingly. We use a mask-group size of M = 32 throughout and a grid-resolution curriculum of [4, 8, 14], with each stage occupying one third of training. We have also corrected Appendix A.2.2 to match Section 4.1, specifying 40 total training epochs, including 10 warmup epochs.

Computation Measurements (Q2, W1)
We appreciate the reviewer's request for more complete measurement details. We measure FLOPs, throughput, and peak memory using each method's reference implementation on a single NVIDIA RTX A6000 with 48 GB of memory. Throughput is measured in clips per second, in evaluation mode with bf16 autocast, using the largest power-of-two batch size that fits in memory for each configuration. We have added these details to Appendix A.1 under "Measuring Computation."

LookWhen and our method share the same inference architecture and differ in how selection is trained, explaining their equivalent computational requirements at matched token budgets. We have clarified this architectural equivalence in Appendix A.1.

Figure and Table Layout (W3)
We appreciate the reviewer's care for clarity, and have adopted their suggestions to improve the presentation of our work.

Layout. The plots and tables in Figures 2 and 7-9 now span the full text width and share the same outer boundaries. The tables in Figures 7-9 use evenly spaced columns, including between LookWhen and Ours.

Captions. Figures 2, 4, and 7-10 now follow a consistent caption style, with a bold title followed by a description.

Units. Figure 7 now labels sparsity in %. Accuracy differences in Figures 4 and 10, Appendix A.2.1, and the main text are expressed in percentage points.

Figure 1. We have corrected the caption and method overview to state that the extractor uses selected patches together with the selector's register and frame tokens, which summarize a downsampled copy of the full video.

[R1] Sreehari Rammohan et al. LookThere! Sparse Vision by Reinforced Selection. NeurIPS 2026.

Dear Reviewer 7DCD,

Thank you again for your detailed feedback. We have provided responses to your concerns, including additional experimental results. As the discussion period is approaching its end, we would appreciate your thoughts on whether our responses and additional results have addressed your concerns. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Reviewer_JxbXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a select-then-extract approach for efficient video recognition with ViTs. A lightweight selector looks at a low-resolution copy of the video and picks frames first, then patches; the extractor processes only the selected tokens. The selector is pretrained with RL on K400+SSv2 with feature reconstruction as the reward, and is then fine-tuned jointly with the extractor on each downstream task with GRPO, using the task loss as the reward. The method extends LookThere to video and follows LookWhen's training pipeline. Experiments cover six action recognition datasets, three ViT-B backbones (DINOv3, InternVideo2, VideoMAEv2) and 90/95/99% sparsity, plus text-conditioned selection on A2D and CLEVRER. The main result is higher accuracy than LookWhen at the same FLOPs, with the largest gains on the two video backbones.

Critical strengths

Because tokens are chosen before the extractor runs, the FLOP savings translate into actual speed: about 485 clips/s at 95% sparsity versus 58.7 for the dense model (Fig. 2), with higher throughput and lower memory than token merging methods. I see this as the paper's main practical value. The evaluation is broad (six datasets, three backbones, four baselines). On InternVideo2 and VideoMAEv2 the method improves over LookWhen by about 7 to 16 points on average (Fig. 4). If these results hold up after clarification, learned selection appears more robust on video backbones whose patch features suit heuristics less well.

Critical weaknesses
  1. The central claim is not isolated. Most components come from LookThere and LookWhen, so the contribution rests on task-optimized selection and temporal selection, and where the gain comes from matters. Apart from the selector's training objective, the method also differs from LookWhen in hierarchical temporal selection and other components, so the current comparison cannot attribute the gain to task-optimized selection. The selector is also pretrained with a reconstruction objective, which sits awkwardly with the paper's criticism of reconstruction-based heuristics. Since the claim in the title depends on this attribution, this point weighs most in my rating. 2 .. The evidence for temporal selection is weak. Temporal selection is the main addition over prior work, yet its only quantitative ablation is on A2D. A2D samples are built by concatenating semantically different clips, and the question names the actor, so picking the right segment is close to category recognition. That says little about the method's ability to localize in time, and none of the six action recognition datasets has such an ablation.
Questions for authors

For DINOv3 on SSv2 at 1% of tokens, Fig. 7 shows the method below LookWhen, while Fig. 10 reports a gain of about +12. Which is correct? If Fig. 7 is correct, the average gain on DINOv3 drops to roughly zero. In the VideoMAEv2 visualizations (Figs. 31 to 34), LookWhen uses half as many tokens as the proposed method, yet Fig. 9 reports identical FLOPs for both. Are the token budgets matched in the quantitative comparison? How does the full method compare with a variant in which the pretrained selector is frozen and only the extractor is fine-tuned downstream? How do results on Diving48 or SSv2 change when the frame head is removed?

4 comments
AuthorsOpenReview ↗

Response to Reviewer JxbX (1/2)

We thank the reviewer for their thoughtful feedback and for recognizing the practical value of our work and the breadth of our evaluation. We address their concerns below.

Pre-trained Reconstruction Objective
We appreciate the reviewer raising this point and would like to clarify that reconstruction is used only for pre-training, while downstream training directly optimizes for task performance. We pretrain to initialize the selector to identify informative regions and the extractor to process sparse inputs. Without a task-specific objective at this stage, latent reconstruction provides a natural proxy for learning to reduce inputs while preserving their semantic content. We note that even during pre-training, selection is learned through joint, end-to-end optimization with the extractor, using the feature-reconstruction loss from distillation as the RL reward rather than relying on selection heuristics.

During downstream training, we jointly fine-tune the selector through RL and the extractor using the task objective, allowing selection to adapt directly to prediction. We view this task-specific, end-to-end optimization as our central contribution, and it helps most, as the reviewer notes, on video backbones whose patch features suit heuristics less well. Accordingly, we believe only using reconstruction for initialization remains consistent with our criticism of relying on reconstruction alone to determine token importance.

Evidence for Temporal Selection
We appreciate the reviewer's suggestion to strengthen the evidence for temporal selection. To assess the frame head's contribution, we compared our model with and without it on 20K subsets of SSv2, Jester, and EK100, and the full Diving48 dataset. At a 10% token budget, DINOv3 achieves 58.88%/95.49%/30.29%/86.45% accuracy with the frame head, compared with 59.23%/96.05%/30.08%/85.98% without it. These comparable results suggest limited benefit from explicit frame selection when most frames are informative, as in these action-recognition tasks. Its benefit is greater when the information needed for prediction occupies only a short interval, as in A2D. We have included these results in Appendix A.11, referenced in the ablation discussion in Sec. 4.3.

Action-conditioned A2D. During rebuttal, we extended our frame-head ablations to an action-conditioned variant of A2D and a longer-context Diving48 task in which the relevant information occupies only a short interval. For A2D, instead of naming the actor and asking for the action, we name the action and ask for the actor ("Who is walking?" instead of "What is the dog doing?"). This removes the named-actor cue and requires selection based on the queried action. Using 0.32%/0.64%/1% of tokens, our method with a frame head achieves 63.2%/66.4%/69.0% accuracy, compared with 48.0%/51.7%/60.1% without it. When prompted with a different randomly selected action, both models adapt their selections and maintain similar accuracies. We include and discuss these results in Appendix A.8 and Sec 4.3.

Long-context Diving48. The second task uses longer Diving48 clips in which the discriminative information occupies only a short interval. We identify the core dive segment and place it at a random position in a 64-frame clip, surrounded by less informative footage from the beginning and end of the same video. Our selector processes these clips in two windows of 32 frames, producing 64 frame scores. We select the 16 highest-scoring frames and pass them to the frozen extractor from the paper to predict the label. We compare this with a patch-only baseline that trains a selector and extractor over all 64 frames.

At 90%/95%/99% sparsity within the selected 16 frames, the frame-head model achieves 81.2%/76.9%/41.6% accuracy using 95/80/68 GFLOPs, compared with 80.7%/75.9%/38.7% accuracy and 109/93/81 GFLOPs for the patch-only model. Figure 6d plots these accuracy–FLOP curves, showing that the frame-head model achieves higher accuracy at lower computational cost across the reported operating points.

We believe these results strengthen the evidence that the frame head learns useful temporal selection when only a short part of the video is informative, supporting the frame head's contribution.

AuthorsOpenReview ↗

Response to Reviewer JxbX (2/2)

Isolation of the Central Claim
We thank the reviewer for this suggestion and agree that the contribution of task-specific optimization can be more clearly isolated. During the rebuttal period, we ran additional ablations of the spatial hierarchy, frame head, RL algorithm, selector adaptation, reward gating, and warmup. We refer the reviewer to our response to Reviewer PpHL for the full results and discuss the requested comparisons below.

Hierarchical/temporal selection. We adopted LookThere's [R1] spatial hierarchy to reduce sampling variance in the multivariate Bernoulli policy. To assess its contribution separately, we compared hierarchical and flat policies on SSv2, Jester, and EK100 (20K subsets), and the full Diving48 dataset. At a 10% token budget, DINOv3 achieves 58.75%/95.71%/33.02%/85.92% accuracy with the hierarchy, compared with 59.06%/95.76%/28.70%/85.87% without it.

Although the hierarchy produces cleaner selection masks, flat policies perform similarly, suggesting that spatial cascading is not necessary for accuracy in these settings. We hypothesize that the hierarchy is more useful at the higher resolutions used by LookThere, where the larger token count increases the action space and sampling variance. Its benefit appears reduced at the lower resolutions we use for video. We discuss these findings in Sec. 4.3 for ablations, and provide additional comparative visualizations in Appendix A.7.

Comparison to frozen selector. Following the reviewer's suggestion, we compared our method with freezing the pretrained selector (Sec. 3.3) and fine-tuning only the extractor. This keeps the selection architecture fixed while testing whether task-specific RL improves on reconstruction-based pretraining. We train and evaluate at 10%, 5%, and 1% token budgets, reporting final-epoch accuracy below. Adapting the selector improves accuracy at every evaluated budget, with gains of 6.04–10.31 percentage points on Diving48 and up to 3.76 points on SSv2. We have included both these results in Sec. 4.3 for ablations, as well as Appendices A.4 and A.5.

Table 1. Frozen vs. jointly adapted selector at varying token budgets

MethodDiving48 10%Diving48 5%Diving48 1%SSv2 10%SSv2 5%SSv2 1%Jester 10%Jester 5%Jester 1%
Frozen selector79.1972.2846.3558.5153.5733.0695.4093.9181.89
GRPO86.4582.5952.3958.7956.0836.8295.7995.3582.22
Gain (%)+7.26+10.31+6.04+0.28+2.51+3.76+0.39+1.44+0.33

DINOv3 on SSv2 at 1%
We thank the reviewer for their careful reading and for identifying this inconsistency. Figure 7 reports the correct results, and we have corrected the reporting error in Figure 10. Our method remains competitive with LookWhen on DINOv3 and shows stronger generalization and robustness to the video backbones InternVideo2 and VideoMAEv2.

VideoMAEv2 Visualizations
We thank the reviewer for catching this error. The LookWhen visualizations were generated incorrectly for VideoMAEv2, and we have corrected them. We confirm that token budgets are matched in the quantitative comparison and the reported numbers have not changed.

[R1] Sreehari Rammohan et al. LookThere! Sparse Vision by Reinforced Selection. NeurIPS 2026.

Dear Reviewer JxbX,

Thank you again for your detailed feedback. We have provided responses to your concerns, including additional experimental results. As the discussion period is approaching its end, we would appreciate your thoughts on whether our responses and additional results have addressed your concerns. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Thank you for the detailed responses and additional experiments. The frozen-selector ablation addresses my concern about the contribution of task-specific adaptation, and the additional temporal-selection experiments clarify where the frame head is useful. I also accept the explanation of reconstruction pretraining and the clarifications regarding the figure inconsistencies.

The rebuttal has addressed my main concerns. Given these additions and the practical efficiency gains, I am raising my rating from 2 (Weak rejection) to 3 (Weak acceptance). Please ensure that the corrections and updated conclusions are reflected consistently throughout the revised paper.

Reviewer_PpHLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a task-driven framework for sparse video understanding. A lightweight selector observes downsampled video and hierarchically selects frames and spatial patches; a deeper extractor processes only the chosen high-resolution tokens. Its main contributions are replacing proxy-based selection with reinforcement learning, introducing coarse-to-fine temporal-spatial selection, using distillation for stable initialization, and extending selection to text-conditioned tasks. The selector and extractor are jointly optimized with a gated accuracy-sparsity reward and GRPO. Validation covers six action-recognition datasets, three ViT-B backbones, and text-conditioned retrieval and question answering. At 90–99% sparsity, the method generally achieves a strong accuracy-efficiency frontier, particularly on video-pretrained backbones. Overall, the results support the central claim that task relevance is a better selection objective than generic diversity, although broader scalability remains unproven.

Critical strengths
  1. Task-driven selection is both well motivated and empirically consequential. Diversity, motion, and reconstruction are imperfect proxies for downstream relevance. Directly optimizing selection for task performance is therefore reasonable. The consistent gains across DINOv3, InternVideo2, and VideoMAEv2—especially when heuristic selection transfers poorly—suggest that the method addresses a genuine limitation rather than exploiting one favorable representation.

  2. The validation convincingly demonstrates extreme-sparsity efficiency. The experiments cover multiple datasets, backbones, sparsity levels, and text-conditioned tasks while reporting throughput and memory alongside FLOPs. The method preserves much of the dense model’s performance with very few tokens, and qualitative results show selections following relevant motion, objects, and query-specific events. This breadth strongly supports acceptance because it connects the conceptual contribution to measurable hardware-level benefits.

Critical weaknesses

1、The proposed components are insufficiently disentangled. The full method combines GRPO, reward gating, temporal grouping, spatial cascading, distillation, warm-up, and joint extractor adaptation. Yet crucial comparisons—GRPO versus simpler policy gradients, hierarchical versus flat selection, and gated versus ungated rewards—are missing. This weakens the methodological conclusion: the experiments validate the complete recipe but do not establish which claimed innovations are necessary.

Questions for authors

Under identical training budgets, what are the individual contributions of GRPO, reward gating, temporal hierarchy, spatial cascading, and joint extractor adaptation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer PpHL (1/2)

We thank the reviewer for their constructive feedback and for recognizing the breadth of experimentation, practical value and empirical strength of our work. We address the concerns below.

Additional Ablations
We appreciate the reviewer's suggestion to better disentangle the components of our method to strengthen the methodological conclusions. During the rebuttal period, we ran additional ablations of the spatial hierarchy, frame head, RL algorithm, joint adaptation, and reward gating. All comparisons use the DINOv3 variant under matched training budgets, following the downstream protocol described in the paper with 10 warmup epochs followed by 30 epochs of RL fine-tuning. To keep these experiments tractable during the limited rebuttal period, we use 20K-example subsets of the action-recognition datasets and the full Diving48 training set, which contains approximately 15K examples.

RL algorithm. We thank the reviewer for suggesting a simpler policy-gradient baseline. We compared GRPO with REINFORCE using a running reward baseline. At a 10% token budget, GRPO outperforms REINFORCE on every evaluated task.

Table 1. Comparison of RL algorithms at a 10% token budget

MethodDiving48SSv2 (20K)EK100 (20K)
REINFORCE81.6257.0524.67
GRPO86.4558.7929.70
Gain (%)+4.83+1.74+5.03

Joint adaptation. To isolate the benefit of adapting the selector with GRPO during post-training, we compared joint training with freezing the pretrained selector (Sec. 3.3) and fine-tuning only the extractor. We train and evaluate at 10%, 5%, and 1% token budgets, reporting final-epoch accuracy below. RL adaptation of the selector improves accuracy at every budget, with gains of 6.04–10.31 percentage points on Diving48 and up to 3.76 points on SSv2. This isolates the beneficial contribution of selector adaptation within joint training. We have included both these results in Sec. 4.3 for ablations, as well as Appendices A.4 and A.5.

Table 2. Frozen vs. jointly adapted selector at varying token budgets

MethodDiving48 10%Diving48 5%Diving48 1%SSv2 10%SSv2 5%SSv2 1%Jester 10%Jester 5%Jester 1%
Frozen selector79.1972.2846.3558.5153.5733.0695.4093.9181.89
GRPO86.4582.5952.3958.7956.0836.8295.7995.3582.22
Gain (%)+7.26+10.31+6.04+0.28+2.51+3.76+0.39+1.44+0.33

Reward gating. We adopt reward gating from LookThere [R1] to prevent the selector from indiscriminately discarding patches to increase reward at the cost of task performance. Extended to distillation pretraining, a mask receives a sparsity bonus only if its loss falls below the running median. This creates a self-paced curriculum that gradually increases sparsity as the model learns to perform the task.

To assess gating during pretraining, we compared gated and ungated rewards. With gating, the selector retains 32.7%/15.5% of patches and achieves distillation losses of 1.078/1.188. Without gating, it retains 8.5%/3.9% of patches but converges to higher losses of 1.217/1.302. These results suggest that gating prevents premature sparsification, preserving informative content for distillation.

We also ablated gating during downstream RL post-training on EK100 (20K) and the full Diving48 dataset. The table reports accuracy across token budgets. Across both datasets and all budgets, the gated variant achieves the highest accuracy or falls close within it. The benefit is most pronounced on Diving48 at a 1% budget, where gating improves accuracy by 3.40 points. These results suggest that gating can help when sparsity pressure is strong, although it does not improve accuracy in every setting. We have referenced these results in Sec. 4.3 for ablations, and included them in Appendix A.6.

Table 3. Reward gating ablation during RL post-training at varying token budgets

MethodDiving48 10%Diving48 5%Diving48 1%EK100 10%EK100 5%EK100 1%
No gate, λ=284.6781.9356.7045.6145.2226.96
Ours, λ=284.9282.5460.1046.6145.6426.56
Δ (λ=2)+0.25+0.61+3.40+1.00+0.42−0.40
AuthorsOpenReview ↗

Response to Reviewer PpHL (2/2)

Warmup. Following LookThere [R1], we warm up the extractor to help it provide useful feedback for RL selection. During rebuttal, we evaluated warmup on Diving48, SSv2, and EK100. Under the same total epoch budget, we compared our standard protocol of 10 warmup epochs followed by 30 RL epochs with 40 RL epochs without warmup. Warmup improves final accuracy on all three datasets, achieving 86.45%/48.73%/21.42% on Diving48/SSv2/EK100, compared with 86.24%/48.24%/19.59% without warmup. We have included these results in Appendix A.10 with reference in Sec. 4.3 discussing ablations.

Frame head for temporal selection. During rebuttal, we extended our frame-head ablations to a harder variant of A2D and a longer-context Diving48 task that emphasizes locating relevant frames.

Instead of our previous A2D questions, which named the actor and asked for the action, we now name the action and ask for the actor ("Who is walking?" instead of "What is the dog doing?"). This requires greater temporal understanding for selection based on the queried action rather than a named actor. Using 0.32%/0.64%/1% of tokens, our method with a frame head achieves 63.2%/66.4%/69.0% accuracy, compared with 48.0%/51.7%/60.1% without it. When prompted with a different randomly selected action, both models adapt their selections and maintain similar accuracies. We include and discuss these results in Appendix A.8 and Sec 4.3.

The second task uses longer Diving48 clips in which the discriminative information occupies only a short interval. We identify the core dive segment and place it at a random position in a 64-frame clip, surrounded by less informative footage from the beginning and end of the same video. Our selector processes these clips in two windows of 32 frames, producing 64 frame scores. We select the 16 highest-scoring frames and pass them to the frozen extractor from the paper to predict the label. We compare this with a patch-only baseline that trains a selector and extractor over all 64 frames. We include and discuss these results in Appendix A.9 and Sec 4.3.

At 90%/95%/99% sparsity within the selected 16 frames, the frame-head model achieves 81.2%/76.9%/41.6% accuracy using 95/80/68 GFLOPs, compared with 80.7%/75.9%/38.7% accuracy and 109/93/81 GFLOPs for the patch-only model. Figure 6d plots these accuracy–FLOP curves, showing that the frame-head model achieves higher accuracy at lower computational cost across the reported operating points. We believe these results strengthen the evidence that the frame head learns useful temporal selection when only a short part of the video is informative, supporting the frame head's contribution.

Hierarchical selection. We adopted LookThere's [R1] spatial hierarchy to reduce sampling variance in the multivariate Bernoulli policy. To assess its contribution separately, we compared hierarchical and flat policies on SSv2, Jester, and EK100 (20K subsets), and the full Diving48 dataset. At a 10% token budget, DINOv3 achieves 58.75%/95.71%/33.02%/85.92% accuracy with the hierarchy, compared with 59.06%/95.76%/28.70%/85.87% without it.

Although the hierarchy produces cleaner selection masks, flat policies perform similarly, suggesting that spatial cascading is not necessary for accuracy in these settings. We hypothesize that the hierarchy is more useful at the higher resolutions used by LookThere, where the larger token count increases the action space and sampling variance. Its benefit appears reduced at the lower resolutions we use for video. We discuss these findings in Sec. 4.3 for ablations, and provide additional comparative visualizations in Appendix A.7.

[R1] Sreehari Rammohan et al. LookThere! Sparse Vision by Reinforced Selection. NeurIPS 2026.

Dear Reviewer PpHL,

Thank you again for your detailed feedback. We have provided responses to your concerns, including additional experimental results. As the discussion period is approaching its end, we would appreciate your thoughts on whether our responses and additional results have addressed your concerns. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.