acceptodds
Withdrawn from ICLR 2027

TokenGazer: Learning Where to Look in Token Space for Visuomotor Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Using dense tokens from pretrained vision transformers in robot policies preserves rich spatial structure and enables strong visuomotor reasoning, but dense representations are computationally expensive and contain substantial task-irrelevant information such as background clutter and distractors. We introduce TokenGazer, a policy for selecting a small subset of task-relevant tokens from frozen dense visual representations while preserving the information needed for control. Rather than training the selector with reconstruction, which favors visual fidelity instead of action relevance, TokenGazer scores candidate subsets according to how accurately a downstream action-prediction model can recover expert actions from the selected tokens. We optimize the discrete selection policy using Group Relative Policy Optimization (GRPO) and compare it with a Gumbel-Softmax relaxation. A downstream diffusion policy is then trained using only the selected tokens, reducing its conditioning sequence length without fine-tuning the pretrained vision encoder. Across PushT, Cube, cluttered variants, and language-conditioned LIBERO, TokenGazer matches or exceeds dense full-token policies while using substantially fewer visual tokens. On PushT, learned selection outperforms the full 256-token policy and same-budget baselines, while on Cube it reaches parity with dense conditioning using a fraction of the tokens. TokenGazer also shows improved robustness to visual clutter and reduces the recurring cost of downstream policy training and deployment. Code is available at https://anonymous.4open.science/r/tokengazer-release-E075.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, visuomotor control, token selection, vision transformers, imitation learning, diffusion policy, reinforcement learning, GRPO, efficient robotics, visual representations

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tp6AOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TokenGazer, which conditions a visuomotor policy on a small subset of frozen DINOv2 patch tokens rather than on all tokens. In Stage 1, a lightweight transformer gaze policy scores each token and samples K of them. It is trained with GRPO, using as reward how accurately a frozen action-prediction head predicts the expert action from the selected tokens alone, and a Gumbel-softmax variant is also compared. In Stage 2, a diffusion policy is trained on an even mix of the gaze’s top-K tokens and random subsets. Only the top-K tokens are used at deployment. On PushT, PushT-Distract, OGBench Cube, and LIBERO-Goal, TokenGazer with 8 to 64 tokens outperforms the dense policy (256 or 512 tokens) on PushT (0.85 vs. 0.75) and LIBERO (0.82 vs. 0.72) and matches it on Cube (0.88 vs. 0.89 cubes placed). Stage 2 training is also cheaper than dense policy training.

Critical strengths
  • The problem is clearly formulated. The paper keeps the dense tokens of a frozen encoder and frames the task as selecting only the tokens the policy needs. Selecting tokens by action-prediction accuracy rather than reconstruction quality is well suited to robot learning.

  • The method matches or exceeds the dense policy with far fewer tokens. Without modifying the encoder, the policy receives only 8 to 64 of the 256 tokens (512 on LIBERO), and it outperforms the dense policy on PushT and LIBERO and matches it on Cube. Because the policy’s sequence length scales with K, Stage 2 training time also drops by a factor of 1.46 to 3.2.

Critical weaknesses
  • The performance gains are small and statistically uncertain. TokenGazer only matches the dense policy on Cube (0.88 vs. 0.89) and Cube-Distract (0.53 vs. 0.51).

  • Inference efficiency, a central motivation of the paper, is not measured. Only training time is reported (Table 3), although at inference DINOv2 still encodes the full image and the gaze policy attends to all N tokens. With Stage 1 included, end-to-end training on PushT takes longer than for the dense policy.

  • The experimental scope is limited. All experiments are in simulation, and only DINOv2 is tested as the encoder. Removing proprioception for all methods also places the dense baseline well below the published Patch Policy results (LIBERO 0.72 vs. 0.98), which raises concerns about the reported dense policy results.

  • The manuscript would also benefit from experiments at VLA/WAM scale and from a comparison with the VLA token pruning method it cites (Pei et al., 2026).

  • Several results are incomplete, and the text is inconsistent in places. Pending cells remain (Table 16), and Gumbel-softmax is not run on Cube or LIBERO. The abstract describes alternating training, whereas the method trains in two stages with a frozen reward head.

Questions for authors
  • Does the same trend hold with encoders other than DINOv2, such as SigLIP, CLIP, or WebSSL?

  • Could the authors provide quantitative results comparing reconstruction-based and action-based rewards?

  • Could the authors report per-step inference latency broken down into the encoder, the gaze policy, and the diffusion policy? Since the gaze policy attends to all N tokens, is its cost smaller than the savings in the diffusion policy?

  • Can the method be applied at VLA scale? Results on a small VLA such as SmolVLA, together with a comparison with Pei et al. (2026), would strengthen the paper.

  • Why are the dense results lower than the published Patch Policy numbers? Could the authors compare TokenGazer and the dense policy under the Patch Policy protocol, with proprioception and the original action horizon?

  • With proprioception, TokenGazer drops to 0.31 at K = 8 on LIBERO, well below the dense policy (0.70) and random selection (0.48). Why is a policy conditioned on selected tokens more prone to the proprioception shortcut, and does removing proprioception favor selection methods?

Reviewer_m7UuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

TokenGazer identifies which frozen DINOv2 patch tokens a visuomotor policy should condition on. A lightweight gaze policy evaluates all patch tokens and samples K tokens per frame; it is trained via GRPO (or a Gumbel-Softmax relaxation) using as reward the action-prediction error of a frozen action head that sees only the selected tokens (on PushT, blended with an attention term). A diffusion policy is subsequently trained on these selected tokens using a 50/50 mixture of gaze-selected and random token subsets. Benchmark experiments across PushT (including a distractor variant), OGBench Cube (including a cluttered variant), and LIBERO-Goal (vision-only) demonstrate that sparse token subsets of just 8 to 64 tokens match or outperform full dense representations (e.g., 0.85 vs. 0.75 on PushT and 0.82 vs. 0.72 on LIBERO-Goal), while yielding a 1.5× to 3.2× speedup in Stage-2 training compute.

Critical strengths

The problem is well posed: choosing frozen tokens by their usefulness for action prediction, rather than by saliency or reconstruction, is a sensible way to reduce the cost of dense-feature policies. The paper also reports experimental protocols, per-seed results, and compute resources in exceptional detail (Appendices A and F, Table 3), including cases where the method does not help.

Critical weaknesses
  1. The experimental results fail to support the primary claim that learned token selection matches or exceeds full-token conditioning. The paper's dense baselines fall well short of the published Patch Policy benchmarks cited in the paper itself: 0.691 vs. 0.81 coverage on PushT (Section 5), 0.89 vs. 1.24 cubes on OGBench Cube (Table 7), and 0.72 vs. 0.98 on LIBERO-Goal (Table 9). On both Cube and LIBERO-Goal, TokenGazer also lags significantly behind published metrics. Furthermore, many entries are single-seed, the Cube result is parity with overlapping confidence intervals, and LIBERO results depend strongly on the seed for random selection (0.53 to 0.835 at K=8, Table 9) and, under an earlier protocol, for the learned gaze as well (0.70 to 0.035, Appendix E.4). Under the original Stage-2 setup, the learned gaze at K=64 scored between 0.59 and 0.64 on PushT, underperforming both random selection (0.784) and the dense baseline (0.750) (Table 14). This suggests that, at least on PushT, much of the gain comes from the Stage-2 recipe, which trains on varied subsets instead of the gaze's top-K, rather than from the learned selection itself. Parts of the manuscript also look unfinished, e.g., Appendix E.4 still calls Table 9 pending, and Appendix E.2 says the PushT-Distract table uses the original Stage-2 recipe, contrary to Section 5.1.

  2. The empirical scope remains narrow relative to the paper's broad claims. Evaluation is limited to PushT, OGBench Cube, and LIBERO-Goal using compact diffusion policies whose individual training stages take under two hours on a V100 (Table 3). The core motivating claim, that reconstruction-based token selection is actively misleading, is asserted without quantitative evidence (Section 4 and Appendix E.3). Because task-driven token selection and glimpse-style attention mechanisms are well-established concepts (e.g., TokenLearner in RT-1), the contribution depends on a convincing empirical case, which the current results do not make.

Questions for authors

Clarifications Requested:

  1. Is the random-selection baseline across Tables 4 to 9 trained using the identical 50/50 random-subset Stage-2 recipe as TokenGazer?
  2. What accounts for the performance gap between your dense policy reproduction and the published Patch Policy numbers (e.g., differences in evaluation protocol, proprioceptive state inputs, or overall training compute budget)?

Key Experiments Needed:

  1. Provide comparative evaluations for dense, random, and TokenGazer configurations under the exact published Patch Policy protocol, making comparisons against the 0.98 and 1.24 benchmarks meaningful.
  2. Include an ablation study isolating the contribution of the learned gaze policy from the random-subset training mixture, alongside empirical, quantitative comparisons against a reconstruction-based reward mechanism.
  3. Report multi-seed means and standard deviations for all evaluated token budgets.
Reviewer_dtZxOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TokenGazer, a method for selecting task-relevant visual tokens from frozen pretrained vision encoders for visuomotor control. Rather than conditioning a robot policy on all image patch tokens, the method trains a token-selection policy using an action-prediction-based reward, optimized through GRPO or Gumbel-Softmax. A downstream diffusion policy is then trained on the selected tokens. Experiments on sim benchmarks demonstrate comparable or improved performance over full-token conditioning while reducing computational costs, with additional benefits under certain visual distractor settings.

Critical strengths
  • Clear presentation and organization. The paper is well written, with a clear motivation, intuitive method description, and informative visualizations.

  • Interesting perspective and valid problem formulation. The paper makes an interesting connection between visual token selection research and robot policy learning, highlighting how action supervision can guide representation selection more effectively than reconstruction-based objectives.

Critical weaknesses
  • Insufficient discussion and comparison with closely related methods. Although the paper discusses TokenLearner and general token-pruning approaches, it omits relevant recent work such as Compressor-VLA and VLA-IAP, which also investigate task-aware visual token reduction for robot manipulation. The experiments lack comparisons against other learning based approaches, hence it is unclear how much improvement comes from the proposed optimization formulation.

  • All experiments are conducted in simulation, on simple manipulation tasks. PushT is a highly structured 2D benchmark, while the visual rendering in PushT and LIBERO is substantially simpler than real-world scenes. Hence it's unclear the method can reliably identify task-relevant visual information in complex, cluttered real-world environments. Real-robot experiments or more visually realistic and diverse tasks would strengthen the practical significance.

  • The method assumes access to individual patch embeddings from a frozen vision encoder and trains a separate diffusion policy conditioned on the selected subset. This architecture does not directly integrate with modern pretrained VLA models such as π₀, which employ tightly coupled vision-language-action architectures and their own tokenization and training schemes. The paper does not discuss how the approach would generalize to these increasingly common robot policy architectures.

Questions for authors
  1. Comparison with learned token selection. How does TokenGazer compare against methods that learn token selection or aggregation directly from downstream action-prediction losses? What specific benefits does the proposed two-stage optimization provide over end-to-end training?

  2. If the control policy observes multiple historical frames, could random token sampling provide sufficient spatial coverage across time without learning a separate selector? The relatively strong Random and Pooled baselines in Table 1 suggest this may be plausible. Have the authors evaluated temporal conditioning or token selection across multiple frames?

  3. How would the proposed method compare against simply reducing the input image resolution to produce fewer visual tokens? This would preserve global scene coverage while reducing token count, potentially offering a much simpler alternative. A comparison at matched token counts and computational budgets would help establish the value of sparse token selection over uniform spatial compression.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.