acceptodds
Under review as a conference paper at ICLR 2027

Separating Free Token Geometry from Raw Feature-Energy Routing: An OAPC Diagnostic Study

Abstract

Fixed-grid tokenization couples uniform allocation with a factorized spatial geometry, so an apparent gain from adaptive placement may instead come from simply removing the grid constraint. We separate three questions: whether free-position geometry helps, whether content conditioning adds value beyond content-independent free geometry, and whether raw feature-energy routing is semantically useful. OAPC (Optimization-driven Adaptive Patch Centroids) is a deliberately non-amortized probe that optimizes N continuous centroids per input using raw-L2-energy-weighted spatial quantization. Randomized synthetic controls show that geometric and content-conditioned gains are budget-dependent rather than universal. On a stratified 300-image COCO diagnostic subset, raw energy improves the same projected optimizer over constant-density routing in 23 of 24 backbone-budget settings, yet projected OAPC beats a Lloyd-optimized uniform centroidal Voronoi control in only one setting by mean per-image relative MSE reduction. In a 1,500-image, two-backbone mask-prediction stress test, raw-energy routing is statistically unresolved against uniform-CVT on ResNet-50 and underperforms it on ConvNeXt-T; channel standardization removes the resolved deficit without establishing an advantage. Diagnostic GT-mask routing improves micro-IoU through architecture-matched decoders trained under the same protocol on both backbones, showing that aligned guidance can exploit the continuous placement mechanism. Free positions are flexible, but raw feature magnitude is not a stable semantic routing signal under this spatial-quantization objective across the representations tested.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
visual tokenization, adaptive token placement, centroidal Voronoi tessellation, feature reconstruction, semantic routing, representation learning, vision transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yiHfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether gains from adaptive visual token allocation come from free-position geometry or image-content guidance. The authors propose OAPC, which optimizes token positions using raw feature magnitude, and compare it with fixed grids and content-independent free layouts. The results show that feature-magnitude guidance does not consistently outperform a strong uniform-layout baseline or provide consistent gains in object-mask prediction.

Critical strengths
  1. The control design addresses a useful distinction. Separating free geometry from content guidance helps avoid attributing all gains to content adaptation.

  2. The reporting is transparent. The authors include a same-optimizer control and openly report negative results and methodological limitations.

Critical weaknesses
  1. The importance of the central hypothesis is not established. Feature magnitude does not inherently represent semantic importance, and the paper does not sufficiently explain which practical methods or important debates rely on this assumption. Demonstrating that this signal is unreliable therefore seems more like diagnosing a particular design than resolving a broadly significant question.

  2. The experimental coverage is insufficient to support broader conclusions. Reconstruction experiments use only 300 images and ResNet-50, ConvNeXt-T, and ViT-B/16; semantic experiments use 1,500 images and two backbones. Model scale and diversity are limited, with no evidence from larger or more recent visual encoders. Since the results depend strongly on the backbone, their relevance to current vision-language models remains unclear.

  3. The reasons for underperformance are not sufficiently disentangled. Raw energy improves over constant density with the same optimizer in 23 of 24 settings, yet both generally underperform Lloyd uniform-CVT. This suggests that the optimizer itself may be an important limitation, so overall underperformance cannot simply be attributed to the guidance signal. The paper does not yet turn these findings into clear, generalizable design insights.

Questions for authors
  1. Which existing methods specifically rely on raw feature magnitude for token allocation? What important prior understanding do these findings revise?

  2. Given the backbone-dependent results, which conclusions are supported beyond the models tested here?

  3. Based on the existing controls, how can optimizer limitations be distinguished from guidance-signal limitations? Which is the main bottleneck in the reported results?

Reviewer_NwEkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Standard Vision Transformers (ViTs) partition images into a uniform, non-overlapping patch grid, assigning equal token budget across all regions regardless of content density. Existing dynamic tokenization strategies attempt to improve efficiency via post-hoc token pruning or merging. However, when adaptive token placement yields performance gains, previous studies fail to separate whether improvement stems from free continuous geometry (removing grid constraints), content-conditioned density routing, or whether raw feature magnitude is a semantically meaningful signal.

To isolate these factors, the authors introduce Optimization-driven Adaptive Patch Centroids (OAPC)—a non-amortized, training-free diagnostic probe. It evaluates three explicit hypotheses:

H1 (Geometry): Continuous token positions represent non-grid-aligned structure better than fixed rectangular grids.

H2 (Content Conditioning): Routing free positions by raw feature energy improves global feature reconstruction over uniform density.

H3 (Semantic Routing): Feature-energy spatial quantization places tokens where they benefit downstream object-mask prediction

Critical strengths

Meticulous Diagnostic Isolation: The paper systematically disentangles continuous geometry, density-based routing, and solver implementation mechanics using tightly matched controls.

Statistical Rigor & Reproducibility: Employs 20,000 crossed bootstrap resamples across images, data splits, initial seeds, and decoders alongside Holm family-wise error adjustments, full procedural code releases, hardware metrics, and transparent AI-use disclosures.

Critical weaknesses

Limited significance:

Small effects: Reconstruction differences are mostly under 1% MSE, and raw-energy IoU differences are 0.002–0.007.

Narrow setting: one layer per backbone, 28×28×28 maps, square-resized inputs, subsets excluding 4–7-annotation images, and N≤64

No token-consuming model: the readout is a small broadcast conv decoder (possibly biased toward uniform coverage), not a transformer.

Not a usable tokenizer: about 950 ms/image (Table 18).

Expected conclusion: task-aligned routing beating raw norm is what practitioners would predict, and the oracle uses the target itself.
Questions for authors

Does the decoder's 3×3 conv head favor uniform coverage? A small transformer readout (e.g., a multi-label linear probe on the N tokens) would test the motivating use case.

Reviewer_CRrYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents an in-depth diagnostic study into the effectiveness of adaptive tokenisation in Visual Transformers (ViT). Existing adaptive methods typically involve pruning or merging after a fixed grid, or learning end-to-end routing networks. This paper proposes decoupling three factors that are often conflated: ‘free-position geometry’, ‘content conditioning’ and ‘semantic routing’. To achieve this, the authors have designed a training-free, non-amortised diagnostic probe called OAPC (Optimisation-driven Adaptive Patch Centroids). For each image, OAPC optimises consecutive 2D centroids, utilising raw feature-energy as density weights for spatial quantisation. Through extensive reconstruction experiments on synthetic datasets and a COCO subset, as well as a mask prediction stress test involving 1,500 images, the paper draws several counterintuitive yet highly valuable conclusions: (1) the advantage of free geometry is highly dependent on the token budget; (2) after controlling for optimiser factors, raw feature-energy routing fails to outperform content-free Uniform-CVT (uniform centre-of-mass Voronoi tessellation) in most settings; (3) raw feature magnitude is not a stable or universal semantic routing signal, and its performance varies significantly across different backbones (ResNet vs. ConvNeXt).

Critical strengths
  1. The research motivation is profound and of great scientific value: this paper investigates when and why adaptive placement is effective. By rigorously decoupling geometric freedom, content conditioning and semantic alignment, this debunking and diagnostic research offers extremely valuable guidance to the computer vision community.

  2. Rigorous experimental controls: Rather than simply comparing OAPC with Fixed-grid, which could easily allow OAPC to ‘cheat’ its way to victory by breaking the aspect ratio constraints of the grid, the authors introduced Uniform-CVT and Matched constant-density OAPC . This ablation design lends immense persuasiveness to the paper’s conclusion.

Critical weaknesses
  1. Although the authors claim that OAPC is a diagnostic probe, each image requires 200 steps of gradient descent (taking approximately 900 ms, see Table 18). Does this extreme, non-amortised optimisation—which ‘seeks global or local optima at any cost’—truly reflect the potential of lightweight predictors, such as Attention-score-based Token Merging or MLP-based Token Pruning in practical applications? If OAPC fails to outperform Uniform-CVT on ConvNeXt even with an unlimited computational budget to fit feature energies, this does indeed highlight the shortcomings of the energy signal; however, if OAPC becomes trapped in a poor local optimum due to the optimisation landscape, then such a conclusion may be misleading.

  2. Feature representation and network layer selection. The paper only tested specific intermediate layers such as ResNet-50 C4, ConvNeXt-T C4, ViT-B block 8. It is well known that shallow features in CNNs and ViTs contain a large amount of high-frequency texture and edges, where energy may be concentrated in the background or irrelevant textures, whilst deeper features are more semantic. Testing on a single layer alone makes it difficult to draw a universal conclusion that the energy of raw features is not a semantic signal. Furthermore, more discriminative self-supervised features (such as DINOv2 or CLIP features) were not included; the energy distribution of these features may differ markedly from that of traditional ImageNet pre-trained features.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.