acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Geometric Perception Kernels

Abstract

Modern vision architectures typically construct visual representations using predefined spatial structures, such as fixed patch grids or convolutional receptive fields. We introduce , a vision architecture that learns a compact set of content-dependent anisotropic Gaussian kernels. Each kernel adapts its location, scale, and orientation to the input image and directly aggregates a feature map into a visual token, allowing them to be processed by a standard Transformer architecture. Across image-classification benchmarks, AdaGPK consistently outperforms fixed-patch Vision Transformer baselines and is competitive with or exceeds the evaluated convolutional baselines on ImageNet-scale classification. Compared with unconstrained adaptive pooling, Gaussian parameterization preserves classification performance while producing more spatially structured representations and substantially stronger cross-dataset zero-shot localization. Analysis of the learned representation shows that increasing the kernel budget produces progressively smaller, less overlapping spatial supports and more differentiated token features, while the learned geometry remains stable across augmentations and training seeds. Together, these results suggest that learned spatial decomposition provides an effective way to construct compact visual representations with explicit spatial support.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Adaptive tokenization, Visual representation learning, Gaussian kernels, Vision Transformers, Image classification

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_odxXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduced AdaGPK, which builds visual tokens by pooling convolutional features with image-dependent Gaussian kernels that adapt their position, size, and orientation. The model also achieved competitive results, while the Gaussian constraint improves zero-shot localization compared with free-form adaptive pooling.

Critical strengths
  1. Each token comes from a region described by a small set of geometric parameters, making its pooling region easy to inspect. The method also fits into a standard Transformer without requiring a specialized attention mechanism.

  2. This paper shows that free-form pooling can match or exceed classification accuracy, while Gaussian kernels provide better localization on CUB-200 and single-object Pascal VOC. This is a more convincing argument than relying on classification gains alone, and the three-seed experiments and kernel analyses add useful support.

Critical weaknesses
  1. This work moves quickly into equations without clearly explaining the motivation for the design choices or what each component is meant to achieve. In particular, the authors should explain why moment-based Gaussian reconstruction is needed and what benefit it is intended to provide over directly pooling with the predicted heatmaps.

  2. AdaGPK uses a convolutional feature extractor, whereas the ViT and TokenLearner comparisons use patch-based models. The matched free-form ablation addresses part of this concern, but a fixed-pooling baseline with the same convolutional backbone is still needed. Comparisons with additional hybrid or hierarchical models would make the performance claims stronger.

  3. The evaluation uses class-agnostic relevance maps and restricts Pascal VOC to single-object images. Since Gaussian reconstruction directly produces smooth spatial maps, it remains unclear how much of the gain reflects better learned spatial representations versus the shape of the maps used for evaluation.

Questions for authors
  1. How does AdaGPK compare with fixed-grid pooling or TokenLearner-style pooling using exactly the same convolutional backbone, token count, and Transformer?

  2. Would the localization advantage remain if the free-form and baseline relevance maps were smoothed before evaluation? This would help separate the effect of Gaussian map shape from what the model learns.

  3. How does the method behave on images with multiple objects, small objects, or objects with irregular shapes? Could you include results beyond the single-object setting?

  4. Could you please report inference latency, throughput, and peak memory at different token budgets? The paper acknowledges higher FLOPs than the ViT baselines, so these measurements would clarify the practical benefit of using fewer tokens.

Reviewer_gCnPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaGPK, a vision model that uses adaptive Gaussian kernels to create image tokens based on image content, instead of fixed patches. The authors evaluate it on four image classification benchmarks and compare it with ResNet and ViT models. Results show that AdaGPK outperforms the tested ViT baselines and is competitive with ResNet on ImageNet. Experiments on CUB-200 and Pascal VOC also show better zero-shot object localization than the compared methods.

Critical strengths
  1. AdaGPK uses adaptive Gaussian kernels to build image tokens based on image content. This gives each token a clear spatial region and makes the method different from standard ViT models.

  2. The paper tests AdaGPK on four classification datasets and evaluates zero-shot object localization on CUB-200 and Pascal VOC. The localization results suggest that Gaussian kernels provide benefits beyond classification accuracy.

Critical weaknesses
  1. The unconstrained model achieves similar or better classification accuracy in several settings. For example, on ImageNet-100, the Wide model achieves 88.09% accuracy without Gaussian constraints, compared with 86.77% for AdaGPK. This makes the added value of Gaussian kernels less clear for classification.

  2. The localization experiments use only CUB-200 and single-object images from Pascal VOC. Testing on more datasets and tasks would help show whether the localization benefits generalize to other settings.

Questions for authors
  1. The unconstrained model achieves better classification accuracy in some settings. Can the authors explain why the Gaussian constraint is still beneficial and provide clearer evidence of its advantages?

  2. How sensitive are the localization results to the choice of evaluation method? Would the advantage of AdaGPK remain under other localization metrics?

  3. Testing AdaGPK on more complex localization tasks, such as multi-object images, would help show whether its benefits generalize beyond the current experiments.

Reviewer_kh1SOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

AdaGPK replaces fixed patch tokenization with N content-dependent anisotropic Gaussian kernels. A kernel head predicts heatmaps that define the Gaussian parameters (center, scale, and orientation), which are then used to pool a content feature map into a single token. The authors demonstrate competitive classification performance against ViT/ResNet baselines, and zero-shot localization on CUB-200/VOC.

Critical strengths
  1. The explicit geometric parameterization of token support is interpretable, and the formulation, (comprising moment estimation, analytic Gaussian, and normalized pooling) is elegant.
  2. The paper openly acknowledges that adaptive pooling accounts for most of the classification gain and that the Gaussian constraint does not consistently improve accuracy (Table 1). I commend the authors for reporting this transparently rather than concealing it.
Critical weaknesses
  1. Table 1 shows that the unconstrained tokenizer outperforms AdaGPK in most configurations, such as all results on ImageNet-100. This indicates that the Gaussian parameterization compromises classification accuracy, with the remaining benefit confined to a single proxy localization metric.
  2. The paper claims to "exceed the evaluated convolutional baselines," yet it lacks many modern CNNs, such as ConvNeXt and subsequent models. All comparable methods are 5 years or even 10 years old.
  3. Moment-based geometric estimation from heatmaps is standard in coordinate regression (the soft-argmax line of work), and Gaussian heatmaps are ubiquitous in pose estimation and detection. The paper frames the parameterization as novel without adequately situating it within this literature.
Questions for authors

None

Reviewer_nKncOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Geometric Perception Kernels (AdaGPK), which constructs visual tokens through content-dependent anisotropic Gaussian pooling. A convolutional backbone produces dense features, and predicted spatial heatmaps are converted into Gaussian parameters using their first- and second-order moments. The reconstructed Gaussian maps pool content features into tokens for a standard Transformer. Experiments cover image classification, class-agnostic cross-dataset localization, and analyses of token geometry, diversity, and stability. The central empirical finding is that Gaussian constraints preserve competitive classification performance while improving the evaluated localization metrics relative to unconstrained adaptive pooling.

Critical strengths
  1. A simple mechanism for spatially structured token construction. Moment-based Gaussian parameterization provides an understandable connection between spatial weighting maps and pooled tokens. This makes the tokenizer amenable to inspection and controlled modification.

  2. A useful architecture-matched ablation. The comparison with unconstrained adaptive maps helps distinguish the effects of adaptive pooling from those of Gaussian constraints. I appreciate the explicit acknowledgment that Gaussian parameterization does not consistently improve classification accuracy. The cross-dataset localization results provide a potentially useful direction for investigating its benefits.

Critical weaknesses
  1. The distinctive contribution of Gaussian tokenization is insufficiently established, both experimentally and relative to prior work.

Table 1 shows lower mean classification accuracy for AdaGPK in 9 of the 12 comparisons with unconstrained maps, including all four ImageNet-100 configurations. For the Wide model, ImageNet-100 accuracy decreases from 88.09% to 86.77%. Consequently, the classification improvements over the evaluated ViTs do not establish a benefit attributable specifically to Gaussian parameterization.

The TokenLearner comparison in Appendix A.3 does not resolve this attribution issue. AdaGPK includes a convolutional feature extractor, whereas the comparison describes TokenLearner configurations matched to the patch-based ViTs. The manuscript does not establish that these models share the same feature extractor and downstream architecture. A shared-backbone comparison with fixed spatial pooling and an otherwise identical adaptive tokenizer would more directly establish which component produces the gains. Controls using input-independent learned Gaussians and restricted covariance structures would also clarify the necessity of content-dependent anisotropic geometry.

The related-work discussion also needs substantial updating. In particular, [1] already introduces spatially adaptive tokens parameterized by 2D Gaussians, including their positions and shapes. Its reconstruction/generation objective and parameter-refinement procedure differ from AdaGPK, but the conceptual overlap is sufficiently close that the novelty claim requires an explicit technical comparison. [2] studies compact anisotropic-Gaussian image representations for visual understanding. Although its representation pipeline differs, it is relevant to the broader claim that Gaussian spatial primitives provide useful alternatives to conventional image representations. [3][4] provide relevant context for preserving and exploiting 2D spatial structure in vision models. Their contribution concerns spatial propagation and efficient context mixing, while AdaGPK concerns token construction. This distinction should be articulated when positioning the paper’s spatial inductive bias and efficiency motivation. [5] is an additional direct antecedent for learning token regions with adaptive positions and scales.

These references matter because they help identify the specific advance: moment-derived Gaussian pooling for discriminative token construction, beyond previously studied adaptive regions and Gaussian representations. My concern is the unresolved technical positioning and evidence for that advance, rather than citation coverage alone.

  1. The localization evaluation does not sufficiently separate learned spatial grounding from the imposed attribution-map geometry.

Section 4.4 constructs localization maps by weighting each tokenizer’s spatial maps with token-level prediction sensitivity. AdaGPK therefore produces relevance maps through its smooth, unimodal Gaussian kernels, whereas the unconstrained tokenizer uses free-form maps and ViT assigns scores uniformly within patches. The comparison changes both the learned representation and the spatial basis used to express attribution.

This leaves an important alternative explanation unresolved: part of the localization improvement may arise from imposing a spatially coherent map during attribution. The reported results establish better localization under the proposed readout, but do not adequately isolate how much improvement comes from learning more spatially grounded representations.

A particularly informative control would apply moment-matched Gaussian reconstruction to the unconstrained tokenizer’s maps only at evaluation time, while retaining its trained model and token-importance scores. Comparisons with appropriately smoothed baseline maps would further test this explanation. Independent localization or attribution evaluations would strengthen the conclusion.

There is also a distinction between Gaussian support on the content-feature grid and support in the input image: each content feature already aggregates information through the convolutional backbone. The manuscript should qualify how directly its kernel maps describe the input evidence supporting a prediction.

This issue is consequential because localization is the main demonstrated advantage of the Gaussian constraint once the classification ablation is taken into account. The current evidence does not sufficiently establish the interpretation underlying the paper’s central contribution.

[1] Zhang et al. GPSToken: Gaussian Parameterized Spatially-adaptive Tokenization for Image Representation and Generation. NeurIPS 2025

[2] Omri et al. GaussianVision: Vision-Language Alignment from Compressed Image Representations using 2D Gaussian Splatting. CVPR 2026

[3] Hongjun Wang et al. Parallel Sequence Modeling via Generalized Spatial Propagation Network. CVPR 2025

[4] Hongjun Wang et al. GSPN-2: Efficient Parallel Sequence Modeling. NeurIPS 2025

[5] Chen et al. DPT: Deformable Patch-based Transformer for Visual Recognition. ACM Multimedia 2021

Questions for authors
  1. Contribution attribution: With the convolutional backbone, content head, token budget, and Transformer held fixed, how does AdaGPK compare with fixed spatial pooling and input-independent learned Gaussian pooling? These comparisons would materially affect my assessment of whether content-dependent geometry is necessary.

  2. Localization confound: How much of the localization advantage remains when the unconstrained tokenizer’s maps are replaced by moment-matched Gaussians only during evaluation, keeping the trained model and token-importance scores fixed? Can a common attribution procedure corroborate the ordering?

  3. Implementation clarification: Section 4.3 states that the unconstrained ablation retains “moment-derived geometry fusion,” but Sections 3.3–3.4 describe pooled content tokens, and Appendix A.12 states that the geometry-aware extension is excluded from the main experiments. What geometry information is actually supplied to the main models, and is this identical across the matched ablation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.