Rare but Sufficient: Extremely Sparse Native Supports in Frozen Vision Representations

Frozen vision foundation models provide dense representations for downstream segmentation, yet these representations are typically used in full. Random-subset analyses reveal substantial diffuse redundancy. Their average-case results, however, leave open whether rare native supports can sustain unusually strong segmentation performance at extreme sparsity. To study this question without changing the representation basis, we define a support as a fixed dataset-level subset of the original spatial feature maps. We physically remove all other channels, discard the discovery model, and train a fresh decoder from scratch. Because exhaustive enumeration is infeasible, we use a coarse-to-fine backward search with fresh-decoder acceptance to identify candidate supports. On ACDC with frozen DINOv3 features, four of 384 native channels achieve a development Dice of 0.787, compared with 0.807 for the full representation. None of 512 uniformly sampled four-channel supports reaches this score. On unseen patients, the fixed support retains 95.2% of full-decoder Dice. Across ten additional discovery runs, independently selected supports contain only five to eight channels, each retaining at least 95% of its corresponding full-decoder test Dice. After excluding the four coordinates of the primary ACDC support from the search space, a separate search recovers a 13-channel support with comparable performance, demonstrating that high-performing native supports are not unique. The same qualitative sparse-support phenomenon recurs on COCO, M&Ms, MSD Liver, and G1020, spanning distinct anatomical targets, imaging settings, and visual domains. With a freshly trained decoder, the ACDC support also transfers directly to M&Ms, showing that sparse native-support structure can persist across related datasets. Together, these results indicate that redundancy in frozen dense representations is highly structured: broadly distributed task information can coexist with rare, non-unique, highly capable native supports. Code will be released on GitHub after the review process.