acceptodds
Under review as a conference paper at ICLR 2027

Rare but Sufficient: Extremely Sparse Native Supports in Frozen Vision Representations

Abstract

Frozen vision foundation models provide dense representations for downstream segmentation, yet these representations are typically used in full. Random-subset analyses reveal substantial diffuse redundancy. Their average-case results, however, leave open whether rare native supports can sustain unusually strong segmentation performance at extreme sparsity. To study this question without changing the representation basis, we define a support as a fixed dataset-level subset of the original spatial feature maps. We physically remove all other channels, discard the discovery model, and train a fresh decoder from scratch. Because exhaustive enumeration is infeasible, we use a coarse-to-fine backward search with fresh-decoder acceptance to identify candidate supports. On ACDC with frozen DINOv3 features, four of 384 native channels achieve a development Dice of 0.787, compared with 0.807 for the full representation. None of 512 uniformly sampled four-channel supports reaches this score. On unseen patients, the fixed support retains 95.2% of full-decoder Dice. Across ten additional discovery runs, independently selected supports contain only five to eight channels, each retaining at least 95% of its corresponding full-decoder test Dice. After excluding the four coordinates of the primary ACDC support from the search space, a separate search recovers a 13-channel support with comparable performance, demonstrating that high-performing native supports are not unique. The same qualitative sparse-support phenomenon recurs on COCO, M&Ms, MSD Liver, and G1020, spanning distinct anatomical targets, imaging settings, and visual domains. With a freshly trained decoder, the ACDC support also transfers directly to M&Ms, showing that sparse native-support structure can persist across related datasets. Together, these results indicate that redundancy in frozen dense representations is highly structured: broadly distributed task information can coexist with rare, non-unique, highly capable native supports. Code will be released on GitHub after the review process.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual Representation, Sparsity, Foundation Models, Semantic Segmentation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nFaPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether frozen vision representations contain rare, fixed subsets of native channels that remain highly predictive for segmentation. It proposes Dense-STG followed by Patience-Annealed Backward Feature Elimination (PA-BFE). After selection, all other channels are removed and a fresh decoder is trained from scratch. On ACDC with frozen DINOv3 features, a 4/384-channel support achieves 0.772 test Dice versus 0.811 for the full representation, while random four-channel supports perform substantially worse. The paper further studies repeated discovery, disjoint supports, deletion audits, decoder seeds, cross-dataset transfer, and multiple datasets, encoders, and decoder architectures.

Critical strengths
  1. Clean evaluation protocol. The discovery model is discarded, unselected channels are physically removed, and a fresh decoder is trained from scratch. This substantially reduces concerns about selector–decoder co-adaptation and directly tests the predictive value of the retained channels.
  2. Strong empirical validation. The selected supports clearly outperform same-width random supports. Repeated searches, disjoint alternatives, deletion audits, seed replications, cross-setting experiments, and ACDC-to-M&Ms transfer provide broad evidence that the phenomenon is real and non-unique.
Critical weaknesses
  1. The slice-level medical Dice convention may make Full-relative retention look more favorable. The medical metric averages hard Dice over 2D slices and foreground classes, assigns a score of one to both-empty prediction/target pairs, and retains background-only slices. Shared both-empty terms can mechanically pull the Selected/Full ratio toward one. The 128×128 experiment addresses label resolution but retains the same aggregation convention, while the patient bootstrap quantifies sampling uncertainty without changing the image-weighted estimand. Reporting class-wise per-volume Dice on the original-resolution masks, together with results restricted to present structures, would clarify how much of the headline 95.2% retention persists under a volumetric evaluation.

  2. The exact four-channel endpoint is not fully robust. The main search uses one decoder seed and repeatedly queries the same development split. With seeds 23 and 41, the fixed four-channel support has development gaps of −0.032 and −0.041, exceeding the stated 0.02 tolerance. Thus, the broader sparse-support finding is convincing, but the precise four-channel sufficiency and local-irreducibility claims appear seed-sensitive.

Questions for authors

1.Can the authors report class-wise patient- or volume-level Dice on the original-resolution ACDC and M&Ms masks, with absent-class handling clearly specified?

2.What support size is obtained when deletion acceptance is based on multiple decoder seeds? Do all three-channel children of the four-channel support remain below the threshold across seeds?

3.Can the authors repeat discovery across multiple patient-level train/development splits and report endpoint size, performance, and coordinate overlap?

4.Under a matched search budget, how does PA-BFE compare with Dense-STG alone or standard greedy backward elimination?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nFaP

We sincerely thank the reviewer for recognizing the fresh-decoder evaluation protocol and the evidence from repeated searches, disjoint supports, and cross-dataset transfer. We address the questions about medical Dice aggregation, the stability of the selected support, and the contribution of the search procedure below.

Summary of Key Concerns and Main Responses

Key ConcernsMain Responses
Medical Dice aggregation (W1, Q1)High retention persists when Dice is computed on complete volumes and only structures present in the reference mask are scored.
Support stability (W2, Q2–Q3)Multi-seed acceptance and alternative patient splits recover compact supports, with variation in their exact coordinates and widths.
Search strategy (Q4)Compared Dense-STG alone and alternative deletion strategies from the same confirmed ACDC starting support.

W1. Please see Q1.

W2. Please see Q2–Q3.


Q1. Class-wise volumetric Dice on ACDC and M&Ms

Thank you for highlighting the effect of both-empty slices on the original Dice metric. The reported 95.2% retention on ACDC uses slice-averaged Dice, and the finer-resolution evaluation in Supplement A.4.1 retains that aggregation rule. Following your suggestion, we have added class-wise volumetric Dice on the original-resolution ACDC and M&Ms masks (Supplement A.10.1, Table S10).

For each cardiac phase, we compute Dice over the complete volume for each structure present in the reference mask. A slice on which both masks are empty therefore contributes no separate score. We average phase scores within each patient and give each patient equal weight.

Table R1. Original-resolution volumetric Dice. Each Dice cell gives the full-decoder score followed by the selected four-channel score. Confidence intervals for mean retention use 10,000 paired patient-bootstrap resamples.

DatasetLeft ventricular cavityMyocardiumRight ventricular cavityMean DiceMean retention [95% CI]
ACDC0.798 / 0.7850.590 / 0.5880.754 / 0.7320.714 / 0.70298.26% [97.18, 99.26]
M&Ms0.759 / 0.7650.589 / 0.5770.741 / 0.7280.696 / 0.69099.11% [98.20, 99.97]

The selected supports retain over 97% of full-decoder volumetric Dice for each of the three structures on both datasets. High retention therefore persists when both-empty slices receive no separate credit (Sec. 4.5).


Q2–Q3. Stability across decoder seeds and patient splits

Thank you for distinguishing the stability of the exact support from the recurrence of compact, predictive supports. We have tested both sources of variation (Supplement A.10.2, Table S11).

1. Multi-seed acceptance (Q2). Using mean development Dice across three decoder seeds, the search selected four channels. All four three-channel subsets of the original support fell below the acceptance threshold at every seed (Table S11, panel A).

2. Patient-level splits (Q3). Repeating discovery on two alternative splits yielded four- and eight-channel supports. They shared three and four channels with the original support, respectively, and both retained over 97% of matched full-decoder Dice on the same held-out patients. High retention therefore persisted across splits even as the width and membership of the support varied (Table S11, panel B).


Q4. Search strategy under a common budget

Thank you for suggesting this comparison. We evaluated Dense-STG alone and compared deletion strategies from its confirmed 17-channel ACDC support (Supplement A.8, Table S7, panel A). Dense-STG already provided a predictive starting point. Within the same 17-check budget, PA-BFE reduced it to four channels, while exhaustive greedy still retained 16, with similar test Dice. The benefit of PA-BFE here is how quickly it reaches a narrow support while preserving predictive performance.

We thank the reviewer again for the constructive feedback and welcome further discussion.

Thank you for authors' response and clarifications which address several of my concerns. I would keep its original rate.

Reviewer_KegyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether there exist extremely sparse combinations of feature channels in frozen visual encoders that are still sufficient to support downstream segmentation tasks. It proposes the PA-BFE algorithm to search for such channel combinations, and through extensive experiments demonstrates the existence and non-uniqueness of high-performance native support sets. The authors argue that broadly distributed task information coexists with extremely sparse, highly predictive native supports.

Critical strengths
  1. The research question is clearly defined, the method design is reasonable, and the core finding is thought-provoking.
  2. The experimental design is fairly comprehensive and provides some support for the core claims. The study also considers datasets from different domains, as well as the effect of different encoders and decoders.
Critical weaknesses
  1. The boundary of the core claim is unclear.
  2. Insufficient experimental rigor and inadequate validation.
Questions for authors
  • The novelty and additional insights of the paper need to be more clearly articulated. SCC (Sparse Concept Channels in Frozen 3D CT Vision Encoders) has already shown that a small number of native embedding dimensions in frozen visual representations can support strong clinical finding detection. The paper's sparsity finding therefore overlaps with prior work to some extent. Extending the study object to feature maps that preserve spatial layout, and further analyzing the non-uniqueness of the support set as well as the dependence of support-set width on the task, encoder, and decoder, does add research value. However, at present these results mostly stay at the level of empirical demonstration and do not sufficiently clarify what new understanding the paper contributes beyond existing sparse-channel research. For example, why can completely non-overlapping feature-map combinations achieve comparable segmentation performance, and how does the spatial aggregation capability of different decoders affect the required support-set width and channel combination? I would suggest the authors more clearly position the paper's unique contribution and add targeted analyses around the observed phenomena to strengthen its originality and persuasiveness. If the authors can reveal the underlying principles behind these phenomena through such targeted analyses, I would significantly raise my assessment of the paper.

  • The boundary of the core claim is not clear enough. In addition to medical datasets, the paper evaluates natural-image datasets such as COCO and ADE20K (Table 2). For these two datasets, the selected channels account for 38.54% and 27.60% of the full set, respectively, while still retaining most of the full representation's performance. This is hard to call "extremely sparse." It shows that extreme sparsity does not hold for every dataset. Although the authors note that the support-set width depends on the specific setting, I would still recommend more carefully describing the applicable boundary of the finding.

  • The method section lacks direct baseline comparisons.

    • The core claim is that broadly distributed task information can coexist with rare, non-unique, and highly predictive native supports, and PA-BFE is the central method designed to discover these supports. The supplementary material (Section A.5.1) compares with approaches that rank channels by RMS activation and variance on training-set features and pick the top-4. However, both RMS and variance are single-channel statistics and do not directly reflect each channel's association with the labels, nor the joint predictive value of multiple channels combined. These results are therefore insufficient to determine whether the paper's finding depends on a particular search strategy. The authors should add direct comparisons against standard greedy backward feature elimination and other search algorithms, comparing the final channel count and predictive performance under the same decoder-training budget.
    • The authors should also separately report the performance of the Dense-STG support set when validated with a new decoder, and clarify, through ablations, the respective contributions of Dense-STG initialization and the PA-BFE search.
  • The advantage of the selected support set over random support sets across settings lacks sufficient test-set quantitative evidence. Table 2 fairly comprehensively presents the gap between selected and random support sets on the development set, while Table 1 mainly reports the retention rate on the test set, without systematically presenting the test-set advantage of the selected support set over random support sets of the same width. Fig. 4 and the supplementary figure S1 only show results on a few test samples, which are insufficient to reflect the average test-set performance gap and its variability. The authors should add aggregate test-set quantitative comparisons of Selected vs. Random support sets at the same width on representative datasets, along with the corresponding uncertainty, to verify whether the advantage of the selected support set generalizes stably to the test set.

  • The non-uniqueness of the channel combination is only verified on the ACDC dataset. The paper excludes all channels of the main support set and finds an alternative support set that does not overlap with it at all but achieves comparable performance, providing valuable evidence for the non-uniqueness of the support set. However, this conclusion is currently based mainly on a single ACDC setting, and it is unclear whether the phenomenon can be reproduced under other datasets, encoders, or decoder configurations. The authors should add experiments to address this.

  • The robustness of the experimental results still needs to be strengthened.

    • In Fig. 2(a), apart from the 4- and 8-channel widths, only 8 random support sets are sampled at every other width; in Table 2, except for ACDC, the other settings typically evaluate only 64 random support sets. The authors should justify the choice of the number of random samples, increase the sampling at key widths or representative settings, and report the corresponding variance or confidence intervals to further verify the advantage of the selected support set over random ones.
    • The paper conducts fairly thorough repeated searches on ACDC, but for other datasets it mainly reports single-search results. The authors should add multi-seed independent searches on at least one or two representative other datasets, and report the variation in the final support-set width and test-set performance, to further verify the reproducibility of the phenomenon.
  • Search cost and practical efficiency gains are not sufficiently quantified. PA-BFE validates each candidate support set by training a new decoder before deciding whether to accept a channel removal. This design helps assess the predictive ability of the support set itself, but may incur a substantial search cost. The paper defines the notion of an oracle call in the supplementary material and allows reusing eligible existing evaluations, but does not adequately report the actual search cost under each setting. The authors should report the number of decoder-training rounds during the search, the total search time, or the corresponding compute budget, so that readers can assess the actual overhead of the method.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Kegy (Part 1/3)

Response to Reviewer Kegy (Part 1/3)

We thank the reviewer for recognizing the clarity of the research question, the method design, and the evaluation across datasets, encoders, and decoders. We address the scope of the sparse-support finding and report additional comparisons of spatial decoders, search strategies, and support discovery beyond ACDC.

Summary of Key Concerns and Main Responses

Key ConcernsMain Responses
Scientific contribution and scope (W1, Q1–Q2)Examined anatomical information in disjoint supports and the role of spatial decoding, and specified where extreme sparsity is observed.
Search strategy and cost (W2, Q3, Q7)Evaluated Dense-STG alone on six datasets and compared four deletion strategies from the same ACDC starting support.
Validation and reproducibility (W2, Q4–Q6)Added held-out random controls, disjoint-support searches, and repeated discovery on M&Ms and MSD Liver.

W1. Please see Q1–Q2.

W2. Please see Q3–Q7.


Q1. Scientific contribution beyond sparse concept channels

Thank you for raising this point. We agree that SCC established the predictive value of sparse native channels for clinical finding detection (Sec. 2.1). Our contribution is to show that broadly distributed segmentation information can coexist with rare, non-unique combinations of native spatial maps that retain most of the full decoder’s performance.

On ACDC, the selected four maps achieve 0.787 development Dice, comparable to the mean for random supports containing 48–64 maps. None of 512 uniformly sampled four-map supports reaches that score (Sec. 4.2, Fig. 2a). Thus, many subsets carry information useful for segmentation, but particular combinations make it accessible with far fewer maps.

Following your suggestion, we have added two analyses to examine what the disjoint supports carry and how spatial aggregation affects the use of a sparse support.

1. Anatomical information in disjoint supports. Both supports exceed their respective width-matched random controls in mean within-heart AUROC (Supplement A.7.1, Table S5). Although the supports share no channels, each contains maps whose responses distinguish structures within the cardiac foreground.

2. Spatial aggregation and support width. The original comparison showed that the pointwise MLP uses a broader support than U-Net on the same frozen ACDC features (Table 2). To examine the role of spatial context, we compared a pointwise MLP and a local convolutional decoder of similar parameter count on identical input maps. We also evaluated the support selected independently for each decoder.

Spatial aggregation on ACDC (Supplement A.7.2, Table S6). Retention is relative to each decoder’s full-width test Dice.

DecoderAll 384 maps, test DiceSame four maps, test DiceIndependently selected widthTest Dice retention
Pointwise MLP0.7760.3166795.09%
Local convolutional0.7970.7142996.14%

The two decoders perform similarly with all 384 maps. Restricting them to the same four maps widens the test Dice gap from 0.021 to 0.398 in favor of the local decoder. Its independently selected support is also narrower at comparable retention. Together, these results support the interpretation that access to neighboring locations helps a decoder use fewer native maps.


Q2. Scope of extreme sparsity

Thank you for this helpful observation. We agree that the degree of sparsity varies across tasks. The four-channel supports on ACDC and M&Ms illustrate the extreme sparsity observed in the cardiac MRI settings. On COCO and ADE20K, the selected supports are substantially broader, yet they retain most of the full decoder’s test performance and outperform all evaluated width-matched random supports on the development set (Tables 1–2).

Following your suggestion, we now state explicitly in the Abstract that “the degree of sparsity varies across tasks.” Sec. 4.5 discusses how the cardiac and natural-image results support this distinction.

AuthorsOpenReview ↗

Response to Reviewer Kegy (Part 2/3)

Q3. Search baselines and the contribution of Dense-STG

Thank you for this valuable suggestion. To test how the search strategy affects the result, we have added a direct comparison with exhaustive greedy, first-improvement greedy, and random-order deletion, together with a separate fresh-decoder evaluation of Dense-STG alone across six datasets (Supplement A.8, Table S7).

For the ACDC comparison, all four deletion strategies start from the same confirmed 17-channel Dense-STG support and use the same development-Dice acceptance rule. We compare the last accepted support within 17 candidate checks and the endpoint of each completed search. Each new decoder fit uses the same training budget.

Table R1, panel A. Deletion strategies from the same ACDC starting support. Times cover the deletion stage on one GH200. Random-order entries are means over five runs.

MethodWidth within 17 checksTest Dice within 17 checksFinal widthFinal test DiceTotal checksMinutes
Dense-STG alone——170.7800—
PA-BFE40.77240.772923.1
Exhaustive greedy160.78160.775138368.8
First-improvement greedy50.76850.7681948.65
Random-order deletion8.20.7767.40.77422.656.36

Table R1, panel B. Dense-STG alone across datasets. Each Dense-STG support is evaluated with a fresh decoder. Retention is relative to its matched full-width decoder. Final support widths are from Table 1.

MeasureACDCM&MsMSD LiverG1020COCOADE20K
Dense-STG width17723211350195
Final support width441121148106
Dense-STG test Dice0.7800.8030.7350.7840.6350.502
Dense-STG retention96.18%97.60%96.19%100.66%99.92%96.58%

Dense-STG already identifies predictive supports. With a fresh decoder, these supports retain over 95% of full-width test Dice on all six datasets. On G1020, COCO, and ADE20K, however, Dense-STG retains more than half of the original channels (panel B). Thus, Dense-STG alone does not consistently identify supports that are both sparse and highly predictive.

The ACDC comparison shows what the deletion stage adds. Starting from the same predictive support, greedy and random-order deletion also recover compact supports with test Dice close to that of PA-BFE. The difference is clearest under the common budget: exhaustive greedy still retains 16 of the 17 starting channels, while PA-BFE has completed its four-channel search at comparable test Dice (panel A). Thus, the existence of a compact, predictive ACDC support holds across search strategies, while the choice of strategy affects how efficiently such a support is reached.


Q4. Held-out comparison with width-matched random supports

Thank you for suggesting a direct test-set comparison. The original Table 2 compares selected and width-matched random supports on the development sets, while Table 1 reports test performance relative to the full decoder. Following your suggestion, we have added a comparison on held-out patients from M&Ms and MSD Liver, evaluating the selected support against 256 random supports of the same width on each dataset (Supplement A.9.1, Table S8).

Table R2. Test Dice of selected and width-matched random supports on held-out patients. The SD measures variation across random supports. The 95% confidence interval is obtained by resampling patients for the difference between the selected score and the random mean.

DatasetWidthRandom test Dice (mean ± SD)Selected − random mean [95% CI]Random ≥ selected
M&Ms40.707 ± 0.0200.100 [0.092, 0.107]0/256
MSD Liver110.634 ± 0.0260.111 [0.078, 0.122]0/256

On both datasets, the selected support outperformed all 256 width-matched random supports on held-out patients. The patient-bootstrap interval was wider on MSD Liver, but both intervals remained above zero. Together with the ACDC test comparison in Supplement A.3.1, Table S2, these results show that the advantage of the selected coordinates persists on independent test patients across all three datasets evaluated.

AuthorsOpenReview ↗

Response to Reviewer Kegy (Part 3/3)

Q5. Non-uniqueness beyond ACDC

Thank you for suggesting that we test whether non-uniqueness extends beyond ACDC. We have repeated the exclusion experiment from Sec. 4.4 on M&Ms and MSD Liver (Supplement A.9.2, Table S9, panel A). Before each search, we excluded every channel in that dataset’s original support and searched the remaining native maps for an alternative.

Table R3. Disjoint supports on M&Ms and MSD Liver. Retention is relative to the matched full-width decoder on the same test set.

DatasetOriginal widthDisjoint widthShared channelsDisjoint test DiceFull test DiceRetention
M&Ms4600.7910.82396.04%
MSD Liver116500.7270.76495.20%

On both datasets, the decoder retained over 95% of full-width test Dice without access to any originally selected channel. This extends the evidence for non-uniqueness from ACDC to a second cardiac MRI dataset and to liver CT.


Q6. Random-support sampling and independent discovery

Thank you for suggesting larger random-support comparisons and repeated searches beyond ACDC. We have increased the number of random supports evaluated on held-out patients and repeated the full discovery procedure on M&Ms and MSD Liver.

Random-support sampling. The smaller samples at intermediate widths in Fig. 2(a) characterize the development-set trend. The primary comparison uses 512 random supports at the selected width of four channels to estimate how often a random draw matches the selected support. On the ACDC test set, the revised Table S2 (Supplement A.3.1) now reports 128 random supports at each of 4, 8, 16, and 32 channels. Even at 32 channels, the random supports average 0.753 test Dice, below the 0.772 achieved by the selected four maps. On M&Ms and MSD Liver, the 256-support comparisons show that the selected supports also retain their advantage on held-out patients. Table R2 in Q4 reports both variation across random supports and patient-level confidence intervals for that advantage.

Independent discovery. We added two complete searches on each dataset using seeds 23 and 41 alongside the original seed 17. Each run searches over all 384 channels and evaluates its selected support against its own matched full-width decoder (Supplement A.9.2, Table S9, panel B).

Table R4. Support width and held-out test performance across independent searches.

DatasetSeedSelected widthSelected test DiceFull test DiceRetention
M&Ms1740.8080.82298.29%
M&Ms2330.7980.82396.98%
M&Ms4120.7960.82097.11%
MSD Liver17110.7450.76497.48%
MSD Liver23260.7240.76095.24%
MSD Liver41100.7320.76296.08%

Every search retained over 95% of its matched full-decoder test Dice. The selected width varied more on MSD Liver than on M&Ms. Thus, high test retention recurred across independent searches on both datasets even when the searches settled on different support widths.


Q7. Search cost and practical efficiency

Thank you for this helpful suggestion. We have added measured runtimes and candidate-check counts for all four ACDC deletion strategies (Supplement A.8, Table S7, panel A). All timings start from the same confirmed 17-channel Dense-STG support and cover the deletion stage on one GH200 GPU.

As reported in Table R1, panel A, PA-BFE reached four channels in 9 checks and 23.1 minutes. Exhaustive greedy reached six channels in 138 checks and 368.8 minutes, with comparable final test Dice (0.772 for PA-BFE and 0.775 for exhaustive greedy). Thus, both strategies recovered predictive supports, but PA-BFE reached a narrower support in substantially less search time.

This cost is incurred offline, once per dataset-level support. The selected channel indices are then reused for all evaluation images. At inference, the frozen encoder computes the full representation and the decoder receives only the retained maps (Supplement A.1 and Sec. 5).

We thank the reviewer again for the constructive feedback and welcome further discussion.

Reviewer_wovgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigated the redundancy of frozen vision foundation model representations for image segmentation. The main idea is to find extremely sparse subsets of feature channels that can preserve most of the segmentation performance. The authors proposed a feature elimination strategy to select the channels and then trained new decoders using only the selected features. Experiments on medical and natural image datasets demonstrated that very few channels can achieve competative performance compared with the full representations.

Critical strengths
  1. This p1aper studies an interesting and meaningful problem. Although the redundancy of foundation model features has been investigated before, it is still interesting to understand whether only a few original channels can support downstream segmentation.

  2. The results are impressive. For example, the authors demonstrated that only four out of 384 channels can preserve more than 95% of the full segmentation performance on ACDC. This is an interesting observation.

  3. The experiments also investigated different datasets, feature encoders, and alternative channel subsets. The results provide useful insights into the structure of pretrained vision representations.

Critical weaknesses
  1. My main concern is the generality of the extremely sparse representations. Although four channels are sufficient for ACDC, the results on COCO and ADE20K require much more channels. Therefore, the main conclusion may strongly depend on the dataset and downstream task.
  2. The proposed feature selection requires repeatedly training new decoders. This process can be computationally expensive, and it is unclear whether the method is practically useful for feature compression.
  3. Another concern is the explanation of the selected channels. The authors demonstrated that some channel combinations are effective, but it is still unclear why these channels contain sufficient information and whether they correspond to meaningful visual or anatomical features.
  4. Although the decoder only uses a few channels, the frozen encoder still needs to generate the complete representations. Therefore, the claimed sparsity does not necessarily lead to significant inference acceleration.
Questions for authors
  1. Please clarify whether the main contribution is understanding the structure of pretrained representations or developing an efficient feature compression method. The current results seem to support the former more strongly.
  2. Discuss why the required number of channels differs significantly across datasets and whether the current observations can be generalised.
  3. Please clarify the computational limitations of the proposed feature selection and distinguish channel sparsity from end-to-end inference efficiency.
3 comments
AuthorsOpenReview ↗

Response to Reviewer wovg (Part 1/2)

We sincerely thank the reviewer for recognizing the significance of our research question and the breadth of our evaluation across datasets and encoders. In response, we discuss how support width varies across tasks and decoders, examine the anatomical information carried by the selected maps, and report the cost of offline support discovery.

Summary of Key Concerns and Main Responses

Key ConcernsMain Responses
Main contribution (Q1)The results show that broad redundancy in frozen representations can coexist with rare, non-unique combinations of native maps that retain most of the full decoder’s segmentation performance.
Generality across tasks (W1, Q2)The advantage over width-matched random supports recurs across settings, while the degree of sparsity varies with the task and decoder.
Anatomical interpretation (W3)Added comparisons with cardiac structure labels showing that maps in both disjoint ACDC supports distinguish cardiac structures.
Search cost and inference efficiency (W2, W4, Q3)Reported deletion-stage checks and runtime and specified which maps the encoder computes and the decoder uses at inference.

W1. Please see Q2.


W2. Please see Q3.


W3. Anatomical interpretation of the selected channels

Thank you for this valuable suggestion. We have added an anatomical alignment analysis comparing the selected channel responses with cardiac structure labels on the ACDC development set (Supplement A.7.1, Table S5, p. 20).

Among the four channels in the primary support, channel 257 showed the strongest full-image discrimination for both the left and right ventricular cavities (one-vs-rest AUROC 0.949 and 0.907, respectively). Within the cardiac foreground, both the primary support and its disjoint alternative also exceeded their width-matched random controls in mean channel-wise AUROC (Table R1).

Table R1. Mean within-heart AUROC on the ACDC development set. Scores are averaged across channels and cardiac structures. Each random mean uses 512 width-matched supports.

SupportSelectedRandom meanDifference [95% CI]
Primary support (4 channels)0.6100.584+0.026 [0.007, 0.045]
Disjoint support (13 channels)0.6200.585+0.035 [0.028, 0.043]

These results show that both disjoint supports contain maps whose responses distinguish cardiac structures. We discuss this finding alongside the disjoint-support result in Sec. 4.4.


W4. Please see Q3.

AuthorsOpenReview ↗

Response to Reviewer wovg (Part 2/2)

Q1. Main contribution

Thank you for highlighting the distinction between understanding frozen representations and using sparse features for compression. Our contribution is to show that broadly distributed task information in frozen vision representations can coexist with rare, non-unique combinations of native feature maps that retain most of the full decoder’s segmentation performance. PA-BFE locates candidate supports, and training a fresh decoder on the retained maps tests their predictive value independently of the discovery model (Sec. 3.1, Fig. 1).

The original ACDC results illustrate this finding. The selected four maps perform comparably to the average random support containing 48–64 maps, while none of 512 random four-map supports matches their development Dice (Sec. 4.2, Fig. 2a). Thus, task information is accessible through many subsets, but some combinations preserve it at much smaller widths. Excluding all four selected maps still yields a disjoint support with comparable test Dice (0.773 versus 0.772), showing that this predictive capacity is accessible through more than one combination of native coordinates (Sec. 4.4, Fig. 3).


Q2. Generality and setting-dependent support width

Thank you for raising this question about generality. Across the evaluated settings, selected supports retain most of the full decoder’s test performance and outperform width-matched random supports on the development set, while the degree of sparsity varies across tasks and decoders (Tables 1–2).

In the original cross-dataset comparison, with the encoder and decoder held fixed, the search selects broader supports on COCO and ADE20K than on the cardiac MRI datasets. These broader supports retain 96.7% and 95.8% of full-decoder test Dice, respectively (Table 1). On both datasets, every evaluated random support of the same width scores below the selected support on the development set (Table 2). Thus, the choice of native maps continues to matter even where the search selects a broader support. The revised Abstract and Sec. 4.5 make this distinction explicit.

To examine how decoder design contributes to this variation, we have added a controlled comparison between a local convolutional decoder and a pointwise MLP of similar size (Supplement A.7.2, Table S6, p. 21). With the ACDC task and frozen features held fixed, the local decoder retained over 95% of its full-width test Dice with 29 maps, while the pointwise MLP used 67 maps for comparable retention. The local decoder can combine responses from neighboring positions, whereas the pointwise MLP predicts from each position independently. This comparison shows that the width recovered on the same task and representation also depends on how the decoder uses spatial information.


Q3. Discovery cost and end-to-end efficiency

Thank you for raising this practical concern. We have added the measured cost of the ACDC deletion stage in Supplement A.8, Table S7 (p. 21) and specified how the selected support is used at inference in Sec. 5.

  • Offline discovery. Starting from the confirmed 17-channel ACDC support, PA-BFE reached four channels in 9 candidate checks and 23.1 minutes on one GH200. From the same starting support, exhaustive greedy deletion took 138 checks and 368.8 minutes. These timings begin after initialization and confirmation of the starting support. A check can reuse a previous decoder evaluation when the support, feature cache, decoder seed, and training configuration all match (Supplement A.1).
  • Inference. Once a dataset-level support is selected, its channel indices are fixed across evaluation images. In the primary ACDC setting, the frozen encoder computes all 384 maps and the decoder receives four. The reduction therefore applies to the number of maps passed to the decoder, while the encoder still runs at full width (Sec. 5).

We sincerely thank the reviewer again for the constructive feedback. We hope the anatomical analysis, controlled decoder comparison, and cost measurements address these questions, and we welcome further discussion.

Many thanks to the answers from the authors. I will keep my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

I. Acknowledgments

We thank the AC and all three reviewers for their careful evaluation and constructive feedback. Their comments helped us clarify the contribution of the work and strengthen the evidence for our findings across tasks and evaluation settings. Below, we summarize the strengths recognized by the reviewers, our main responses, and the revisions to the manuscript.


II. Key Strengths

The reviewers highlighted three strengths.

  • Research significance. The reviewers found the question of native-channel redundancy meaningful and the segmentation performance of sparse channel sets striking (wovg, Kegy).
  • Clean evaluation protocol. Discarding the discovery model and training a fresh decoder on the retained channels directly tests their predictive value (nFaP).
  • Strong empirical validation. The reviewers valued the comparisons across datasets, encoders, and decoders (wovg, Kegy), together with the random controls, repeated searches, and disjoint supports that substantiate the finding (nFaP).

III. Key Concerns and Main Responses

Key ConcernsReviewersMain Responses
Scientific contributionwovg, KegyClarified that broad redundancy in frozen vision representations can coexist with rare, non-unique combinations of native maps that preserve most of the full decoder’s segmentation performance.
Generality across taskswovg, KegyExtended the random-support and disjoint-support comparisons to M&Ms and MSD Liver, strengthening the evidence that predictive supports recur at task-dependent widths.
Anatomical interpretation and spatial decodingwovg, KegyAdded anatomical alignment measurements and compared local and pointwise decoders on the same selected maps.
Dependence on the search strategyKegy, nFaPCompared PA-BFE with Dense-STG alone and showed that greedy and random-order deletion also recover compact, predictive supports.
Search costwovg, KegyReported candidate checks and elapsed time for each deletion strategy from the same ACDC starting support.
Robustness of sparse supportsKegy, nFaPHigh retention persists under volumetric Dice, and compact supports recur across decoder seeds and patient splits.

IV. Additional Clarifications

We also clarify three aspects of the evaluation and search protocols.

  • Random sampling (Kegy). The smaller samples in Fig. 2(a) trace the trend across widths, while the primary four-channel comparison uses 512 random supports.
  • Search-budget accounting (Kegy, nFaP). A candidate check counts toward the budget even when a matching cached evaluation can be reused (Supplement A.1).
  • Inference efficiency (wovg). The frozen encoder still computes all feature maps at inference, and only the selected maps are passed to the decoder (Sec. 5).

V. Revisions and Updates

The revised manuscript incorporates the following changes.

  • Scope of the finding — Abstract and Sec. 4.5. Clarified how the degree of sparsity varies across tasks.
  • Anatomical alignment and spatial decoding — Secs. 4.4–4.5, Supplement A.7, Tables S5–S6 (pp. 20–21). Added anatomical comparisons and a decoder comparison on fixed input maps.
  • Search strategies and computational cost — Secs. 4.2 and 5, Supplements A.1 and A.8, Table S7 (p. 21). Added search baselines and deletion-stage costs, and clarified which computations use the selected channels at inference.
  • Cross-dataset validation — Sec. 4.5, Supplement A.9, Tables S8–S9 (p. 22). Added held-out random controls, disjoint supports, and repeated searches on M&Ms and MSD Liver.
  • Volumetric evaluation and stability — Secs. 4.3–4.5, Supplement A.10, Tables S10–S11 (p. 23). Added class-wise volumetric Dice and support comparisons across decoder seeds and patient splits.

We thank the AC and reviewers again for their time and valuable feedback. We hope these clarifications and additions provide a clearer basis for assessing our paper, and we welcome further questions and discussion.

Area_Chair_MeX4OpenReview ↗

Discussion Reminder

Dear reviewers, please review the authors’ responses and leave your comments on whether your concerns have been addressed. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.