acceptodds
Under review as a conference paper at ICLR 2027

PACO-SAE: Evaluating Concept Grounding and Feature Splitting in Vision Sparse Autoencoders

Abstract

Sparse autoencoders (SAEs) are used to interpret vision models, but predicting a concept does not establish whether a feature activates on the object it represents. We introduce PACO-SAE, an object-level benchmark and evaluation protocol that evaluates concept grounding as semantic segmentation. Built from exhaustively annotated PACO-LVIS objects, it combines grounding F1 with separate feature selection and testing across 63 object concepts. Across four SAE families and four sparsity levels, image-level probing and peak localization select features with lower test grounding F1 than grounding-based selection, showing that neither substitutes for evaluating the entire activated region. PACO-SAE also enables object-level analysis of feature splitting: different features best ground different objects of one concept, substantially outperforming a single feature selected for the highest average grounding. To examine whether these splits reflect visual specialization or residual splitting, we introduce Pairwise Visual Separation (PVS), which compares visual differences between groups of objects with variation within each group, without additional subclass labels. A human study of 200 feature pairs supports PVS as an indicator of recognizable visual differences. Together, PACO-SAE establishes an object-level evaluation foundation, reveals the limits of prediction and localization as grounding criteria, and connects the extent of feature splitting to its visual meaning.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoder, mechanistic interpretability, interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_R6CvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PACO-SAE, an object-level benchmark for evaluating whether vision SAE features are truly grounded in the visual concepts they represent, rather than merely predicting concept presence. It further analyzes feature splitting across object instances and proposes Pairwise Visual Separation (PVS) to distinguish meaningful visual specialization from residual splitting.

Critical strengths
  1. The paper is motivated by a clear and important limitation of existing vision SAE evaluation: strong concept prediction does not necessarily imply correct spatial grounding.
  2. The proposed benchmark and evaluation protocol are well designed, with exhaustive object annotations, held-out feature selection/testing, and object-level grounding metrics that make the conclusions convincing.
  3. The experiments are comprehensive and insightful, especially the analysis showing extensive feature splitting and the human-supported PVS metric for examining whether such splits correspond to meaningful visual specialization.
Critical weaknesses
  1. The evaluation is limited to a single visual backbone and specific layers. Experiments were conducted solely on the 11th block of DINOv3 ViT-B/16. Evaluating additional backbones (such as CLIP-based models) and various layers would significantly strengthen the generalizability of the conclusions.
  2. The feature splitting analysis based on an Oracle relies on ground-truth masks, potentially overstating the practical significance of the splitting phenomenon. The metric improved substantially by utilizing ground-truth segmentation masks to select the optimal feature for each test object. While this demonstrates that distinct features can localize different instances, it does not show whether such instance-specific feature selection is achievable without Oracle supervision. Therefore, the reported 39%–43% performance gain should be interpreted with caution regarding its real-world implications.
  3. PVS measures the degree of specialization within the backbone's representation space rather than directly within the SAE feature space. PVS is calculated based on the distances between object representations derived from backbone features prior to the SAE stage. Thus, a high PVS value merely indicates that the backbone has visually distinguished between two sets of objects; it does not necessarily prove that the SAE has learned semantically meaningful, specialized features. The quality of the paper would be enhanced by supplementary analysis demonstrating that the observed specialization is specifically induced or captured by the SAE, rather than simply inherited from the backbone's representations.
Questions for authors

See the weakness above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer R6Cv (1/2)

Thank you for recognizing the importance of distinguishing concept prediction from spatial grounding, the benchmark's annotation and evaluation protocol, and the feature-splitting analysis. We address your three concerns below.

W1. The evaluation needs additional visual backbones and network layers.

We extended the evaluation to CLIP ViT-B/16 across all four SAE families and k∈{32,64,128,256}k \in \{32, 64, 128, 256\}, and to DINOv3 layers 9 and 10 with BatchTopK-128, using the same selection/test protocol.

Additional settingTest F1 gain over probing selectionTest F1 gain over mSLA selection
CLIP, layer 11: 16 settings+0.018 to +0.058+0.014 to +0.037
DINOv3, layer 9+0.059+0.049
DINOv3, layer 10+0.043+0.020

The grounding advantage generalizes to every additional setting. CLIP also reproduces substantial splitting: object-wise oracle F1 exceeds fixed-feature F1 by 0.187–0.244 across its 16 settings. (Revised Tables 1–2, §4.1–4.2, Figures A4, A6, and A8.)

W2. Ground-truth reliance and practical implications of the oracle analysis.

Thank you for emphasizing the practical interpretation of the oracle analysis. Our analysis asks whether one feature selected for a concept grounds its different objects well, or whether different objects are better grounded by different features. Ground-truth masks let us measure this difference. The reported 39–43% relative F1 improvement quantifies how much a single concept-selected feature misses relative to other existing features; it is a diagnostic upper bound, not a demonstrated test-time improvement. We agree that this distinction matters for real-world implications and have emphasized it in the revised §4.2.

Inspired by your emphasis on real-world implications, we conducted a proof of concept using Grounding DINO [1] and SAM [2] to recover part of the oracle gain without ground-truth test masks for feature selection. We randomly sampled 1,260 test objects—20 from each of the 63 concepts—and evaluated DINOv3 ViT-B/16 BatchTopK at k=128k = 128. For each existing benchmark crop, we provide the known concept name to Grounding DINO and use the predicted box nearest the crop center to prompt SAM. From the same 32 candidates fixed on the selection set, we select the feature whose activation mask has the highest F1 against the resulting pseudo mask. Feature choices are fixed before evaluation against ground-truth test masks. If no box is detected or the projected pseudo mask contains no foreground patch, we retain the concept-selected feature, jgroundj_{\text{ground}}.

For each evaluation subset SS, we define the oracle-gain recovery as

Recovery(S)=Gpseudo(S)−Gfixed(S)Goracle(S)−Gfixed(S)×100\text{Recovery}(S) = \frac{G_{\text{pseudo}}(S) - G_{\text{fixed}}(S)}{G_{\text{oracle}}(S) - G_{\text{fixed}}(S)} \times 100\\%.

Each GG is grounding F1 averaged over objects within each concept and then equally over the 63 concepts. All three terms use the same objects in SS.

Evaluation subsetObjectsFixed-feature F1Pseudo-selected F1GT-oracle F1Oracle-gain recovery
All sampled test objects1,2600.5220.5690.73821.92%
Objects with usable pseudo masks1,0270.5440.6050.75928.33%

The usable-mask subset includes all objects for which pseudo-mask matching was possible, including inaccurate masks. Recovery is calculated from unrounded F1 scores. Thus, pseudo masks recover 28.33% of the oracle gain when usable masks are available, and 21.92% overall, including the 233 cases without a usable mask. Although this recovers only part of the gap, it shows that automatically generated masks can support per-object feature selection without ground-truth test masks. We believe improving this approach is an interesting direction for future work, which we highlight in the revised §5.

[1] Liu et al. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV 2024.

[2] Kirillov et al. Segment Anything. ICCV 2023.

AuthorsOpenReview ↗

Response to Reviewer R6Cv (2/2)

W3. PVS may reflect backbone geometry rather than specialization captured by SAE features.

Response: Thank you for this insightful comment on the relationship between the SAE backbone and the PVS representation, and for motivating a comparison using SAE feature parameters.

1. SAE features determine the object groups; PVS evaluates their visual differences. Under our oracle protocol, objects of each concept are grouped by the SAE feature with the highest grounding F1 against their masks. PVS evaluates the groups' visual separation relative to within-group variation. Although pretrained representations already contain visual structure, the groups are determined by the SAE's grounding behavior. Different SAE features can best ground visually similar objects and yield low PVS, illustrating residual splitting. We therefore use PVS to assess the degree of visual specialization captured by SAE features.

2. PVS remains informative when its evaluation space differs from the SAE backbone’s representation space. We make this distinction explicit in revised §3.4. We conducted an additional human evaluation of CLIP-trained BatchTopK at k=128k=128, with DINOv3 fixed as the PVS extractor (revised Figure 6). The retained samples comprise 196 CLIP pairs and 195 DINOv3 pairs:

SAE backboneQ1 (lowest PVS)Q2Q3Q4Q5 (highest PVS)
CLIP0.08 [0.03, 0.21]0.33 [0.20, 0.48]0.30 [0.18, 0.45]0.31 [0.19, 0.46]0.56 [0.41, 0.71]
DINOv30.10 [0.04, 0.24]0.28 [0.16, 0.43]0.33 [0.21, 0.49]0.46 [0.32, 0.61]0.63 [0.47, 0.77]

Majority-Yes specialization rates; reported intervals in brackets. DINOv3 is the PVS extractor for both rows.

CLIP shows higher specialization in Q5 than Q1, with similar rates in Q2–Q4. Thus, the human-supported association persists when the SAE backbone and PVS extractor differ.

3. Comparison with SAE decoder directions. Following your suggestion, we compared PVS with decoder cosine distance, 1−cos⁡(dj,dk)1-\cos(\mathbf d_j,\mathbf d_k), between SAE decoder directions in the input representation space. On the same 195 DINOv3 pairs, Pearson correlation with the human Yes fraction is 0.363 for PVS versus 0.175 for decoder cosine distance (revised Table 3). This higher observed association supports using the rich visual information in the objects each feature grounds to evaluate specialization beyond decoder similarity alone.

Reviewer_KDp5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PACO-SAE, an object-level benchmark covering 63 concepts, and PVS, a metric for assessing visual differences underlying feature splitting. Experiments across four SAE families show that concept prediction and peak localization do not ensure accurate object grounding, while concepts are commonly distributed across multiple features. Human evaluation supports PVS as an indicator of meaningful visual specialization.

Critical strengths
  1. The object-level evaluation is carefully built. Annotations are exhaustive, ignore regions are explicit, and the selection and test splits are disjoint at the source-image level, which makes me more willing to trust the grounding numbers (Sec. 3.2, App. B).

  2. Table 1 and Figure 3 make a clear point: a feature can predict the concept and peak in the right place and still miss most of the object. Because all criteria are compared on the same test objects, the argument is easy to follow.

  3. I particularly liked the random SAE baseline. It gets strong concept-prediction scores but grounds poorly, which exposes a weakness of probing-style evaluation and helps separate what the SAE learns from what the backbone already provides (Tables 1–2).

  4. The feature-splitting analysis goes beyond counting splits. PVS is tied to visible differences, and the human evaluation supports this (Sec. 4.3).

Critical weaknesses
  1. The interpretation of feature splitting needs stronger calibration. Object-wise maximization guarantees nonnegative oracle gains, while winner diversity may reflect competition among near-equivalent candidates. The larger gains and higher diversity of the untrained SAE further motivate caution. Reporting winner margins, tie handling, resolved-object coverage, and sensitivity to candidate-set size would strengthen the interpretation.

  2. PVS is validated in one configuration only. It is computed in the backbone space the SAE reads, and the human study covers only BatchTopK k=128. The untrained SAE has a higher PVS (0.057 vs. 0.052) and was not human-checked. Inter-annotator agreement, the Fig. 6 error-bar definition, and dependence among pairs sharing features are not reported. There is no comparison with simpler distance measures.

  3. Grounding F1 lacks a reference point. The 0.02–0.04 gain (Table 1) is partly expected from selecting and evaluating on the same metric. Crops are object-centered (App. B), so a centered-region predictor may already reach non-trivial F1. Precision/recall would show whether gains come from coverage or from fewer background activations.

  4. Variant comparisons rest on single seeds. Differences of 0.003–0.005 (pp. 7, 9) are interpreted without seed variance. Matryoshka/C²R hyperparameters and PVS pair-eligibility rules are not given.

Questions for authors
  1. Oracle splitting calibration. Section 4.2 reports that training reduces splitting, but the untrained SAE has larger oracle gains and higher winner diversity (Table 2). Please report tie and near-tie frequencies, winner-to-runner-up margins, resolved coverage per concept, and the permutation details for the threshold. Are the findings robust to a small winner tolerance and to changes in candidate-set size?

  2. Baseline calibration. Given the object-centered crops, what F1 does a feature-agnostic centered-region predictor achieve under the same protocol? Please report precision and recall for all three selectors to explain whether the roughly 0.02–0.04 F1 gain reflects better target coverage or fewer non-target activations.

  3. PVS robustness. Please clarify the minimum group size, validity rules, pair-to-concept aggregation, and the selection of the common 58 concepts. Do trends survive controls for group size and object scale? The paper notes that the untrained SAE has a higher PVS (Table 2, p.9); what do you think explains this?

  4. Human-study reliability. Please clarify pairs per concept, task assignment and blinding, inter-annotator agreement, and how the error bars in Figure 6 are computed. Does the association between PVS and human judgments hold after accounting for concept?

4 comments
AuthorsOpenReview ↗

Response to Reviewer KDp5 (1/4)

Thank you for your careful review and for highlighting the annotation protocol, the comparison on common test objects, the random-SAE control, and the analysis of visible differences between split features. Your questions prompted the following calibration and reporting checks.

W1/Q1. Could oracle gains and winner diversity arise from maximization, near ties, or the candidate-pool size?

We agree that an oracle gain requires calibration. First, allowing the best single feature to be chosen using test labels raises BatchTopK F1 only from 0.519 to 0.523, whereas choosing a feature separately for each object reaches 0.739. Thus, 0.216 of the 0.219 oracle gap remains beyond even the best fixed test-set feature.

We also checked ties and winner margins, with the following results for 32 candidates:

DiagnosticBatchTopK-128
Resolved-object coverage95.5%
Exact winner ties among resolved objects6.7%
Mean winner-to-runner-up F1 margin0.070
Near ties: margin ≤0.01\le 0.0115.1%
Resolved objects retained at margin >0.02> 0.0274.9%
Effective winner count after this exclusion12.21

Rates are concept-macro averages, and margin statistics use resolved objects. BatchTopK's per-concept resolved coverage ranges from 84.7% to 100% (median 96.4%). After excluding margins ≤0.02\le 0.02, the four trained families still use 11.2–12.7 effective features, so splitting persists beyond near ties.

Exact ties use the first feature in the frozen selection ranking. The resolution threshold is the 95th percentile of maximum candidate F1 from 50 target-mask permutations for each of 2,000 concept-balanced selection objects; every candidate sees the same permuted mask. Resolved coverage is the fraction of positive test objects whose oracle F1 exceeds this threshold.

With 8/16/32/64 candidates, BatchTopK oracle F1 is 0.685/0.716/0.739/0.758, while fixed-feature F1 remains 0.519. We recalibrate the threshold at each pool size. The untrained SAE has higher winner diversity but lower fixed/oracle grounding F1 (0.348/0.674 versus 0.519/0.739). We therefore report splitting extent separately from grounding quality. (Revised Appendix K, Figure A9.)

W3/Q2. What does a centered-region predictor achieve, and do the F1 gains reflect target coverage or fewer background activations?

We agree that a centered-region predictor provides a useful reference for grounding F1. Although target objects are not always centered, many crops place them near the center. We therefore select one square size on the selection set and use it for every test image and concept. The selected square is 6×6 patches on the 14×14 grid, evaluated with the same target and ignore masks as the SAE features.

Predictor / selectorPrecisionRecallF1
Global centered square0.3760.7050.455
Probing selection0.4840.6270.483
mSLA selection0.5470.5750.499
Grounding selection0.5310.6340.519

SAE rows use DINOv3 BatchTopK-128; all values use the same 63 test concepts. All three selectors achieve higher F1 than the global centered square, showing better grounding than a fixed center prediction. Relative to probing selection, grounding selection reduces background-positive patches from 13.0 to 9.7 per object while preserving target coverage. Relative to mSLA selection, the gain comes mainly from higher recall, 0.575 → 0.634. The center and untrained-SAE (Table 1; F1 0.348) references help interpret absolute grounding scores and complement our main comparison of the three selectors on identical test objects. (Revised Table 1 caption, §4.1, Appendix I.)

AuthorsOpenReview ↗

Response to Reviewer KDp5 (2/4)

W2/Q3. How are PVS groups and pairs defined, and what explains the untrained SAE's higher PVS?

Group definition, validity rules, and aggregation. Each feature group contains the resolved objects of a concept for which that feature is the best-grounding candidate. A group must contain at least five objects from at least three source images; a pair requires finite distances and positive between-group distance, and a concept requires at least one valid pair. PVS uses target-only DINOv3 object embeddings and excludes self-distances from within-group means. Pair scores are weighted by nAnBn_A n_B within a concept, then concepts are averaged equally. The common 58 concepts have valid PVS in every configuration included in the corresponding paper comparison.

Comparison with simpler distance measures. On the same 195 pairs, PVS has Pearson r=0.363r=0.363 with the human Yes fraction, compared with r=0.343r=0.343 for the simpler mean inter-group L2L_2 distance and r=0.175r=0.175 for decoder cosine distance, which uses SAE weights rather than backbone object embeddings; the revised Table 3 (§4.3) reports the full comparison, including four further alternatives.

Untrained SAE. We explain the higher PVS of the untrained SAE through three points (Revised §4.3).

  1. Untrained SAEs still operate on pretrained visual representations. In revised Table 2, untrained versus trained BatchTopK at k=128k=128 has PVS 0.057 versus 0.052 on DINOv3 and 0.036 versus 0.039 on CLIP. Although the SAE is untrained, its inputs retain the visual structure learned by the pretrained backbone. Random encoder directions can respond selectively to this structure, allowing the resulting object groups to exhibit visual differences without SAE training. This interpretation is consistent with prior evidence that fixed random encoder directions can yield interpretable activation patterns on pretrained representations [1].

  2. PVS remains informative about the visual differences between untrained feature groups. Figure A11 presents untrained-SAE examples for the concept fan, following the qualitative comparison in Figure 5. The higher-PVS pair contains visually distinct object groups, consistent with specialization, whereas the lower-PVS pair assigns visually similar objects to separate features, consistent with residual splitting. These examples illustrate how PVS can characterize the visual differences underlying a split even when the SAE itself is untrained.

  3. PVS complements grounding and splitting metrics. As emphasized in revised §4.3, PVS measures relative visual separation between object groups and should be interpreted jointly with grounding and feature splitting. In Table 2, the untrained SAE has substantially lower grounding GtestG_{\mathrm{test}} than trained BatchTopK: 0.348 versus 0.519 on DINOv3, and 0.273 versus 0.441 on CLIP. Thus, competitive PVS does not imply comparable overall SAE quality. Grounding measures how accurately features localize objects, splitting metrics quantify how objects are distributed across features, and PVS describes the visual distinctions between those groups.

[1] Korznikov et al. Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer KDp5 (3/4)

W2/Q3/Q4. How reliable is the human study, and does the PVS association persist after accounting for concept?

Study design and blinding. The original DINOv3 study samples 200 pairs, 20 per PVS decile, spanning 60 concepts with 1–4 pairs per concept (median 3). Annotators see four randomly sampled objects per group, with target outlines and labels A/B; feature IDs, PVS scores, and deciles are not displayed. Pairs are presented in random order while balancing the number of answers per pair, and annotators can stop at any time. The 17 annotators contributed 1,217 responses, with at least five per pair and uneven workloads of 1–200 pairs per annotator. Five pairs with at least three NA votes were excluded, leaving 195 pairs; remaining NA votes are omitted and Yes/No ties count as No. (Revised Appendix L: Task and pair sampling; Annotation protocol; Response aggregation.)

Inter-annotator agreement. Two annotators of the same pair agree on 62% of answers, versus 50% expected by chance (Fleiss' κ=0.23\kappa = 0.23). Binary answers split on borderline pairs, which limits individual agreement, but the pair-level Yes rate used in our analysis is consistent across annotators: splitting each pair's annotators into two random halves gives Spearman ρ=0.45\rho = 0.45 between the halves' Yes rates. (Revised Appendix L: Inter-annotator agreement.)

Figure 6 error bars. Error bars are 95% Wilson intervals for the majority-Yes rate over the 38–40 pairs in each quintile. (Revised Figure 6 caption.)

Dependence among pairs sharing features. Among the 195 pairs, 103 share a feature group with another pair. We link such pairs into 133 clusters of one to four pairs and compare pair-level with cluster-level bootstrap (10,000 resamples). Cluster resampling barely widens the 95% interval of the Spearman correlation between PVS and the Yes rate (0.21–0.46 to 0.20–0.46), and the top-quintile specialization rate exceeds the bottom-quintile rate in all resamples. (Revised Appendix L: Dependence among pairs sharing features.)

Controls for concept, group size, and object scale. We regress the human judgments on PVS while adding three controls: concept, the sizes of the two groups and their difference, and the area fraction of the target object in each crop, averaged per group, and its difference between groups. With the concept control, PVS is compared only among pairs of the same concept. Outcomes are the Yes fraction of each pair and the majority-Yes label used in Figure 6. 95% CIs use 2,000 concept-bootstrap resamples.

ControlsYes fractionMajority-Yes
None0.109 [0.069, 0.149]0.172 [0.109, 0.236]
Concept0.088 [0.038, 0.138]0.153 [0.069, 0.237]
+ group size0.088 [0.039, 0.140]0.154 [0.073, 0.236]
+ object scale0.092 [0.041, 0.145]0.162 [0.080, 0.243]
+ object scale of displayed images0.082 [0.029, 0.138]0.147 [0.066, 0.227]

Each entry is the increase in the outcome when PVS rises by one standard deviation. If these factors drove the association, the coefficient would fall to zero once they are controlled for. Instead, it stays positive with CIs above zero under every control: controlling for concept reduces it only moderately (0.109 to 0.088 for the Yes fraction), and adding group size and object scale leaves it nearly unchanged. The last row measures object area on the four images per group that annotators actually saw.

Group size can also be equalized directly rather than only controlled for. We recompute PVS using four objects from each group, averaged over 100 draws. Its Pearson correlation with the human Yes fraction is 0.368 [0.235, 0.495], versus 0.363 with full groups.

Thus, the PVS–human association is not explained by the measured concept, group-size, or object-scale differences.

Second configuration with a separate PVS encoder. We repeated the human study on a BatchTopK SAE (k=128k=128) trained on CLIP ViT-B/16, forming object groups with the same protocol but computing PVS from DINOv3 object representations, so that the representation used to measure separation differs from the one the SAE reads. With the same question, interface, and worked examples, 15 annotators gave 1,386 answers on 200 feature pairs across 55 concepts (4–9 per pair), sampled 20 per PVS decile. The specialization rate rises from 0.08 [0.03, 0.21] in the lowest PVS quintile to 0.56 [0.41, 0.71] in the highest, compared with 0.10 [0.04, 0.24] and 0.63 [0.47, 0.77] for the DINOv3 SAE. (Revised Figure 6, §4.3, Appendix L.)

AuthorsOpenReview ↗

Response to Reviewer KDp5 (4/4)

W4. Do the small variant differences survive training-seed variation, and are the training configurations reproducible?

We evaluated three training seeds for each family at k=128k=128:

SAEGrounding F1Winner diversity Doracle\boldsymbol{D}_{\textbf{oracle}}PVS
TopK0.5189 ± 0.00030.9076 ± 0.00160.0542 ± 0.0009
BatchTopK0.5202 ± 0.00090.9054 ± 0.00070.0522 ± 0.0003
+ Matryoshka0.5245 ± 0.00040.9019 ± 0.00070.0475 ± 0.0015
+ C$^2$R0.5220 ± 0.00170.9072 ± 0.00170.0506 ± 0.0013

Entries are mean ± sample SD across three seeds per family. F1 and winner diversity use 63 concepts; PVS uses the same 60 valid concepts across all 12 runs. (Revised Appendix F, Figure A5.)

Matryoshka uses cumulative prefix widths of 1,536, 4,608, 10,752, 23,040, and 49,152, with the five reconstruction losses summed using unit weights. C$^2$R uses coefficient 5, α=1\alpha=1, and randomized chunks of 8,192 features for decoder-neighbor search; it applies the raw penalty every fifth step with an interval-adjusted multiplier of 25. (Revised Appendix C, Table A2.)

All four families retain substantial splitting across seeds, and the variant effects remain modest. The PVS pair-eligibility rules are specified in W2/Q3 above.

Reviewer_UHULOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PACO-SAE, an object-level benchmark for evaluating concept grounding and feature splitting in vision sparse autoencoders (SAEs). Using semantic segmentation annotations, the authors demonstrate that conventional concept probing and peak localization do not reliably measure spatial grounding. The paper further investigates feature splitting across objects of the same concept and proposes Pairwise Visual Separation (PVS) to distinguish visual specialization from residual splitting. Experiments across multiple SAE variants and sparsity levels, together with human evaluations, support the proposed analysis.

Critical strengths
  1. Meaningful benchmark contribution. PACO-SAE provides an object-level evaluation protocol for concept grounding, addressing limitations of image-level probing and localization metrics.

  2. Comprehensive experimental analysis. The experiments cover multiple SAE variants and sparsity levels, providing useful evidence of grounding gaps and feature splitting.

  3. Interesting analysis of feature specialization. The proposed PVS metric provides a quantitative perspective on feature splitting, with supporting evidence from human evaluations.

Critical weaknesses
  1. Limited experimental diversity. All evaluated SAEs use the same DINOv3 backbone and layer, limiting conclusions about generalization across different visual representations.

  2. Limited novelty in evaluation methodology. Grounding F1 relies on standard segmentation metrics, while PVS builds on existing feature-distance measures. The main contribution is therefore primarily empirical.

  3. Insufficient validation of PVS. Although human evaluations support a relationship between PVS and visual specialization, the metric does not reliably distinguish specialization from residual splitting at the individual feature-pair level.

Questions for authors
  1. How well do the proposed evaluation metrics generalize to SAEs trained on other vision backbones, such as CLIP or DINOv2, and different network layers?

  2. Can the authors compare PVS with alternative cluster separation metrics to demonstrate its advantages in measuring visual specialization?

  3. How sensitive are the grounding and feature-splitting results to different feature-selection settings, activation thresholds, and SAE training seeds?

2 comments
AuthorsOpenReview ↗

Response to Reviewer UHUL (1/2)

Thank you for your thoughtful review and for recognizing the object-level benchmark, the comprehensive experiments, and the human-supported analysis of feature specialization. We address your concerns below.

W1/Q1. Do the grounding and feature-splitting findings generalize to other backbones and network layers?

We extended the evaluation to CLIP ViT-B/16 and DINOv3 layers 9 and 10 using the same selection/test protocol. CLIP covers all four SAE families at k∈{32,64,128,256}k \in \{32, 64, 128, 256\}. The grounding advantage holds in every additional configuration:

Backbone / layerSettingsTest F1 gain over probing selectionTest F1 gain over mSLA selection
DINOv3, layer 1116+0.028 to +0.048+0.017 to +0.078
CLIP, layer 1116+0.018 to +0.058+0.014 to +0.037
DINOv3, layer 9BatchTopK-128+0.059+0.049
DINOv3, layer 10BatchTopK-128+0.043+0.020

Values are concept-macro means over the same 63 test concepts; ranges are across settings. Splitting also persists: CLIP's object-wise oracle exceeds its fixed concept-level feature by 0.187–0.244 F1 across all 16 settings. Both findings therefore extend beyond the original backbone/layer. (Revised Tables 1–2, §4.1–4.2, Figures A4, A6, and A8.)

W2/Q2. What is novel in the evaluation contribution, and how does PVS compare with alternative separation metrics?

We agree that grounding F1 is a standard metric and that PVS adapts the energy distance of DISCO. Our contribution lies not in new formulas but in an evaluation design that makes previously overlooked properties (such as grounding and splitting) of SAE features measurable. First, PACO-SAE combines exhaustive annotations, explicit ignore regions, and independent selection/testing to evaluate whether concept-associated features ground new objects. This evaluation shows that concept prediction and peak localization do not ensure whole-object grounding. Second, PACO-SAE groups objects by their best-grounding feature, and PVS compares the visual differences between these groups to assess whether a split reflects visual specialization. Whereas existing evaluations assess splitting through the benefit of using multiple features for concept prediction, this reveals how a split divides the objects of a concept and whether the division is visually meaningful.

To examine whether PVS is an effective choice among separation measures for this purpose, we compared PVS with six alternatives on the same 195 evaluated feature pairs:

MetricPearson rr with human Yes fraction
PVS0.363
Mean inter-group L2L_20.343
Silhouette0.323
RBF MMD2\mathrm{MMD}^20.322
1-NN accuracy0.300
Linear-probe accuracy0.217
Decoder cosine distance0.175

These correlations use individual pairs, without binning. PVS has the largest observed coefficient in this comparison. The metrics also differ in what they measure: PVS contrasts between-group distance with within-group distances, whereas mean inter-group L2L_2 considers between-group distance alone and thus cannot distinguish group separation from the intrinsic variability within each group. PVS is also computed directly from object distances, without the classification protocol of 1-NN accuracy or the classifier training of linear-probe accuracy. Finally, all object-group metrics outperform decoder cosine distance, indicating that specialization is better captured by comparing the objects each feature grounds than by comparing decoder weights. (Revised Table 3, §4.3.)

AuthorsOpenReview ↗

Response to Reviewer UHUL (2/2)

W3. How reliably does PVS indicate specialization for individual feature pairs?

Additional human validation of PVS. We repeated the human study of Figure 6 on a second SAE (BatchTopK, k=128k=128, trained on CLIP ViT-B/16), using the same question, interface, and worked examples on 200 new feature pairs across 55 concepts (15 annotators, 1,386 answers; PVS computed from DINOv3 object representations). As in the original study, the specialization rate rises from the lowest to the highest PVS quintile. (Revised Figure 6, §4.3, Appendix L.)

Pair-level reliability. As noted in §4.3 and our Limitations, we use PVS as a continuous indicator of visual specialization rather than as a classifier of individual feature pairs. The original human study supports this use: PVS achieves a pair-level AUROC of 0.709 against majority judgments, and the specialization rate rises from the lowest to the highest PVS quintile (Figure 6). When a binary decision is needed, a threshold can be chosen from the ROC curve; the point maximizing Youden's JJ is PVS =0.056= 0.056, which yields a balanced accuracy of 0.67 (sensitivity 0.60, specificity 0.74) and 0.65 under leave-one-concept-out validation. PVS thus separates specialization from residual splitting above chance, though not reliably for every pair. Validating whether a stable threshold transfers across concepts and backbones requires a larger human study, which we leave to future work. (Revised Limitations, §5.)

Q3. Are grounding and splitting sensitive to feature-selection settings, activation thresholds, or SAE training seeds?

Training seeds. Across three seeds at DINOv3 layer 11 and k=128k=128, both the grounding advantage and oracle gap remain consistent:

SAEGrounding minus probing F1Oracle minus fixed-feature F1
TopK0.038 ± 0.0010.221 ± 0.002
BatchTopK0.039 ± 0.0020.219 ± 0.001
+ Matryoshka0.034 ± 0.0040.204 ± 0.002
+ C$^2$R0.040 ± 0.0020.217 ± 0.002

Entries are mean ± sample SD across training seeds. (Revised Appendix F, Figure A5.)

Activation thresholds. We tested cutoffs at the 5th, 10th, and 25th percentiles of each feature's positive selection-set activations, in addition to the default z>0z>0, and recomputed grounding selection. Across the four trained families at k=128k=128, the advantage remains +0.034–0.042 F1 over probing and +0.021–0.036 over mSLA; effective winner counts remain 10.9–11.5 at the strictest cutoff. (Revised Appendix H, Figure A7.)

Feature-selection settings. We varied the number of candidates (8/16/32/64) to test whether feature splitting persists across candidate-pool sizes. For BatchTopK-128, oracle F1 is 0.685/0.716/0.739/0.758 across these candidate pools, compared with a fixed-feature F1 of 0.519. Thus, substantial splitting persists across the tested settings. (Revised Appendix K, Figure A9.)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all three reviewers for their thoughtful and constructive feedback. Their comments helped us strengthen the evidence, clarify our findings, and improve the presentation. We have uploaded a revised manuscript incorporating these improvements.

We appreciate Reviewer UHUL's recognition of a “Meaningful benchmark contribution,” Reviewer KDp5's assessment that “The object-level evaluation is carefully built,” and Reviewer R6Cv's description of the experiments as “comprehensive and insightful.” Reviewer KDp5 also emphasized that our feature-splitting analysis “goes beyond counting splits.” These assessments highlight PACO-SAE's contribution to evaluating whole-object grounding and the visual distinctions underlying feature splitting.

To address the reviewers' concerns and strengthen our work, we conducted additional experiments and provided detailed clarifications:

1. Generalization and robustness (UHUL, KDp5, R6Cv). We extended evaluation to CLIP and additional DINOv3 layers, and tested training seeds, activation thresholds, and candidate-set sizes. The additional backbone and layer settings reproduce the grounding advantage over probing and peak-localization selection, while the sensitivity analyses further support the stability of the grounding and splitting findings. (§§4.1–4.2; Appendices F, H, K.)

2. Grounding and splitting calibration (KDp5). A centered-region baseline and precision–recall analysis establish a reference for grounding scores and explain the sources of improvement. Comparisons against the best fixed feature selected using test labels, together with checks of ties, winner margins, and resolved-object coverage, show that substantial splitting persists beyond near ties and fixed-feature selection effects. (§4.1; Appendices I, K.)

3. PVS validation and interpretation (UHUL, KDp5, R6Cv). We compared PVS with six alternative measures, including SAE decoder similarity, on the same human-evaluated pairs; PVS had the largest observed correlation with human judgments. This association persists after controlling for concept, group size, and object scale, and accounting for dependence among pairs sharing features. A new CLIP human study using DINOv3 representations for PVS further supports this association when the SAE backbone and evaluation space differ. We clarified that PVS measures visual differences between groups defined by SAE grounding behavior, with limits at the individual-pair level. The untrained-SAE results also highlight why PVS should be interpreted alongside grounding quality. (§§3.4, 4.3, 5; Table 3; Figure 6; Appendix L.)

4. Oracle interpretation and practical follow-up (R6Cv). We clarified the oracle gain as a diagnostic upper bound on what a single concept-selected feature misses (§4.2). In our response to Reviewer R6Cv, we also reported a Grounding DINO + SAM proof of concept that recovered approximately 22% of the oracle gain on 1,260 sampled test objects without ground-truth test masks for feature selection. This provides an initial practical direction for using automatically generated masks to select features for individual objects, which we discuss as future work in §5.

5. Contribution and reproducibility (UHUL, KDp5). We clarified our contribution as an object-level evaluation design combining exhaustive annotations, independent selection/testing, and analysis of the visual structure underlying feature splitting. We also expanded the reporting of training configurations, evaluation rules, and human-study procedures, including hyperparameters, sampling and blinding, inter-annotator agreement, and uncertainty estimates, making the protocol easier to reproduce and assess. (Appendices C, K, L; Figure 6.)

We believe these revisions substantially address the reviewers' concerns and strengthen PACO-SAE as a rigorous, reproducible resource for vision SAE evaluation and mechanistic interpretability. We thank the reviewers again for helping improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.