acceptodds
Under review as a conference paper at ICLR 2027

DisParQ: Self-Supervised Part Concepts for Interpretable Vision Foundation Models

Abstract

Concept-based vision models represent images through an intermediate layer of human-inspectable concepts, so what a model relies on can be traced to those concepts. However, those models are often limited to fixed categories or depend on language to define their concepts. We introduce DisParQ (Discrete Parts with Quantized attributes), a method that learns spatially grounded, discrete concept representations from a powerful frozen vision-only self-supervised backbone. It requires no class labels and no language supervision. Each image patch is assigned to exactly one concept from a learnable prototype dictionary and only a sparse subset of concepts may activate per image. To capture how each concept varies across images (e.g., the type of a “wheel”), we learn continuous residuals alongside the concepts, and then quantize them into discrete attributes. A spatial decoder reconstructs the backbone's representation from the concepts and attributes alone, so successful reconstruction means that the discrete representation preserves the backbone's information. We evaluate DisParQ across seven datasets, from general recognition (ImageNet, PartImageNet, Places) to fine-grained benchmarks (CUB, Cars, Dogs, Flowers). We show that DisParQ closely matches its frozen DINOv2 teacher on ImageNet linear probing (83.2% top-1), achieves higher concept consistency than language-aligned models, remains competitive on fine-grained recognition, and enables cross-category part-based retrieval.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, self-supervised learning, concept-based models, computer vision, vision foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1oxqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DisParQ, a framework that learns discrete, spatially grounded part concepts from a frozen vision foundation model without class labels, part annotations, or language supervision during concept learning. Each image selects a small subset of concepts from a shared dictionary, assigns each spatial location to one concept, and represents within-concept appearance variation using product-quantized residual attributes. A spatial decoder reconstructs the backbone's global representation from these discrete quantities. The evaluation covers seven datasets, several frozen backbones, classification, part quality, retrieval, and a human study.

The results are encouraging: DisParQ achieves 83.2% ImageNet linear-probe accuracy compared with 83.4% for its DINOv2 teacher, while improving several part-quality metrics over the evaluated self-supervised baselines. The appendix substantially strengthens the paper through controlled quantization experiments, decoder and objective ablations, attribute analyses, and detailed evaluation protocols.

Critical strengths
  1. Research significance and method design: The paper addresses the important challenge of retaining foundation-model performance while exposing spatially grounded concepts. Separating concept identity, spatial assignment, and appearance variation provides a clear representation to inspect. The decoder receives only the discrete description at inference, without a direct continuous-feature bypass.

  2. Strong representation retention:The small linear-probe gaps on ImageNet and Places, together with competitive fine-grained recognition results, demonstrate that the discrete representation preserves substantial task-relevant information. The part-quality and retrieval results provide evidence beyond classification accuracy.

  3. Extensive supporting experiments: Appendix E examines feature fusion, upsampling, concept budgets, prototype maintenance, attribute quantization, decoder depth, gradient routes, and training objectives. Many component comparisons report three-seed means and standard deviations. The equal-duration continuation experiments in Figure 16 provide a useful control for evaluating quantization.

  4. Evidence that the representation itself is useful: Under the continuous-stage ablation protocol, replacing the three-block decoder with mean pooling reduces linear-probe accuracy by only 1.26 percentage points (Figure 17). This weakens the concern that performance is primarily due to a powerful decoder. Appendices G.6–G.7 also provide evidence that quantized attributes preserve neighborhoods and meaningful variation within a concept.

  5. Human evaluation and transparent reporting: The user study uses hidden method identities, randomized presentation, and paired statistical tests. The appendix reports code length, codebook size, evaluation exceptions, and limitations of spatial locality and intervention scores. These disclosures help distinguish supported conclusions from stronger interpretations.

Critical weaknesses
  1. Classification comparisons are not fully controlled: Appendix D.5 reports differences in preprocessing and probe-time augmentation across methods. A unified evaluation protocol is needed to determine how much of the accuracy gain comes from the representation itself.

  2. The scope of interpretability needs clarification: Attribute visualizations show meaningful variation, but do not establish that attributes encode only the displayed part. Moreover, the explanations concern the DisParQ-based classifier; their fidelity to the original teacher remains unverified.

  3. Cross-category retrieval is not evaluated separately: The retrieval protocol excludes the query image but retains same-category candidates. Excluding these candidates would provide more direct evidence for cross-category part consistency.

  4. The benefit of private codebooks is unclear: Shared PQ performs nearly as well as private PQ in Figure 16. Further evidence is needed to justify the substantially larger private codebooks, particularly in terms of interpretability.

  5. Some conclusions need stronger statistical support: Main results use a single seed, making small improvements difficult to assess. The user study evaluates selected important concepts, so its conclusions should not be generalized to the entire dictionary.

Questions for authors
  1. Regarding classification comparability: How much of the linear-probe advantage remains under identical image preprocessing, probe-time augmentation, and optimization? Can a representative SAE baseline be evaluated with a comparable global distillation objective and readout?

  2. Regarding attribute semantics: Do attributes remain stable when content outside their assigned hard region is changed? Can swapping attributes between occurrences of the same concept produce predictable changes while preserving concept identity?

  3. Regarding explanation fidelity: Can the authors report teacher–student prediction agreement or evaluate the reconstructed representation using a fixed teacher classifier? Please clarify whether the intended explanation target is the original backbone or the DisParQ-based prediction system.

  4. Regarding cross-category retrieval: What happens when all same-category gallery images are excluded? Please report both R@1 and mAP, and distinguish identity-only retrieval from same-concept appearance ranking.

  5. Regarding private codebooks: Given the near-matching performance of shared PQ in Figure 16, what semantic or interpretability benefit justifies private codebooks? A direct comparison on these properties would strengthen the architectural motivation.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer 1oxq, Part 1/3

We thank the reviewer for their positive review and constructive feedback. We are glad that they find the problem "important", the results "encouraging", the supporting experiments "extensive", and the reporting "transparent". Below we answer the weaknesses and questions together where they overlap. Please also see our global comment (https://openreview.net/forum?id=40Arf70gfe&noteId=SlKp38nbyE) for the questions shared across the reviews.


Summary

  • W1/Q1 (classification fairness). DisParQ leads under a standardized probing protocol. With the same decoder and training objective, it scores 88.5% LP on PartImageNet versus 84.2% for PatchSAE and 75.4% for MSAE.
  • W2/Q2 (attribute scope). Noise outside a concept’s region preserves its identity in 91.5% of CUB images, though 36.4% of attribute components change. Swapping attributes preserves about 89% of predictions with same-class donors and 19% with other-class donors.
  • Q3 (explanation target). The explanations describe DisParQ’s classifier, which agrees with the backbone’s predictions on 91.0–93.7% of test images.
  • W3/Q4 (cross-category retrieval). With the query category excluded, concept identity alone leads in recall@1 on PartImageNet and CUB. Ranking by attribute similarity further improves mAP by 27.6 and 18.8 points, respectively.
  • W4/Q5 (codebooks). Private and shared codebooks yield similar accuracy, but private codebooks preserve the original attributes more faithfully.
  • W5 (seeds and user study). Gains exceed three-seed variation on CUB, Cars, and Dogs. The user study covers concepts sampled from the four most important for each prediction.

W1 and Q1: Classification under one protocol; an SAE with our readout.

DisParQ still leads under a standardized protocol. The original results follow the probing protocol of each baseline (where available). Every dictionary baseline re-probed under our protocol, with the mean of the raw DINOv2 patch tokens as a control (LP / kNN):

MethodPartImageNetCUBCarsDogsFlowers
TopK SAE83.9 / 77.743.0 / 26.249.1 / 27.068.3 / 51.099.6 / 99.5
PatchSAE86.2 / 80.972.8 / 40.084.5 / 44.781.9 / 63.599.6 / 99.5
MSAE86.4 / 81.879.1 / 48.087.5 / 51.782.8 / 65.699.7 / 99.6
CFM (reconstruction)72.1 / 56.047.5 / 16.471.9 / 36.663.1 / 41.094.6 / 85.0
PIP-Net stage 176.9 / 71.577.8 / 68.482.3 / 63.677.8 / 69.199.1 / 98.4
ProtoQuant stage 185.2 / 83.980.1 / 67.787.0 / 60.282.2 / 66.999.7 / 99.5
DINOv2 raw patch mean87.2 / 81.879.2 / 48.388.0 / 52.583.2 / 65.799.6 / 99.6
DisParQ, description without the decoder87.7 / 85.987.9 / 86.487.9 / 76.682.6 / 77.899.6 / 99.0
DisParQ89.2 / 88.789.4 / 85.591.0 / 83.487.8 / 85.899.7 / 99.7
DINOv2 CLS token (our target)89.3 / 88.990.4 / 86.793.2 / 85.688.6 / 86.899.7 / 99.7

DisParQ leads the best self-supervised baseline by 2.8 / 9.3 / 3.5 / 5.0 LP and 4.8 / 17.1 / 19.8 / 16.7 kNN on PartImageNet / CUB / Cars / Dogs. Without the decoder, its description still leads by 2.0 to 18.0 kNN on all four datasets and by 0.4 to 7.8 LP on three. A pooled reconstruction is bounded by the patch-mean control, whereas DisParQ decodes the CLS token. This is a strength of the method: the decoder's input is exactly the discrete description (concept identities, their regions and quantized attributes) that our method learns, with no continuous feature bypassing it.

An SAE with our readout and objective (Q1). We feed the concept maps of the frozen PatchSAE and MSAE, 16 concepts per image, to our decoder, train it with our CLS distillation loss and probe it identically. They reach 84.2 and 75.4 LP on PartImageNet, against 88.5 for DisParQ (all probed on precomputed features to keep the evaluation identical, hence not the 89.2 above). Removing the per-image budget moves them only to 85.2 and 76.9. The difference lies in the representation, not in the readout or the budget. Both tables are in App. F.3 and F.5.

AuthorsOpenReview ↗

Reply to Reviewer 1oxq, Part 2/3

W2, Q2 and Q3: Scope of the attributes; explanation target and fidelity.

Target (Q3): the DisParQ classifier, now compared with the backbone's. The explanations describe a linear classifier on the decoded CLS token; Sec. 4.3 now says so. It agrees with the same classifier on the backbone's own CLS token on 92.8 / 93.7 / 93.2 / 91.0 % of test images (PartImageNet / CUB / Cars / Dogs), and on 96 to 97 % of the images the backbone classifies correctly (App. F.18).

Stability (Q2a): the concept is stable, its attribute only partly. We run the spatial misalignment benchmark [1] on CUB: noise is added outside the box of the concept most important for the prediction. PLC (Prototypical part Location Change) is the change in that concept's location and PAC (Prototypical part Activation Change) the drop in its peak activation (both in %):

MethodPLC ↓PAC ↓clean accuracyperturbed accuracy
CFM30.573.9955.1650.09
TopK SAE18.891.1847.7446.91
PIP-Net stage 115.632.8177.3074.39
ProtoQuant stage 114.164.9162.7261.60
DisParQ, without the decoder12.960.0788.8787.78
DisParQ8.850.0790.0188.82

DisParQ's concept moves least and keeps its activation, and DisParQ stays the most accurate (all methods in Table 26, App. F.9). Its attribute is less stable (Table 27):

outcome for that concept, share of imagesDisParQ
concept retained91.5 %
retained, attribute identical2.1 %
retained, attribute of the same class63.2 %
components of the attribute changed, of 1636.4 %

About a third of the attribute's 16 components change, and it mostly stays in the same class. As App. B states, the backbone's features are contextual, so an attribute is not causally confined to its pixels.


W2 and Q2, continued: attribute swap.

Swap (Q2b): the decoder reads the attributes, and they carry class-specific appearance. We keep the concepts and their assignment and replace the attributes by those of the same concepts in another image:

predictions retained, all attributes replacedsame-class donorother-class donorrandom attributes
PartImageNet89.6 %18.9 %53.7 %
CUB89.2 %19.0 %42.5 %

Attributes from a same-class image keep the prediction in about 89 % of cases. Attributes from another class keep it in only 19 %, less than random attributes (54 % and 43 %), so the decoder reads class-specific appearance from them. Concept identity and the spatial map are unchanged by construction (App. F.16).


W3 and Q4: Cross-category retrieval.

Added: DisParQ leads the self-supervised baselines in cross-category retrieval. Thank you for the suggestion. The retrieval gallery now excludes every image of the query's category: the ImageNet class on PartImageNet, the species on CUB.

category excludedPartImageNet R@1PartImageNet mAPCUB R@1CUB mAP
CFM30.115.48.97.4
TopK SAE (strongest SAE baseline)49.516.315.69.3
ProtoQuant stage 1 (best self-supervised baseline on R@1)69.620.630.38.1
DisParQ, concept identity only76.329.136.49.3
DisParQ, identity + attribute ranking79.656.740.028.1

On PartImageNet, concept identity alone leads every self-supervised baseline by at least 6.7 R@1 and 3.6 mAP (PIP-Net stage 1, not shown, has the best baseline mAP at 25.5); on CUB it leads by 6.1 R@1 at similar mAP. The last row ranks candidates of the same concept by attribute similarity, which separates identity from appearance, and adds 3.3 R@1 and 27.6 mAP on PartImageNet (3.6 and 18.8 on CUB). The full tables are in App. F.14.

AuthorsOpenReview ↗

Reply to Reviewer 1oxq, Part 3/3

W4 and Q5: Private vs shared codebooks.

Accuracy and part metrics are similar; private attributes are more faithful and slightly more stable. A model with one shared codebook, trained from the same stage-1 model with the same budget, is within 0.4 LP of the private one on all five datasets and has identical part metrics, as in Fig. 17. The attributes themselves differ (App. F.16):

attribute propertyPartImageNet privatePartImageNet sharedCUB privateCUB shared
fidelity: cosine of the quantized to the continuous attribute0.430.290.460.25
components kept under flip / crop / color jitter, %72.870.167.366.2

A private attribute stays closer to the continuous attribute it quantizes (0.43 to 0.46 against 0.25 to 0.29) and keeps one to three percentage points (p.p.) more of its 16 components when the image is flipped, cropped or color-jittered. A private attribute is also scoped to its concept ("variant j of part c"), and therefore can be analysed and pruned per concept without affecting other concepts.


W5: Seeds, and the scope of the user study.

Seeds. The revision reports means and standard deviations over three training seeds on the four fine-grained datasets, for DisParQ and five baselines (Table 3; part-to-part retrieval on CUB in Table 31). On CUB, DisParQ reaches 89.54 ± 0.16 LP, 85.34 ± 0.20 kNN, 54.50 ± 1.06 PIP purity and 37.49 ± 0.61 part-to-part R@1; its margins over the best self-supervised baseline (+9.1, +16.6, +17.5 and +6.3) are far larger than the standard deviations, as on Cars and Dogs. Three seeds for PartImageNet will be in the final version.

User study. Agreed. Prototypes were sampled among the four most important for an image's prediction (for all methods), so the study speaks about the concepts a user sees in an explanation, not the whole dictionary. Sec. 4.3 now states this. Within this scope, both prefer DisParQ: participants rate its concepts as significantly more consistent than those of PIP-Net and CFM (p < 0.001 for both, Fig. 5 right) and match them to the correct region most often (69 % against 60 % and 52 %; App. F.19).

We hope our response adequately addresses the reviewer's concerns and provides clarifications to their questions.

References [1] Sacha, Jura, Rymarczyk, Struski, Tabor and Zieliński, Interpretability Benchmark for Evaluating Spatial Misalignment of Prototypical Parts Explanations, In AAAI, 2024.

Reviewer_cSD3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DisParQ, a self-supervised method that learns discrete, spatially grounded visual concepts from frozen DINOv2 features. It assigns image regions to learned concepts and uses quantized attributes to capture differences in their appearance. The authors report improved part consistency and retrieval over several baselines while preserving nearly all of DINOv2's classification accuracy.

Critical strengths
  • The paper offers an interesting way of learning discrete concepts that correspond to image regions, without needing labels.
  • DisParQ retains nearly all of DINOv2's classification accuracy despite introducing a concept bottleneck.
  • The authors evaluate the method across several datasets and provide a thorough set of ablations.
  • The human study offers some support for the claim that the discovered concepts are meaningful and consistent.
Critical weaknesses
  • It's not really clear what DisParQ adds beyond the semantic concepts that DINOv2 already learns.

  • The paper lacks a straightforward baseline that clusters DINOv2's patch features, which would help establish whether the proposed architecture is necessary. For instance, DINO's original paper already shows strong conceptual features.

  • The classification results are difficult to compare fairly, since DisParQ learns to reproduce DINOv2's CLS token, whereas the baselines use different representations and training objectives. Although I'm not too concerned about this.

  • There is little evidence that the concepts are actually more useful in practice, for example through better generalization, improved downstream performance, or more reliable interventions.

Questions for authors

I am a bit confused with regards to the actual contribution of the paper. While I understand that DisparQ is supposed to learn better concepts, the method itself does not outperform DINOv2's CLS token representation. Also the authors don't really show any downstream task where the newly learned concepts might be useful. I think in this sense Figure 5 is quite valuable and I commend the authors for human evaluation, but it still does not prove that these concept representations are superior to DINO's for instance.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer cSD3, Part 1/2

We thank the reviewer for finding our work "interesting", the ablations "thorough", and the human study "quite valuable". We address the concerns and the question below. Please also see our global comment (https://openreview.net/forum?id=40Arf70gfe&noteId=SlKp38nbyE) for the questions shared across the reviews.


Summary

  • W1/Q1 (what DisParQ adds). On a frozen self-supervised backbone it learns part concepts, which are: spatially grounded, exclusive per patch, with attributes for how each concept varies, and without class labels or language supervision. It keeps the backbone's accuracy within 0.0-2.2 LP across five datasets, which neither clustering nor the best self-supervised baseline does (PartImageNet: k-means 76.8 and MSAE 86.4 LP, against 89.2).
  • W2 (clustering baseline). At the same dictionary size and concept budget, DisParQ beats k-means on every reported PartImageNet metric. Larger k-means dictionaries surpass it only on purity and NMI.
  • W3 (classification fairness). Under a standardized protocol, DisParQ leads the best self-supervised baseline by 2.8-9.3 linear-probe points. Without the decoder, its discrete description still leads in kNN accuracy on all four datasets.
  • W4/Q1 (practical usefulness). A prediction becomes a short list of parts that are more human interpretable (in the user study, rated most consistent and matched in 69 % of cases, against 60 % and 52 % for the baselines) and that can be edited. For example, removing the background concepts improves worst-group accuracy on Waterbirds by 3.9 p.p. (App. F.18).

W1 and Q1: What DisParQ adds beyond the concepts DINOv2 already learns.

A shared global discrete part-concept vocabulary, which raw features do not have. DINOv2's patch features are continuous and image-specific: nothing in them says that a region in one image is the same part as a region in another. DisParQ adds (learns) 1,024 concepts, without labels, whose identities persist across images. Part-to-part retrieval (Table 4), explanations as a short list of parts (Sec. 4.3) and the user study (Fig. 5) rely on this. Methods that group the same DINOv2 tokens without it (three sparse autoencoders, PCA and now k-means) trail DisParQ; against the best sparse autoencoder it scores 81.5 vs 76.2 foreground purity, 69.8 vs 60.1 NMI, 62.3 vs 36.0 ARI and 35.9 vs 25.0 mIoU. We do not claim to beat the CLS token, which is the ceiling by construction; the claim is that the description keeps its accuracy (within 0.0 to 2.2 LP) across five datasets while exposing the parts. The introduction now says this explicitly.


W2: A clustering baseline on DINOv2's patch features.

Agreed and added. Thank you for the suggestion. k-means with K = 1,024 (our dictionary size) on the DINOv2 patch tokens, with the same budget of 16 concepts per image, the same evaluation code, and LP / kNN probed as for the other dictionary baselines:

PartImageNetLPkNNPur_fgNMI_mARI_mmIoUp2p R@1LocCons
PCA, class-wise (uses class labels)n/an/a57.953.434.414.252.422.94.3
k-means, K = 1,02476.865.476.967.254.531.971.630.39.9
DisParQ89.288.781.569.862.335.979.736.416.3

At the matched size DisParQ leads on every column. Across K from 16 to 8,192 (App. F.8), k-means stays below DisParQ in accuracy at every K; from K = 2,048 it passes DisParQ on purity and NMI, which favor large dictionaries, and stays behind on ARI, mIoU, locality, consistency and retrieval. On CUB, a fine-grained dataset, DisParQ's lead over k-means is wider: 89.4 vs 40.0 LP and 47.5 vs 24.8 foreground purity. The K = 1,024 row is now in Tables 2 to 4.


W3: Fair comparison of classification results.

Reconstructing a self-supervised CLS token is a strength of the method. The target needs no labels, and the decoder reads only the discrete description. Under one protocol DisParQ leads the best self-supervised baseline by 2.8 to 9.3 LP (see Table in global comment); even without the decoder, probed on the mean of its discrete patch tokens, it still leads by 2.0 to 18.0 kNN on all four datasets and by 0.4 to 7.8 LP on three. The rest of the gap is the target: a pooled reconstruction is bounded by the backbone's patch mean (87.2 on PartImageNet), while DisParQ decodes the CLS token (89.3). A sparse autoencoder given our decoder and objective does not close it (84.2 LP for PatchSAE against 88.5 under the same evaluation protocol; App. F.5).

AuthorsOpenReview ↗

Reply to Reviewer cSD3, Part 2/2

W4 and Q1: Evidence that the concepts are useful in practice.

We provide three new measurements, in addition to the two from the paper.

(1) Test-time intervention. Waterbirds [1] is a standard benchmark for spurious correlations: CUB birds are pasted onto land or water backgrounds, and in training 95 % of the birds sit on their typical background, so a classifier can learn the background in place of the bird. It is scored on the worst of the four bird / background groups. We apply the CUB model unchanged and train one linear head on its decoded CLS token, without group labels. At test time we delete the background concepts, identified with CUB's masks, and decode again:

Waterbirdsworst groupmean
frozen DINOv2 CLS token (no concepts, nothing to edit)89.995.0
DisParQ decoded CLS, plain92.796.2
DisParQ, background concepts deleted at test time, same head96.6 (+3.9)98.5 (+2.3)

A linear head trained on the frozen DINOv2 CLS token reaches 89.9 worst-group accuracy [2] and offers nothing to delete. DisParQ's decoded CLS token (from the discrete description) reaches 92.7 and is a 2.8 percentage points (p.p.) improvement already.

Dropping the background concepts, with the same head and no retraining, improves worst-group accuracy by 3.9 p.p. and the mean by 2.3 p.p. The discrete description that DisParQ learns, makes this edit direct: a concept is either part of it or not.

(2) Zero-shot segmentation. DisParQ averages 40.9 mIoU over the eight benchmark settings of CFM, against 40.0 for CFM at the same concept budget (reply to E3Wy W3).

Following CFM's protocol, one DisParQ model is trained on CC12M with the CLIP-DINOiser backbone. Its decoder predicts the backbone's feature at every patch from the discrete description, and each patch takes the class whose text embedding is closest. No segmentation head is trained. Both methods use 16 concepts per window (mIoU):

16 concepts / windowVOC20Context-59COCO-StuffCityscapesADE20KContext-60COCO ObjectVOC21Avg.
CFM80.636.622.535.418.733.730.362.440.0
DisParQ81.536.523.537.218.733.432.763.540.9

DisParQ is best or tied on six of the eight settings; CFM leads on the two PASCAL Context settings by a small margin. The results are in Sec. 4.2, Fig. 6 and App. H.

(3) Deletion. We delete concepts one after another and decode again, in three orders (PartImageNet, share of predictions changed, %):

concepts deleted148
most important first (the explanation's order)7.215.523.2
random order1.23.66.7
least important first0.20.20.2

The concepts and explanations rank highest change three to six times more predictions than random ones, and the least important change almost none. CUB, Cars and Dogs behave the same (see Table 42 and Fig. 25, App. F.18 for additional results and concept deletion plots).

(4) In the paper. Part-to-part retrieval across categories (R@1 79.7), and a user study with 120 participants and two questions (Fig. 5, App. F.19). Part consistency: participants rate DisParQ's concepts as significantly more consistent than those of PIP-Net and CFM (mean rating 3.23 against 3.07 and 2.87 on a scale of 1 to 5; p < 0.001 for both). Part retrieval: they predict the model's localization from DisParQ's concepts in 69 % of cases, against 60 % for PIP-Net and 52 % for CFM (chance 25 %).

We agree the user study does not show superiority over DINOv2's representation and do not claim it; it shows that the parts are consistent and recognizable to people, which is the focus of our work. We have clarified this in the paper (Sec. 4.3 and App. F19).

We hope our response adequately addresses the reviewer's concerns and provides clarifications to their questions.

References [1] Sagawa, Koh, Hashimoto and Liang, Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization, In ICLR, 2020. [2] Darbinyan, Harutyunyan, Markosyan and Khachatrian, Identifying and Disentangling Spurious Features in Pretrained Image Representations, arXiv:2306.12673, 2023.

While I appreciate the authors' substantial efforts during the rebuttal, I remain unconvinced by the evidence for the broader utility of the learned representations. The newly introduced downstream experiments are encouraging, but they focus on rather specific settings, and particularly the Waterbirds intervention lacks comparisons with alternative methods that could achieve similar outcomes. Although the additional segmentation results provide a more direct comparison, the improvements are relatively modest. My central concern therefore remains: it is unclear whether DisParQ provides a generally useful improvement over existing representations, or whether its advantages are limited to the particular experimental settings considered.

Overall, I think there is potential here, but as I mentioned in my original review the actual showcase needs to be more focused and centralised in my opinion. I think the authors have done good strides in the rebuttal, I would suggest to iterate through this paper and resubmit. I also understand that time for the rebuttal is limited, but I think coming up with a coherent concept for evaluation is a bit too much for a rebuttal.

In any case, I'm also not opposed to this paper being accepted, I think the methodology is strong but let down by its current evaluation.

Reviewer_E3WyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a method for learning concept-based interpretable image representation using frozen features from vision foundation models (VFMs). The paper observes that current models give up one of the 4 desirable properties of such representations: (1) spatial grounding, (2) one concept per location, (3) attributes describing how a concept varies across images and (4) no supervision. It then proposes DisParQ, a method that learn part concepts with all four properties.

DisParQ consists of an encoder that maps upsampled patch features from VFMs to a small set of learnable prototypes, learned concept-specific codebooks that quantises concept residuals into discrete attributes, and a concept decoder that predicts the image CLS token given by the VFMs from the its concept-based representation.

DisParQ is evaluated on image classification and part concepts tasks. The paper also proposes a new part-to-part retrieval benchmarks that evaluates part consistency across images and categories. Finally, a user study completes the paper’s experimental validation. Overall, the paper shows good results in all benchmarks compared to baselines.

Critical strengths

The paper tackles interpretable image representation, an interesting problem that has attracted a lot of research efforts. It motivates the method well by pointing out limitations of current works (lacking one of the 4 properties). The proposed method which consists of mostly existing components is reasonable and shown to satisfy all 4 properties. The proposed learned attributes are interesting.

The paper provides substantial experimental validation of DisParQ, showing strong results compared to baselines such as CFM or PIP-Net. The interpretability of the learned representation is shown both qualitatively, and quantitatively through the new part-to-part retrieval benchmark. The user study presented in the paper, which is quite rare to see, is also interesting.

The paper is in general well written and easy to follow.

Critical weaknesses

A major weakness of DisParQ is the evaluation protocol in classification. This protocol is unfair to competitors since the DisParQ is trained to recover the CLS token produced by VFMs and use the decoded CLS in linear probing and Knn classification. As such, DisParQ has a clear advantage compared to other methods which are not trained to decoder the VFM’s CLS token. This evaluation protocol also invalidates the paper’s claim that “the discrete description costs almost no accuracy” because the concept-based representation is not used in these evaluation. How DisParQ fares against baselines when the concept-based representation is used?

There is no direct evaluation of the learned attributes which is a major advertised contribution. In the paper, attributes are only used to decode the CLS.

The paper lacks semantic segmentation benchmarks that are considered in other works such as CFM. For example, the paper can decode the concept-based representation of each patch to a semantic class with a linear classifier. Adding results on these benchmarks would give strong signals on the quality of the concept-based representation.

The comparison to CFM is uncontrolled because CFM uses another backbone and its dictionary was trained on a different dataset.

Questions for authors

The paper trains one model per dataset. Since the training method is unsupervised, the model could be trained on the combination of several datasets or a very large in-the-wild dataset. It would be interesting to observe how results would change in this case compared to the current one-model-per-dataset setting.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer E3Wy, Part 1/2

We thank the reviewer for recognizing that the method is well motivated and satisfies all four properties, that "the proposed learned attributes are interesting", that the paper "provides substantial experimental validation", and that its user study is "quite rare to see". We address the concerns and the question below. Please also see our global comment (https://openreview.net/forum?id=40Arf70gfe&noteId=SlKp38nbyE) for the questions shared across the reviews.


Summary

  • W1 (classification fairness and concept use). Under the same probing protocol, DisParQ leads by 2.8–9.3 LP points. Its discrete description is the decoder’s only input and still leads in kNN accuracy on all four datasets when probed directly.
  • W2 (attributes). Swapping attributes while keeping concept identities fixed preserves about 89% of predictions with same-class donors and 19% with other-class donors, showing that attributes encode class-specific appearance.
  • W3 (segmentation). At matched evaluation, DisParQ averages 40.9 mIoU across eight zero-shot benchmarks versus CFM’s 40.0.
  • W4 (CFM comparison). On CFM’s backbone, DisParQ leads CFM and MSAE on 9 of 11 metrics.
  • Q1 (transfer). With only a new linear head, the ImageNet-trained model comes within 2.5 accuracy points of per-dataset training on four datasets (without any hyperparameter tuning).

W1: The classification protocol and the use of the concept-based representation.

Is the protocol unfair? DisParQ still leads under a standardized protocol. The original results follow the probing protocol of each baseline (where available). With every method under the same protocol (see Table in global comment), DisParQ leads the best self-supervised baseline by 2.8 / 9.3 / 3.5 / 5.0 LP on PartImageNet / CUB / Cars / Dogs. Probed without the decoder, on the mean of its discrete patch tokens (identities + attributes, table below), it still leads by 2.0 to 18.0 kNN on all four datasets and by 0.4 to 7.8 LP on three. The remaining difference is the target: a pooled reconstruction is bounded by the backbone's patch mean (87.2 on PartImageNet), while DisParQ decodes the CLS token (89.3). Thus, even at the patch level, DisParQ's discrete description costs no accuracy against the backbone: it is at most 0.6 LP below the raw DINOv2 patch mean (8.7 above on CUB) and 4.1 to 38.1 kNN above it. Finally, giving the dictionaries of PatchSAE and MSAE our decoder and objective does not close the gap: 84.2 and 75.4 LP against 88.5 for DisParQ under the same evaluation protocol (App. F.5).

Note: All part-concept and part-retrieval results remain unchanged: they were already computed on the concept representations, under a standardized protocol across every method. Flowers is saturated, with most methods close to 100 %, and its results barely change (table in the reply to Reviewer 1oxq).

Is the concept-based representation used? Yes: it is the decoder's only input (Sec. 3.4), and we now probe it directly. Without the decoder (LP / kNN):

PartImageNetCUBCarsDogs
concept identities only69.4 / 66.866.2 / 57.158.5 / 41.757.8 / 45.5
attributes only87.1 / 88.687.3 / 86.587.6 / 81.680.9 / 79.0
identities + attributes87.7 / 85.987.9 / 86.487.9 / 76.682.6 / 77.8
decoded CLS token (DisParQ)89.2 / 88.789.4 / 85.591.0 / 83.487.8 / 85.8
DINOv2 CLS token (our target)89.3 / 88.990.4 / 86.793.2 / 85.688.6 / 86.8

The attributes carry most of the class information; concept identities alone are a weak global descriptor, as expected of a category-agnostic part vocabulary. The decoder adds 1.5 to 5.2 LP by reading concepts, attributes and layout jointly. Both tables are in App. F.3 and F.4.

AuthorsOpenReview ↗

Reply to Reviewer E3Wy, Part 2/2

W2: Direct evaluation of the attributes.

The paper evaluated them in two ways, and the revision adds three more. (i) Ablation: removing the attributes costs 12.2 LP (Fig. 17). (ii) Retrieval: ranking same-concept regions by attribute lifts part-to-part mAP from 30.5 to 57.3 on PartImageNet (Table 31). (iii) Structure: among the 10 nearest occurrences of a concept in attribute space, 31.9 % share the image class against 11.2 % by chance (PartImageNet; similar on CUB, Cars and Dogs; App. F.15). (iv) Replacement: we keep the concepts and their assignment, replace the attributes and decode again.

predictions retained, all attributes replacedsame-class donorother-class donorrandom attributes
PartImageNet89.6 %18.9 %53.7 %
CUB89.2 %19.0 %42.5 %

The decoder reads the attributes, and they carry class-specific appearance: attributes from a same-class image are interchangeable, attributes from another class change the prediction (App. F.16). (v) Stability: with noise added outside the box of an image's most important concept (CUB), the concept is retained on 91.5 % of images, 36.4 % of its attribute's 16 components change, and on 63.2 % of images the attribute stays in the same class (App. F.9).

Together, these tests show that the attributes carry the class-specific appearance of each part: the prediction depends on them (i, iv), they order the occurrences of a concept by appearance (ii, iii), and under changes elsewhere in the image the concept is retained and its attribute mostly keeps its class (v).


W3: Semantic segmentation benchmarks.

Agreed, added: the zero-shot segmentation benchmarks of CFM. Following CFM's protocol, one DisParQ model is trained on CC12M with the CLIP-DINOiser backbone. Its decoder predicts the backbone's feature at every patch from the discrete description, and each patch takes the class whose text embedding is closest. No segmentation head is trained. Both methods use 16 concepts per window (mIoU):

16 concepts / windowVOC20Context-59COCO-StuffCityscapesADE20KContext-60COCO ObjectVOC21Avg.
CFM80.636.622.535.418.733.730.362.440.0
DisParQ81.536.523.537.218.733.432.763.540.9

DisParQ is best or tied on six of the eight settings; CFM leads on the two PASCAL Context settings by a small margin. The results are in Sec. 4.2, Fig. 6 and App. H.


W4: The comparison to CFM.

Controlled comparison on CFM's backbone. CFM releases a single dictionary, on CLIP-DINOiser and fitted on CC12M. A refit would no longer be CFM and needs the CC12M feature dump (about 4.5 TB) and 119k optimization steps. We keep it as released, reproduce its published results under its own evaluator (App. D.6), and control both differences. DisParQ is trained on CFM's backbone. MSAE, a Matryoshka sparse autoencoder like CFM and of the same size, is fitted on PartImageNet with that backbone, as the analogue of a refitted CFM:

CLIP-DINOiser, 16 concepts / imageLPkNNPur_fgNMI_mARI_mmIoUFG-ARILocConsImp ↓p2p R@1
CFM (released, CC12M)72.369.154.751.424.522.712.522.912.40.37831.6
MSAE (PartImageNet)74.257.859.152.130.123.414.317.49.20.50433.5
DisParQ (PartImageNet)79.875.760.962.245.324.713.728.713.30.39341.7

DisParQ leads on 9 of the 11 metrics; MSAE is ahead on FG-ARI and CFM on impurity. The table is in App. F.12.


Q1: One model per dataset vs joint or in-the-wild training.

One configuration everywhere, and the ImageNet-1k model is the in-the-wild case. We transfer our ImageNet-trained model (on the 1.28M training images) without retraining it; only a linear head is fit per dataset (App. F.1). It reaches 89.0 LP on PartImageNet (a subset of ImageNet) against 89.2 for the model trained on it, and 87.1 / 88.5 / 99.6 on CUB / Cars / Flowers, within 2.5 percentage points of per-dataset training. The parts transfer less well in this case than the accuracy: foreground purity is 69.0 against 81.5 on PartImageNet. These initial results are promising, but they use our default hyperparameters, which may need adapting to larger datasets such as ImageNet-1k; as training at this scale is expensive, we will update them in the final version.

We hope our response adequately addresses the reviewer's concerns and provides clarifications to their questions.

AuthorsOpenReview ↗

Follow up to Reviewer E3Wy

Dear Reviewer E3Wy,

Thank you again for your constructive feedback. As the author–reviewer discussion period is nearing its end, we wanted to kindly check whether any concerns remain that we could address. If our response has resolved the points you raised, we would be grateful if you would consider whether your assessment has changed.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Follow-up Reminder to Reviewer E3Wy: Discussion Period Closing within 24 Hours

Dear Reviewer E3Wy,

With less than 24 hours remaining in the discussion period, we kindly ask if you could take a moment to read our rebuttal and take a look at the revised manuscript.

We believe that we have answered all of your questions and addressed all of your concerns (standardized classification protocol, use of the concept-based representation, direct evaluation of the attributes, semantic segmentation benchmarks, comparison to CFM, one model per dataset vs in-the-wild training on a large dataset).

Could you please let us know if you are satisfied with our responses or if any issue remains unresolved. If we have successfully resolved your questions, we would be grateful if you would consider reflecting this in your final evaluation.

We remain available to clarify any remaining questions before the deadline. Thank you for your time and consideration.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global response to common concerns [summary]

We thank all reviewers for their detailed and constructive feedback. We appreciate them finding the problem important and the method well motivated, the evaluation extensive, and the user study valuable. Three questions are shared across the reviews, we answer them here and give the details in the individual replies.

Summary.

  • Classification: DisParQ still leads under a standardized protocol, by 2.8 to 9.3 LP and 4.8 to 19.8 kNN over the best self-supervised baseline on four datasets; its description leads on kNN even without the decoder.
  • Added value: a shared vocabulary of global discrete part concepts that persist across images and categories. It is useful in practice: deleting background concepts raises worst-group accuracy on Waterbirds by 3.9 percentage points, and zero-shot segmentation reaches 40.9 mIoU against 40.0 for CFM.
    This is in addition to DisParQ achieving highest scores on part to part retrieval and part concept consistency metrics compared to prior work.
  • Attributes: the prediction depends on them. Removing them costs 12.2 LP; attributes from a same-class image keep about 89 % of the predictions, attributes from another class only 19 %.

All additional results fully corroborate our initial findings and show the superiority of DisParQ on classification accuracy, part-concept metrics and zero-shot segmentation.

AuthorsOpenReview ↗

Global response to common concerns [details]

1. Is the classification comparison fair? (E3Wy W1, cSD3 W3, 1oxq W1 / Q1.) The original results follow the probing protocol of each baseline (where available). We re-probed every dictionary baseline under exactly our protocol (App. D.5) and added a control, the mean of the raw DINOv2 patch tokens (LP / kNN):

MethodPartImageNetCUBCarsDogs
best dictionary baseline (MSAE)86.4 / 81.879.1 / 48.087.5 / 51.782.8 / 65.6
prototype baseline (PIP-Net stage 1)76.9 / 71.577.8 / 68.482.3 / 63.677.8 / 69.1
best prototype baseline (ProtoQuant stage 1)85.2 / 83.980.1 / 67.787.0 / 60.282.2 / 66.9
DINOv2 raw patch mean87.2 / 81.879.2 / 48.388.0 / 52.583.2 / 65.7
DisParQ, description without the decoder87.7 / 85.987.9 / 86.487.9 / 76.682.6 / 77.8
DisParQ89.2 / 88.789.4 / 85.591.0 / 83.487.8 / 85.8
DINOv2 CLS token (our target)89.3 / 88.990.4 / 86.793.2 / 85.688.6 / 86.8

DisParQ still leads under a standardized protocol: by 2.8 to 9.3 LP and 4.8 to 19.8 kNN over the best self-supervised baseline. Without the decoder, its description still leads by 2.0 to 18.0 kNN on all four datasets and by 0.4 to 7.8 LP on three, and exceeds the raw DINOv2 patch mean by 4.1 to 38.1 kNN. A pooled reconstruction is bounded by the patch-mean control, whereas DisParQ decodes the CLS token from its discrete description, the decoder's only input. An SAE given our decoder and objective does not close the gap (reply to 1oxq W1).

Note: All part-concept and part-retrieval results remain unchanged: they were already computed on the concept representations, under a standardized protocol across every method. Flowers is saturated, with most methods close to 100 %, and its results barely change (table in the reply to 1oxq).

2. What does DisParQ add over the backbone, and is it useful? (cSD3 W1 / W4 / Q1, E3Wy W3, 1oxq W2.) The backbone's patch features are continuous and image-specific. DisParQ learns a shared global vocabulary of 1,024 discrete concepts, each with a quantized attribute, whose identities persist across images and categories. New evidence in the revision: (a) a k-means baseline (cSD3 W2), (b) concept deletion and a test-time intervention on Waterbirds (cSD3 W4), (c) agreement of the explained classifier with the backbone's (1oxq Q3), (d) zero-shot semantic segmentation on the benchmarks of CFM (40.9 vs 40.0 mIoU at the same concept budget; E3Wy W3), and (e) transfer of an ImageNet-trained model (E3Wy Q1).

3. Are the attributes evaluated? (E3Wy W2, 1oxq W2 / Q2 / W4 / Q5.) The paper already presented the ablation (removing the attributes costs 12.2 LP, Fig. 17) and the retrieval descriptors (Table 31). The revision adds (a) an attribute-replacement test, (b) a stability test under changes outside a concept's region, and (c) a comparison of private and shared codebooks (replies to E3Wy W2 and 1oxq W2 / W4). Together, they show that the attributes carry the class-specific appearance of each part, and that the prediction depends on them: attributes from a same-class image keep about 89 % of the predictions, attributes from another class only 19 %.

Revised PDF. All changes are in blue. All section, table and figure numbers in our responses refer to the revised PDF.

  • Classification: every row under one probe protocol, the description probed without the decoder, and SAEs given our decoder (App. F.3 to F.5); the measured cost of the bottleneck (Sec. 4.2).
  • Baselines: k-means in Tables 2 to 4 and swept over K (App. F.8); CFM compared on its own backbone (App. F.12).
  • Attributes: replacement and private vs shared codebooks (App. F.16); stability under changes outside the region (App. F.9).
  • Usefulness: deletion, Waterbirds and agreement with the backbone's classifier (App. F.18); zero-shot segmentation (Sec. 4.2, Fig. 6 and App. H); ImageNet transfer (App. F.1).
  • Retrieval and seeds: part retrieval with the query's category excluded (App. F.14); three-seed means and standard deviations on the four fine-grained datasets (Table 3).
  • Text: what DisParQ adds over the backbone's features (Sec. 1); the explanation target and the scope of the user study (Sec. 4.3).
AuthorsOpenReview ↗

AC Summary

Dear Area Chair,

We thank you and the reviewers for their time and constructive feedback leading to better shape of our work. We briefly summarize the discussion of Submission 8140.


Reviewer positions.


Concerns raised and how they were addressed (all in the revised PDF, changes in blue).

  1. Fairness of the classification protocol (all three reviewers). Every baseline was re-probed under one protocol: DisParQ leads the best self-supervised baseline by 2.8–9.3 LP and 4.8–19.8 kNN on four datasets, and still leads in kNN on all four when its discrete description is probed without the decoder. Sparse autoencoders given our decoder and objective do not close the gap (84.2 / 75.4 vs 88.5 LP). DisParQ stays within 0.0–2.2 LP of its DINOv2 teacher on five datasets and 0.2 on ImageNet-1k (App. F.3–F.5).
  2. What DisParQ adds over the backbone, and its usefulness (cSD3). Added a k-means baseline on DINOv2 patch tokens: DisParQ leads it on every metric at the same dictionary size (App. F.8). Added a Waterbirds intervention compared against the backbone, prototype models, k-means and sparse autoencoders under one protocol: DisParQ has the best mean and worst-group accuracy before and after deleting background concepts (worst group 96.6), and even without any test-time mask (96.0) it exceeds the backbone given the mask (95.5) (App. F.18). Sec. 4 is now organised around one claim and three questions (accuracy, part consistency, usability).
  3. Direct evaluation of the attributes (E3Wy, 1oxq). Replacing attributes with those of a same-class image keeps ~89% of predictions, of another class only ~19%; stability under changes outside a concept's region and private vs shared codebooks are also reported (App. F.9, F.16).
  4. Semantic segmentation and a controlled CFM comparison (E3Wy). On CFM's eight zero-shot benchmarks at a matched concept budget, DisParQ reaches 40.9 vs 40.0 mIoU (best or tied on six); on CFM's own backbone, DisParQ leads CFM and a matched MSAE on 9 of 11 metrics (App. H, F.12).
  5. Cross-category retrieval, explanation fidelity and seeds (1oxq). With the query's whole category excluded, DisParQ leads part-to-part retrieval (R@1 79.6 vs 69.6); the explained classifier agrees with the backbone's on 91–94% of images; fine-grained results are reported over three seeds, with margins far above the standard deviation (App. F.14, F.18, Table 3).

To the best of our knowledge, DisParQ is the first method to learn part concepts that are spatially grounded, exclusive per patch, with discrete attributes, and fully self-supervised on a frozen backbone, while retaining the backbone's accuracy.

We hope this summary helps in reaching a decision.

Yours faithfully,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.