- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DisParQ, a framework that learns discrete, spatially grounded part concepts from a frozen vision foundation model without class labels, part annotations, or language supervision during concept learning. Each image selects a small subset of concepts from a shared dictionary, assigns each spatial location to one concept, and represents within-concept appearance variation using product-quantized residual attributes. A spatial decoder reconstructs the backbone's global representation from these discrete quantities. The evaluation covers seven datasets, several frozen backbones, classification, part quality, retrieval, and a human study.
The results are encouraging: DisParQ achieves 83.2% ImageNet linear-probe accuracy compared with 83.4% for its DINOv2 teacher, while improving several part-quality metrics over the evaluated self-supervised baselines. The appendix substantially strengthens the paper through controlled quantization experiments, decoder and objective ablations, attribute analyses, and detailed evaluation protocols.
Critical strengths
-
Research significance and method design: The paper addresses the important challenge of retaining foundation-model performance while exposing spatially grounded concepts. Separating concept identity, spatial assignment, and appearance variation provides a clear representation to inspect. The decoder receives only the discrete description at inference, without a direct continuous-feature bypass.
-
Strong representation retention:The small linear-probe gaps on ImageNet and Places, together with competitive fine-grained recognition results, demonstrate that the discrete representation preserves substantial task-relevant information. The part-quality and retrieval results provide evidence beyond classification accuracy.
-
Extensive supporting experiments: Appendix E examines feature fusion, upsampling, concept budgets, prototype maintenance, attribute quantization, decoder depth, gradient routes, and training objectives. Many component comparisons report three-seed means and standard deviations. The equal-duration continuation experiments in Figure 16 provide a useful control for evaluating quantization.
-
Evidence that the representation itself is useful: Under the continuous-stage ablation protocol, replacing the three-block decoder with mean pooling reduces linear-probe accuracy by only 1.26 percentage points (Figure 17). This weakens the concern that performance is primarily due to a powerful decoder. Appendices G.6–G.7 also provide evidence that quantized attributes preserve neighborhoods and meaningful variation within a concept.
-
Human evaluation and transparent reporting: The user study uses hidden method identities, randomized presentation, and paired statistical tests. The appendix reports code length, codebook size, evaluation exceptions, and limitations of spatial locality and intervention scores. These disclosures help distinguish supported conclusions from stronger interpretations.
Critical weaknesses
-
Classification comparisons are not fully controlled: Appendix D.5 reports differences in preprocessing and probe-time augmentation across methods. A unified evaluation protocol is needed to determine how much of the accuracy gain comes from the representation itself.
-
The scope of interpretability needs clarification: Attribute visualizations show meaningful variation, but do not establish that attributes encode only the displayed part. Moreover, the explanations concern the DisParQ-based classifier; their fidelity to the original teacher remains unverified.
-
Cross-category retrieval is not evaluated separately: The retrieval protocol excludes the query image but retains same-category candidates. Excluding these candidates would provide more direct evidence for cross-category part consistency.
-
The benefit of private codebooks is unclear: Shared PQ performs nearly as well as private PQ in Figure 16. Further evidence is needed to justify the substantially larger private codebooks, particularly in terms of interpretability.
-
Some conclusions need stronger statistical support: Main results use a single seed, making small improvements difficult to assess. The user study evaluates selected important concepts, so its conclusions should not be generalized to the entire dictionary.
Questions for authors
-
Regarding classification comparability: How much of the linear-probe advantage remains under identical image preprocessing, probe-time augmentation, and optimization? Can a representative SAE baseline be evaluated with a comparable global distillation objective and readout?
-
Regarding attribute semantics: Do attributes remain stable when content outside their assigned hard region is changed? Can swapping attributes between occurrences of the same concept produce predictable changes while preserving concept identity?
-
Regarding explanation fidelity: Can the authors report teacher–student prediction agreement or evaluate the reconstructed representation using a fixed teacher classifier? Please clarify whether the intended explanation target is the original backbone or the DisParQ-based prediction system.
-
Regarding cross-category retrieval: What happens when all same-category gallery images are excluded? Please report both R@1 and mAP, and distinguish identity-only retrieval from same-concept appearance ranking.
-
Regarding private codebooks: Given the near-matching performance of shared PQ in Figure 16, what semantic or interpretability benefit justifies private codebooks? A direct comparison on these properties would strengthen the architectural motivation.
5 comments
Reply to Reviewer 1oxq, Part 2/3
W2, Q2 and Q3: Scope of the attributes; explanation target and fidelity.
Target (Q3): the DisParQ classifier, now compared with the backbone's. The explanations describe a linear classifier on the decoded CLS token; Sec. 4.3 now says so. It agrees with the same classifier on the backbone's own CLS token on 92.8 / 93.7 / 93.2 / 91.0 % of test images (PartImageNet / CUB / Cars / Dogs), and on 96 to 97 % of the images the backbone classifies correctly (App. F.18).
Stability (Q2a): the concept is stable, its attribute only partly. We run the spatial misalignment benchmark [1] on CUB: noise is added outside the box of the concept most important for the prediction. PLC (Prototypical part Location Change) is the change in that concept's location and PAC (Prototypical part Activation Change) the drop in its peak activation (both in %):
| Method | PLC ↓ | PAC ↓ | clean accuracy | perturbed accuracy |
|---|---|---|---|---|
| CFM | 30.57 | 3.99 | 55.16 | 50.09 |
| TopK SAE | 18.89 | 1.18 | 47.74 | 46.91 |
| PIP-Net stage 1 | 15.63 | 2.81 | 77.30 | 74.39 |
| ProtoQuant stage 1 | 14.16 | 4.91 | 62.72 | 61.60 |
| DisParQ, without the decoder | 12.96 | 0.07 | 88.87 | 87.78 |
| DisParQ | 8.85 | 0.07 | 90.01 | 88.82 |
DisParQ's concept moves least and keeps its activation, and DisParQ stays the most accurate (all methods in Table 26, App. F.9). Its attribute is less stable (Table 27):
| outcome for that concept, share of images | DisParQ |
|---|---|
| concept retained | 91.5 % |
| retained, attribute identical | 2.1 % |
| retained, attribute of the same class | 63.2 % |
| components of the attribute changed, of 16 | 36.4 % |
About a third of the attribute's 16 components change, and it mostly stays in the same class. As App. B states, the backbone's features are contextual, so an attribute is not causally confined to its pixels.
W2 and Q2, continued: attribute swap.
Swap (Q2b): the decoder reads the attributes, and they carry class-specific appearance. We keep the concepts and their assignment and replace the attributes by those of the same concepts in another image:
| predictions retained, all attributes replaced | same-class donor | other-class donor | random attributes |
|---|---|---|---|
| PartImageNet | 89.6 % | 18.9 % | 53.7 % |
| CUB | 89.2 % | 19.0 % | 42.5 % |
Attributes from a same-class image keep the prediction in about 89 % of cases. Attributes from another class keep it in only 19 %, less than random attributes (54 % and 43 %), so the decoder reads class-specific appearance from them. Concept identity and the spatial map are unchanged by construction (App. F.16).
W3 and Q4: Cross-category retrieval.
Added: DisParQ leads the self-supervised baselines in cross-category retrieval. Thank you for the suggestion. The retrieval gallery now excludes every image of the query's category: the ImageNet class on PartImageNet, the species on CUB.
| category excluded | PartImageNet R@1 | PartImageNet mAP | CUB R@1 | CUB mAP |
|---|---|---|---|---|
| CFM | 30.1 | 15.4 | 8.9 | 7.4 |
| TopK SAE (strongest SAE baseline) | 49.5 | 16.3 | 15.6 | 9.3 |
| ProtoQuant stage 1 (best self-supervised baseline on R@1) | 69.6 | 20.6 | 30.3 | 8.1 |
| DisParQ, concept identity only | 76.3 | 29.1 | 36.4 | 9.3 |
| DisParQ, identity + attribute ranking | 79.6 | 56.7 | 40.0 | 28.1 |
On PartImageNet, concept identity alone leads every self-supervised baseline by at least 6.7 R@1 and 3.6 mAP (PIP-Net stage 1, not shown, has the best baseline mAP at 25.5); on CUB it leads by 6.1 R@1 at similar mAP. The last row ranks candidates of the same concept by attribute similarity, which separates identity from appearance, and adds 3.3 R@1 and 27.6 mAP on PartImageNet (3.6 and 18.8 on CUB). The full tables are in App. F.14.
Reply to Reviewer 1oxq, Part 3/3
W4 and Q5: Private vs shared codebooks.
Accuracy and part metrics are similar; private attributes are more faithful and slightly more stable. A model with one shared codebook, trained from the same stage-1 model with the same budget, is within 0.4 LP of the private one on all five datasets and has identical part metrics, as in Fig. 17. The attributes themselves differ (App. F.16):
| attribute property | PartImageNet private | PartImageNet shared | CUB private | CUB shared |
|---|---|---|---|---|
| fidelity: cosine of the quantized to the continuous attribute | 0.43 | 0.29 | 0.46 | 0.25 |
| components kept under flip / crop / color jitter, % | 72.8 | 70.1 | 67.3 | 66.2 |
A private attribute stays closer to the continuous attribute it quantizes (0.43 to 0.46 against 0.25 to 0.29) and keeps one to three percentage points (p.p.) more of its 16 components when the image is flipped, cropped or color-jittered. A private attribute is also scoped to its concept ("variant j of part c"), and therefore can be analysed and pruned per concept without affecting other concepts.
W5: Seeds, and the scope of the user study.
Seeds. The revision reports means and standard deviations over three training seeds on the four fine-grained datasets, for DisParQ and five baselines (Table 3; part-to-part retrieval on CUB in Table 31). On CUB, DisParQ reaches 89.54 ± 0.16 LP, 85.34 ± 0.20 kNN, 54.50 ± 1.06 PIP purity and 37.49 ± 0.61 part-to-part R@1; its margins over the best self-supervised baseline (+9.1, +16.6, +17.5 and +6.3) are far larger than the standard deviations, as on Cars and Dogs. Three seeds for PartImageNet will be in the final version.
User study. Agreed. Prototypes were sampled among the four most important for an image's prediction (for all methods), so the study speaks about the concepts a user sees in an explanation, not the whole dictionary. Sec. 4.3 now states this. Within this scope, both prefer DisParQ: participants rate its concepts as significantly more consistent than those of PIP-Net and CFM (p < 0.001 for both, Fig. 5 right) and match them to the correct region most often (69 % against 60 % and 52 %; App. F.19).
We hope our response adequately addresses the reviewer's concerns and provides clarifications to their questions.
References [1] Sacha, Jura, Rymarczyk, Struski, Tabor and Zieliński, Interpretability Benchmark for Evaluating Spatial Misalignment of Prototypical Parts Explanations, In AAAI, 2024.
Reply to Reviewer 1oxq, Part 1/3
We thank the reviewer for their positive review and constructive feedback. We are glad that they find the problem "important", the results "encouraging", the supporting experiments "extensive", and the reporting "transparent". Below we answer the weaknesses and questions together where they overlap. Please also see our global comment (https://openreview.net/forum?id=40Arf70gfe¬eId=SlKp38nbyE) for the questions shared across the reviews.
Summary
W1 and Q1: Classification under one protocol; an SAE with our readout.
DisParQ still leads under a standardized protocol. The original results follow the probing protocol of each baseline (where available). Every dictionary baseline re-probed under our protocol, with the mean of the raw DINOv2 patch tokens as a control (LP / kNN):
DisParQ leads the best self-supervised baseline by 2.8 / 9.3 / 3.5 / 5.0 LP and 4.8 / 17.1 / 19.8 / 16.7 kNN on PartImageNet / CUB / Cars / Dogs. Without the decoder, its description still leads by 2.0 to 18.0 kNN on all four datasets and by 0.4 to 7.8 LP on three. A pooled reconstruction is bounded by the patch-mean control, whereas DisParQ decodes the CLS token. This is a strength of the method: the decoder's input is exactly the discrete description (concept identities, their regions and quantized attributes) that our method learns, with no continuous feature bypassing it.
An SAE with our readout and objective (Q1). We feed the concept maps of the frozen PatchSAE and MSAE, 16 concepts per image, to our decoder, train it with our CLS distillation loss and probe it identically. They reach 84.2 and 75.4 LP on PartImageNet, against 88.5 for DisParQ (all probed on precomputed features to keep the evaluation identical, hence not the 89.2 above). Removing the per-image budget moves them only to 85.2 and 76.9. The difference lies in the representation, not in the readout or the budget. Both tables are in App. F.3 and F.5.