- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors claim that retrieval accuracy is not a guarantee for a disentanglement in frozen foundation models and other metric should be involved. The disentanglement is specifically examined on separating style and content. The metric they suggest is cross Nullspace Projection (CNP), which aim to find the projection of one group on another's group nullspace given a set of labeled data.
Critical strengths
-
relatively fluent paper, the goal is clear and the introduction and proposition of it is fluent. I like that each derivation is followed by an intuitive explanation, it makes complicated derivations be more digestible.
-
The metrics used for comparison are important and analyze and improve them is important as well.
-
The non-trainable phase, which aim to orthogonalize the spaces is highly appreciated.
-
The metric is extensively examined on leading representation methods.
Critical weaknesses
-
The scope of specifically analyze the style vs. content is useful for some extent, but it would be highly beneficial to extend this to general disentangled representations and not limited to some degree of disentanglement (the style vs content)
-
The combination between the metrics is somewhat arbitrary, why do you select the geometric mean for the mAPs? why do you multiply it with the disentanglement purity? why is the purity 1 minus the leak and not 1over the leak for example? those are all heuristic picks, which I believe that new metric should be simple and clean such that it would be a good measure for disentanglement but still easy to understand. Even though you elaborate on the rationale (which is highly appreciated), I still believe it is too heuristic for a broad usage.
-
The method still need a portion of labeled data, which especially in self-supervised regimes can be hard to collect. depends on how much labels the method need to be accurate.
-
I think that the amount of labeled samples should be ablated as well (couldnt find this kind of ablation).
Questions for authors
-
The paper uses many acronyms without explaining them. What LDA stands for? (first shown by its full name only in Line 240) The related work full of many paradigms which are three letters, but for the general reader which do not familiar with each and every metric it is highly recommended to elaborate the full name for any acronym before using it, and even with more care when you keep using an acronym along the paper. It is fine that well known acronyms such as mAP or KNN will not be defined, but less familiar ones are more encouraged to be elaborated for readability.
-
why the metric is not mutual? you wrote (line 175-176): "probe leakage (information level) asks whether style labels are linearly predictable from content vectors." - why not also vice versa? measure whether a content label is predictable out of the style vector? As I understand from keep reading, equation 2 do take into consideration both sides. If so, I believe you should be more accurate on these lines.
-
what is that unfamiliar sign in Eq. 1? ⊮? please explain it
-
If you made a large amount of experiments, 5 per seed, then add the std, t get a notion of stability for the experiments.
-
highly relevant citation that havent included [1].
I like the idea of question the current metrics, however I think that a good metric should be a one that do involve the least hyperparameters possible, and obviously do not sensitive to them much. I believe it should be simple, such that it would be easy to use it as a probe but still understand it well. Moreover, I think that it should be as free as possible from heuristic choices.
refs:
[1] The Linear Representation Hypothesis and the Geometry of Large Language Models. Park et al.
3 comments
Response to Reviewer woDZ
4. Writing, dispersion, and one missing citation
Acronyms.
LDA, DCI, MIG, SAP, INLP and LEACE are now expanded at first use.
The probe is bidirectional.
You are right that the sentence described one direction while Eq. 2 evaluates both. It now states that style is probed from the content vector and content from the style vector, matching Eq. 2.
The symbol in Eq. 1.
⊮[·] is the indicator function: 1 when the condition holds, 0 otherwise. It
will be defined immediately after the equation.
Standard deviations.
Thank you for this suggestion. For CLIP+CNP, the seed-to-seed standard deviation of DScore is approximately 0.003. We will report this in the main text and provide full means and standard deviations for the label-budget ablation. This context is particularly important for small differences, such as the leading DomainNet DScore gap of 0.023.
The citation.
Park et al. (2023), The Linear Representation Hypothesis and the Geometry of Large Language Models, will be cited in the related-work discussion of linear concept directions.
5. The design goal you state
DScore introduces no fitted weights: the utility term is a geometric mean of two retrieval numbers, the purity term is a linear discount, and neither is tuned on data. The aggregation still involves design choices, including the functional form and equal weighting of the two leakage terms; the analysis in §2 above examines sensitivity to the functional form. The component diagnostics do have settings of their own — K, the probe configuration — which Appendix H specifies.
We agree that the scalar should not be the sole interface. The revision will report the retrieval and leakage components alongside DScore. These quantities are already visible separately in Table 1: Qwen2.5-VL has the highest style mAP (0.890) and the highest kNN leakage (0.645), while DINOv2 has the lowest kNN leakage in the table (0.434) at lower style mAP (0.789). This retrieval-purity trade-off is visible before any combination is applied.
Response for authors
Thank you for your dedication, the thorough response, and the new ablations. Particularly the label-budget analysis and the clarifications.
However, my two primary conceptual concerns remain unresolved:
-
Narrow Scope: Restricting the formulation to a two-factor (style vs. content) setting limits the generality of the proposed metric for broader disentanglement research.
-
Scalar Aggregation & Metric Design: For a metric to be widely adopted as a diagnostic tool, it should prioritize simplicity, interpretability, and faithfulness to the underlying properties (at least to my view). Allowing practitioners to debug their models rather than having to debug the metric itself. Combining distinct, competing aspects (utility and leakage) into a single scalar inherently masks trade-offs. In fact, your 16-variant sensitivity analysis reinforces this concern: changing the functional form of the purity term flips the top-ranked encoder between DINOv2 and Qwen2.5-VL. While I appreciate that you will report the decomposed retrieval and leakage components alongside DScore in the revision, I remain unconvinced by the formulation of DScore itself as a primary standalone metric.
Response to Reviewer woDZ
Thank you — your points are grouped into five headings, in the order you raised them.
1. Scope: style versus content only
We agree, and we would rather state the limit than claim generality we have not shown. Extending the framework to three or more factors is a different experiment: the two-factor case admits a single cross-projection between two subspaces, while three factors require a different construction and a different set of null models. We will state this explicitly in the scope paragraph.
2. "The combination is arbitrary"
We use multiplication to express joint utility and purity, rather than an additive trade-off between them. Writing
U = √(mAP_c · mAP_s)andP = 1 − leak, both in[0,1], the productU · Pis zero if either term is zero and cannot exceed either term. High retrieval utility alone therefore cannot yield a high DScore when purity is low, or vice versa. For fixed utility, purity acts as a proportional discount. The geometric mean similarly penalizes imbalance between content and style retrieval. These are explicit design choices, not a claim that the product is the uniquely correct functional form.We ran the analysis your question implies. Four utility aggregations (geometric, arithmetic, harmonic, minimum) crossed with four purity forms (
1−leak,1/(1+leak),exp(−leak),(1−leak)²), on all three benchmarks (Spearman against the paper's ranking):1−leakHolding the purity term at
1−leak, the geometric, arithmetic, and harmonic means leave the top-ranked encoder unchanged on all three benchmarks. The minimum aggregation preserves the DomainNet top-1, but changes the OfficeHome and NICO++ top-1 from DINOv2 to Qwen2.5-VL. The top choice is therefore stable across the three mean-based utility forms, but not invariant to every aggregation.The purity form does matter, and we would rather say so than claim otherwise. Under reciprocal purity the OfficeHome top-1 exchanges between DINOv2 and Qwen2.5-VL; on DomainNet some variants using the minimum aggregation exchange it with DINOv3. We therefore read the leaderboards as indicative orderings and no longer claim invariance to either the utility aggregation or the purity form.
On
1−leakversus1/leak: purity should be bounded, 0 at maximal leakage and 1 at none.1−leakis the affine map with those boundary conditions;1/leakis unbounded as leak → 0 and assigns 1 to maximal leakage. That is why we keep it — not because the ranking is insensitive to the alternative.3. Labelled data, and how much of it
CNP requires labelled estimation data and does not fine-tune the encoder. The abstract identifies it as supervised post-processing; we will make the labelled data requirement more explicit.
You also asked for the ablation. We varied the fraction of the 70% estimation split, refitted the LDA subspaces at each fraction, and evaluated retrieval and DScore on the held-out 30%. Five seeds per row. Here 100% denotes the full 70% estimation split, not the full benchmark.
Performance exhibits clear diminishing returns. 25% of the labelled estimation data (≈17k images) already retains 89% of the full-data DScore, and 50% retains 95%; the remaining half therefore contributes only the final five percentage points. Style mAP rises from 0.582 at 5% to 0.726 at 10%, while content mAP improves more gradually. We will include the curve to quantify the labelled data requirement rather than leaving it qualitative.