acceptodds
Withdrawn from ICLR 2027

High Retrieval Accuracy Does Not Imply Disentanglement: Diagnosing and Repairing Pseudo-Disentanglement in Frozen Visual Encoders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieval accuracy is the default criterion for choosing a visual encoder, yet it is silent about whether content and style are actually separated: discriminable class boundaries suffice for high mAP even when the two factors are thoroughly mixed. We define this failure mode operationally—a representation is pseudo-disentangled when its retrieval mAP is high while its cross-factor leakage is not low—and make it measurable. Our metric, DScore, combines retrieval utility with two subspace-level leakage diagnostics, a kNN leakage proxy and a probe leakage rate; unlike the axis-aligned metrics (DCI, MIG, SAP), it does not assume each factor aligns with one latent dimension. Across nine frozen representations and three benchmarks, "high accuracy ≠ high disentanglement" holds throughout: the best style retriever falls to third once leakage is penalized, while a self-supervised encoder ranking eighth by style mAP ranks first. We then ask whether this entanglement can be repaired without fine-tuning the encoder, and show that it can. Cross-Nullspace Projection (CNP) is a closed-form, supervised post-processing step with two parts—a factor-specific LDA readout, and a bidirectional cross-projection that makes the two readouts approximately orthogonal. Most of its DScore gain (mean +0.15; probe leakage cut by 60–86%) comes from the readout, not the projection, but the projection is what makes the two factor vectors independently usable, lifting compositional retrieval from 0.19 to 0.45 mAP (2.4×). Across the benchmarks studied, content-style entanglement in frozen encoders is thus substantially linear-remediable rather than a deep semantic fusion.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
content-style disentanglement, frozen visual encoders, representation evaluation, disentanglement metrics, style representation, vision foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_woDZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors claim that retrieval accuracy is not a guarantee for a disentanglement in frozen foundation models and other metric should be involved. The disentanglement is specifically examined on separating style and content. The metric they suggest is cross Nullspace Projection (CNP), which aim to find the projection of one group on another's group nullspace given a set of labeled data.

Critical strengths
  • relatively fluent paper, the goal is clear and the introduction and proposition of it is fluent. I like that each derivation is followed by an intuitive explanation, it makes complicated derivations be more digestible.

  • The metrics used for comparison are important and analyze and improve them is important as well.

  • The non-trainable phase, which aim to orthogonalize the spaces is highly appreciated.

  • The metric is extensively examined on leading representation methods.

Critical weaknesses
  • The scope of specifically analyze the style vs. content is useful for some extent, but it would be highly beneficial to extend this to general disentangled representations and not limited to some degree of disentanglement (the style vs content)

  • The combination between the metrics is somewhat arbitrary, why do you select the geometric mean for the mAPs? why do you multiply it with the disentanglement purity? why is the purity 1 minus the leak and not 1over the leak for example? those are all heuristic picks, which I believe that new metric should be simple and clean such that it would be a good measure for disentanglement but still easy to understand. Even though you elaborate on the rationale (which is highly appreciated), I still believe it is too heuristic for a broad usage.

  • The method still need a portion of labeled data, which especially in self-supervised regimes can be hard to collect. depends on how much labels the method need to be accurate.

  • I think that the amount of labeled samples should be ablated as well (couldnt find this kind of ablation).

Questions for authors
  • The paper uses many acronyms without explaining them. What LDA stands for? (first shown by its full name only in Line 240) The related work full of many paradigms which are three letters, but for the general reader which do not familiar with each and every metric it is highly recommended to elaborate the full name for any acronym before using it, and even with more care when you keep using an acronym along the paper. It is fine that well known acronyms such as mAP or KNN will not be defined, but less familiar ones are more encouraged to be elaborated for readability.

  • why the metric is not mutual? you wrote (line 175-176): "probe leakage (information level) asks whether style labels are linearly predictable from content vectors." - why not also vice versa? measure whether a content label is predictable out of the style vector? As I understand from keep reading, equation 2 do take into consideration both sides. If so, I believe you should be more accurate on these lines.

  • what is that unfamiliar sign in Eq. 1? ⊮? please explain it

  • If you made a large amount of experiments, 5 per seed, then add the std, t get a notion of stability for the experiments.

  • highly relevant citation that havent included [1].

I like the idea of question the current metrics, however I think that a good metric should be a one that do involve the least hyperparameters possible, and obviously do not sensitive to them much. I believe it should be simple, such that it would be easy to use it as a probe but still understand it well. Moreover, I think that it should be as free as possible from heuristic choices.

refs:

[1] The Linear Representation Hypothesis and the Geometry of Large Language Models. Park et al.

3 comments
AuthorsOpenReview ↗

Response to Reviewer woDZ

Thank you — your points are grouped into five headings, in the order you raised them.

1. Scope: style versus content only

We agree, and we would rather state the limit than claim generality we have not shown. Extending the framework to three or more factors is a different experiment: the two-factor case admits a single cross-projection between two subspaces, while three factors require a different construction and a different set of null models. We will state this explicitly in the scope paragraph.

2. "The combination is arbitrary"

We use multiplication to express joint utility and purity, rather than an additive trade-off between them. Writing U = √(mAP_c · mAP_s) and P = 1 − leak, both in [0,1], the product U · P is zero if either term is zero and cannot exceed either term. High retrieval utility alone therefore cannot yield a high DScore when purity is low, or vice versa. For fixed utility, purity acts as a proportional discount. The geometric mean similarly penalizes imbalance between content and style retrieval. These are explicit design choices, not a claim that the product is the uniquely correct functional form.

We ran the analysis your question implies. Four utility aggregations (geometric, arithmetic, harmonic, minimum) crossed with four purity forms (1−leak, 1/(1+leak), exp(−leak), (1−leak)²), on all three benchmarks (Spearman against the paper's ranking):

datasetgeometric / arithmetic / harmonic, with 1−leakworst of the 16 variants
DomainNet1.000 / 0.950 / 1.000, top-1 unchanged0.883
OfficeHome1.000 / 1.000 / 1.000, top-1 unchanged0.817
NICO++1.000 / 1.000 / 0.983, top-1 unchanged0.933

Holding the purity term at 1−leak, the geometric, arithmetic, and harmonic means leave the top-ranked encoder unchanged on all three benchmarks. The minimum aggregation preserves the DomainNet top-1, but changes the OfficeHome and NICO++ top-1 from DINOv2 to Qwen2.5-VL. The top choice is therefore stable across the three mean-based utility forms, but not invariant to every aggregation.

The purity form does matter, and we would rather say so than claim otherwise. Under reciprocal purity the OfficeHome top-1 exchanges between DINOv2 and Qwen2.5-VL; on DomainNet some variants using the minimum aggregation exchange it with DINOv3. We therefore read the leaderboards as indicative orderings and no longer claim invariance to either the utility aggregation or the purity form.

On 1−leak versus 1/leak: purity should be bounded, 0 at maximal leakage and 1 at none. 1−leak is the affine map with those boundary conditions; 1/leak is unbounded as leak → 0 and assigns 1 to maximal leakage. That is why we keep it — not because the ranking is insensitive to the alternative.

3. Labelled data, and how much of it

CNP requires labelled estimation data and does not fine-tune the encoder. The abstract identifies it as supervised post-processing; we will make the labelled data requirement more explicit.

You also asked for the ablation. We varied the fraction of the 70% estimation split, refitted the LDA subspaces at each fraction, and evaluated retrieval and DScore on the held-out 30%. Five seeds per row. Here 100% denotes the full 70% estimation split, not the full benchmark.

fraction of labelled estimation datanC-mAPS-mAPDScore
5%3,4920.526 ± 0.0080.582 ± 0.0200.392 ± 0.009
10%6,9850.567 ± 0.0090.726 ± 0.0060.441 ± 0.005
25%17,4620.591 ± 0.0090.783 ± 0.0040.536 ± 0.005
50%34,9250.616 ± 0.0070.801 ± 0.0060.571 ± 0.004
75%52,3870.623 ± 0.0080.836 ± 0.0050.594 ± 0.003
100%69,8500.640 ± 0.0060.857 ± 0.0060.601 ± 0.003

Performance exhibits clear diminishing returns. 25% of the labelled estimation data (≈17k images) already retains 89% of the full-data DScore, and 50% retains 95%; the remaining half therefore contributes only the final five percentage points. Style mAP rises from 0.582 at 5% to 0.726 at 10%, while content mAP improves more gradually. We will include the curve to quantify the labelled data requirement rather than leaving it qualitative.

AuthorsOpenReview ↗

Response to Reviewer woDZ

4. Writing, dispersion, and one missing citation

Acronyms.

LDA, DCI, MIG, SAP, INLP and LEACE are now expanded at first use.

The probe is bidirectional.

You are right that the sentence described one direction while Eq. 2 evaluates both. It now states that style is probed from the content vector and content from the style vector, matching Eq. 2.

The symbol in Eq. 1.

⊮[·] is the indicator function: 1 when the condition holds, 0 otherwise. It will be defined immediately after the equation.

Standard deviations.

Thank you for this suggestion. For CLIP+CNP, the seed-to-seed standard deviation of DScore is approximately 0.003. We will report this in the main text and provide full means and standard deviations for the label-budget ablation. This context is particularly important for small differences, such as the leading DomainNet DScore gap of 0.023.

The citation.

Park et al. (2023), The Linear Representation Hypothesis and the Geometry of Large Language Models, will be cited in the related-work discussion of linear concept directions.

5. The design goal you state

DScore introduces no fitted weights: the utility term is a geometric mean of two retrieval numbers, the purity term is a linear discount, and neither is tuned on data. The aggregation still involves design choices, including the functional form and equal weighting of the two leakage terms; the analysis in §2 above examines sensitivity to the functional form. The component diagnostics do have settings of their own — K, the probe configuration — which Appendix H specifies.

We agree that the scalar should not be the sole interface. The revision will report the retrieval and leakage components alongside DScore. These quantities are already visible separately in Table 1: Qwen2.5-VL has the highest style mAP (0.890) and the highest kNN leakage (0.645), while DINOv2 has the lowest kNN leakage in the table (0.434) at lower style mAP (0.789). This retrieval-purity trade-off is visible before any combination is applied.

Reviewer_woDZOpenReview ↗

Response for authors

Thank you for your dedication, the thorough response, and the new ablations. Particularly the label-budget analysis and the clarifications.

However, my two primary conceptual concerns remain unresolved:

  • Narrow Scope: Restricting the formulation to a two-factor (style vs. content) setting limits the generality of the proposed metric for broader disentanglement research.

  • Scalar Aggregation & Metric Design: For a metric to be widely adopted as a diagnostic tool, it should prioritize simplicity, interpretability, and faithfulness to the underlying properties (at least to my view). Allowing practitioners to debug their models rather than having to debug the metric itself. Combining distinct, competing aspects (utility and leakage) into a single scalar inherently masks trade-offs. In fact, your 16-variant sensitivity analysis reinforces this concern: changing the functional form of the purity term flips the top-ranked encoder between DINOv2 and Qwen2.5-VL. While I appreciate that you will report the decomposed retrieval and leakage components alongside DScore in the revision, I remain unconvinced by the formulation of DScore itself as a primary standalone metric.

Reviewer_mwzkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that evaluating frozen visual encoders by retrieval accuracy alone can lead to “pseudo-disentanglement,” where retrieval performance is high even though content and style are not actually separated. To diagnose this, the authors propose DScore, which combines retrieval utility with kNN leakage and uses the LDA subspace overlap ρ\rho for raw single-vector encoders and probe leakage for decoupled representations. They also propose Cross-Nullspace Projection (CNP), a supervised post-processing step consisting of a factor-specific LDA readout and a bidirectional cross-projection. Across nine frozen encoders and three datasets (DomainNet, OfficeHome, NICO++), they report that rankings by retrieval accuracy and by DScore disagree and that CNP generally reduces leakage. Improvements in factorized compositional retrieval are shown for CLIP on DomainNet.

Critical strengths

S1.

The problem setting is practical and important. Given the common practice of selecting frozen encoders by retrieval performance alone, evaluating retrieval utility and disentanglement purity separately is a useful perspective, and the diagnose–reveal–repair structure is clear.

S2.

The evaluation is relatively broad, covering nine representations and three datasets. The authors also describe their limitations fairly transparently in Appendices A, F, and G, including that the supervised LDA readout accounts for most of the improvement, label dependence, the lack of a common scale between raw and decoupled DScore, and the absence of dispersion. Using compositional retrieval to assess the usability of factor-specific representations beyond leakage reduction is also a positive aspect.

Critical weaknesses

W1.

Raw probe leakage measures how well xx encodes each factor, not entanglement. For a raw encoder, zc=zs=xz_c=z_s=x (Sec. 3.1), so by Eq. 2 probe leakage is the mean accuracy of predicting the style and content labels from the same xx. It is therefore high even when content and style occupy orthogonal subspaces of xx. The authors acknowledge that a raw single-vector encoder “makes self-probing meaningless” and use ρ\rho in the raw DScore (Sec. 3.2), yet raw probe leakage serves as evidence of entanglement in Tables 1 and 3 and Sec. 4.7, and the abstract’s “probe leakage cut by 60–86%” is computed against these raw values. This figure therefore compares two different quantities; only the post-CNP value measures cross-factor predictability.


W2.

The cross-projection does not guarantee approximately orthogonal factor subspaces, and the resulting overlap is not reported. Let cc be a principal cosine between the two LDA subspaces (ρ\rho is the largest). Eq. 6 gives cλ=c ∣1−2λ+λ2c2∣/(1−λ(2−λ)c2)c_\lambda=c\,|1-2\lambda+\lambda^2c^2|/(1-\lambda(2-\lambda)c^2): at λ=1\lambda=1 the overlap returns to its original value, and at the chosen λ=0.7\lambda=0.7 it falls to about 40% of it. The paper reports no post-projection ρ\rho, and Propositions 1–2 do not imply approximate orthogonality, so the claim that the two vectors are “approximately orthogonal” and hence “independently usable” is not supported.


W3.

The experiment behind the abstract’s compositional-retrieval claim is missing. The abstract states that the projection makes the two factor vectors independently usable, lifting compositional retrieval from 0.19 to 0.45 mAP. However, the authors acknowledge that the λ=0\lambda=0 dual-LDA baseline is not evaluated for compositional retrieval (App. G) and that the raw baseline uses a joint cosine similarity rather than the composite score (App. H). Given these acknowledged gaps, the abstract’s claim should be softened.


W4.

A small ρ\rho is not by itself evidence of factor separation. The paper argues that ρ≪1\rho\ll1 (typically < 0.3) means the factors are largely non-collinear, and uses this to justify content preservation (Sec. 3.3, App. B). However, randomly drawn subspaces with the same effective ranks (kc=64k_c=64, ks=5k_s=5) in D=768D=768–1024 already give ρ≈0.30\rho\approx0.30–0.34 (our computation), comparable to the reported 0.122–0.406. Without such a reference, differences in ρ\rho across encoders cannot be attributed to separation rather than to dimension and LDA estimation.


Questions for authors

Check Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mwzk

Thank you for identifying the ambiguities in the leakage and overlap measurements and for requesting the missing controls. We address W1–W4 below, distinguishing the measured results from the claims they support.

W1: distinguishing raw probe accuracy from cross-factor leakage.

For a raw encoder z_c = z_s = x, so Eq. 2 evaluates to the mean accuracy with which a probe recovers either label from a single vector: information content, not cross-factor mixing. It is high for any useful encoder, and low for Gram (0.468) only because Gram carries little category information at all. We will remove the "60–86%" figure from the abstract and relabel the Table 1 column as single-vector probe accuracy.

For decoupled representations, Eq. 2 measures cross-factor predictability in both directions. Reporting the directions separately makes this interpretation explicit, particularly because their class counts and chance references differ (6 domains vs 345 categories). For DomainNet with CLIP:

proberawafter CNPuniform-guessing reference
style from content0.8970.2971/6 = 0.167
content from style0.6970.0091/345 = 0.003

The directional breakdown reveals an asymmetry: after CNP the content vector retains style predictability of 0.297, above the uniform-guessing reference of 0.167, while content predictability from the style vector is much lower (0.009, with a uniform-guessing reference of 0.003). For CLIP, CNP is therefore not symmetric: it removes content from the style representation more effectively than it removes style from the content representation.

We will report both directions alongside the aggregate leakage, with their respective chance references. The fixed aggregation is used consistently across encoders, while the directional results make asymmetric residual predictability visible.

DScore provides a joint assessment of retrieval utility and purity; its component diagnostics explain the trade-offs underlying the aggregate. These roles are complementary. The clarification above corrects the interpretation of raw probe accuracy without changing DScore's utility-purity design. Raw probe accuracy will not be presented as evidence of entanglement.

W2 — orthogonality was asserted, not measured.

You are right. You wrote that Eq. 6 makes the overlap fall "to about 40%". On DomainNet at λ = 0.7, with both bases truncated to their first five discriminative directions and QR-orthonormalised after the basis cross-projection:

encoderρ beforeρ afterratiorandom nullbelow null
ResNet-500.1340.0530.400.080✓
Qwen2.5-VL0.1270.0510.400.061✓
DINOv20.1960.0770.390.114✓
SigLIP20.2430.0950.390.108✓
CLIP0.2840.1100.390.130✓
CSD0.2880.1120.390.131✓
Gram0.3660.1390.380.162✓
DINOv30.5250.1860.350.115✗
IntroStyle0.2300.0900.390.071✗

Mean ratio 0.386, spread 0.35–0.40. In seven of nine encoders the post-projection overlap is below the mean of the null distribution for two random subspaces of the same shapes. The exceptions are DINOv3 (0.525 → 0.186, above its 0.115 null) and IntroStyle (0.230 → 0.090, above its 0.071 null).

We will clarify the claim itself: Propositions 1–2 bound attenuation, not orthogonality. The table measures reduced geometric overlap for these five-dimensional cross-projected bases, rather than establishing orthogonality as a consequence of the propositions.

AuthorsOpenReview ↗

Response to Reviewer mwzk

W3 — the compositional claim was not isolated, and adding the row changes what the abstract can say.

Under the same composite-scoring rule, the raw joint embedding obtains 0.186, dual-LDA 0.392, LEACE 0.403, and CNP 0.451 mAP. Cross-projection therefore contributes +0.059 mAP over dual-LDA, rather than the entire 0.186 → 0.451 improvement. We will add these controls, identify the raw baseline as a no-factorisation reference rather than a tuned competitor, and narrow the abstract sentence accordingly. Dual-LDA and LEACE already provide combinable factor vectors; the comparison supports improved retrieval with cross-projection, not its necessity for factorized scoring.

W4 — interpreting subspace overlap against a random reference.

We agree that overlap must be compared with a dimension- and rank-matched null. We also clarify two statistics: Table 1 uses the normalized Frobenius overlap ρ_F = ‖WcᵀWs‖_F/√kk at rank 32, whereas Proposition 2 uses the spectral norm ρ_2 = ‖WcᵀWs‖₂. The reviewer's 0.32–0.34 value corresponds to the spectral-norm random baseline at rank 32, while the reported 0.122–0.406 values are ρ_F; the two are therefore not directly comparable.

For random rank-kk subspaces in R^D, the root-mean-square null for ρ_F is √(kk/D). At rank 32 this gives 0.204 for D = 768, 0.176 for D = 1024, and 0.250 for D = 512. Eight of nine encoders exceed their null, while ResNet-50 (0.122 versus 0.125) does not. We will therefore remove the claim that ρ < 0.3 alone establishes sufficient separation and report ρ_F with its null in Table 1.

W2 separately reports the post-projection five-direction ρ_2 diagnostic and its random-subspace null: seven of nine encoders fall below the null mean. We present this as empirical evidence of reduced overlap, not as a consequence of Propositions 1–2.

Reviewer_2D7LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between retrieval accuracy and content-style disentanglement in frozen visual encoders, arguing that high retrieval performance does not necessarily indicate effective factor separation. To evaluate disentanglement, the authors introduce DScore, a metric that combines content/style retrieval accuracy with cross-factor leakage measurements, including kNN-based leakage and subspace-level diagnostics. The paper further proposes Cross-Nullspace Projection (CNP), a supervised linear post-processing method that uses factor-specific Linear Discriminant Analysis (LDA) to identify content and style subspaces, followed by bidirectional cross-projection to reduce their overlap without fine-tuning the encoder. Experiments across nine visual representations and three benchmarks (DomainNet, OfficeHome, and NICO++) reveal discrepancies between retrieval-based and disentanglement-aware rankings. The authors report that CNP reduces cross-factor leakage, improves DScore, and enhances factorized compositional retrieval performance.

Critical strengths

The paper provides a systematic evaluation of nine frozen visual encoders across three benchmarks, offering useful empirical insights into the relationship between retrieval performance and cross-factor information leakage.

Critical weaknesses
  1. Limited methodological novelty and insufficient comparisons. The paper lacks comparisons with closely related disentanglement methods, such as Ngweta et al. (ICML 2023) and LEACE. Moreover, DScore is not compared with content-style disentanglement metrics used in prior work, such as SCD in Ngweta et al. (ICML 2023), leaving its effectiveness insufficiently validated. Most of the reported DScore improvement also comes from supervised LDA rather than the proposed cross-nullspace projection.

  2. Insufficient validation of practical benefits. Despite improved DScore, content retrieval accuracy decreases for several encoders. The paper does not sufficiently demonstrate the practical usability of the disentangled features, lacking evaluations on applications such as style transfer or domain generalization. Furthermore, the absence of a dual-LDA baseline in compositional retrieval makes it difficult to attribute the reported gains to CNP.

Questions for authors
  1. The paper would benefit from direct comparisons with closely related disentanglement methods, such as Ngweta et al. (ICML 2023) and LEACE. Additionally, DScore should be compared with existing content-style disentanglement metrics, such as SCD in Ngweta et al. (ICML 2023), to better validate its effectiveness.

  2. What is the compositional retrieval performance of dual-LDA without cross-nullspace projection (λ = 0)? This comparison is important for assessing the actual contribution of the proposed projection.

  3. Although CNP improves DScore, content retrieval accuracy decreases for several encoders. Additional evaluations on practical applications, such as style transfer or domain generalization, would help demonstrate the usability and benefits of the disentangled features.

  4. DScore uses different leakage definitions for raw and disentangled representations, making the reported improvements difficult to interpret. The authors should clarify this discrepancy and provide a consistent evaluation to support their conclusions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2D7L

We thank the reviewer for the constructive comments. We have added the requested comparisons and clarify them below; the four questions are answered in order.

1. Comparisons with LEACE, PISCO, and Ngweta et al.'s SCD

LEACE and PISCO. We ran LEACE with the official concept-erasure implementation (Belrose et al.), applied bidirectionally — erase style, take the content readout; erase content, take the style readout — followed by an LDA readout. Because the original PISCO protocol is driven by image transformations rather than our six domain labels, we additionally evaluate a supervised natural-pair adaptation of Ngweta et al.: same-category, different-domain pairs over the full CLIP feature set, their -Gram + λ·DeltaGram objective, and five one-vs-rest style directions. All rows are reported with the same metric definitions:

methodC-mAPS-mAPprobekNNDScoresource
LEACE, official package, bidirectional0.6690.7930.1820.2260.580new experiment
PISCO-natural (λ = 1)0.4960.7230.1800.2570.468new experiment
PISCO-natural (λ = 10)0.2380.7280.0910.2340.349new experiment
CNP (λ = 0.7)0.6400.8570.1530.2240.601submitted Table 2

The CNP row reproduces the submitted Table 2 values; the LEACE and PISCO rows are newly computed. Lower probe and kNN values are better. For the PISCO comparison, we adapt its experimental construction to our DomainNet setting.

Under the aligned natural-pair comparison, CNP achieves the best DScore among the compared methods (0.601). PISCO at λ = 10 reduces probe leakage further (0.091), but its content mAP falls to 0.238 and its DScore to 0.349.

SCD. We computed the statistic Ngweta et al. define, SCD = ‖corr(ẑ_F_C, ẑ_F_S)‖_F, adapted to our setting: their style factors are continuous transformation parameters, whereas ours are discrete domain labels, so we use the one-hot style label as the style factor.

SCD is a complementary purity diagnostic rather than a retrieval utility measure. For this adaptation, we compute the correlation between raw feature coordinates and one-hot style labels for all nine encoders, normalised by √D to account for the number of feature coordinates:

encoderDSCD/√Dcontent mAP
Gram5120.0870.212
ResNet-5020480.2240.442
DINOv210240.3090.686
DINOv310240.3360.725
CSD7680.3480.668
CLIP7680.3510.666
SigLIP211520.3650.716
IntroStyle25600.4620.210
Qwen2.5-VL35840.5000.616

Gram has the lowest adapted correlation (0.087) despite a content mAP of 0.212, while IntroStyle has similarly low content mAP (0.210) and a correlation of 0.462. Low correlation alone therefore does not establish that a representation retains useful content. This motivates reading purity together with retrieval utility, as DScore does; it does not invalidate SCD as a purity diagnostic. We deliberately do not compare these values against the numbers reported in that paper: ours is an adaptation with a different factor representation, so only the within-table comparison is meaningful.

For CLIP, an equal-dimensional comparison of the two 64-dimensional content readouts gives adapted SCD/√d = 0.115 for dual-LDA (λ = 0) and 0.106 for CNP (λ = 0.7), a reduction of approximately 7.8%. This provides complementary evidence of lower style-label correlation under this diagnostic; the compositional retrieval contribution is evaluated separately below.

Attribution of the gain. We agree that the supervised readout is an important part of CNP and should be isolated from the cross-projection. The existing λ ablation provides this control:

configurationDScore
dual-LDA readout (λ = 0, Table 6 first column)0.574
CNP (λ = 0.7)0.601

At λ = 0 the projection term vanishes identically, so that column is the dual-LDA baseline: two separately fitted LDA readouts. Within the same DScore_probe definition, cross-projection adds 0.027 (0.574 → 0.601). We do not express this as a fraction of the gain over raw features, because the raw and decoupled DScore definitions differ, as discussed in answer 4 below. Both dual-LDA and CNP provide factor-specific vectors that can be weighted separately; the compositional comparison below tests whether projection improves their use. Our claim is a closed-form linear post-processing demonstration, not a new state-of-the-art separator.

AuthorsOpenReview ↗

Response to Reviewer 2D7L

2. Dual-LDA compositional retrieval

Thank you for requesting this control. The dual-LDA row is needed to isolate cross-projection from the supervised readout. DomainNet with CLIP:

settingcontent-keep + style-swapstyle-keep + content-swapsource
raw joint embedding0.1860.396Table 5
dual-LDA readout (λ = 0)0.3920.782new — computed for this response
LEACE, bidirectional (official package)0.4030.798new — computed for this response
CNP (λ = 0.7)0.4510.842Table 5

The difference between CNP and dual-LDA is +0.059 mAP (0.392 → 0.451) on content-keep + style-swap, rather than the whole 0.19 → 0.45 difference from raw. Dual-LDA and bidirectional LEACE already support factorized retrieval; CNP achieves higher mAP in the displayed comparison. The appropriate claim is that cross-projection improves compositional retrieval, not that it is necessary to produce combinable vectors. On provenance: the dual-LDA and LEACE rows are computed for this response using the same features and compositional scoring rule; the CNP and raw rows are the published Table 5 values. We will add the controls and clarify this attribution next to Table 5.

3. Content retrieval decreasing

We agree that content retrieval is not uniformly preserved. Proposition 2 lower-bounds retained content energy by (1 − λρ)² for a content-dominated vector, where ρ = ‖W_cᵀW_s‖₂ is the largest principal cosine between orthonormal bases. This describes the projection's energy trade-off, but it is not a bound on mAP and does not establish that the observed retrieval losses are within a theoretical guarantee.

The reported relative C-mAP changes at λ = 0.7 on DomainNet are:

encoderobserved C-mAP change
SigLIP2+3.4%
DINOv3−6.6%
Qwen2.5-VL+6.7%
CSD−4.5%
CLIP−3.9%
DINOv2−12.8%
ResNet-50−27.1%
IntroStyle+54.8%
Gram+19.8%

Four encoders improve and five lose content retrieval accuracy, with the largest loss for ResNet-50. We will clarify that CNP trades off factor suppression and retrieval utility rather than guaranteeing improved content retrieval.

We take the point that style transfer and domain generalisation are the applications readers will ask about. Neither, however, requires two separately usable factor vectors: domain generalisation is achievable with domain-invariant features, and style transfer needs a decoder. Compositional retrieval is the task that demands using one factor while excluding the other — the property CNP is meant to provide — and on that task it beats both the dual-LDA readout (0.392) and bidirectional LEACE (0.403), at 0.451. We therefore treat it as the downstream evidence, rather than a substitute for one, and we do not claim the other two.

4. Different leakage definitions for raw and decoupled representations

We agree. For a raw encoder, z_c = z_s = x, so cross-direction probe accuracy measures how much label information is present rather than cross-factor mixing. Raw DScore therefore uses the geometric overlap between the content and style LDA subspaces. After CNP, z_c and z_s occupy separate representation spaces; probe leakage then tests whether one factor is predictable from the other. These quantities have different scales and chance baselines, so their absolute DScore values are not numerically interchangeable.

We will name the two instantiations DScore_ρ for raw representations and DScore_probe for decoupled representations. We will also remove cross-regime ΔDScore values, including the mean +0.15 improvement in the abstract, and will not present the 60–86% probe reduction as a calibrated improvement over raw features.

For before/after evaluation, we use metrics with fixed definitions in both regimes: content mAP, style mAP, and kNN leakage. For CLIP on DomainNet:

representationC-mAPS-mAPkNN leakage
raw CLIP0.6660.8410.522
CLIP + CNP0.6400.8570.224

This supports the claim we can defend: CNP substantially reduces kNN style leakage and preserves useful content and style retrieval, with a modest content-retrieval trade-off. We no longer treat the mixed-definition DScore difference as a calibrated improvement.

The raw-encoder leaderboard and its rank reversal remain valid because all raw encoders use the same DScore_ρ definition. Likewise, post-CNP method comparisons use only DScore_probe.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.