acceptodds
Withdrawn from ICLR 2027

Revealing and Employing Oblique Semantic Geometry in Vision Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision foundation models learn rich representation spaces that encode a wide range of visual semantics. Yet, understanding the organization of this semantic structure often relies on predefined concepts, supervision, or training additional models. Thus, it remains unclear which semantic factors emerge directly from the representation distribution itself, and how these factors relate to one another. In this work, we first examine, in a controlled manner, the relations among semantic attributes in known datasets. We observe that similar geometry emerges across both vision-only and vision-language encoders, suggesting that this structure is largely data-driven. Since semantic directions are generally non-orthogonal, we develop a geometric framework for analyzing their organization. To recover the underlying factors without labels or semantic supervision, we seek a complete linear decomposition that separates distinct sources. Independent Component Analysis (ICA) provides a natural construction for this purpose by separating statistically independent, non-Gaussian factors through a complete and invertible transformation. Its dual representations form a biorthogonal pair, enabling precise feature disentanglement while exposing the geometry between semantic factors. We show across several measures and datasets that ICA recovers semantic factors more effectively than PCA and SAEs, with less identifiable sources being harder to recover. We further find that geometrically aligned factors form coherent semantic groups, revealing structure from fine-grained attributes to broader concepts. Finally, we employ this structure through controlled interventions, demonstrating its utility for image retrieval and mitigating spurious correlations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Independent Component Analysis, Representation Geometry, Oblique Coordinate System, Feature Disentanglement, Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R5d8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper fits a full-rank ICA (whitening followed by FastICA) on frozen embeddings and maps the components back to the native space, where the directions therefore become non orthogonal. The authors make four claims: (i) the relative geometry of semantic factors directions is shared across encoders; (ii) ICA recovers more annotated factors than PCA and SAEs; (iii) the native-space geometry of the ICA directions is semantically structured, so related concepts end up closer together; (iv) the coordinates support useful interventions (retrieval, collateral damage, Waterbirds worst-group accuracy).

Critical strengths

S1 : The paper is clearly written. The method is simple and training-free, and it gives an exactly invertible map, which makes interventions easy to define.

S2 : The negentropy analysis (Fig. 3, 9) is a sensible connection to what ICA can identify: selective, presence-like factors are recovered better than continuous ones.

S3 : The qualitative visualizations (Fig. 4, 11) are nice, although they read as illustration more than evidence.

Critical weaknesses

W1 : Novelty. ICA has already been extensively studied and shown to yield interpretable axes. The oblique/biorthogonal framing is standard linear algebra (A = P⁻¹Rᵀ, dual basis B = A⁻ᵀ), and the paper itself says obliqueness is expected after inverse whitening. The remaining contribution is the claim that the cosine structure of the latent factors in embedding space is semantically meaningful. That claim carries the paper, and it is the least supported.

W2 : Illusory structure ? The paper starts from the observation that the geometric structure of ground truth latent factors is shared across models. Since this structure is based on the correlation between mean-diff vectors, any functional correlation (label co-occurence) will naturally lead to a geometric correlation. The paper needs to show that this effect goes beyond simple functional correlation (several controls are possible here, e.g., compare the cosine matrix given by DTDD^T D (where DD is the dictionary) to the correlation matrix ZTZZ^T Z (where ZiZ_i is the label of sample ii), or take DmD_m from model mm and DrdD_{rd} from a randomly initialized model). Without proper control, the whole motivation underlying this work is null.

W3 : The central geometric claim (Table 2, Table 5) is under supported. From App. B, ATA=RΛRTA^TA = R\Lambda R^T, so the cosines between the ICA directions depend on the covariance spectrum and the rotation. In an anisotropic embedding space, I would expect related concepts to be closer under many rotations. You need a control to show that the apparent structure you find goes beyond the covariance structure and reflects the success of ICA. Take RICAR_{ICA} the rotation given by ICA and RrdR_{rd} an arbitrary rotation. Keep PP and the rest of the pipeline, and report the same statistics. Otherwise, it is impossible to tell whether the paper reports structure discovered by ICA or mere anysotropy. Section 3.3 promises controls in section 4, but the only comparison is against SAE. Direct comparison is hard to interpret due to several confounding factors.

  • SAE atoms are not trained to be geometrically organised. If anything, the top-k sparsity constraint introduces well known inductive biases, biasing atoms towards mutual repulsion, since atoms compete for the same activation. The within vs across ratio in cosim is therefore not comparable for ICA and SAE. For example, if you look at the similarity between multiclass and multilabel probes for the same classes, and SAE atoms, you will find that one case is a much better fit to the SAE’s dictionary, due to their respective inductive biases that change the geometric organisation of the atoms recovered.
  • Dictionary size is a massive confound, since finer grained features tend to be less correlated with the class average.
  • Only ratios of mean cosines are reported. Please give the raw within and across values with confidence intervals, and state whether absolute or signed cosines are used (Table 5 uses |cos|, Table 2 is unclear). The ratio might be extremely sensitive due to the small denominator.
  • Controlled references exist (mean-diff, LogReg) but are not used in this analysis.

Even if you are able to provide satisfactory controls for ICA, the existence of the same structure in SAE, plus possibly in other baselines, more comparable to ICA (dense or complete, like K-Means, NMF, SVD, etc … which might not all be exactly orthogonal in embedding space) would be a result on the hierarchical organisation of latent factors across methods than a claim about ICA itself.

W4 : Baselines are insufficient throughout. The paper claims that ICA is better than standard methods, most developed after ICA (e.g. sparse PCA, NMF, K-Means, many SAE architectures). Despite this very strong claim, the paper remains extremely lacking in its comparison to these methods …

  • On the geometric organisation of the latent factors mentioned above, it compares only to SAEs.
  • On subclass separability (figure 5, table 4), it does not compare to anything. Interpretability (Figure 4) is anecdotal and well known.
  • Semantic coverage (Table 1) is compared against PCA and SAE only.
  • Collateral damage (figure 6) is compared against supervised baselines but no other method (apart from PCA).
  • Retrieval intervention (figure 7) and spurious correlation removal (table 3) are not compared against any baseline or competing method. About comparison to SAEs : for such geometric claims at the level of the method, you need to report several architectures, not several expansion factors. The latter is part of the ablations unless you have a specific point about dimensionality. The method is what is being tested here.

W5 : Subclass separability (Fig. 5, Table 4) is anecdotal. The metric is not defined in the main text. From appendix D.4 it seems to be a binary classifier trained on the n most extreme samples of two ICs (e.g. 100 fedora and 100 beret), with manual verification of the samples. This is a much easier task than detecting a subclass within its parent class or within the whole population. It covers only four subclasses with no explanation of how they were chosen, and 0.98 -- 1.00 on easy pairs (fedora vs. beret, light aircraft vs. fighter jet) would probably be reached by almost any embedding direction. No baseline is given. No comparison to other methods is proposed.

W6 : Retrieval intervention (Fig. 1, 7, 13, 14) is anecdotal. Same as above, this would usually be a major weakness. It is a compelling narrative illustration but does not serve any scientific purpose if not backed by quantitative results


minor points :

Spurious mitigation (table 3) is undersupported ? In all 4 out of 4 columns that report stds, the effect fall within the confidence interval. You claim that for the ERM columns std is negligible. Can you still report the values ? Assuming that they are indeed negligible and the effect is statistically significative for ERM, do you have any explanation as to why it would be in ERM but not in CA and DFR ? Moreover, you do not explain the selection algorithm for the number of components to remove. The number of removed ICs and of extreme samples was selected per setting as the best-performing configuration (3 to 30 removed ICs). If this selection used test worst-group accuracy, the conclusion is null.

And again, you do not compare to any baseline or other method.

Table 1 is hard to interpret.

  • Coverage is defined only in words. As I understand it, for each annotated factor fif_i the best-aligned direction djd_j is selected, and coverage is the number of unique djd_j divided by the number of factors. Please define it formally.
  • It is unclear whether matching is done in whitened or native space, and whether the measurement direction (rows of W) or the intervention direction (columns of A) is used. App. B shows these have different geometry.
  • Matching each attribute to the best of D directions using labels inflates AUC, and a random rotation would show how much of ICA’s apparent results are just best-of-D selection.
  • It is unclear whether AUC is computed on data held out from the ICA fit, and no variance across seeds is given although FastICA is stochastic.
  • The “highest separability” claim holds for AUC, but SAEs match or beat ICA on coverage in some settings.
Questions for authors

see weaknesses.

5 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for the detailed critique and concrete experimental recommendations. We agree that standard ICA and biorthogonal algebra are not, by themselves, novel, and that the main claim requires controls beyond qualitative examples. We have substantially expanded our experiments and the manuscript to better support our claims. We address each point below.

W1. Novelty and scope of the claims.

We agree that ICA and biorthogonal linear algebra are established tools, and do not claim either as a methodological novelty. Our central contribution is the finding that semantic factors in vision foundation models exhibit a coherent, non-orthogonal geometric organization, where semantically related factors are geometrically aligned yet remain individually controllable. We demonstrate that ICA provides a complete, unsupervised biorthogonal coordinate system that reveals this organization and supports selective semantic interventions.

Importantly, the semantic organization is not merely a consequence of covariance anisotropy. We added new random-rotation controls (Table 2) that preserve these properties but exhibit substantially weaker hierarchical organization than ICA across all three encoders. This is further supported by raw within- and across-parent similarities (Table 9) and comparisons with randomly initialized encoders (Figure 10).

Beyond these geometric controls, we have substantially expanded the evaluation with additional decomposition baselines (Tables 1 and 2, Figure 6), a complementary semantic recovery evaluation (Table 4), quantitative identity-preserving retrieval (Figure 16), concept-removal baselines (Figure 17), and additional disentanglement analyses (Figures 18–19). Together, these experiments strengthen both the evidence for meaningful semantic geometry and its practical value for selective interventions. We discuss the individual additions in detail below.

W2. Could cross-encoder agreement reflect only co-occurrence or architecture?

We agree that correlations between annotated attributes contributes to the observed similarity between mean-difference RSMs. Importantly, this is a motivating observation, not evidence that the underlying factors are statistically independent. Our hypothesis is that the geometry of learned representations reflects the semantic relationships present in the data, including, but not limited to, attribute co-occurrence.

To further examine the role of architecture and training, we added randomly initialized encoder controls (Figure 10). These exhibit weaker agreement with pretrained representations than the agreement observed between pretrained encoders. The moderate correlations (r=0.50r=0.50 for CLIP and r=0.53r=0.53 for DINOv2) indicate that data and architectural biases contribute to this structure, while representation learning further reinforces it. Thus, while attribute co-occurrence may contribute to the shared geometry, the stronger agreement among pretrained encoders suggests that this structure is not solely explained by data correlations or architectural biases.

AuthorsOpenReview ↗

Response 2

W3. Is hierarchical geometry attributable to ICA rather than anisotropy?

We thank the reviewer for suggesting this important control. We implemented the proposed random-rotation baseline (RandRot), keeping the whitening and inverse-whitening transformations fixed while replacing the ICA rotation with a random orthogonal rotation. We repeat the experiment three times and report the best result, providing an optimistic comparison for this control. In Table 2, ICA achieves within/across-parent similarity ratios of 2.82/5.72/4.05 across CLIP/DINO/I-JEPA, compared with 1.64/1.21/1.26 for RandRot. We additionally report the raw mean absolute cosine similarities (Table 9), showing that ICA consistently exhibits both higher within-parent and lower across-parent alignment than RandRot. These results directly demonstrate that embedding covariance and anisotropy alone cannot explain the hierarchical organization recovered by ICA.

We further expanded the evaluation to include Sparse PCA, NMF, and K-Means, alongside SAE baselines. While K-Means achieves a particularly high hierarchical ratio on DINO, its performance is substantially weaker on CLIP and I-JEPA. In contrast, ICA consistently reveals strong hierarchical organization across all three encoders. We also added supervised MeanDiff and LogReg references (Table 10). ICA exhibits stronger hierarchical organization than LogReg on CLIP and DINO, while remaining below MeanDiff. Importantly, these supervised methods provide useful references rather than direct competitors to unsupervised discovery methods such as ICA.

We recognize that different decomposition methods have distinct inductive biases affecting their geometric organization. Therefore, our central evidence comes from the covariance-controlled ICA–RandRot comparison, while the broader baselines establish the consistency and relative strength of the recovered semantic structure.

Table 2 (in the revised paper)

ModelICARandRotPCASPCANMFK-MeansSAE-DSAE-2DSAE-4DSAE-8DSAE-16D
CLIP2.8211.637--0.1881.0661.2521.5041.4671.3261.2431.203
DINO5.7211.206--1.3081.0049.3631.2791.4391.4541.3301.220
I-JEPA4.0461.255--0.2711.0771.0391.1391.0811.0741.0441.029

Table 9 (in the revised paper):

MethodCLIP WithinCLIP AcrossDINO WithinDINO AcrossI-JEPA WithinI-JEPA Across
ICA0.2280.0810.1290.0230.2030.050
RandRot0.1650.1010.0730.0610.0930.074
PCA0.0000.0000.0000.0000.0000.000
Sparse PCA0.0010.0050.0000.0000.0000.000
NMF0.1830.1720.1530.1530.1730.161
K-Means0.8790.7020.4680.0500.9590.923
SAE-D0.0520.0350.0370.0290.0270.024
SAE-2D0.0470.0320.0400.0280.0250.023
SAE-4D0.0420.0320.0390.0270.0240.023
SAE-8D0.0380.0300.0340.0260.0230.022
SAE-16D0.0360.0300.0310.0250.0230.022

Table 10 (in the revised paper):

ModelICA WithinICA AcrossICA RatioMeanDiff WithinMeanDiff AcrossMeanDiff RatioLogReg WithinLogReg AcrossLogReg Ratio
CLIP0.2280.0812.8210.6230.1683.7010.1770.0692.573
DINO0.1290.0235.7210.2470.0357.1520.1400.0373.799
I-JEPA0.2030.0504.0460.5860.0946.2620.3080.0575.362
AuthorsOpenReview ↗

Response 3

W4 and W6. Breadth and fairness of baselines.

We substantially expanded the comparison suite. The revised factor-recovery evaluation (Table 1) now includes RandRot, Sparse PCA, NMF, K-Means, and an additional SAE architecture (SAE-Best), alongside PCA and the original SAE variants. The geometric analysis (Tables 2 and 9) also includes these additional decomposition methods. Importantly, the semantic intervention analysis (Figure 6, 18) demonstrates that ICA achieves substantially lower collateral semantic drift than competing unsupervised decompositions and supervised reference directions, approaching the zero-drift oracle. We further added LEACE and SpLiCE as concept-removal baselines (Figure 17), where ICA again achieves stronger preservation of non-target semantics, and quantitative retrieval comparisons against PCA and SAEs (Figure 16), demonstrating improved identity preservation.

Importantly, the expanded baselines reveal a distinction between recovering many apparently distinct concepts and recovering semantically predictive factors. For example, K-Means achieves the highest raw coverage in Table 1, but with relatively low AUC. Under the stricter evaluation in the new Table 4, which requires distinct directions with AUC ≥ 0.8, its coverage decreases substantially, whereas ICA achieves the highest thresholded coverage across all six dataset and encoder settings. This demonstrates that ICA combines broad semantic coverage with strong separability. These additions provide a substantially broader comparison across semantic recovery, geometric organization, selective interventions, and identity preservation.

Table 1 (in the revised paper):

MethodCelebA CLIP Cov. (%)CelebA CLIP AUCCelebA DINO Cov. (%)CelebA DINO AUCCelebA I-JEPA Cov. (%)CelebA I-JEPA AUCMSCOCO CLIP Cov. (%)MSCOCO CLIP AUCMSCOCO DINO Cov. (%)MSCOCO DINO AUCMSCOCO I-JEPA Cov. (%)MSCOCO I-JEPA AUC
ICA70.00.79870.00.77362.50.80473.80.93183.80.90273.80.878
RandRot57.50.76557.50.76765.00.78563.80.87676.20.85865.00.843
PCA20.00.73610.00.7417.50.75115.00.82736.20.85012.50.783
SPCA22.50.73610.00.7427.50.75113.80.82437.50.85212.50.783
NMF12.50.73025.00.74110.00.75541.20.85865.00.87147.50.806
K-Means85.00.75082.50.73977.50.74587.50.87392.50.91376.20.839
SAE-D62.50.77430.00.76815.00.73363.80.90678.80.85733.80.796
SAE-2D60.00.78037.50.76622.50.74166.20.91677.50.87632.50.829
SAE-4D72.50.73532.50.73920.00.74265.00.91680.00.87038.80.812
SAE-8D67.50.75440.00.74825.00.76572.50.91777.50.87441.20.833
SAE-16D65.00.76850.00.75717.50.76367.50.92076.20.87350.00.828
SAE-Best-D64.30.76645.80.75837.50.78572.50.93576.60.87271.60.853
SAE-Best-4D71.80.78961.80.76240.80.79573.80.93483.40.88772.90.860

Table 4 (in the revised paper):

MethodCelebA CLIP Cov. (%)CelebA CLIP AUCCelebA DINO Cov. (%)CelebA DINO AUCCelebA I-JEPA Cov. (%)CelebA I-JEPA AUCMSCOCO CLIP Cov. (%)MSCOCO CLIP AUCMSCOCO DINO Cov. (%)MSCOCO DINO AUCMSCOCO I-JEPA Cov. (%)MSCOCO I-JEPA AUC
ICA65.00.81857.50.81060.00.81796.20.93993.80.91688.80.890
RandRot45.00.78050.00.79055.00.79386.20.89178.80.86871.20.842
PCA32.50.74025.00.73742.50.75270.00.84670.00.86050.00.790
SPCA30.00.73630.00.74142.50.75267.50.84372.50.86350.00.793
NMF42.50.75345.00.77445.00.77280.00.87781.20.88362.50.831
K-Means37.50.76142.50.77050.00.76881.20.88692.50.92371.20.846
SAE-D57.50.80255.00.80440.00.76686.20.91481.20.87752.50.806
SAE-2D57.50.80455.00.80545.00.76990.00.92381.20.88265.00.836
SAE-4D52.50.78255.00.81045.00.77587.50.92280.00.88560.00.820
SAE-8D47.50.79955.00.79952.50.78291.20.92381.20.88771.20.841
SAE-16D52.50.79447.50.79350.00.78590.00.92283.80.88968.80.837
Reviewer_H7NaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates the geometric organization of semantic factors in pretrained vision foundation models and proposes a biorthogonal representation framework based on Independent Component Analysis. The authors apply whitening and FastICA to frozen image embeddings to recover a complete and invertible coordinate system, where individual coordinates represent potentially meaningful semantic factors. A biorthogonal formulation separates measurement from intervention axes and it makes it possible to manipulate coordinates in a more precise manner. The authors evaluate the semantic disentanglement of coordinates using annotated attributes and measure semantic separability, coverage, and selectivity. The proposed framework is evaluated on multiple datasets (CelebA, MS-COCO, ImageNet-100) using representations from CLIP, DINOv2, and I-JEPA. ICA-based representation outperforms PCA and SAE in semantic separability and selectivity.

During the rebuttal, the author's responses including experimental results on semantic presentation and the clarification on evaluation procedures address my major concerns. I think the paper has merit in terms of improving understanding geometric organization and semantic properties of vision foundation models.

Critical strengths
  • The paper provides a clear distinction between algebraic coordinate disentanglement and empirical semantic disentanglement. Experiments (including analysis of non-target semantic drift and comparisons) offer evidence of semantic selectivity.

  • ICA-based representations outperform baselines including PCA, NMF, and SAE in semantic separability and selectivity across multiple datasets.

  • Experimental results also show that selective suppression of factors can preserve image identity, which verifies practical value of method.

  • Clarification regarding ImageNet-100 evaluation addresses concerns on sample-selection bias.

Critical weaknesses
  • The method relies on standard ICA. The contribution of method is mostly in demonstrating and evaluating sematic properties of ICA-based biorhogonal representation (not introducing new ICA algorithm).

  • The subclass separability experiments select both training and test samples based on extreme ICA activations, potentially inflating the scores. Moreover, the disentanglement evaluation relies on mean-difference directions as semantic ground truth, which may capture correlated attributes rather than independent factors.

Questions for authors
  • Do improvements in geometric disentanglement correspond to actual semantic preservation?
  • What is the precise meaning of semantic independence and completeness in the proposed framework?
  • Can the authors clarify the semantic matching and evaluation protocol used in Table 1?
  • Does the subclass separability performance generalize beyond extreme-activation samples?
5 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for the thoughtful assessment and constructive feedback. We have substantially revised the manuscript and added quantitative experiments to better establish the semantic structure and operational properties of the proposed framework. We address each concern below.

W1. Novelty and evidence of semantic disentanglement.

We agree that invertibility alone does not guarantee semantic disentanglement, and distinguish between exact algebraic disentanglement of the recovered factors and empirical semantic disentanglement of annotated features. As derived in Appendix B, illustrated in Figure 8, and validated in Figure 19, the biorthogonal formulation guarantees that modifying one coordinate leaves all others unchanged. This property can be achieved for any linearly independent set of semantic directions, including the 40 CelebA attributes, yielding an oracle with zero collateral drift (Figure 6). The central challenge is therefore recovering semantically disentangled coordinates without supervision, rather than constructing the biorthogonal system itself. We evaluate this directly: Table 1 (expanded) and Table 4 (new) show that ICA coordinates capture distinct, meaningful semantic factors, while Figure 6 (with additional extended baselines for comparisons, and the new Figures 17, 18) demonstrates that manipulating these coordinates induces substantially less non-target semantic drift than competing unsupervised decompositions and supervised reference directions, approaching the oracle. Together, these results provide quantitative evidence of stronger semantic disentanglement relative to other approaches, beyond the algebraic guarantees of the framework. Our contribution is not a new ICA algorithm, but demonstrating that ICA can recover a semantically meaningful biorthogonal system whose geometry and independent coordinates support selective semantic manipulation.

W2 and Q4. Subclass selection and reliance on mean-difference directions.

We acknowledge that selecting extreme-activation test samples makes subclass discrimination easier. We have therefore clarified that experiment serves as a qualitative illustration of fine-grained factors (Figure 5), while the broader evidence for hierarchical organization comes from the quantitative ImageNet-100 evaluation (Table 2), where there is known benchmark separation between validation and test data. Notably, while K-Means achieves a particularly high hierarchical ratio for DINO, its performance is substantially weaker for CLIP and I-JEPA. In contrast, ICA consistently exhibits strong hierarchical organization across all three encoders, demonstrating that the observed structure is not specific to a particular representation model. Table 9 further reports the raw within- and across-parent similarities, supporting this finding beyond the ratios alone.

Table 2:

ModelICARandRotPCASPCANMFK-MeansSAE-DSAE-2DSAE-4DSAE-8DSAE-16D
CLIP2.8211.637--0.1881.0661.2521.5041.4671.3261.2431.203
DINO5.7211.206--1.3081.0049.3631.2791.4391.4541.3301.220
I-JEPA4.0461.255--0.2711.0771.0391.1391.0811.0741.0441.029

Table 9:

MethodCLIP WithinCLIP AcrossDINO WithinDINO AcrossI-JEPA WithinI-JEPA Across
ICA0.2280.0810.1290.0230.2030.050
RandRot0.1650.1010.0730.0610.0930.074
PCA0.0000.0000.0000.0000.0000.000
Sparse PCA0.0010.0050.0000.0000.0000.000
NMF0.1830.1720.1530.1530.1730.161
K-Means0.8790.7020.4680.0500.9590.923
SAE-D0.0520.0350.0370.0290.0270.024
SAE-2D0.0470.0320.0400.0280.0250.023
SAE-4D0.0420.0320.0390.0270.0240.023
SAE-8D0.0380.0300.0340.0260.0230.022
SAE-16D0.0360.0300.0310.0250.0230.022

Mean-difference directions are used in semantic coverage and separability experiments and in intervention evaluation. We emphasize that mean-difference directions are used only as semantic evaluation probes, not as ground-truth independent factors or as the coordinates used for intervention. They provide a consistent, label-based measure of target suppression and non-target semantic drift across all methods (Figure 6). While annotated attributes may be correlated, this does not invalidate their use as evaluation probes, however, it does limit the interpretation to selectivity with respect to annotated semantics. Importantly, ICA directions and interventions are obtained without using these attribute labels.

AuthorsOpenReview ↗

Response 2

Q1. Do improvements in geometric disentanglement correspond to actual semantic preservation?

Yes. Our biorthogonal formulation guarantees exact algebraic disentanglement, where modifying one coordinate leaves all others unchanged (illustrated in the new Figure 8 and empirically demonstrated in the new figure 19). However, semantic preservation depends on how well the recovered coordinates align with human based semantic factors. We evaluate this empirically in Figure 6, measuring non-target attribute drift under matched target interventions. ICA achieves the closest approximation to the zero-drift oracle, outperforming alternative decompositions and supervised reference directions. Additional comparisons with SAEs (new Figure 18) and with LEACE and SpLiCE (Figure 17) confirm stronger preservation of unrelated semantics. Finally, quantitative controlled retrieval experiments (Figure 16) demonstrate that suppressing nuisance factors using ICA also better preserves image identity, showing that the improved semantic selectivity translates into a practical downstream benefit.

Q2. What is the precise meaning of semantic independence and completeness?

We distinguish algebraic independence from semantic disentanglement. Completeness means that all embedding dimensions are retained, yielding an exactly invertible transformation. Biorthogonality ensures that modifying one coordinate leaves the remaining recovered coordinates unchanged. This does not imply that the corresponding human-interpretable semantics are statistically or causally independent. For this we empirically evaluate semantic selectivity and show that these interventions largely preserve unrelated attributes (Figure 6, 16, 17).

Q3. Can the authors clarify the semantic matching and evaluation protocol used in Table 1?

ICA and the competing decompositions are fitted without semantic labels. Discovered directions are subsequently matched to annotated attributes using a validation split and their predictive separability is evaluated using AUC on held-out test data. Coverage measures the fraction of annotated attributes assigned to distinct discovered directions. We have formalized this definition in Section 4.2, enhances the experimental setting description in Appendix D.3 and added Table 4, which reports the fraction of attributes represented by distinct features achieving validation AUC ≥ 0.8.

Table 4:

MethodCelebA CLIP Cov. (%)CelebA CLIP AUCCelebA DINO Cov. (%)CelebA DINO AUCCelebA I-JEPA Cov. (%)CelebA I-JEPA AUCMSCOCO CLIP Cov. (%)MSCOCO CLIP AUCMSCOCO DINO Cov. (%)MSCOCO DINO AUCMSCOCO I-JEPA Cov. (%)MSCOCO I-JEPA AUC
ICA65.00.81857.50.81060.00.81796.20.93993.80.91688.80.890
RandRot45.00.78050.00.79055.00.79386.20.89178.80.86871.20.842
PCA32.50.74025.00.73742.50.75270.00.84670.00.86050.00.790
SPCA30.00.73630.00.74142.50.75267.50.84372.50.86350.00.793
NMF42.50.75345.00.77445.00.77280.00.87781.20.88362.50.831
K-Means37.50.76142.50.77050.00.76881.20.88692.50.92371.20.846
SAE-D57.50.80255.00.80440.00.76686.20.91481.20.87752.50.806
SAE-2D57.50.80455.00.80545.00.76990.00.92381.20.88265.00.836
SAE-4D52.50.78255.00.81045.00.77587.50.92280.00.88560.00.820
SAE-8D47.50.79955.00.79952.50.78291.20.92381.20.88771.20.841
SAE-16D52.50.79447.50.79350.00.78590.00.92283.80.88968.80.837

We sincerely appreciate the reviewer's constructive feedback, which has helped us substantially strengthen the manuscript. We believe the revised experiments provide evidence for the strong semantic disentanglement and hierarchical organization of the recovered factors, while directly addressing the concerns regarding evaluation protocols and semantic preservation. Given these substantial additions and clarifications, we hope the reviewer will reconsider their assessment and update their score accordingly. We thank the reviewer again for their time and effort dedicated to this review.

Reviewer_H7NaOpenReview ↗

Follow-up question

Thank the authors for their efforts to address previous concerns. Most of them have been addressed well, and I have one more point for clarification: Could the authors clarify whether the ImageNet-100 evaluation uses all validation/test samples without any filtering (e.g., using ICA activations)?

Reviewer_c2zAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper applies Independent Component Analysis (ICA) to embedding spaces from different vision models to find disentangled semantic factors without supervision. The main novelty is that mapping the factors back to the original space leads to semantically meaningful distances, and this can be used to erase specific concepts for retrieval. The experiments show that class-aggregated embeddings have similar cosine similarities across vision models (Exp. 4.1), ICA factors are often semantically interpretable (Exp. 4.2), mapped-back factors are structured geometrically meaningfully (Exp. 4.3), and some factors can be edited independently without changing other factors a lot for retrieval (Exp. 4.4).

The new experiments during the rebuttal look promising, but the main claims still need deeper study and a clearer focus/structure, so I maintain my weak rejection.

Critical strengths

The method is simple and performs better than Sparse Autoencoders and PCA in their evaluated settings.

Critical weaknesses

Most of the experiments just confirm established findings. Exp. 4.1 confirms that angles are similar across modalities, which was established by the Platonic Representation Hypothesis (Huh et al. 2024) and surrounding work. Exp. 4.2 confirms that ICA components correspond to semantic factors: “Prior work has shown that individual ICA components can capture interpretable semantics in text and image embeddings (Yamagiwa et al., 2023), as well as in language-model activations (Liu & Han, 2026). We similarly find that many recovered components in vision foundation models correspond to coherent semantic factors” (ll. 70-74). Similar to Exp. 4.4, ICA Lens (Liu et al. 2026, E.4) already shows concept removal with a fitted ICA inverse for language models. The main remaining novelty seems to be the semantic relation between recovered factors (Exp. 4.3) and that concepts can also be removed with ICA for vision models (Exp. 4.4). However, the evidence in Exp. 4.3 is limited and Exp. 4.4 doesn’t compare to previous methods. For Exp. 4.3, one qualitative experiment is shown involving “cowboy hat” and “cap”. For two categories (”Wearing Hat” and “airplane”), subclasses can be classified with a linear classifier. Also, directions associated with the same superclass are closer than directions across superclasses on ImageNet-100. However, it is not clear whether this comes from ICA or the underlying geometry from the embeddings. In Exp. 4.4, it is shown that a concept can be removed without changing other concepts a lot. These results should be compared to other methods like LEACE (Belrose et al. 2023), SpLiCE (Bhalla et al. 2024), etc.

Huh et al. 2024: “The Platonic Representation Hypothesis”

Liu et al. 2026: “ICA Lens: Interpreting Language Models Without Training Another Dictionary”

Belrose et al. 2023: “LEACE: Perfect linear concept erasure in closed form”

Bhalla et al. 2024: “Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)”

Questions for authors
  1. One control to check if results from Exp. 4.3 come from ICA or the underlying geometry could be to evaluate a random rotation in the whitened space followed by the same inverse whitening. How does such a random rotation perform in the same setting as Exp. 4.3?
  2. How does the method compare to other concept removal methods in Exp. 4.4?
  3. “These methods either rely on human-defined semantics or use them to organize and interpret the discovered concepts. We instead recover semantic factors without supervision, directly from the pre-trained embedding distribution.” (ll. 125 - 127). The method introduced in the paper also uses human-defined semantics to interpret the discovered concepts. Other methods (Ghorbani et al., 2019; Fel et al., 2023) also recover semantic factors without supervision from the embeddings. What is the explicit difference from those other methods?
  4. In Sec. 4.2, it is stated that “ICA consistently achieves strong coverage and the highest separability” (ll. 393 - 394). Coverage measures whether concepts select different features. However, it doesn’t measure how strong each match is, and even weak matches count toward increased coverage if they correspond to different semantic directions. AUC might overlook predictive SAE features because they are selected by angular similarity and not predictability. To solve this, one could design an additional experiment by selecting each concept’s best feature using validation AUC and evaluating its activation on held-out test data. One could then report how many concepts have distinct features that are higher than a predefined performance threshold. How does SAE compare to iCA for such an experiment?
4 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for the thoughtful assessment and the concrete suggestions. We agree that the individual interpretability of ICA components, the ICA transform, and its algebraic invertibility are established. We have revised the paper to clarify that our contribution is the semantic organization of the recovered directions in the native representation space, together with quantitative evidence that this structure supports selective interventions. We also added the requested controls and comparisons. Below we respond to each question in details.

Q1. Does the geometry arise from ICA or from the embedding covariance?

Thank you for suggesting the decisive control. We added RandRot, which replaces the ICA rotation by a random orthogonal rotation in the whitened space while keeping the whitening and inverse-whitening transformations unchanged. To provide a conservative comparison, we repeat the random-rotation procedure 3 times and report the best-performing rotation for each encoder. In the ImageNet-100 hierarchical experiment (revised Table 2), ICA achieves within/across-parent cosine ratios of 2.82, 5.72, and 4.05 for CLIP, DINO, and I-JEPA, compared with 1.64, 1.21, and 1.26 for RandRot. Revised Table 9 also reports the raw within- and across-parent similarities. Thus, the covariance anisotropy may contribute, but alone does not explain the semantic organization recovered by ICA.

Table 2 (in the revised paper):

ModelICARandRotPCASPCANMFK-MeansSAE-DSAE-2DSAE-4DSAE-8DSAE-16D
CLIP2.8211.637--0.1881.0661.2521.5041.4671.3261.2431.203
DINO5.7211.206--1.3081.0049.3631.2791.4391.4541.3301.220
I-JEPA4.0461.255--0.2711.0771.0391.1391.0811.0741.0441.029

Table 9 (in the revised paper):

MethodCLIP WithinCLIP AcrossDINO WithinDINO AcrossI-JEPA WithinI-JEPA Across
ICA0.2280.0810.1290.0230.2030.050
RandRot0.1650.1010.0730.0610.0930.074
PCA0.0000.0000.0000.0000.0000.000
Sparse PCA0.0010.0050.0000.0000.0000.000
NMF0.1830.1720.1530.1530.1730.161
K-Means0.8790.7020.4680.0500.9590.923
SAE-D0.0520.0350.0370.0290.0270.024
SAE-2D0.0470.0320.0400.0280.0250.023
SAE-4D0.0420.0320.0390.0270.0240.023
SAE-8D0.0380.0300.0340.0260.0230.022
SAE-16D0.0360.0300.0310.0250.0230.022

Q2. Comparison with concept-removal methods.

We added direct comparisons with LEACE and SpLiCE (new Figure 17). At matched magnitudes of target-concept suppression, ICA consistently produces lower drift in non-target semantics. For a target change of −0.15, ICA achieves non-target drift of approximately 0.0007 on CelebA, compared with 0.00125 for LEACE and 0.00185 for SpLiCE, representing reductions of ~40% - 60%, respectively. On MSCOCO, ICA achieves approximately 0.00115, compared with 0.0030 and 0.00315, corresponding to reductions of ~ 60%. The main intervention study (Figure 6) also includes alternative decompositions and supervised reference directions, and ICA remains the method with the least affect on non-target factors. Together, these results demonstrate ICA's effectiveness in selective semantic control.

Q3. Difference from prior unsupervised concept-discovery methods.

We agree that unsupervised concept discovery is not unique to our approach, and have clarified this distinction in the related-work discussion (Lines 130–132). Our contribution goes beyond discovering individual semantic factors in two directions. First, we study the geometric organization of the recovered factors in the native embedding space, revealing meaningful relations between them. Second, we exploit the complete, invertible biorthogonal coordinate system derived using ICA to enable precise manipulation of individual semantic factors while preserving unrelated semantics. We have further emphasized these distinct contributions in the revised abstract and introduction.

AuthorsOpenReview ↗

Response 2

Q4. Coverage, feature selection, and held-out separability.

We added the suggested complementary evaluation (new Table 4): for each concept, we select its most predictive feature using validation AUC and evaluate it on held-out test data. We then count only concepts represented by distinct features with validation AUC ≥ 0.8. We also expanded Table 1 with RandRot, Sparse PCA, NMF, K-Means, and an additional SAE architecture, and formalized the coverage definition. The new evaluation reveals that high raw coverage does not necessarily imply strong semantic recovery. This is particularly evident for K-Means, which achieves the highest raw coverage despite relatively low AUC, but its coverage drops substantially under the stricter evaluation. In contrast, ICA consistently achieves the highest thresholded coverage across all six settings, demonstrating that its recovered directions combine semantic diversity with strong predictive separability. These findings directly address the reviewer's concern that raw coverage may count weak semantic matches.

Table 1 (in the revised paper):

MethodCelebA CLIP Cov. (%)CelebA CLIP AUCCelebA DINO Cov. (%)CelebA DINO AUCCelebA I-JEPA Cov. (%)CelebA I-JEPA AUCMSCOCO CLIP Cov. (%)MSCOCO CLIP AUCMSCOCO DINO Cov. (%)MSCOCO DINO AUCMSCOCO I-JEPA Cov. (%)MSCOCO I-JEPA AUC
ICA70.00.79870.00.77362.50.80473.80.93183.80.90273.80.878
RandRot57.50.76557.50.76765.00.78563.80.87676.20.85865.00.843
PCA20.00.73610.00.7417.50.75115.00.82736.20.85012.50.783
SPCA22.50.73610.00.7427.50.75113.80.82437.50.85212.50.783
NMF12.50.73025.00.74110.00.75541.20.85865.00.87147.50.806
K-Means85.00.75082.50.73977.50.74587.50.87392.50.91376.20.839
SAE-D62.50.77430.00.76815.00.73363.80.90678.80.85733.80.796
SAE-2D60.00.78037.50.76622.50.74166.20.91677.50.87632.50.829
SAE-4D72.50.73532.50.73920.00.74265.00.91680.00.87038.80.812
SAE-8D67.50.75440.00.74825.00.76572.50.91777.50.87441.20.833
SAE-16D65.00.76850.00.75717.50.76367.50.92076.20.87350.00.828
SAE-Best-D64.30.76645.80.75837.50.78572.50.93576.60.87271.60.853
SAE-Best-4D71.80.78961.80.76240.80.79573.80.93483.40.88772.90.860

Table 4 (in the revised paper):

MethodCelebA CLIP Cov. (%)CelebA CLIP AUCCelebA DINO Cov. (%)CelebA DINO AUCCelebA I-JEPA Cov. (%)CelebA I-JEPA AUCMSCOCO CLIP Cov. (%)MSCOCO CLIP AUCMSCOCO DINO Cov. (%)MSCOCO DINO AUCMSCOCO I-JEPA Cov. (%)MSCOCO I-JEPA AUC
ICA65.00.81857.50.81060.00.81796.20.93993.80.91688.80.890
RandRot45.00.78050.00.79055.00.79386.20.89178.80.86871.20.842
PCA32.50.74025.00.73742.50.75270.00.84670.00.86050.00.790
SPCA30.00.73630.00.74142.50.75267.50.84372.50.86350.00.793
NMF42.50.75345.00.77445.00.77280.00.87781.20.88362.50.831
K-Means37.50.76142.50.77050.00.76881.20.88692.50.92371.20.846
SAE-D57.50.80255.00.80440.00.76686.20.91481.20.87752.50.806
SAE-2D57.50.80455.00.80545.00.76990.00.92381.20.88265.00.836
SAE-4D52.50.78255.00.81045.00.77587.50.92280.00.88560.00.820
SAE-8D47.50.79955.00.79952.50.78291.20.92381.20.88771.20.841
SAE-16D52.50.79447.50.79350.00.78590.00.92283.80.88968.80.837

We thank the reviewer again for the thoughtful feedback and valuable suggestions, which have helped us meaningfully improve the paper. We believe the additional experiments and clarifications directly address the reviewer's main concerns and provide stronger support for our contributions. We hope these revisions will encourage the reviewer to positively reconsider their evaluation, and we appreciate their time and consideration.

Thank you again for your feedback. We have addressed your comments and questions and would appreciate your reassessment of the revised paper. Please let us know if any concerns remain or further clarifications are needed.

Reviewer_c2zAOpenReview ↗

Promising new results, but the paper still lacks focus and depth on its main claims

Thank you for the detailed response and for the additional experiments. The results look promising, especially the comparisons with random rotations, LEACE, and SpLiCE. However, the paper is not focused enough in its current state. The authors agree that their contribution is to identify the “semantic organization” of mapped-back ICA directions and to use it for “selective intervention”, but the experiments corresponding to these contributions are only one of the eight pages in the paper. “Semantic organization” is assessed only by comparing within- and between-class cosine similarities on ImageNet-100. These results are interesting but support a class hierarchy rather than a general semantic organization. “Selective intervention” is an established field, with many methods and benchmarks for evaluation. During the rebuttal, the authors compare against two baselines, but many other baselines and evaluation settings (e.g., MANCE, INLP) are missing, even though this is one of their main contributions. Meanwhile, Sec. 3.1, Sec. 4.1, and Sec. 4.2 are discussed in detail while mainly repeating results from previous work. Altogether, the paper needs to be restructured significantly to explore and emphasize its two main contributions more while cutting unnecessary parts. Therefore, I keep my weak reject score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.