acceptodds
Under review as a conference paper at ICLR 2027

Fisher-IRG: Fisher-Induced Local Invariant Representation Geometry across Language and Vision Models

Abstract

Semantic-preserving transformations can induce substantial motion in learned representations, while small changes may strongly affect model predictions, raising a basic question: what local metric best captures semantically consequential variation? We propose Fisher-induced invariant representation geometry (Fisher-IRG), which measures local representation directions through their predictive sensitivity. Around each representation, we construct semantic-preserving and semantic-changing neighborhoods, aggregate their local Fisher information, and recover invariant directions through a contrastive generalized eigenvalue problem. Controlled displacement analyses first show that comparable Euclidean motion can have substantially different predictive consequences, supporting the need for a predictive geometry. Across language and vision models, Fisher-IRG yields stronger semantic-versus-nuisance predictive selectivity and generally more reproducible subspaces than covariance-based geometry, while recovering systematically distinct local directions. Representation interventions further localize semantic effects to the Fisher-derived subspace, and held-out separation and retrieval show that the recovered geometry generalizes beyond the discovery neighborhoods. These results support Fisher-IRG as a principled framework for characterizing local invariant representation geometry.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Invariant representation learning, Fisher information, information geometry, representation geometry, semantic invariance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BNZdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces Fisher-IRG, a framework for analyzing local invariant representation geometry in pretrained language and vision models. The key idea is to replace covariance-based characterization of representation displacement with a Fisher-information-based geometry that measures how local representation directions affect the model’s predictive distribution. Given semantic-preserving (SP) and semantic-changing (SC) neighborhoods around an anchor, the method aggregates local Fisher operators for the two types of variations and solves a regularized generalized eigenvalue problem to identify directions that are more predictive-sensitive to semantic changes than to meaning-preserving variations.

Critical strengths
  1. The motivation is interesting, although it could be further strengthened.
  2. The work addresses a fundamental representation-analysis problem while making an effort to connect with modern large language and vision models.
  3. The empirical evaluation covers multiple models and several validation aspects.
Critical weaknesses
  1. The primary evaluation criterion is closely aligned with the objective used to construct the Fisher-IRG subspace. Since Fisher-IRG explicitly selects directions according to predictive sensitivity to semantic-changing versus semantic-preserving variation, evaluating the resulting subspace using the same predictive KL-based criterion provides limited independent evidence of its superiority over Cov-IRG.
  2. The methodological novelty appears limited. The proposed framework largely follows the existing covariance-based invariant representation analysis paradigm, while replacing covariance-based representation geometry with a Fisher-based predictive geometry and retaining a similar generalized eigenvalue formulation. Although this replacement is conceptually reasonable, the paper does not sufficiently demonstrate that it constitutes a substantial methodological advance. Moreover, the improvements on more independent evaluations, such as held-out semantic retrieval, are relatively modest, making the practical benefit of the added complexity less compelling.
  3. The validity of the semantic-preserving / semantic-changing neighborhoods is insufficiently justified. The entire framework relies heavily on the assumption that the constructed SP and SC variants cleanly isolate meaning-preserving and meaning-changing variations. However, the paper provides limited independent validation that these variants are free from confounding factors such as lexical choice, syntax, sentence structure, visual appearance, or other low-level cues. Since the Fisher operator directly measures predictive sensitivity, such confounds could substantially influence the recovered subspaces. Stronger human validation or confound-controlled experiments would be necessary to support the claim that the discovered geometry genuinely reflects semantic structure.
  4. The empirical evaluation is largely confined to synthetic or tightly controlled local semantic neighborhoods constructed specifically for this analysis. While this setup is useful for isolating the proposed geometric effect, it remains unclear whether Fisher-IRG provides meaningful benefits on realistic downstream representation-learning or semantic understanding tasks.
  5. The paper suffers from significant presentation and visualization issues. For example, Fig. 1 is visually cluttered and does not clearly convey the key methodological intuition. The overall formatting and visual presentation of the manuscript could also be substantially improved.
Questions for authors

See the weaknesses above, especially 1-4.

6 comments

We thank the reviewer for recognizing the importance of the representation-analysis problem and the breadth of our language and vision evaluations. We appreciate the concerns regarding methodological novelty, evaluation independence, semantic-neighborhood validity, and practical utility. In response, we have substantially clarified the contributions and their theoretical scope, added independent semantic evaluations, and expanded the evidence for neighborhood validity and functional relevance. We address the concerns individually below.

W1. Evaluation circularity: Does the predictive-KL criterion provide independent evidence?

We agree that KL-based selectivity is aligned with the Fisher discovery objective and should not, by itself, establish semantic superiority. We have clarified the distinct roles of our evaluations and strengthened the evidence using criteria that are not optimized during Fisher subspace discovery.

In particular, the experiments establish different properties:

  • Predictive selectivity: Whether Fisher-IRG identifies directions with greater semantic-versus-nuisance predictive sensitivity, as intended by its geometric construction.
  • Grassmann reproducibility and Fisher–Cov geometry: Whether the recovered subspaces are reproducible under resampling and geometrically distinct from covariance-derived subspaces.
  • Held-out semantic discrimination and retrieval: Whether the resulting representations preserve useful semantic distinctions under cosine-distance evaluations, rather than the predictive-KL objective.

The third category provides more independent functional evidence. Fisher-IRG achieves the largest held-out semantic separation margin in 21/21 language model–layer comparisons and 9/9 vision layer–difficulty comparisons. In seven-model semantic retrieval, its macro-average pairwise accuracy improves from 0.949 to 0.959, while Hit@1 improves from 0.873 to 0.885 relative to the original representations.

We additionally introduced paired bootstrap analyses with 2,000 replicates. The 95% confidence intervals for the aggregate improvements exclude zero, although several individual-model comparisons remain statistically inconclusive.

To further reduce dependence on the original generated neighborhoods, we added semantic matching on SNLI, transferring Fisher-derived subspaces learned from synthetic discovery data without re-estimation on SNLI labels, as well as instance retrieval under visual corruptions. These provide additional evidence under independently defined semantic or instance-matching criteria.

We therefore interpret KL-based selectivity as validation of the intended predictive geometry, while treating cosine-based discrimination and retrieval as complementary evidence of semantic utility.

Manuscript Change: Sections 4.3–4.6; Appendix G.1, Statistical Reliability of Semantic Retrieval, Tables 28–29; Appendix G.3–G.4, natural semantic and visual retrieval evaluations; Appendix I.

W2. Methodological novelty: Is Fisher-IRG merely covariance replacement in an existing generalized eigenproblem?

The novelty is not the Fisher matrix or the generalized eigenproblem individually, but their integration into a contrastive framework for discovering local predictive-selective representation subspaces. We have revised the introduction and contributions to make this distinction explicit.

Our contribution follows three connected steps.

First, we establish the motivating geometric phenomenon. Controlled distance-matched SP/SC experiments show that comparable Euclidean displacements can have substantially different predictive consequences. This establishes an existence claim: displacement magnitude alone does not always determine the behavioral significance of representation variation.

Second, we formulate a different object of geometric measurement and make it computationally operational. Cov-IRG analyzes the directional covariance of sampled representation displacements. Fisher-IRG instead measures local predictive sensitivity through the second-order KL geometry,

DKL(p(⋅∣z)∥p(⋅∣z+δz))=12δz⊤F(z)δz+o(∥δz∥22).D_{\mathrm{KL}}(p(\cdot\mid z)\Vert p(\cdot\mid z+\delta z)) = \frac{1}{2}\delta z^\top F(z)\delta z + o(\lVert\delta z\rVert_2^2).

We evaluate Fisher forms at individual SP and SC neighborhood representations, aggregate them separately, and discover directions through the contrastive generalized eigenproblem

FSCv=λ(FSP+ϵI)v.F_{\mathrm{SC}}v=\lambda(F_{\mathrm{SP}}+\epsilon I)v.

First-order score computation, top-qq output approximation, and low-rank eigendecomposition make this construction feasible for large pretrained models. The approach does not assume that finite anchor-to-variant displacements are infinitesimal tangent vectors.

Third, we establish that this geometric change has measurable consequences across modalities. Across seven language and three vision models, Fisher-IRG exhibits stronger predictive selectivity and generally greater Grassmann reproducibility than Cov-IRG. More importantly, the two methods recover substantially different local subspaces:

Geometric comparisonObserved result
Fisher–Cov projection distance0.992–0.998
Mean principal cosine0.056–0.080
Cross-method separation exceeding within-method variability100% of evaluated semantic groups

These findings establish that the Fisher and covariance constructions do not merely yield different numerical estimates of the same subspace. They select different directions because they measure different geometric quantities. Near-orthogonality alone does not prove semantic superiority; the causal interventions and held-out semantic evaluations provide the complementary functional evidence.

We do not claim to invent Fisher–Rao geometry or generalized eigendecomposition. Our methodological contribution is a theoretically grounded and computationally practical framework for contrastive local predictive geometry, with empirical characterization across language and vision.

Manuscript Change: Revised abstract and Section 1 contributions; Sections 3.1–3.2; Section 4.2, Displacement magnitude does not uniquely determine predictive consequence; Section 4.4; Appendix B and Appendix E, Table 20.

W3. Semantic-neighborhood validity and possible confounds

We agree that the SP/SC construction is central to the semantic interpretation of the recovered geometry. We have expanded Appendix A to document how variants are generated, screened, and partitioned, and added two validation experiments.

For language, we construct 1,800 neighborhoods using semantically constrained generation with ChatGPT and Claude. Each contains 10 SP and 64 SC variants. For vision, we construct 1,000 Visual Genome neighborhoods, using appearance transformations for SP and localized object edits with GroundingDINO, SAM, and Stable Diffusion inpainting for SC. Generated variants underwent manual quality screening.

We further conducted an AI-assisted semantic-label audit of 200 randomly sampled language variants:

Assigned labelConfirmedAmbiguousIncorrect
SP (100)81145
SC (100)8596
Total (200)1662311

The overall confirmation rate is 83%. This is not an independent human annotation study, and we explicitly acknowledge the remaining ambiguity rather than assuming perfectly controlled labels.

To directly test lexical confounding, we evaluated 200 held-out groups in which SC variants are lexically closer to the anchor than SP variants, reversing the usual relationship between surface similarity and semantic consistency.

RepresentationPairwise accuracy ↑Semantic separation ↑
Original0.710.011
PCA0.780.023
cPCA0.740.018
Cov-IRG0.810.029
Fisher-IRG0.860.037

Fisher-IRG retains the strongest discrimination under this control, suggesting that its semantic selectivity cannot be explained solely by lexical overlap. Nevertheless, other potential confounds, including syntax and visual appearance, are not completely eliminated.

Manuscript Change: Appendix A, dataset generation, quality control, and discovery/evaluation separation; Appendix A.1, Semantic Neighborhood Validity and Confound Controls, Tables 10–11; Section 4.6; Appendix I.

W4. Generalization beyond synthetic neighborhoods and practical utility

We agree that a foundational geometric analysis should be distinguished from a complete downstream application benchmark. We have expanded the functional evaluation beyond the original controlled neighborhoods to examine whether the recovered subspaces retain useful information under more natural conditions.

Natural language variation (SNLI). We evaluate Mistral-7B and LLaMA-3-8B on naturally collected entailment and contradiction pairs. Fisher subspaces are estimated from the original synthetic discovery data and transferred without using SNLI labels for subspace estimation.

ModelOriginalCov-IRGFisher-IRG
Mistral-7B pairwise accuracy0.64670.64670.6833
LLaMA-3-8B pairwise accuracy0.58000.66000.6967

Both experiments use compact 16-dimensional Fisher-derived representations rather than the original full-dimensional features.

Visual corruption retrieval. We also evaluate 600 corrupted-image queries from 200 Visual Genome images, applying Gaussian noise, defocus blur, and JPEG compression at severity 3.

RepresentationHit@1 ↑MRR ↑
Original0.9900.995
cPCA0.4450.556
Cov-IRG0.4200.549
Fisher-IRG0.5300.624

The original full-dimensional representation remains substantially better than the compressed representations. The relevant finding is therefore better preservation of retrieval utility under dimensionality constraints, not superiority over the original features.

Together with the representation-intervention results, these experiments establish functional relevance beyond KL-based selectivity and the original generated evaluation variants. They do not constitute state-of-the-art application benchmarks.

Our primary contribution is to establish and characterize a local predictive geometry; developing task-specific retrieval, steering, robustness, or disentanglement applications is a separate research direction that builds on this foundation.

Manuscript Change: Section 4.5–4.6; Appendix F.1, extended causal interventions; Appendix G.3, SNLI evaluation and Figure 5; Appendix G.4, visual retrieval and Figure 6; Appendix I, Functional validation and potential applications.

Reviewer_aENaOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a method to get locally meaningful subspaces per sample (as opposed to directions). They are constructed using contrastive samples close to the original one, departing from standard covariance based geometry by focusing on behaviorally important (Fisher-IRG) rather than high variance directions (e.g. Cov-IRG Dasgupta et al.).

The paper validates Fisher IRG over covariance-based counterparts as follows. (1) Section 4.3. shows that Fisher IRG is more semantically selective than Cov IRG. It additionally shows that (2) corresponding subspace is slightly more stable. Section 4.4. shows that (3) the subspaces recovered by both methods are nearly orthogonal, so Fisher IRG is not just a better approximation of the same subspace. Then, section 4.5. shows (4) that the small dimensional Fisher-IRG subspaces capture more semantically changing information than its high dimensional complement. Finally, section 4.6. shows that the local inner product induced by Fisher IRG (5) separates semantic differences better than the euclidean inner product or the local one induced by Cov IRG, and (6) outperforms all baselines in terms of retrieval.

Points (5) and (6) are the strongest points showing that their local metric is an impactful contribution.

Critical strengths

S1 : the paper measures robustness of their inner product against sample pool. The results are a bit hard to interpret, but they are transparent. The paper is honest in its reportings and interpretations, outliers are explained.

S2 : the problem of finding a meaningful inner product is very important, and the paper has a clear positioning in the literature in this regard.

S3 : the paper has enough evidence that their proposed inner product outperforms previous proposals on meaningful metrics (points (5) and (6)) though more transparent comparisons for other points would significantly strengthen the conclusion (there is no conclusion btw).

S4 : the experiments include a wide range of models across both vision and language.

Critical weaknesses

W1 : Circularity. The Fisher IRG method is optimised for its effects on the KL and on the output distribution. Metrics in points (1) and (4) rely on KL. It is not surprising that Fisher IRG outperforms Cov IRG on these since it was built for them.

W2 : Baselines. Section 4.2. seems mostly narrative. It shows that euclidean geometry isn’t always enough. I expected to see a comparison to other methods (random, pca, cpca, covirg and fisher irg), at the very least to show that they suffer less from this issue, and at best that FisherIRG suffers the least. It seems that neither in the main body nor in the appendix is rho reused for other methods. Points (1) through (3) seem to compare only fisher and cov irg, (4) doesn’t compare to any baseline at all. Points (5) and (6) are the strongest and do compare to baselines.

W3 : Layout. The paper is very hard to read due to very strong negative vspaces. There is no conclusion or discussion at all. I recognize that the authors were tight on space and barely made it to 9 pages, but I ask that at the very least they provide the text of the conclusion they intend to put in the camera ready version during rebuttal, and the modifications to the layout they intend to make for the paper to be more readable.

minor :

  • Stability remains poor. The values reported in table 3 remain high, as confirmed by table 4, which shows it through Delta geo. I suspect this is due to the small number of samples leading to a poor estimator of local geometry. More investigation would strengthen the paper.
  • Contrastive samples. This limitation is not inherent to this paper but inherited from the line of work. It does not affect my score. Still, the need for local contrastive samples is a real limitation of possible applications that has to be acknowledged.
  • Table 3 should bold the best value per row. It would make it much more readable.
Questions for authors

Q1 : Could the authors clarify what they wanted to show with point (3) ? Is it simply that both methods recover different subspaces, or was there a point I missed ?

Q2 : Could the authors explain the procedure for the generation of contrastive samples around an anchor ? This seems particularly important but I can’t find any mention of it in the paper.

Q3 : Why is Cov IRG below random in table 7 ? This suggests a poorly tuned baseline, or that Cov IRG is actually super bad.

Q4 : It looks like there is a link to be made with adversarial attacks here. Samples estimate the tangent to the local manifold if they are “close” compared to the Lipschitz constant of the model. Given that adversarial attacks are possible, it means that variant samples should be extremely close to the anchor. You do mention that “we do not approximate every anchor-to-variant displacement as an infinitesimal tangent vector”, but I wondered if you had any insights into how good of an approximation this is.

3 comments

We thank the reviewer for the constructive assessment and questions concerning geometric distinctness, semantic-neighborhood construction, evaluation rigor, and local geometric interpretation. We have revised the manuscript and added experiments addressing these concerns.

Q1. Fisher-IRG and Cov-IRG recover nearly orthogonal subspaces?

Our purpose is to establish geometric non-equivalence, not semantic superiority from orthogonality alone.

Cov-IRG characterizes the directional statistics of representation displacements, whereas Fisher-IRG measures local predictive sensitivity. We test whether they recover genuinely different subspaces by comparing cross-method separation against within-method variability from independent estimations.

The cross-method projection distances are 0.992–0.998, with mean principal cosines of 0.056–0.080. Moreover, cross-method separation exceeds within-method variation for 100% of evaluated semantic groups across seven language and three vision models.

These results establish that the two methods recover distinct geometric structures. Near-orthogonality alone does not establish semantic superiority; causal interventions and independent cosine-based semantic evaluations provide complementary functional evidence.

Manuscript Changes: Section 4.4; Appendix E, Table 20; Appendix I.

Q2. How are contrastive samples generated?

We expanded Appendix A to document dataset construction, quality control, and discovery/evaluation separation.

Language: We construct 1,800 anchor neighborhoods using ChatGPT and Claude, each containing 10 semantic-preserving (SP) and 64 semantic-changing (SC) variants. SP variants change wording or syntax while preserving meaning; SC variants modify semantic content, with 32 easy and 32 hard cases.

Vision: We use 1,000 Visual Genome anchors. SP variants apply appearance transformations; SC variants use GroundingDINO, SAM, and Stable Diffusion inpainting for localized object modifications across three difficulty levels.

For both modalities, 7 SP and 50 SC variants are used for discovery, while 3 SP and 14 SC variants are held out for evaluation. Generated samples underwent manual quality screening.

We additionally conducted an AI-assisted audit of 200 language variants, confirming 81% of SP and 85% of SC labels (83% overall). We acknowledge the remaining ambiguity.

To examine lexical confounding, we evaluated 200 held-out groups where SC variants are lexically closer to the anchor than SP variants. Fisher-IRG achieved 0.86 pairwise accuracy and 0.037 semantic separation, compared with 0.81 and 0.029 for Cov-IRG. Thus, the observed advantage is not explained solely by lexical overlap, although other confounds may remain.

Manuscript Changes: Appendix A; Appendix A.1, Tables 10–11; Section 4.6.

Q3. Why does Cov-IRG sometimes perform below Random?

We repeated the Random baseline using 20 independent seeds, with matched data, projection dimension (k=16k=16), and retrieval protocol.

RepresentationPairwiseMRRHit@1
Random (mean ± SD).908 ± .009.864 ± .012.788 ± .017
Cov-IRG.914.872.791
Fisher-IRG.959.931.885

The 20-seed Random mean is slightly below Cov-IRG on all three metrics, although their differences are small relative to random variability.

Neither Cov-IRG nor Fisher-IRG directly optimizes held-out cosine retrieval, so performance on this criterion is not guaranteed by their discovery objectives. Nevertheless, Fisher-IRG retains the strongest aggregate retrieval performance.

Manuscript Changes: Section 4.6, Table 7; Appendix G.2, Table 30.

Q4. Do finite SP/SC displacements accurately approximate tangent directions? What about adversarial sensitivity?

Our method does not require finite anchor-to-variant displacements to approximate infinitesimal tangent vectors.

Fisher information is estimated separately at each sampled representation, where it describes the local second-order predictive KL response. The neighborhood operators aggregate these local measurements:

Fa=1na∑i=1naF(zia),a∈{sp,sc}.F_a=\frac{1}{n_a}\sum_{i=1}^{n_a}F(z_i^a),\qquad a\in\{\mathrm{sp},\mathrm{sc}\}.

Finite SP/SC transformations therefore determine where predictive sensitivity is measured, rather than serving as infinitesimal tangent vectors in a single anchor-based Taylor expansion. All operators are expressed in common layer coordinates.

We also agree that predictive sensitivity is not necessarily semantic sensitivity. Adversarial directions may influence predictions without corresponding semantic change. The SP/SC contrast provides semantic constraints but does not guarantee adversarial robustness or perfect semantic disentanglement.

Our claim is therefore limited to locally predictive-selective subspaces under specified neighborhoods and readouts.

Manuscript Changes: Section 3.2; Appendix H, local Fisher formulation; Appendix I.

W1. Evaluation circularity

We agree that predictive-KL selectivity is closely related to the Fisher discovery objective and should not independently establish semantic superiority.

We therefore distinguish three types of evidence: predictive selectivity tests the intended geometric criterion; Grassmann and cross-method comparisons assess reproducibility and geometric distinctness; and held-out cosine-based discrimination and retrieval provide more independent semantic evaluation.

Fisher-IRG achieves the highest held-out separation in 21/21 language model–layer and 9/9 vision layer–difficulty comparisons. In seven-model retrieval, macro-average pairwise accuracy improves from 0.949 to 0.959, and Hit@1 from 0.873 to 0.885, relative to original representations. Paired bootstrap intervals support positive aggregate gains, although some individual comparisons remain statistically inconclusive.

We further added SNLI semantic matching and Visual Genome retrieval under ImageNet-C-style corruptions, evaluating the recovered subspaces beyond the original generated neighborhoods. These experiments provide functional evidence under criteria different from predictive KL without claiming general downstream superiority.

Manuscript Changes: Section 4.6; Appendix G.1–G.4, Tables 28–33; Appendix I.

W2. Scope of the displacement–consequence diagnostic

The distance-matched SP/SC analysis in Section 4.2 establishes an existence result: representation perturbations with comparable Euclidean magnitudes can produce substantially different predictive consequences. It does not imply universal inadequacy of Euclidean geometry.

The reported KL ratio measures predictive change under the original frozen model, not the quality of a recovered subspace. Applying the same diagnostic to PCA, cPCA, Random, Cov-IRG, and Fisher-IRG without changing the evaluation would therefore not meaningfully compare their subspaces.

Instead, we compare these methods through geometric and functional evaluations, particularly held-out semantic separation and retrieval. We do not claim that covariance-based geometry is incapable of capturing useful semantic structure.

Manuscript Changes: Section 4.2; Appendix B; Sections 4.3–4.6.

W3. Presentation, conclusion, and readability

We appreciate this criticism. Our original intention was to provide sufficient technical detail for evaluating a new geometric framework, but we recognize that the resulting density reduced readability. We have used the additional pages permitted during the discussion stage to improve spacing and layout, carefully reviewed the manuscript for clarity and precision, and organized the expanded evidence into dedicated appendix sections.

Minor concerns: Subspace stability and dependence on contrastive samples

We clarify that Fisher-IRG demonstrates greater relative reproducibility, not perfect recovery of an identical subspace. Fisher-IRG obtains lower Grassmann distances than Cov-IRG in 58/63 language and 26/27 vision configurations, although absolute variability remains appreciable.

Subspace estimation depends on sampled neighborhoods, regularization, operator approximation, and generalized eigengaps. Our experiments do not establish population-subspace convergence guarantees.

We also acknowledge that constructing reliable SP/SC neighborhoods remains an operational requirement and limitation of the framework.

Manuscript Changes: Section 4.3; Appendix D; Appendices A and I.

We appreciate the reviewer's feedback, which has helped us clarify the distinction between geometric non-equivalence, empirical reproducibility, and independent semantic utility, while improving the scientific scope and presentation of the manuscript. Should any questions or concerns remain, we would be grateful for the opportunity to clarify them promptly during the discussion period.

Thank you for this discussion. The major concerns about presentation have been addressed. Some technical issues remain, and I could keep asking for more baselines and more comparisons, but the results are well positioned in the literature and are strong enough. I believe this paper makes a valuable scientific contribution that should be accepted. I will raise my score accordingly.


Weaknesses (addressed or minor) :

  • W1 : The circularity weakness is indeed limited to only a subset of your experiments, with the most compelling ones being free from this issue. I should not have flagged this as a major weakness. It does not support rejection, but I should have clarified that what I wanted was more transparency in the main text.
  • W2 : thank you for the clarifications. I understand better the purpose of this experiment, and the reason it could not be applied to other methods.
  • W3 : this has been addressed using the additional page.

Questions : Thank you for the detailed answers. I have no further questions.

Reviewer_hu6KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks what local metric best captures semantically consequential variation in learned representations. Authors proposed Fisher-IRG , which uses Fisher information to weight local representation directions by their predictive sensitivity. For each anchor, they construct semantic-preserving and semantic-changing neighborhoods, then aggregate local Fisher matrices within each neighborhood, and solve a regularized generalized eigenproblem to recover directions with greater predictive sensitivity to semantic change. Authors compared it against Cov-IRG on seven language models and three vision models. Based on the analysis, authors claimed several key findings.

  1. distance-matched SP/SC perturbations can induce 2–187× differences in predictive KL, though the mismatch rate is only 5.8–24.1% of matched pairs;
  2. Fisher-IRG consistently improves predictive selectivity over Cov-IRG across all 10 models;
  3. Fisher and Cov recover geometrically near-orthogonal subspaces;
  4. Causal interventions at layer 10 localize semantic effects to the Fisher-derived subspace;
  5. The held-out retrieval shows modest gains over the original representation, with a few per-model exceptions where cPCA or Cov-IRG match or beat Fisher-IRG.
Critical strengths
  1. The paper has conceptually clean distinction between displacement geometry and predictive geometry. It starts from a real observation and builds a principled fix. The Fisher information is used to weight local directions by their predictive consequence rather than by how often representations. The comparison against Cov-IRG isolates the contribution cleanly. The finding that Fisher and Cov recover near-orthogonal subspaces is an important result. It shows that predictive geometry and displacement geometry are not two estimates of the same thing but genuinely different structures. 2.The representation swapping experiment goes beyond correlational subspace analysis. putting SP activations into the Fisher subspace barely changes output, while putting SC activations in changes output dramatically. This 1000× effect size is strong evidence that the Fisher subspace carries functional semantic information.
Critical weaknesses
  1. The practical value is under-demonstrated, and the "consistent improvement" claim has exceptions. All evaluations are internal, i.e., same-generation held-out variants and 1-vs-5 retrieval from the same pool. The absolute gain over the original representation is about 1% on a task where the original representation already achieves 95% accuracy. More importantly, per-model results in Appendix Table 23 show the Fisher-IRG underperforms the original representation on LLaMA-3 pairwise accuracy, with cPCA also beating Fisher. On Qwen2.5, Cov-IRG beats Fisher-IRG on Hit@1. No downstream application is shown where Fisher-IRG produces a qualitatively better outcome.
  2. Readout choice is under-justified, and vision readouts are inconsistent across models. The main text implies a uniform classification head readout, but Appendix H reveals three different vision readouts. The consistent Fisher advantage across ViT/DINOv2/CLIP is actually measured against three different output geometries, and no sensitivity analysis varies the readout, temperature, or probe. It is difficult for readers to tell whether Fisher-IRG recovers a robust semantic geometry or one tightly coupled to the specific readout chosen per model.
Questions for authors
  1. Appendix H shows three different vision readouts: ViT-B/16 uses the pretrained ImageNet classifier, DINOv2 uses a self-fit linear probe, and CLIP uses text-candidate cosine similarity. Based on this, I would like to know:
  • How sensitive is the recovered subspace to these choices? What happens if DINOv2 uses a different random probe split, or if CLIP's temperature is changed from 0.5 to 1.0?
  • Have authors tested a unified readout? For example, the same linear probe trained on identical labels across all three vision models.
  1. Appendix F states that donors are drawn from the same local semantic neighborhood. Does the causal effect persist when donors come from a completely unrelated semantic group? Additionally, why is layer 10 the only layer reported for causal interventions? Do results hold at earlier or later layers?
  2. No confidence intervals or p-values are reported. Given that Fisher-IRG underperforms the original representation on LLaMA-3 pairwise accuracy and Cov-IRG beats it on Qwen2.5 Hit@1, are the per-model differences statistically significant under bootstrap?
  3. Appendix H mentions a mean vs. last-token pooling ablation but reports no results. What q was used for the main experiments, and how does the Fisher advantage vary with q?
  4. Downstream utility and natural variation. Has Fisher-IRG been used for any downstream task (steering, ablation, disentanglement)? Does the advantage persist on naturally occurring semantic variation (natural entailment pairs, ImageNet-C)?
4 comments

We thank the reviewer for the careful evaluation and encouraging assessment of the distinction between displacement and predictive geometry, as well as the functional evidence from representation interventions. We appreciate the concerns regarding readout dependence, statistical reliability, and practical significance. In response, we have added readout and approximation ablations, extended causal interventions, paired bootstrap analyses, and new evaluations on naturally occurring language data and visually corrupted images. We have also clarified the scientific scope and limitations of the framework.

Question 1. How sensitive is Fisher-IRG to the predictive readout? Have we tested a unified readout?

Fisher-IRG is readout-conditioned by construction, but its observed advantages persist under the tested readout variations. We agree that the distinction between a robust predictive geometry and a readout-independent semantic geometry is important.

The Fisher operator measures the local sensitivity of a specified predictive distribution:

\left[\nabla_z\log p(y\mid z)\nabla_z\log p(y\mid z)^\top\right].$$ Consequently, changing the predictive head, output candidates, or temperature can change the metric. Our three vision readouts reflect the native predictive interfaces of their respective encoders, rather than assuming that their output spaces are numerically identical. We have clarified that our cross-model claim concerns the same **geometric construction and semantic contrast**, not a readout-independent metric. To test sensitivity, we evaluated 200 Visual Genome anchors with $k=16$. For CLIP, we changed the softmax temperature from $\tau=0.5$ to $1.0$ while keeping candidate texts fixed. For DINOv2, we refitted the linear probe using three random training splits and independently recovered the Fisher subspaces. | Model | Readout comparison | Mean principal cosine ↑ | Projection distance ↓ | |---|---|---:|---:| | CLIP | $\tau=0.5$ vs. $1.0$ | 0.84 | 0.54 | | DINOv2 | Split 1 vs. 2 | 0.85 | 0.47 | | DINOv2 | Split 1 vs. 3 | 0.76 | 0.52 | | DINOv2 | Split 2 vs. 3 | 0.79 | 0.49 | Although the recovered subspaces are not identical, they exhibit substantial overlap. More importantly, their predictive and semantic advantages persist: | Configuration | Fisher selectivity | Cov selectivity | Fisher − Cov | Fisher / Cov held-out separation | |---|---:|---:|---:|---:| | CLIP, $\tau=0.5$ | 1.04 | 0.81 | +0.23 | 0.124 / 0.094 | | CLIP, $\tau=1.0$ | 0.93 | 0.76 | +0.17 | 0.118 / 0.094 | | DINOv2, split 1 | 1.29 | 0.87 | +0.42 | 0.151 / 0.119 | | DINOv2, split 2 | 1.34 | 0.82 | +0.52 | 0.145 / 0.119 | | DINOv2, split 3 | 1.28 | 0.79 | +0.49 | 0.139 / 0.119 | Thus, Fisher-IRG maintains positive improvements over Cov-IRG in predictive selectivity and held-out semantic separation for every tested readout configuration. **We have not evaluated a single unified probe trained on identical labels across all three vision models.** Such a comparison would be valuable for isolating readout effects more strictly, but would define a different common downstream prediction task. Our present evidence supports robustness to the tested readout variations, not invariance to arbitrary readout choices. **Manuscript Change:** Section 3.2, *Language and vision readouts*; Appendix H.1, *Predictive Readout Dependence and Robustness*, Tables 34–36; Appendix I, *Scope and Limitations*.

Question 2. Do causal effects persist with unrelated donors and at other layers?

Yes, both causal selectivity and localization remain positive across the additional layers and donor settings evaluated.

We use the original component-replacement intervention, retaining k=16k=16, and report two complementary gaps:

\qquad \Delta_{\mathrm{local}}=D_{\mathrm{SC,inv}}-D_{\mathrm{SC,nuis}}.$$ The first measures whether replacing the Fisher component with SC information has greater predictive effect than replacing it with SP information. The second measures whether an SC donor produces greater effect through the Fisher subspace than through its complement. **Layer-wise interventions.** We extended the original layer-10 evaluation to layers 3, 10, and 20 for Mistral-7B and LLaMA-3-8B, and layers 3, 8, and 10 for ViT-B/16. | Model | Layers evaluated | $\Delta_{\mathrm{causal}}$ range | $\Delta_{\mathrm{local}}$ range | |---|---|---:|---:| | Mistral-7B | 3, 10, 20 | 1.54–1.87 | 0.58–0.89 | | LLaMA-3-8B | 3, 10, 20 | 1.46–1.62 | 0.51–0.69 | | ViT-B/16 | 3, 8, 10 | 0.94–1.24 | 0.47–0.59 | All nine evaluated model–layer configurations retain positive gaps. **Unrelated semantic donors.** At layer 10, we additionally replaced the SC donor with an example from an unrelated semantic group, while retaining the original same-group SP reference. | Model | SC donor | $\Delta_{\mathrm{causal}}$ ↑ | $\Delta_{\mathrm{local}}$ ↑ | |---|---|---:|---:| | Mistral-7B | Same group | 1.63 | 0.58 | | Mistral-7B | Unrelated group | 1.48 | 0.52 | | LLaMA-3-8B | Same group | 1.62 | 0.69 | | LLaMA-3-8B | Unrelated group | 1.39 | 0.61 | | ViT-B/16 | Same group | 1.24 | 0.59 | | ViT-B/16 | Unrelated group | 0.93 | 0.52 | The unrelated-donor gaps are smaller but remain positive. This extends the causal evidence beyond the original within-neighborhood donor protocol and layer 10. It supports functional localization under these additional conditions without implying that a single invariant subspace transfers globally across semantic groups. **Manuscript Change:** Section 4.5; Appendix F.1, *Robustness Across Intervention Layers and Semantic Donors*, Tables 23–24. ### Question 3. Are the per-model retrieval differences statistically significant? **The individual exceptions are not statistically significant under our paired bootstrap; the aggregate improvement is positive with confidence intervals excluding zero.** We added a paired bootstrap using 1,800 semantic groups per model and 2,000 resampling replicates. Each replicate resamples groups while preserving the five distractor comparisons per anchor. We report percentile-based 95% confidence intervals for paired differences, with positive values favoring Fisher-IRG. | Model | Metric | Fisher minus baseline | 95% CI | |---|---|---:|---| | LLaMA-3-8B | Pairwise vs. Original | −0.010 | [−0.017, 0.008] | | LLaMA-3-8B | Pairwise vs. cPCA | −0.004 | [−0.007, 0.006] | | LLaMA-3-8B | MRR vs. Original | +0.004 | [−0.010, 0.009] | | Qwen2.5-7B | Hit@1 vs. Cov-IRG | −0.018 | [−0.023, 0.007] | | Qwen2.5-7B | MRR vs. Cov-IRG | +0.011 | [0.003, 0.022] | The confidence intervals for the highlighted LLaMA pairwise differences and Qwen Hit@1 difference include zero. Therefore, we do not interpret these exceptions as statistically established advantages for either method. At the seven-model macro-average level, Fisher-IRG shows positive improvements over the original representation: | Metric | Original | Fisher-IRG | Difference | 95% CI | |---|---:|---:|---:|---| | Pairwise accuracy | 0.949 | 0.959 | +0.010 | [0.007, 0.019] | | MRR | 0.921 | 0.931 | +0.010 | [0.006, 0.018] | | Hit@1 | 0.873 | 0.885 | +0.012 | [0.006, 0.021] | These results support **modest but statistically supported aggregate improvements**, rather than uniform superiority across individual models and metrics. We have clarified this distinction in the main paper. **Manuscript Change:** Section 4.6; Appendix G.1, *Statistical Reliability of Semantic Retrieval*, Tables 28–29.

Question 4. What was the main-experiment value of qq, and how sensitive are the results to truncation and pooling?

The default Fisher approximation retains q=8q=8 predictive outcomes. We have now made this explicit in Appendix H and added controlled sensitivity analyses for q∈{4,8,16}q\in\{4,8,16\} and mean- versus last-token pooling.

The experiments use 200 Mistral-7B anchors at layer 10 with k=16k=16.

qqFisher − Cov selectivity ΔS\Delta S ↑Fisher held-out Δsep\Delta_{\mathrm{sep}} ↑Principal cosine vs. q=8q=8
40.220.150.81
80.290.231.00
160.250.180.86

The Fisher advantage remains positive under all three settings, although the recovered subspaces and effect magnitudes vary with qq.

PoolingFisher − Cov ΔS\Delta S ↑Fisher Δsep\Delta_{\mathrm{sep}} ↑
Mean pooling0.290.23
Last-token pooling0.250.16

The results therefore do not depend exclusively on the default mean-pooling convention. They also do not imply that q=8q=8 or mean pooling is universally optimal.

The top-qq procedure retains the highest-probability outputs, computes their score gradients under the original predictive distribution, and renormalizes the retained weights. This is an approximation to the full Fisher expectation, not an exact Fisher matrix for a newly restricted softmax.

Manuscript Change: Section 3.2, First-order computation and output truncation; Appendix H, Text experiments; Appendix H.2, Fisher Approximation and Pooling Sensitivity, Tables 37–38.

Question 5. Does Fisher-IRG provide downstream utility and generalize to natural semantic variation?

We have added two evaluations beyond the original controlled SP/SC neighborhoods, while clarifying the distinction between functional validation and application-level superiority.

Natural language variation. We evaluate semantic matching on SNLI using Mistral-7B and LLaMA-3-8B. The experiment samples 300 validation premises per model, pairing each premise with an entailing hypothesis and a contradicting hypothesis. Importantly, the Fisher subspace is discovered using the original synthetic SP/SC data and transferred to SNLI without re-estimating it on SNLI examples.

ModelOriginal pairwiseCov-IRG pairwiseFisher-IRG pairwise
Mistral-7B0.64670.64670.6833
LLaMA-3-8B0.58000.66000.6967

Fisher-IRG achieves the highest reported pairwise accuracy on both models using a 16-dimensional subspace, compared with 4096-dimensional original hidden states. The result provides evidence that semantic discrimination extends to naturally collected sentence relations under this transfer protocol, rather than merely recovering distinctions within the original generation process.

Visual nuisance variation. We also evaluate image instance retrieval on 200 Visual Genome images under Gaussian noise, defocus blur, and JPEG compression at severity 3, yielding 600 queries. These are ImageNet-C-style corruptions applied to Visual Genome, not an evaluation on ImageNet-C itself.

RepresentationHit@1 ↑MRR ↑
Original0.99000.9950
Random0.31000.4303
PCA0.30830.4316
cPCA0.44500.5559
Cov-IRG0.42000.5488
Fisher-IRG0.53000.6240

Here, the original full representation remains substantially better than every projected method. The result is therefore a dimensionality-constrained advantage: Fisher-IRG retains more retrieval utility than the competing compact representations under the tested corruptions, rather than improving upon the uncompressed original.

Regarding steering and disentanglement, our component-replacement experiments already provide a direct causal intervention test: manipulating Fisher-derived directions changes downstream predictive behavior in a semantically selective manner. However, we do not present a task-optimized steering method or a complete disentanglement application.

The paper's primary contribution is a theoretically grounded formulation and cross-modal characterization of local predictive geometry. Retrieval and interventions establish functional relevance, while broader task-specific applications remain important future directions.

Manuscript Change: Section 4.6; Appendix G.3, Natural Semantic Retrieval on SNLI, Table 31 and Figure 5; Appendix G.4, Instance Retrieval under Image Corruptions, Tables 32–33 and Figure 6; Appendix I, Scope and Limitations.

We hope these additional experiments and clarifications address the central concerns. We appreciate the reviewer's suggestions. Should any questions or concerns remain, we would be grateful for the opportunity to clarify them promptly during the discussion period.

Thank you again for your careful review and constructive feedback. We sincerely appreciate the opportunity for scientific discussion. As the discussion period is approaching its end, we wanted to kindly follow up on our November 16 response and the revised manuscript. We hope the additional experiments and clarifications have addressed your concerns. If any questions remain, please let us know, and we will do our best to respond promptly within the remaining discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions During the Discussion Stage

We sincerely thank all reviewers for their careful evaluations and constructive feedback. We have substantially revised the manuscript and conducted additional experiments addressing methodological novelty, evaluation independence, semantic-neighborhood validity, readout dependence, causal robustness, statistical reliability, and generalization.

The revised manuscript has been uploaded, with expanded experimental evidence, theoretical clarifications, and improved presentation throughout the ten-page main paper and appendices.

Clarification of the Scientific Contribution

We have revised the abstract, introduction, and contribution summary to clarify three connected contributions.

First, we identify a displacement–consequence mismatch: comparable Euclidean representation displacements can have substantially different predictive effects, motivating a predictive geometric perspective.

Second, we formulate Fisher-induced local invariant representation geometry (Fisher-IRG) through the local Fisher quadratic form and its predictive KL interpretation. We develop an efficient contrastive generalized-eigenvalue procedure for discovering predictive-selective subspaces using first-order score gradients, output truncation, and low-rank computation.

Third, we validate the resulting geometry across seven language and three vision models, examining predictive selectivity, Grassmann reproducibility, geometric distinctness, causal interventions, and semantic retrieval.

Our contribution is not Fisher information or generalized eigendecomposition individually, but a theoretically grounded, computationally practical framework for contrastive local predictive geometry, supported by cross-modal evidence.

New Experimental Evidence

  • Appendix A.1 (Tables 10–11): Semantic-label auditing and lexical-overlap controls assess neighborhood validity and surface-form confounding. Appendix A also clarifies SP/SC generation, quality control, and discovery/evaluation splits.

  • Appendix F.1 (Tables 23–24): Causal interventions across additional layers and unrelated semantic donors demonstrate persistent functional selectivity and localization under the tested conditions.

  • Appendix G.1–G.2 (Tables 28–30): Paired bootstrap confidence intervals distinguish aggregate improvements from statistically inconclusive individual-model differences. A 20-seed Random baseline clarifies its relationship with Cov-IRG.

  • Appendix G.3–G.4 (Tables 31–33): Natural semantic retrieval on SNLI and corruption-based Visual Genome retrieval provide functional evidence beyond the original controlled neighborhoods and predictive-KL evaluation.

  • Appendix H.1–H.2 (Tables 34–38): CLIP temperature, DINOv2 probe-split, Fisher top-qq, and pooling ablations examine robustness to predictive readout and computational choices.

We additionally clarified relative versus absolute Grassmann reproducibility in Appendix D and Fisher–covariance geometric non-equivalence in Section 4.4 and Appendix E.

Scientific Scope and Limitations

We appreciate the reviewers' questions regarding what Fisher-IRG establishes. We have expanded Appendix I, Scope and Limitations of Fisher-IRG, to clarify the boundaries of our theoretical and empirical claims.

Fisher-IRG provides a complementary predictive geometry, not a universal replacement for displacement-based analysis. Our experiments establish that there exist representation variations with comparable Euclidean magnitudes but substantially different predictive consequences. This motivates the Fisher perspective without claiming universal failure of Euclidean geometry. The recovered geometry remains conditioned on specified semantic neighborhoods and predictive readouts.

Our primary contribution is a theoretically grounded framework for local predictive geometry, with geometric and functional validation across language and vision. Retrieval and interventions demonstrate functional relevance rather than application-level superiority; broader downstream applications remain future directions.

Presentation and Readability

We appreciate the concerns about presentation density and aggressive spacing. Our original intention was to provide sufficient technical detail, but we recognize that the resulting density made the manuscript harder to read.

We have improved the layout, redesigned the main visualization, revised the manuscript sentence by sentence for clarity and precision, strengthened cross-referencing, and added Section 5, Conclusion and Discussion. The revised main paper remains within the ten-page limit.

Overall, we hope these revisions clarify the distinct geometric contribution, stronger independent functional evidence, and scientific scope of Fisher-IRG. We hope the revised manuscript and point-by-point responses address the reviewers' principal concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.