acceptodds
Under review as a conference paper at ICLR 2027

Structure Retention in Embedding Spaces Correlates with Benchmark Performance

Abstract

In this paper, we show that high-performing embedding models organize their embedding spaces in a consistent way. We evaluate 25 contemporary embedding models on five MTEB tasks spanning four diverse task categories (retrieval, bitext mining, pair classification, and summarization) in both English and multilingual settings, and reveal that nearest-neighbor overlap and magnitude differences in independent component analysis (ICA) between paired text instances strongly correlate (even up to 0.97) with performance on the given task. Ultimately, we show that embedding tasks display varying degrees of linearity and reliance on retention of local information. Our results further the understanding of embeddings and their relation to model performance beyond cosine similarity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
text embedding models, ICA, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KvH1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the hypothesis that in embedding models, the structural consistency of the embeddings of the corresponding pairs predicts the final benchmark performance. It provides a big set of evaluations on the MTEB benchmark and proposes two different measures of embedding similarity. It demonstrates that on some tasks they correlate well with the model's performance, although the strength of these correlations varies considerably across tasks.

Critical strengths

Understanding what a good embedding is characterized by geometrically is important.

I also like how this paper presents a very exhaustive list of experimental evaluations of different models and tasks from the point of view of representations.

Critical weaknesses

The generality of empirical findings is limited. They differ by the dataset and utilized method.

The scientific insight beyond the observed correlations is limited. That is, the authors show that quite often some geometric properties are correlated with the performance. They however do not explain why is that the case and when can we expect that to happen.

Questions for authors
  • How were the evaluated models trained?
  • How does Figure 2 look on log-scale?
  • How does varying the prompts used change the experimental results?
  • If you'd look at averaged MTEB performance (per category) instead of per-task performance, would some clearer patterns emerge?
  • Can the authors demonstrate some way in which the insights from this paper can be used to improve text-embeddings algorithms?
  • Are the proposed methods of investigating embeddings better than cosine similarity?
  • Why for some datasets gini-coefficients shows stronger correlation than for others?
3 comments
AuthorsOpenReview ↗

Response to review

Thank you for your review and your patience in waiting for our response while we ran new experiments. We address the weaknesses and questions below:

W1. (Generalizability) The manuscript demonstrates quite a lot of generalizability despite the assumption that tasks are encoded differently: 3 tasks show high correlation, and additionally, the summarisation task also shows medium correlation; the same applies to ICA with some caveats. We also consider finding dataset- and task-level differences in the level of correlation as one of the most interesting results in our paper. We further explain the outliers we see(like WebFAQ:eng, and the RTE-datasets), and the differences between neighborhood retention and ICA are analyzed in its own section (Section 5 “Shuffling ICA”).

W2. (Causality) We agree that our work is purely correlational and we indeed carefully avoided claiming causality until it can be proven in subsequent work; our goal was to analyze two ways to study structures in embedding spaces, and our results are very strong in this regard. Additionally, at the request of reviewer 77Uu, we ran a case study at the query level, looking at how retention relates to retrieval. If you are interested, please have a look at the message directed to all reviewers.

Q1. (Model training) We add the full hyperparameter list of trained ICA models in the appendix. The 25 embedding models we use in the study are all “off-the-shelf” transformer-based encoder models, for which we do no fine-tuning.

Q2. (New Figure) We are happy to include this figure: With a logarithmic x-axis, the models lie more or less on the line. We thank you for the suggestion, this figure will be available in the updated pdf.

Q3. (Varying prompts) We agree this is a reasonable concern; however, comparison across all models, datasets, and multiple prompt options would result in huge increase of the required compute, time, and effort, and we cannot run this analysis on subset of data, as we are looking for a signal across models and datasets. We investigate two options in the current manuscript and the scope of the current study is already resource-heavy and we are unable to expand the experiments in this direction at this moment.

Q4.(Task average scores) One of the bottlenecks in our study is the availability of MTEB scores. Running the full MTEB evaluation takes up to 2-3 GPU days per model, and doing this for the full 25 models is beyond our scope. While we think that studying more datasets per task would be valuable, we unfortunately cannot deliver this request within the time span of the discussion period.

Q5. (Improving embeddings) We think our work gives the community knowledge of properties that are present in models with high performance; this helps us understand the embedding spaces, and most importantly, which characteristics to look for when running causal studies. In this exploratory work, the strong correlations highlight candidate properties for further intervention research and this is genuine future work for us.

Q6. (Comparison to Cosine similarity) Yes, we already compare to cosine similarity. We analyse this in Section 5 shortly and in length in Appendix B.6. We see that our Ret_k method reaches higher correlation than any of the cosine similarity types we tested, from raw cosine similarity to z-standardized cosine similarity. We additionally do partial correlation and demonstrate that retention does not reduce to cosine similarity.

Q7. (Gini-variability) We believe this is due to multiple reasons: first, ICA is unsupervised, meaning it can latch onto any signal found in the data. We find that the same signal is found consistently, and see that in most cases, vocabulary explanations verify the signal is what we expect; despite this, ICA can in some cases latch onto something else, and this is simply a feature of ICA and will be visible in the Gini values. Second, in certain tasks, despite ICA being able to latch onto a correct signal, this might not be in line with what is needed to perform well on the task. This is the reason why we investigate the structure using two methods instead of one.

Thank you again for your review. We believe we have answered your questions; please let us know if we left anything unclear. If not, we hope that you will consider updating your evaluation.

Reviewer KvH1: are you convinced by the rebuttal, particularly the generality of the empirical findings?

Thank you for your responses. While they make things clearer for me, my concern about the generality of empirical findings has not been addressed, I will therefore maintain my score.

Reviewer_vpcZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The authors analyse the embedding spaces of 25 different embedding models, showing a correlation between: kNN overlap between queries and targets and performance on 5 MTEB tasks. The paper investigates the relationship between the structural geometry of embedding spaces and downstream performance on the MTEB benchmark. They introduce a new metric called “Neighborhood Retention”, which measures k-nearest neighbor overlap between paired queries and targets, and use it to show the correlation between the structure of the embedding space and performance. By introducing a target-shuffling intervention, they isolate whether specific MTEB tasks rely on preserving precise local query-target geometry or merely global separation.

Critical strengths
  1. Robust and wide cross-model (25 embedding models, sizes range from 8M to 8B parameters) empirical experiments on 5 different MTEB tasks (belonging to 4 categories).

  2. Disentanglement of local and global task dependencies in embedding spaces, providing a valuable mechanistic tool for auditing what benchmarks are actually testing.

Critical weaknesses
  1. The narrative flow and paper structure are confusing. In particular:
  • The abstract only mentions results, but the motivation is missing
  • Background and Motivation section mixes together related work, their specific hypothesis about linear ICA methods, and a “sneak peek” of the results
  • Methods section blends the methodology with the experimental setup, then the Experiments section gives a high-level overview of the experiments.

In order to improve the flow of the paper, it would be better to have:

  • Method: formal definitions and theoretical explanation of the experiments
  • Experimental Setup: all the implementation details that are now in the methods section (hyperparams, libraries, etc.)
  1. The framework remains purely correlational (no causal interventions have been tested), and it is not totally dataset-agnostic because it needs the dataset to have one-to-one indexing (e.g. not easily adaptable to multi-class classification)
Questions for authors

Clarifications:

  1. Could the authors explicitly clarify in Section 3.4 how the nearest neighbor overlap algorithm mechanically tracks the pairs across the query and target spaces? The mathematical summation over (t,t’)∈D(t,t’) \in D implies an index mapping, introduced briefly in Section 3.1, but the way the neighbours are handled is not clearly explained, making the methodology difficult to grasp fully.

Questions for future work:

  1. Given the strong reliance of RetkRet_k on one-to-one pre-paired datasets, how do you envision adapting this structural diagnostic framework to evaluate the multi-class classification or clustering tasks that make up a large portion of standard embedding evaluations?

  2. Do you have plans to perform targeted causal interventions to prove that explicitly enforcing these structural retentions causally improves model performance, rather than just correlating with it?

4 comments
AuthorsOpenReview ↗

Response to review

Thank you for your review and your patience in waiting for our response while we ran new experiments. We address the weaknesses and questions below:

W1 (Narrative flow). We will add motivation clearly to the abstract. To make the narrative flow even better, we will make an effort to move the current Methods sections 3.1, 3.2, and 3.3 under Experiments. We will be updating the pdf as soon as possible.

W2. (Correlational Framework) See our answers to Q2 and Q3.

Q1. (Clarification to equation) As our datasets have a one-to-one pairing quality, the indices are matched with this pairing, essentially matching the indices of a query and its correct answer one-to-one. We will add a clarifying sentence and update the pdf as soon as possible.

Q2. (Other tasks) Clustering and classification are distinct from our tasks for two reasons: No strictly one-to-one correspondence Both query and target sides use the instructions We do have a preliminary setup that handles both of these issues simultaneously: we use the cluster/class center of mass as the target. However, this is not perfectly comparable to our other tasks, and a proper handling of these experiments would not fit into the manuscript, even with the additional page. We thank you for your interest in our work!

Q3. (Causal Interventions) Yes, one option is the fine-tuning experiment mentioned in the future work section, which would allow us to investigate the causal relationship. Indeed this paper does rely on correlation, a possible candidate for intervention, and in next work, we will actually plan to implement it. We also ran a new case study on the query-level, if you're interested, please have a look at the message directed to all reviewers.

Thank you again for your review. We believe we have answered your questions; please let us know if we left anything unclear. If not, we hope that you consider updating your evaluation.

Reviewer vpcZ: are you convinced by the authors' reply, particularly as regards the flow of the paper?

Thank you to the authors for their response and the revised version of the paper. The motivation in the abstract and the new organisation of the sections make the paper clearer and easier to follow, and I have updated my presentation score accordingly. A remaining suggestion is to separate related work, the ICA hypothesis and the preview of results in the Background and Motivation section, which would further improve the flow.

Regarding my Q1, the added sentence clarifies that each query is index-matched with its target. However, it is still not explicit within which set the neighbours of queries and targets are computed. A sentence stating this would make the method much easier to follow. Additionally, while rereading the section I noticed that the equation sums over pairs and divides by k, whereas the values in Figure 2 suggest a mean Jaccard overlap. I would suggest the authors to double-check this for the camera-ready version.

Thank you for your feedback and for increasing our presentation score!

We agree that the implemented Methodology and Experiments sections' changes make the paper easier to follow. We also now see what you meant with the Background and Motivation section; We will further divide the Background and Motivation section into two, with a clear separation.

Thank you for pointing out the problem with the formula. The formula should calculate the average Jaccard overlap over the paired instances, and not divide by k but by the number of pairs. We will correct this in the formula, and additionally add a matching explanation to App. D. We will also be adding a clarifying sentence about the neighborhoods, akin to "The neighborhoods of query and target are calculated separately using k-nearest neighbors according to cosine distance. After this, for each correct pairing of query and answer, we calculate the Jaccard overlap between their k-neighborhoods, and average over the pairs."

Thank you again for your valuable comments, which have made our manuscript stronger. If time permits, we will still be updating the pdf before the discussion period ends.

Reviewer_77UuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether embedding geometry correlates with downstream performance. It evaluates 25 models on five datasets using two probes: neighborhood retention and the Gini coefficient of paired differences after ICA. Both probes correlate strongly with performance on translation mining and ARCChallenge. Results vary on other tasks. Target-shuffling experiments examine whether ICA captures pair-specific or dataset-level signals.

Critical strengths
  1. Useful empirical scope. The study covers multiple model families, languages, tasks, and prompting settings. It reports task-dependent failures alongside strong correlations.
  2. Informative controls. Target shuffling helps distinguish pair-specific alignment from global distribution differences. Cosine baselines and partial correlations test whether neighborhood retention adds information beyond pairwise similarity.
  3. Complementary probes. The two probes expose different structural properties. Their disagreement provides useful evidence against a single geometric explanation of performance.
Critical weaknesses
  1. Unclear evaluation advantage. Both probes require known text pairs. The paper does not establish an advantage over direct retrieval evaluation on the same subset. A matched comparison of cost and predictive accuracy is missing.
  2. Limited validity of the structural assumption. Similar queries can require different documents because of negation, entities, or constraints. Multiple valid documents can also have different neighborhoods. Low retention can therefore reflect valid task structure rather than poor embeddings.
  3. Mechanistic claims exceed the evidence. ICA concentration does not establish linear task encoding. Weak ICA correlations do not establish nonlinearity. The study also lacks intervention experiments or held-out validation showing that these probes predict performance beyond the selected models and datasets.
Questions for authors

Can the authors provide query-level case studies showing how low neighborhood retention or low ICA difference concentration relates to retrieval errors, including the rankings of relevant and competing documents?

4 comments
AuthorsOpenReview ↗

Response to review

Thank you for your review and your patience while we ran a new case study at your request with positive results. We address the weaknesses and questions below:

W1. (Evaluation advantage) Our goal with the paper was to understand some of the ways in which tasks are encoded in embeddings, not to suggest a new evaluation setup. We therefore treat the MTEB evaluation as the target; hence, we cannot meaningfully compare our methods to it. Knowledge of text pairs is also assumed in MTEB evaluation. See also our new experiment as the answer to Q1.

W2. (Limited validity) Our metrics make it possible to investigate this characteristic: Different queries have different target neighborhoods per task; for example, consider the same query used in translation and question answering tasks: if translation shows high neighborhood retention while QA does not, we can diagnose this with our methods and understand among which kinds of structures the perceived performance consists of. We do not claim that in order to be correctly retrieved, a particular example should always have high retention, rather, we show that neighborhood retention is a model-level mechanism that generally leads to better performance (up to r=0.97 with p<0.001). In the new case study, we look at this relationship in more detail. We also investigate the ICA approach precisely because we do not expect neighborhood retention to be the only source of correct retrieval. Additionally, we have shown that this effect does not reduce to cosine similarity.

W3. (Overclaims of mechanisms) We remove any language that suggests this to be a direct relationship, and reword them to “ICA is able to extract a vector, where the majority of the difference is concentrated”. We also already verify with the dictionary analysis that this corresponds to task-related vocabulary. Our conclusions still hold, and we will be updating the pdf as soon as possible.

Q1. (New case study) We ran a new case study by investigating the ranks of correct retrievals per query against their neighborhood retention. Please have a look at the results in the separate message directed to all reviewers. In summary, models with high average retention make near-miss errors, while models with low retention produce catastrophic errors. This systematic relationship explains why aggregate retention correlates strongly with benchmark scores.

Thank you again for your review. We believe we have answered your questions; please let us know if we left anything unclear. If not, we hope that you will consider updating your evaluation.

Reviewer 77Uu: are you convinced by the reply, particularly as regards the structure of evaluations and the validity of assumptions?

Thank you for the additional experiments. The query-level analysis provides useful evidence linking neighborhood retention to retrieval error severity. However, this correlation does not establish a mechanism or show that higher retention causes better retrieval. The analysis also lacks concrete examples showing queries, relevant documents, and competing documents to explain retrieval successes and failures.

AuthorsOpenReview ↗

Examples of queries, correct targets and competing documents

Thank you for your response.

First, we note that we do not claim causality in our manuscript.

Below, we show a few text examples from the high- and low-retention retrieval errors for ARCChallenge and text-embedding-3-large:

High neigh. retention:

  idx=220  retention=0.6667  rank=6
  Query:            Farmers in Wyoming were concerned because 
                    some of their chickens were being preyed upon
                    by hawks that lived in areas around their
                    ranches. The farmers grouped together and 
                    hunted the hawks until they were no longer in
                    their area. Which would most likely happen next?
  Correct target:   Populations of mice and rats would increase.
  Best match:       Their predators were disappearing.   (idx=498)

  idx=260  retention=0.4286  rank=4
  Query:            The digestive system breaks food down into
                    molecules that are used by our body's cells.
                    Which other body system regulates how fast or
                    slow the digestive system functions?
  Correct target:   nervous system
  Best match:       Cells break down nutrients into usable forms.  (idx=298)

  idx=687  retention=0.4286  rank=6
  Query:            The nucleus of the cell controls cell activity.
                    Which body system performs a similar function 
                    in humans?
  Correct target:   central nervous system
  Best match:       nucleus  (idx=99)

  idx=170  retention=0.4286  rank=2
  Query:            Students observed a turtle in the schoolyard.
                    They measured the distance that the turtle
                    traveled in 30 minutes. If the turtle walked 40
                    meters in that time, what was its average speed?
  Correct target:   80 m/h
  Best match:       0.5 m/s   (idx=519)

  idx=26  retention=0.4286  rank=6
  Query:            Which of the following materials are direct
                    products of photosynthesis?
  Correct target:   oxygen and sugar
  Best match:       make sugar through photosynthesis.  (idx=371)

Low neigh. retention:

  idx=935  retention=0.0000  rank=77
  Query:            Students study how the temperature of a 
                    substance is related to the rate of 
                    molecular movement. The molecules 
                    in which substance are moving the fastest?
  Correct target:   steam rising from boiling water
  Best match:       The molecules inside the balloon 
                    gained energy from the heat.  (idx=776) 

  idx=791  retention=0.0000  rank=3  
  Query:            In the life cycle of a frog, which of the 
                    following best describes what happens 
                    after the tadpole stage?
  Correct target:   The lungs develop and the tail becomes 
                    shorter.
  Best match:       Butterfly life cycles have a pupa stage.  (idx=279) 

  idx=210  retention=0.0000  rank=3
  Query:            The presence of fossil organisms in rocks 
                    can provide information about the 
                    environment at the time the sediments 
                    were deposited because
  Correct target:   different organisms usually inhabit 
                    specific environments.
  Best match:       The remains of plants and animals were trapped.   (idx=479)

  idx=516  retention=0.0000  rank=43
  Query:            Which activity is an example of conserving
                    nonrenewable resources?
  Correct target:   carpooling to work
  Best match:       reduces the number of trees cut down.  (idx=373)

  idx=213  retention=0.0000  rank=7
  Query:            Which characteristic is least likely to be
                    affected by an individual's environment?
  Correct target:   eye color
  Best match:       the type of hair their parents have (idx=321)

We additionally annotated the top 20 and bottom 20 retrieval errors by correctness and topical relevance using Claude Sonnet 5.5 and checked the annotations manually. The distribution of categories is different between high and low retention.

Low retention:

  • incorrect and topically relevant: 17/20
  • partially correct and topically relevant: 2/20
  • correct and topically relevant: 0/20
  • incorrect and topically irrelevant: 1/20

High retention:

  • incorrect and topically relevant: 10/20
  • partially correct and topically relevant: 5/20
  • correct and topically relevant: 5/20
  • incorrect and topically irrelevant: 0/20

If time still permits, we will be updating these results in the pdf.

Comments on the paper

4 comments
Area_Chair_aBSWOpenReview ↗

From AC.

Dear Authors,

Given the deadline on 18 November, please submit your responses to the reviews to facilitate a productive discussion with the reviewers.

Thanks, AC

AuthorsOpenReview ↗

New case-study results

To analyze how retention functions at the individual query level, we ran a case study that looks at each query’s retention score and the rank of retrieval for the correct answer. We omit ICA in this experiment, as the Gini-value is a global, per-dataset feature, unlike retention, which can be calculated for each query separately. For each query, the retention score is calculated with the Ret_k formula, and retrieval rank is simply the rank of the correct target when retrieving with cosine similarity. As some of the dataset-model combinations reach performance close to 1, we intentionally selected models that have less-than-perfect performance (to ensure we have negatives to analyze). Similarly, we leave out RTE3 due to reasons explained in Section 5 and SummEval due to the small size of the dataset. The combinations we selected are:

tatoeba:fra-eng, multilingual-e5-small

web-faq-engl, snowflake-arctic-embed-l-v2.0

ARCChallenge, text-embedding-3-large

Our main focus is the retrieval errors, which we visualize for Tatoeba:fra-eng and multilingual-e5-small. We rank all queries with retrieval errors by their individual neighborhood retention (Ret_k), and see that queries with high retention have the correct target closer to the top of the retrieved targets (low rank of correct target):

Ret_krank of correct target
11.00002
20.81822
30.66672
40.66672
50.66672
60.66672
70.66672
80.66673
90.66672
100.53852
.........
49910.000013
49920.00003
49930.00002
49940.000062
49950.00003
49960.00002
49970.00003
49980.0000666
49990.000010
50000.00008

Clearly, the “severity” of the retrieval error is substantially different when the retention is high, compared to when it is low. In all high-retention failure cases, rank=2 means a single competitor outscored the correct target, meaning the model was close but not precise. In contrast, the low-retention failure at rank=666 suggests the correct target was buried under hundreds of competitors. We see this in all model-dataset combinations studied here (although Tatoeba is the most striking; it also had the best results in the main experiments). This means that while neighborhood retention does not predict perfect retrieval, it is aligned with the overall quality of the retrieval as seen through ranks.

We further verify this by calculating ROC-AUC on the retention scores and retrieval scores with Recall@1 (measuring perfect match) and Recall@10 (measuring the absence of catastrophic failures) with 95% confidence intervals (CI) in parenthesis, calculated with n=2000 bootstrap resamples:

tatoeba:fra-eng, multilingual-e5-small: ROC-AUC with

Recall@1 0.68 (CI: 0.65, 0.72)

Recall@10 0.88 (CI: 0.84, 0.93)

web-faq-engl, snowflake-arctic-embed-l-v2.0: ROC-AUC with

Recall@1 0.61 (CI: 0.59, 0.63)

Recall@10 0.76 (CI: 0.72, 0.81)

ARCChallenge, text-embedding-3-large: ROC-AUC with

Recall@1 0.57 (CI: 0.54, 0.62)

Recall@10 0.66 (CI: 0.62, 0.70)

This means that especially Recall@10 scores (absense of catastrophic failure) can be distinguished by the retention score. Values of Recall@1 are lower, although they are above the random threshold of 0.5 as well. This shows that while retention is not connected to perfect retrieval accuracy, it is connected to the absence of retrieval failures: models with high average retention make near-miss errors; models with low retention produce catastrophic errors. This systematic relationship explains why aggregate retention correlates strongly with benchmark scores even when per-query AUC is moderate.

We are adding these results with visualisations to the pdf, and will notify when the pdf is updated.

AuthorsOpenReview ↗

Updated pdf

Dear reviewers and AC,

We have now updated the pdf with your comments in mind. We made the following changes:

  • added motivation to abstract
  • added the new case study to Appendix with additional figures and tables (Appendix D)
  • added clarification of index matching to the Ret_k formula
  • moved contents of Methods to the better-suited Experiments, and added a new subsection to Experiments for clarity
  • reworded all mentions of explicit linearity by ICA to better reflect the findings; conclusions still hold
  • added ICA hyperparameters explicitly to Appendix A.3.1
  • included a version of Figure 2 with a logarithmic x-axis (Figure 7)
  • renamed the cosine similarity comparison to Appendix C for clarity

Thank you again for your reviews and patience. We hope we have answered all your questions in a satisfactory way; if so, we kindly ask you to consider revising your scores.

AuthorsOpenReview ↗

End of discussion period

Dear AC,

As the discussion period is coming to an end and we have not received engagement from two reviewers, we would like to summarize our new contributions and rebuttals:

TL;DR:

Overall, we managed to explain the causes of weaknesses or resolve them completely with our new experiment, in which we showed that neighborhood retention is connected to the most dire failure cases. We have made changes to the manuscript that demonstrably increase the clarity of presentation, and reworded some claims to better reflect the findings while still supporting our conclusions.

Reviewer 77Uu:

  • We clarified the causes of weaknesses pointed out in the rebuttal:
    • Evaluation advantage: our goal is not to suggest a new evaluation setup.
    • Validity: our methods make it possible to study the qualities mentioned as weaknesses.
  • We resolved a weakness: reworded overclaims about ICA; the conclusions are unaffected.
  • We ran a new experiment that answers the reviewer's question: we showed that structure retention is connected to failure cases. ICA was left out of these experiments because the method is not suited for individual query-target pairs.

Reviewer vpcZ:

  • We thank the reviewer for engagement!
  • We made changes to the structure of the paper and added clarifying sentences, which were deemed to increase the legibility by the reviewer.
  • We responded to the correlational framework weakness with our new experiment and by explaining further possible causal experiments in our future work.

Reviewer KvH1:

  • We addressed weaknesses pointed out in the rebuttal:
    • Generalizability: We pointed out that the amount of generalizability we found in a setting where one would not expect to find it is substantial, and that we further analyze outliers.
    • Causality: While our paper does not aim to prove causality, we responded to the correlational framework weakness with our new experiment and by explaining further possible causal experiments in our future work.
  • We answered all questions:
    • Model training: We added the full hyperparameters of ICA models and clarified that we do not fine-tune the embedding models.
    • New figure: We added a new Figure at the request of the reviewer.
    • More prompts: We noted that due to the scope of the study, adding new prompts would mean running thousands of experiments due to the large number of models and datasets we use.
    • MTEB aggregate score: We note that evaluating the full MTEB is similarly resource-heavy and out of scope during the discussion period.
    • Cosine similarity: We have already compared to cosine similarity in Appendix C.
    • Gini variability: We explain the reasons for variability in the Gini coefficients.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.