acceptodds
Under review as a conference paper at ICLR 2027

Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry

Abstract

Humans cannot always intuit what scenarios are most challenging to LLMs. Hoping to capture challenging edge cases, developers either design problems to be difficult for humans or curate extensive benchmarks. What if we could instead anticipate which scenarios a model will fail on? In this paper, we use an LLM's representational geometry to predict which concept combinations it will fail on. We attribute this compositional failure to interference between salient features. In tasks that require systematic composition - toy programmatic settings, multihop reasoning, multilingual factual recall - we find that when a pair of concepts is encoded near-orthogonally, the model reliably composes them. When their linear encodings are close, producing interference, the model fails to compose them. Our method reliably anticipates failure modes across different compositional tasks, without evaluating specific inputs. These results lay the groundwork to use representational geometry to identify high-risk examples, construct targeted stress tests, and provide a scalable foundation for active learning in real-world deployment.

Primary area:
interpretability and explainable AI
Keywords:
Interpretability, Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_3eKgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors ask whether the combinations of concepts an LLM will get wrong can be predicted from how it represents each concept on its own, without running the combined input, and call this goal Adversarial Concept Search (ACS). Their score, compositional interference (CI), measures how closely the representations of the concepts being combined are aligned, an idea they motivate with compressed sensing. For the LLMs, the representations are first centered within clusters of similar inputs. They test CI on small Transformers trained on SCAN and on two-hop questions and multilingual fact recall, each with three LLMs, checking with accuracy curves and PR-AUC whether the highest-CI combinations are the ones the model gets wrong.

Critical strengths
  1. On SCAN, where the training data are known, the authors test CI against simpler explanations: how often each concept and each pair of concepts appear in training, how long the command is, and whether other models trained on the same data give the same ranking. CI holds up against all three.

  2. To score a combination, CI needs only the representations of the single concepts, not the combined input. The multilingual setup shows why this is useful: an English vector for each fact and a subspace for each language are enough to score every fact-language pair, which would otherwise have to be translated and tested one by one. This also sets the method apart from An & Du (2026) and Blum et al. (2025b), which need representations of the combined examples.

Critical weaknesses
  1. The pooled two-hop result does not separate failing compositions from hard task types. The model succeeds mostly on one cluster pair (str_str), so a ranking that only separates the pairs beats the pooled baseline, and the within-pair results (Fig. 24) use the uncentered CI and show little gain on str_str.

  2. The LLM experiments lack a baseline that does not rely on geometry. The model's confidence on the atomic answers also needs no composed input, and Press et al. (2023) found that it predicts two-hop success, but CI is compared only with an uninformative ranking and other similarity measures. For multilingual recall, a mismatched-language subspace would show whether the language part of CI matters.

Questions for authors
  1. What are PR-AUC and point-biserial r for the centered CI within each cluster pair or dataset? A clear within-dataset gain would resolve my first concern.

  2. Does CI add predictive value over the model's confidence on the atomic answers, and does a mismatched-language subspace predict multilingual failures as well as the matched one?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3eKg

Thank you for recognizing the novelty and practical utility of the work. We provide clarifications and additional experiment results here. All additions will appear in the updated version of the paper.

W1, Q1. Within-cluster correlation Fig. 24 explicitly shows that each cluster group succeeds (high correlations and PR-AUC beating the baselines). Thus, the observation that “The model succeeds mostly on one cluster pair (str_str)” is a misread of the results. We show the within-cluster correlations and the fact that pooling them together hides the correlation between CI and failures, a classic case of Simpson’s paradox. To fix the statistical paradox, we center the representations, making the clusters comparable to each other.

To confirm that our predictions work for all subgroups after cluster mean-centering, we ran additional experiments. See the table below, where CI outperforms the baseline significantly on all individual subgroups except str-year, which might be more challenging because it is extremely imbalanced (LLM fails at 97.2% of combinations), suggesting that the LLM may not be generalizing compositionally on this subset at all. We will add these detailed results into the Appendix.

SubgroupCI PR-AUCRandom baseline
num-num0.928*0.918
num-str0.997***0.919
str-str0.588***0.537
str-year0.9640.972
year-num0.974*0.959
* p < 0.05, *** p < 0.001.

W2, Q2. Separating English fact difficulty from fact-language interference

We test whether using the matching language subspace improves failure prediction. For each model and target language, we evaluate whether using the target language’s subspace to predict the failures works better than using other languages’ subspaces. Although the statistical power of this test is very small, at least for Qwen2.5-14B, predictions from the matching subspace performed significantly better than those from mismatched subspaces. These results were not significant for the smaller Llama models (3B or 7B), which could be due to low statistical power. Another reason for the insignificance might be that smaller models are worse at multilingual transfer generally, weakening the accuracy of predictions that assume compositional transfer.

However, we also report new experiments to confirm that CI captures more information than just English fact difficulty in General Response to All Reviewers. We compare to new baselines such as English fact confidence and English fact representation norm. CI beat confidence and norm baselines in multilingual setup in the majority of the models and languages, suggesting that CI is capturing information beyond English fact difficulty. We will include these details in the updated Appendix.

Please let us know if we have addressed all the concerns raised in the review. If it does, we would appreciate your reconsideration of the score. Let us know if there are additional questions and we are happy to answer them.

Thank you for your rebuttal. My first concern also remains: two-hop success is concentrated in str_str (accuracy about 0.46 against below 0.10 for the other pairs), and the within-pair increase over the baselines are small. Besides, given the overall quality of the paper, I tend to maintain my score.

Reviewer_eByQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Adversarial Concept Search, a method for identifying concept combinations that are likely to cause compositional failures in LLMS without evaluating each composed input directly. The work hypothesize that such failures are associated with interference between salient linear features in the model’s representations. They quantify this using Compositional Interference, derived from the cosine similarities of estimated salient feature directions. The method is evaluated in controlled Transformer experiments on SCAN, followed by experiments on multihop question answering and multilingual factual recall using pretrained language models. The authors report that CI ranks difficult examples more effectively than random ranking and a failure-rate baseline, supported by permutation tests and bootstrap confidence intervals. The results suggest that representation geometry can help prioritize scenarios for targeted model evaluation.

Critical strengths
  1. The paper addresses the practical challenge of finding model failures when the space of possible concept combinations is large. ACS aims to rank abstract scenarios without requiring evaluation of every corresponding composed input, potentially reducing the cost of targeted stress testing.
  2. The authors first study SCAN under varying training coverage and model capacity, then evaluate multihop reasoning and multilingual factual recall. The reported ranking improvements, statistical tests, and analyses of potential confounds provide converging evidence that CI contains useful information about example difficulty.
Critical weaknesses
  1. Demonstrating CI predicts failures does not establish that interference is the mechanism responsible for those failures. Other properties of the representations or underlying tasks could contribute to the observed relationship. A stronger intervention or controlled analysis would help distinguish the proposed explanation from a predictive association.
  2. The reported comparisons establish improvements over random ranking and failure-rate baselines, but do not by themselves establish that CI is superior to other inexpensive predictors of compositional difficulty. Moreover, estimating salient feature directions requires choices about representation layers, context aggregation, and centering. Although the paper investigates some of these choices, stronger systematic comparisons would clarify how dependent the results are on the estimation procedure. The data-statistic baselines exist only for SCAN, not for the LLM tasks, and no learned difficulty predictor is tested.
Questions for authors
  1. What evidence supports the claim that representational interference contributes causally to compositional failures, rather than merely correlating with difficulty? Do interventions that reduce interference also improve compositional performance?
  2. How does CI compare, under the same evaluation budget, with embedding similarity or a learned difficulty predictor? Do these comparisons change ACS's relative advantage?
  3. How sensitive are the results to the representation layer and the procedure used to estimate salient feature directions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer eByQ

Thank you for your detailed review and suggestions. We provide clarifications and additional experimental results below and will incorporate them into the revised paper.

W1, Q1. Causal experiment.

Unfortunately, it is not possible to adjust the representational geometry directly without substantial side effects. We nonetheless have strong evidence for the causal mechanism by controlling for confounders. The strongest evidence is our test of random variation in SCAN, where we control for every possible alternative explanation (scale, data distribution, etc) by only varying random seeds. While independently trained models often make the same mistakes, we still find that a model’s own geometry is more predictive of these mistakes than a different model’s geometry (Appendix D.6).

W2, Q2. Analysis of baselines, representation layer and centering.

(1) Baselines: We discuss existing and additional results for comparisons with atomic-query confidence, entity popularity, and representation norm in the General Response.

(2) Embedding similarity: We ran an embedding similarity baseline to confirm that CI is not just capturing textual similarity between atomic questions. Specifically, for multihop, we take the average representation of the question tokens after the embedding layer and compute the similarity between the two atomic questions. Across all three models, CI achieves a higher failure PR-AUC than embedding similarity with statistical significance: Llama-3B 0.873 vs. 0.849, OLMo-7B 0.836 vs. 0.821, and Qwen-14B 0.700 vs. 0.610. For SCAN, we take the atomic concepts representation after the embedding layer and compute the average similarity and CI achieves significantly higher PR-AUC in all 13/13 models (median PR-AUC 0.126 vs. 0.065 with p<0.05 for all). For multilingual, embedding similarity is undefined because there is only one atomic component that has a meaningful embedding, which is the English question, because there’s no intuitive definition for the embedding of a language.

(3) Representation Layer: We do use a small validation set to choose the representation layer. To investigate how small we could make this set, we have conducted further experiments. We define the PR-AUC of the best layer on the test set as the model’s highest predictive power. With a validation set of only 200 labeled compositional examples, the resulting layer selections can retain 83% of the highest predictive power on average across all tasks. This validation budget corresponds to about 4.8% of the datasets on average, and we repeated the experiment over 200 random validation sets. (Even using only 100 validation examples, we selected layers with, on average, 71% of the highest predictive power.) We repeated this experiment over 200 different validation splits, demonstrating that performance is robust to resampling. In future work, we hope to understand why interference is an issue in some layers more than others, which would allow us to choose layers without a validation set. This goal is, however, out of scope for our paper.

(4) Centering: In Appendix Figure 24, we show that the uncentered scores present a classic Simpson’s paradox: within each cluster, the trend is strong, but the correlation reverses when all clusters are measured together. Fortunately, the paradox is easy to identify because the dominant clusters are obviously visible in visualizations. Mathematical details are given in Appendix C. With the right featurizer, we hope that future work can avoid this explicit centering step.

Please let us know if we have addressed all the concerns raised in the review. If it does, we would appreciate your reconsideration of the score. Let us know if there are additional questions and we are happy to answer them.

Thank you for your responses and for adding the new experiments. I acknowledge the general response as well.
The new results do strengthen the predictive claim, but my main concerns remain: The seed experiments show that CI captures model-specific information, but they do not establish that interference causes the failures. The Representation norm outperforms CI in all three multi-hop QA models. Moreover, the baseline comparison reports counts without providing the full PR-AUC values or confidence intervals. Layer selection requires labeled compositional examples. This additional data requirement and the associated cost should be made clearer in the main paper. I would suggest that the authors use predictive rather than causal language, report the full baseline results, and explicitly discuss these limitations in the main text. I will maintain my rating for now.

Reviewer_r6jROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adversarial Concept Search (ACS), which predicts compositional failures from the geometry of atomic concept representations without evaluating the corresponding composed inputs.

Critical strengths
  1. Predicting difficult concept combinations before constructing or evaluating specific inputs is an interesting approach to targeted model testing.
  2. The evaluation covers multiple tasks and model families.
Critical weaknesses
  1. The theoretical justification does not establish that feature interference causes compositional errors. The recovery bounds in Appendix A concern idealized encoders and sufficient conditions, while the experiments use residual activations as proxies for latent features. In multihop reasoning, CI yields the same ranking as absolute cosine similarity when only two atomic representations are involved (Appendix B). No intervention tests the proposed causal mechanism.
  2. The multilingual experiments do not adequately separate fact difficulty from fact–language interference. Within each language, the language subspace is fixed, so CI may also reflect properties of the English fact representation. The significant coarse-grained correlation for English (Table 9, r = −0.46), which does not involve cross-lingual composition, makes this concern particularly relevant.
  3. The LLM experiments lack strong baselines using the same atomic-level information. The paper does not evaluate whether CI improves actual data collection or downstream active learning.
  4. The results depend heavily on representation preprocessing. Without cluster mean-centering, the multihop correlation reverses from −0.210 to +0.178 (Table 5), and PR-AUC falls below the random baseline (Appendix F). Appendix C describes clusters identified through visualization, whereas Appendix I uses k-means without reporting k. The method also relies on labeled compositional development examples for layer and parameter selection.
  5. The SCAN evaluation does not fully account for the structure of its predictions. CI depends only on the deduplicated primitive set, so commands with different orderings or role bindings can receive identical scores.
Questions for authors

Q1. Equation 3 includes self-similarity, whereas Equation 2 excludes it. Consequently, CI equals 1/∣S(C)∣1/|S(C)| even when all concept representations are mutually orthogonal. How is this dependence on concept count consistent with the normalization claim in footnote 4, and how does it affect comparisons across compositions with different numbers of concepts?

Q2. How are background clusters and representation layers determined for previously unseen tasks without labeled compositional examples? Given the reversal of the correlation without mean-centering, how dependent are the reported results on these choices?

6 comments
AuthorsOpenReview ↗

Response to Reviewer r6jR 1/n

[ Response 1/n ]

Thank you for recognizing the novelty of the work and the generalizability of our results. We provide clarifications and additional experiment results here. All additions will appear in the updated version of the paper.

W1. Clarification on theory

Clarifying idealized encoders: Although all theoretical arguments must make some assumptions, we diverge from the superposition literature precisely because we weaken the assumptions mentioned here. In prior literature (Olah et al., 2022; Gorton & Lewis, 2025), compressive sensing is often cited as theoretical evidence for superposition; the classic results in this area assume an ideal encoder/decoder pair. Our theoretical assumptions are weaker than those used in exact recovery settings. Instead, Appendix A describes and cites results for robust recovery, i.e., recovery with noisy rather than ideal decoding conditions. By contrast, in all the work we cite on exact (non-robust) coding, claims rely on clean ideal encoders and decoders.

Relationship with absolute cosine: We use cumulative coherence because of the recovery bounds in Appendix A, but this metric is equivalent to cosine similarity when only two concepts interact. This is an inherent property of the metric, so there is no reason for their results to differ. We will clarify this relationship in future versions.

Causal claim: If we directly adjust the representational geometry, it would harm the model’s performance as a side effect. However, we do have very strong causal evidence; we tested random variation for SCAN in Appendix D.6 and for multihop queries in Polypythias 400M models of ten random seeds (see General Response). These experiments control for every environmental alternative explanation (scale, data distribution, etc) because they only vary random seeds. While independently trained models often make the same mistakes, we find that the model’s own training run-specific geometry is significantly more predictive than representational geometry from other training runs.

W2. Separation between fact difficulty and fact-language interference. Thank you for pointing this out. We have corrected Table 9 in our updated manuscript as below by using Spearman correlation since we do not assume the linear relationship between CI and the binary label English should have no correlation because we are explicitly filtering out any facts the model cannot retrieve in English.

LanguageSpearman ρp-value
es−0.450.048
fr−0.340.14
hu−0.500.025
ja−0.260.26
ko−0.510.021
nl−0.480.032
ru−0.070.76
uk−0.680.001
vi0.110.65
zh−0.060.81

To separate fact difficulty from fact-language interference, in General Response, we provide comparisons to baselines including English fact confidence and English query representation norm and show that CI provides additional information beyond fact difficulty.

W3.

(1) Baselines. There are no standard existing baselines for our novel ACS task, but thanks to the suggestions of several reviewers we have incorporated several baselines. These baselines are given in the General Response to All Reviewers. Overall, CI outperforms these baselines in most settings.

(2) Improvement for data collection and downstream training. Thank you for the suggestion; we updated the manuscript with new experiments below to demonstrate the practical value for efficient data collection. We study how many combinations we need to curate and test before we have found 25% of all model errors in the search space. A random sampling would on average have to sample 25% of the exhaustive search space. Across LLM tasks, when we sort by CI, we find 25% of errors without having to sample that much.

Fraction of examples needed for evaluation to recover 25% of errorsMulti-hop QAkovijaukzhhufrruesnl
CI21%22%21%21%22%21%18%14%15%14%11%
Perfect-predictor lower bound20%18%16%15%15%15%14%9%8%8%5%

On two-hop QA and half of the languages, the search efficiency is closer to search with a perfect oracle (the theoretical floor) than it is to 25%. Furthermore, the better the model is overall at a language, the more efficient CI typically is at identifying its errors, which is important for frontier models.

These results suggest a direct practical approach to data collection: rank English facts for a target language and then translate the highest-ranked facts for evaluation. Future work should expand to other settings, and incorporate these datasets into active training loops.

AuthorsOpenReview ↗

Response to Reviewer r6jR 2/n

W4. Cluster mean-centering.

In Appendix Figure 24, we show that, for raw CI scores within each cluster, the trends remain strong within each setting, indicating that CI is still predictive of interference and model failure. However, the absolute CI value ranges are shifted across cluster pairs, making raw CI values difficult to compare globally. Pooling those raw scores hides the correlations between interference and failures - this is classic Simpson’s paradox. The correlation is already present within cluster pairs, and preprocessing makes it visible by removing the broader effect instead of adding any additional information. This is a proof of concept for handling interactions at multiple scales. In this setting, the visualization revealed three clusters k=3 so either manual clustering or k-means removes the obvious large-scale structure. Automatically identifying and controlling for this structure is a direction for future work. One possibility is to use concept representations from sparse autoencoders that may have discovered atomic concepts without entangled large-scale features.

W5. Structure of compositions for SCAN.

We agree that CI does not account for errors arising from differences in ordering or role binding. We hope that future work can take advantage of structured inputs, but in our case CI empirically does very well without any structure or order information.

Q1. Equation 3 Typo.

Thank you for pointing out this notation error. Equation 3 should have the subscript i!=j and is fixed in the updated manuscript, resolving this issue.

Q2. Representation layer and centering.

(1) Representation Layer: We do use a small validation set to choose the representation layer. To investigate how small we could make this set, we have conducted further experiments. We define the PR-AUC of the best layer on the test set as the model’s highest predictive power. With a validation set of only 200 labeled compositional examples, the resulting layer selections can retain 83% of the highest predictive power on average across all tasks. This validation budget corresponds to about 4.8% of the datasets on average, and we repeated the experiment over 200 random validation sets. (Even using only 100 validation examples, we selected layers with, on average, 71% of the highest predictive power.) In future work, we hope to understand why interference is an issue in some layers more than others, which would allow us to choose layers without a validation set. This goal is, however, out of scope for our paper.

(2) Centering: In Appendix Figure 24, we show that the uncentered scores present a classic Simpson’s paradox: within each cluster, the trend is strong, but the correlation reverses when all clusters are measured together. Fortunately, the paradox is easy to identify because the dominant clusters are obviously visible in visualizations. Mathematical details are given in Appendix C. With the right featurizer, we posit that future work can avoid this explicit centering step.

References:

Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022b.

Liv Gorton and Owen Lewis. Adversarial examples are not bugs, they are superposition. arXiv:2508.17456, 2025.

Thank you for adding these experiments. I still have some concerns: The experiments on validation set size clarified the amount of labeled data required when selecting layers for a specific model. Given the intermediate layer pattern mentioned in Appendix J, have you tested whether the fixed relative depth rule can be transferred across different model families or scales without the need for recalibration? Both tasks used the Llama-3.2-3B and Qwen2.5-14B models, but the 7B-scale models were different. What was the basis for this selection?

Reviewer_etBrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Compositional Interference, a coherence score computed from atomic concept representations, to predict compositional failures without evaluating the composed input. The proposed mechanism is tested on a synthetic compositional generalization benchmark (SCAN), validating the insights.

Critical strengths
  1. The goal of predicting failures from atomic representations alone is practical and novel.
  2. The paper is easy to read and follow.
Critical weaknesses
  1. In the LLM experiments Compositional Interference is compared only against the failure rate. It needs comparison with atomic-query confidence, entity popularity, representation norm, and uncentered cosine.
  2. Experiments in table 9 show that CI is significant in only 4 of 11 languages. Authors should justify the choice in more detail in the main paper.
  3. The experiments compare the proposed mechanism with only random baseline. There is no comparison with cheap alternatives that also avoid composed inputs, such as atomic-query confidence, entity popularity, or uncentered cosine. Without these, it’s unclear whether CI adds anything over generic difficulty.
Questions for authors
  1. The method is framed as not requiring composed inputs, but layer selection uses labeled compositional examples. How do you see the role of this validation step? How would a practitioner choose a layer with no compositional labels at all?
  2. The coarse-grained results seem to vary considerably across languages. What do you think explains that variation, and how robust do you consider this part of the contribution?
2 comments
AuthorsOpenReview ↗

Response to Reviewer etBr

Thank you for recognizing the novelty of our work. We provide clarifications and additional experiment results here. All additions will appear in the updated version of the paper.

W1, W3. Comparison with baselines

We discuss existing and additional results for comparisons with atomic-query confidence, entity popularity, and representation norm in the General Response to All Reviewers.

For uncentered cosine: Uncentered cosine results are provided in Appendix F. Specifically, Table 5, Figure 21 (a) and (c) show that the cluster mean-centering is crucial for removing dominant background structures (which are irrelevant to the atomic concepts) before measuring interference.

W2, Q2. Moderate coarse-grained concept correlations. Thank you for flagging Table 9 on page 45. We provide the updated Table 9 here, and have switched to Spearman correlation, which is more appropriate in this setting as we do not expect a linear relationship between CI and a binary error label.

LanguageSpearman ρp-value
es−0.450.048
fr−0.340.14
hu−0.500.025
ja−0.260.26
ko−0.510.021
nl−0.480.032
ru−0.070.76
uk−0.680.001
vi0.110.65
zh−0.060.81

This result shows that coarse-grained representations for a topic have negative correlations with its mean accuracy in 9/10 models and reach statistical significance in 5/10 languages. Across 10 random seeds for validation set, the median number of languages with statistically significant results is 5. The very limited number of examples per coarse-grained topic in the validation split (n ≈ 9 on average) limits our ability to estimate each topic subspace, which would make layer selection less stable. Future work should investigate stable subspace construction given very limited data.

Q1. Validation set and layer sensitivity.

Our use of the term “input-free” was intended to distinguish our method from standard error prediction approaches in the uncertainty or active learning literature. Usually, error predictions require the model to process the specific composed input whose error is being predicted. In contrast, our method does not require evaluating the target composed input to predict whether the model will make an error. In Section 2.3 we acknowledge the use of a small development set of composed examples to pick which layer to measure interference in. In future work, we hope to develop more understanding of why interference is more of an issue in some layers or modules than others, but it is out of the scope of this work.

We ran additional experiments to demonstrate that the validation step is lightweight and robust for layer selection. We define the PR-AUC of the best layer on the test set as the model’s highest predictive power. With a validation set of only 200 labeled compositional examples, the resulting layer selections can retain 83% of the highest predictive power on average across all tasks. This validation budget corresponds to about 4.8% of the datasets on average, and we repeated the experiment over 200 random validation sets. Even with only 100 validation examples, the selected layer retained about 71% of its highest predictive power on average. These results show that we can select a layer with only limited validation data and retain close to the best layer’s predictive performance. We will include the full validation-size sweep in the updated appendix. We believe finding the best layer using no compositional input is an important objective for future work.

Please let us know if we have addressed all the concerns raised in the review. If it does, we would appreciate your reconsideration of the score.Let us know if there are additional questions and we are happy to answer them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

Thank you for the reviews. To address whether CI merely reflects generic difficulty, we point to existing results in the appendix and run new experiments. Together, the following results support that CI captures model-specific information about compositional failures.

1. Data Statistics / Entity popularity: This suggested baseline requires access to the training data. One of the major advantages of our approach is that we do not require access to the training data (and we don’t have access in the case of the language models). However, we can compare with this baseline in the case of SCAN. In Appendix D.6 co-occurrence statistics alone underperformed against CI; this is the closest equivalent to entity popularity in a corpus we do have access to.

We also show evidence that no corpus analysis could possibly capture the information in CI, because CI reflects random variation between training runs with the same training data. We test ACS on different random initialization seeds on the same SCAN corpus, so data statistics remain identical while the learned geometry would be different between each model. We then compare error predictions from CI computed from the specific seed model’s own representation to predictions from CI on other model runs, with different seeds. We find that a model’s own interference geometry is more predictive than that of a different model trained on the same corpus. Thus, compositionality failures are associated with model-specific representational geometry, not just training data.

For a similar random variation experiment with LMs, we ran a new experiment on 400M-parameter PolyPythias models with ten different random seeds, all trained on the same corpus. We focused on multi-hop QA because these models are too small to perform multilingual task transfer. In terms of failure PR-AUC, rankings using the model’s own representation are significantly better than imported rankings from another model with the same training data (p = 0.005). Therefore, CI captures more model-specific information beyond data statistics.

2. Atomic Query Confidence and Representation Norm: We thank reviewer etBr for suggesting these baselines, which we have now tested. Following Press et al 2023, we measure atomic query confidence using the average perplexity the model assigns to the correct answers to the atomic questions. We measure representation norm by averaging the norms of the residual-stream representations at the last token of each atomic question. Using all settings (different models for scan and multihop reasoning, and different languages and models for multilingual), we report in how many settings CI achieves higher failure PR-AUC than the baselines.

Settings where CI achieves higher failure PR-AUC (count / total)

TaskAtomic query confidenceRepresentation norm
SCAN13/139/13
Multi-hop QA3/30/3
Multilingual16/3022/30
Overall32/4631/46

Confidence and norm refer to atomic-query confidence and atomic representation norm, respectively.

CI outperforms each baseline in about 70% of the evaluated settings. This suggests that CI is more informative than these simple atomic baselines. Note that part of our objective is to scientifically understand failures, not just to build systems for predicting them, but we are pleased that interference is overall more consistently accurate than these baselines. We leave it to future work to understand the relationship between these different factors, but note that CI is not redundant with them; it is specifically normalized, so only the angle matters, not the representation norm.

Reference:

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and Narrowing the Compositionality Gap in Language Models. In Findings of EMNLP 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.